Search NASA⌕ Search

SEARCH · Search NASA

Results for “chunking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

HAPPA: A Modular Platform for HPC Application Resilience Analysis with LLMs Embedded

High-performance computing (HPC) systems are increasingly vulnerable to soft errors, which pose significant challenges in maintaining computational accuracy and reliability. Predicting the resilience of HPC applications to these errors is crucial for robust code protection and detailed resilience analysis. In this study, we present HAppA, a modular platform designed for HPC Application Resilience Analysis. Embedding Large Language Models (LLMs), HAppA addresses understanding the context information of long code sequences typical in HPC applications. HAppA implements a novel code representation module that chunks the code into fixed-size segments and aggregates the embeddings of these segments. Three aggregation methods have been explored: MeanPooling, MaxPooling, and LSTM-based techniques. We built a DAtaset for REsilience analysis using Fault Injection (FI), named DARE. Using our DARE dataset, HAppA is trained for regression prediction tasks. Our evaluation results demonstrate the predictive accuracy of HAppA compared to other models, particularly noting that the LSTM-based aggregation method -- HAppA-LSTM -- achieves a mean squared error (MSE) of 0.078 for SDC prediction, surpassing the existing state-of-the-art PARIS model, which recorded an MSE of 0.1172. Additionally, HAppA with the KeyBERT model extracts a list of keywords representing the source code. A comprehensive importance analysis of these keywords further elucidates the code patterns contributing to the error rate. These findings highlight the effectiveness of HAppA in analyzing the resilience of HPC applications and establish a new benchmark for predictive accuracy in resilience.

Jiang, Hailong [Kent State University]↗

Physical properties, internal structure, and the three‐dimensional petrography of CI chondrites

physical properties and the nature of their breccation, we investigated nine samples of the Ivuna and Orgueil CI chondrites ranging in size from 1 mm to 4 cm in approximate diameter. The combined mass of unique material investigated in this work is 113 g. For our investigations, we use ideal gas pycnometry, 3-D laser scanning, x-ray computed microtomography (μCT), and accompanying digital data extraction techniques. We found that the bulk density of the samples ranged from 1.61 to 2.10 g cm −3 . Larger samples tend to have a lower bulk density. Grain density (ranging from 2.44 to 2.55 g cm −3 ) is significantly less variable than the bulk density in our samples and the quantity of porosity (ranging from 14.6% to 33.8%) is the dominant factor in determining the bulk density of CI chondrite material. Our μCT results show that the visible porosity across all sizes of our CI chondrite samples is in the form of cracks, but these cracks can account for less than two-thirds of the porosity in the CI chondrites. Other porosity is not visible, even at μCT resolutions of 2.7 μm voxel edge −1 and we conclude that it is sub-micron in nature. It is not clear if the cracks seen in our samples are indigenous to the chondrites or are a result of terrestrial processes. We also find that the CI chondrites are excellent examples of the fractal-like nature of brecciation, where clasts can be observed at all scales we imaged. The breccias are composed of sub-equant-shaped and sub-rounded-textured clasts like melt-free impact breccias on other solar system bodies. From our μCT volume and digital data extraction, we determine that the Ivuna CI chondrite breccia is organized: the mostly sub-equant clasts within our ~2 cm chunk of Ivuna have a mean diameter of 1.33 mm and their aligned longest axes define a lineation structure. We speculate that the lineation was imparted after fragmentation of the clasts by slight shear on the parent asteroid which could be the result of seismic-related granular flow or mild non-axial impact-related compaction. These data will help to place returned asteroidal material from asteroids 162173 Ryugu and 101955 Bennu and the CI chondrites into a mutual geological context.

CI chondrite↗

Vibration atomic layer deposition for conformal nanoparticle coating

A vibration atomic layer deposition reactor was developed for fabricating a conformal thin-film coating on nanosize particles. In this study, atomic layer deposition of 10–15-nm-thick Al{sub 2}O{sub 3} films was conducted on a high-surface-area acetylene black powder with particle diameters of 200–250 nm. Intense vibration during the deposition resulted in the effective separation of particles, overcoming the interparticle agglomeration force and enabling effective diffusion of the precursor into the powder chunk; this phenomenon led to the formation of a conformal film coating on the nanopowder particles. It was also confirmed that the atomic layer deposition Al{sub 2}O{sub 3} films initially grew on the high-surface-area acetylene black powder particles as discrete islands, presumably because chemisorption of the precursor and water occurred only on a few sites on the high-surface-area acetylene black powder surface. Relatively sluggish growth of the films during the initial atomic layer deposition cycles was identified from composition analysis.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Liver segmentation in contrast enhanced CT data using graph cuts and interactive 3D segmentation refinement methods

Purpose: Liver segmentation is an important prerequisite for the assessment of liver cancer treatment options like tumor resection, image-guided radiation therapy (IGRT), radiofrequency ablation, etc. The purpose of this work was to evaluate a new approach for liver segmentation. Methods: A graph cuts segmentation method was combined with a three-dimensional virtual reality based segmentation refinement approach. The developed interactive segmentation system allowed the user to manipulate volume chunks and/or surfaces instead of 2D contours in cross-sectional images (i.e, slice-by-slice). The method was evaluated on twenty routinely acquired portal-phase contrast enhanced multislice computed tomography (CT) data sets. An independent reference was generated by utilizing a currently clinically utilized slice-by-slice segmentation method. After 1 h of introduction to the developed segmentation system, three experts were asked to segment all twenty data sets with the proposed method. Results: Compared to the independent standard, the relative volumetric segmentation overlap error averaged over all three experts and all twenty data sets was 3.74%. Liver segmentation required on average 16 min of user interaction per case. The calculated relative volumetric overlap errors were not found to be significantly different [analysis of variance (ANOVA) test, p = 0.82] between experts who utilized the proposed 3D system. In contrast, the time required by each expert for segmentation was found to be significantly different (ANOVA test, p = 0.0009). Major differences between generated segmentations and independent references were observed in areas were vessels enter or leave the liver and no accepted criteria for defining liver boundaries exist. In comparison, slice-by-slice based generation of the independent standard utilizing a live wire tool took 70.1 min on average. A standard 2D segmentation refinement approach applied to all twenty data sets required on average 38.2 min of user interaction and resulted in statistically not significantly different segmentation error indices (ANOVA test, significance level of 0.05). Conclusions: All three experts were able to produce liver segmentations with low error rates. User interaction time savings of up to 71% compared to a 2D refinement approach demonstrate the utility and potential of our approach. The system offers a range of different tools to manipulate segmentation results, and some users might benefit from a longer learning phase to develop efficient segmentation refinement strategies. The presented approach represents a generally applicable segmentation approach that can be applied to many medical image segmentation problems.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Active workflow system for near real-time extreme-scale science. In: PPAA '14 Proceedings of the first workshop on Parallel programming for analytics applications

In recent years, streaming-based data processing has been gaining substantial traction for dealing with overwhelming data generated by real-time applications, from both enterprise sources and scientific computing. In this work, however, we look at an emerging class of scientific data with Near Real-Time (NRT) requirement, in which data is typically generated in a bursty fashion with the near real-time constraints being applied primarily between bursts, rather than within a stream. A key challenge for this types of data sources is that the processing time per data element is not uniform, and not always feasible to predict. Given the observations on the increasing unpredictability of compute load and system dynamics, this work looks to adapt streaming-based approach to the context of this new class of large experiments and simulations that have complex run-time control and analysis issues. In particular, we deploy a novel two-tier scheme for handling the increasing unpredictability of runtime behaviors: Instead of relying on determining what and where to run the scientific workflows beforehand or partial dynamically, the decision will also be adaptively enhanced online according to system runtime status. This is enabled by embedding workflow along with data streams. Specifically, we break data outputs generated from experiments or simulations into multiple self-describing "chunks", which we call active data objects. As such, if there is a transient hotspot observed, a data object with unfinished workflow pipeline can break its previous schedule and search for a least loaded location to continue the execution. Our preliminary experiment results based on synthetic workloads demonstrate the proposed active workflow system as a very promising solution by outperforming the state-of-the-art semi-dynamic workflow schedulers with an improved workflow completion time, as well as a good scalability.

Zhang, Yanwei↗

Design Considerations and Analysis of Multi-Level Erasure Coding in Large-Scale Data Centers

Multi-level erasure coding (MLEC) has seen large deployments in the field, but there is no in-depth study of design considerations for MLEC at scale. In this paper, we provide comprehensive design considerations and analysis of MLEC at scale. We introduce the design space of MLEC in multiple dimensions, including various code parameter selections, chunk placement schemes, and various repair methods. We quantify their performance and durability, and show which MLEC schemes and repair methods can provide the best tolerance against independent/correlated failures and reduce repair network traffic by orders of magnitude. To achieve this, we use various evaluation strategies including simulation, splitting, dynamic programming, and mathematical modeling. We also compare the performance and durability of MLEC with other EC schemes such as SLEC and LRC and show that MLEC can provide high durability with higher encoding throughput and less repair network traffic over both SLEC and LRC.

Wang, Meng↗

HQ-Sim: High-performance State Vector Simulation of Quantum Circuits on Heterogeneous HPC Systems

Quantum circuit simulations are applied in more and more circum- stances as the quantum computing community becomes broader. It helps researchers to evaluate quantum algorithms and relieve the burden of limited quantum computing resources. However, most of the state-of-the-art quantum simulators utilize either CPU or GPU to store and calculate the state vector, which results in resource starvation. Moreover, the maximum number of qubits supported by the simulator is bounded by the memory, since the memory utilization increases exponentially with the number of qubits. In this study, we leverage Heterogeneous computing to utilize both CPU and GPU to store and update state vectors. We also integrate lossy data compression to reduce memory requirements. Specifically, we develop a heterogeneous framework that has a dynamic scheduler to fully utilize the computing resources. We apply lossy compression to chunked state vector to make the maximum number of qubits higher than the regular simulators, the compression also benefits the data movement between CPU and GPU.

Zhang, Boyuan↗

DDStore: Distributed Data Store for Scalable Training of Graph Neural Networks on Large Atomistic Modeling Datasets

Graph neural networks (GNNs) are a class of Deep Learning models used in designing atomistic materials for effective screening of large chemical spaces. To ensure robust prediction, GNN models must be trained on large volumes of atomistic data on leadership class supercomputers. Even with the advent of modern architectures that consist of multiple storage layers that include node-local NVMe devices in addition to device memory for caching large datasets, extreme-scale model training faces I/O challenges at scale.We present DDStore, an in-memory distributed data store designed for GNN training on large-scale graph data. DDStore provides a hierarchical, distributed, data caching technique that combines data chunking, replication, low-latency random access, and high throughput communication. DDStore achieves near-linear scaling for training a GNN model using up to 1000 GPUs on the Summit and Perlmutter supercomputers, and reaches up to a 6.15x reduction in GNN training time compared to state-of-the-art methodologies.

Choi, Jong Youl↗

ARM Data Integrator

The Atmospheric Radiation Measurement (ARM) Data Integrator (ADI) streamlines the development of scientific algorithms and analysis of time-series NetCDF data, and improves the content and consistency of the output data products produced by these algorithms. The framework automates the process of retrieving and preparing data for analysis, and allows users to design output data products through a graphical interface. It also provides a modular, flexible software development architecture that scientists can use to generate C, Python, and IDL source code templates that embed the pre and post processing logic allowing the scientist to focus on only their science. The input data, preprocessing, and output data specifications of algorithms are defined through a graphical interface and stored in a database. ADI implements workflow for data integration and supports user access to data through a library of software modules. Data preprocess capabilities supported include automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog and allows developers to use existing data product to build new data products. Views of the output data products include an overlay of how the design meets ARM archive's data standards providing the user with a visual cue indicating where their output violates an archive standard. The ADI libraries access the information provided through the GUI via a Postgres database. The ADI framework and its supporting components can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Source record↗

basecRAWller v0.1

This software implements the first base caller for nanopore data that calls bases directly from raw data. The basecRAWller algorithm has two major advantages over current nanopore base calling software: (1) streaming base calling and (2) base calling from information rich raw signal. The ability to perform truly streaming base calling as signal is received from the sequencer can be very powerful as this is one of the major advantages of this technology as compared to other sequencing technologies. As such enabling as much streaming potential as possible will be incredibly important as this technology continues to become more widely applied in biosciences. All other base callers currently employ the Viterbi algorithm which requires the whole sequence to employ the complete base calling procedure and thus precludes a natural streaming base calling procedure. The other major advantage of the basecRAWller algorithm is the prediction of bases from raw signal which contains much richer information than the segmented chunks that current algorithms employ. This leads to the potential for much more accurate base calls which would make this technology much more valuable to all of the growing user base for this technology.

Brown, JamesB↗

Berkeley Nuclear Data Cloud (BDC) v2.3

The software was developed to serve and curate arbitrarily large datasets comprising data acquired from various mobile platforms. The software is contained in a number of server and client libraries. The former manage the ingestion, indexing, querying, and serving of the data. The latter libraries are distributed for Linux, Mac OSX, and Windows and enable users to interact with data downloaded from the service either in the form of an HDF5 file or streamed in a BSON data chunk. Using the Berkeley Data Cloud, researchers from varying fields can collaborate, compare results and curate both their raw data and the derived products of their analysis.

Cholia, Shreyas↗

MLEC-Sim: A Simulator for Evaluating Multi-Level Erasure Coding

We present MLEC-Sim, a sophisticated simulator for Multi-Level Erasure Coding (MLEC), developed in approximately 13 KLOC. The simulator is engineered to analyze the impact of various system configurations and erasure coding policies on system durability and network overhead. It supports a comprehensive range of parameters including disk capacity, disk I/O bandwidth, failure rates, network bandwidth, and system scale, accommodating various erasure coding approaches such as Single-Level Erasure Coding (SLEC), Multi-Level Erasure Coding (MLEC), and Local Reconstruction Codes (LRC). MLEC-Sim provides support for multiple chunk placement policies, including clustered parity and declustered parity, and encompasses a variety of repair methods like Repair-ALL, Repair-FCO, Repair-HYB, and Repair-MIN. It is capable of simulating disk failures through a variety of means, including distribution-based or trace-based mechanisms, and can handle complex multi-level (de)clustered placements and repair processes. A key feature of MLEC-Sim is its adoption of the splitting simulation method for evaluating system durabilities at extremely high levels, which are challenging to assess with traditional simulation approaches. This feature allows for a detailed evaluation of system resilience under a range of conditions, aiding in the selection of appropriate erasure coding solutions for enhancing system durability. MLEC-Sim contributes to the field of data storage and reliability by providing a tool for the detailed evaluation of the durability and efficiency of erasure coding configurations, intended for use by researchers and practitioners in the design and optimization of storage systems.

Wang, Meng↗

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗

V-HAMSTeR v1.0.0

V-HAMSTeR is a bioinformatics software tool designed to predict the hosts of viruses directly from genomic sequences. It can be used by researchers to predict animal, prokaryotic, plant, protist or fungal viral hosts including viruses that may be fragmented or discovered in environmental metagenomic datasets. Features & Uses: The software employs a novel dual-stream deep learning architecture that dynamically fuses implicit sequence embeddings from a genomic foundation model with 13 explicit, handcrafted biological features (e.g., coding density and strand switch rates). To ensure maximum reliability, V=HAMSTeR deploys a 5-fold deep ensemble calibrated via Joint Temperature Scaling, providing users with statistically rigorous confidence probabilities. It also features an automated sequence chunking and mean-pooling module to seamlessly process variable-length contigs. Advantages Over Similar Technologies: Existing tools (e.g., IPEV, RNAVirHost) typically rely on either basic k-mers or isolated neural networks. V-HAMSTeR's hybrid architecture captures both broad genomic context and specific biological motifs that standalone foundation models often miss. Furthermore, unlike competitor tools that struggle with incomplete data or exhibit extreme overconfidence, V-HAMSTeR is explicitly benchmarked and mathematically calibrated for fragmented assemblies (1kb–10kb). This makes it uniquely robust, accurate, and trustworthy for the messy reality of real-world environmental viromics.

Grigson, Susie [Lawrence Berkeley National Laborat↗

Globus service enhancements for exascale applications and facilities

Many extreme-scale applications require the movement of large quantities of data to, from, and among leadership computing facilities, as well as other scientific facilities and the home institutions of facility users. These applications, particularly when leadership computing facilities are involved, can touch upon edge cases (e.g., terabyte files) that had not been a focus of previous Globus optimization work, which had emphasized rather the movement of many smaller (megabyte to gigabyte) files. We report here on how automated client-driven chunking can be used to accelerate both the movement of large files and the integrity checking operations that have proven to be essential for large data transfers. In conclusion, we present detailed performance studies that provide insights into the benefits of these modifications in a range of file transfer scenarios.

97 MATHEMATICS AND COMPUTING↗

Power Distribution Analysis for the ORNL High Flux Isotope Reactor Critical Experiment 3

The mission of the Reduced Enrichment for Research and Test Reactors Program is to minimize and, to the extent possible, eliminate the use of highly enriched uranium (HEU) in civilian nuclear applications by working to convert research and test reactors, as well as radioisotope production processes, to low-enriched uranium (LEU) fuel and targets. Oak Ridge National Laboratory (ORNL) is currently reviewing the design bases and key operating criteria including fuel operating parameters, enrichment-related safety analyses, fuel performance, and fuel fabrication in regard to converting the fuel of the High Flux Isotope Reactor (HFIR) from HEU to LEU. The purpose of this study is to validate Monte Carlo methods currently in use for conversion analyses. The methods have been validated for the prediction offlux values in the reactor target, reflector, and beam tubes, but this study focuses on the prediction of the power density profile in the core. Power distributions were calculated in the fuel elements of the HFIR, a research reactor at ORNL, via MCNP and were compared to experimentally obtained data. This study was performed to validate Monte Carlo methods for power density calculations and to observe biases. A current three-dimensional MCNP model was modified to replicate the 1965 HFIR Critical Experiment 3 (HFIRCE-3). In this experiment, the power profile was determined by counting the gamma activity at selected locations in the core. 'Foils' (chunks of fuel meat and clad) were punched out of the fuel elements in HFIRCE-3 following irradiation, and experimental relative power densities were obtained by measuring the activity of these foils and comparing each foil's activity to the activity of a normalizing foil. This analysis consisted of calculating corresponding activities by inserting volume tallies into the modified MCNP model to represent the punchings. The average fission density was calculated for each foil location and then normalized to the reference foil. Power distributions were obtained for clean core (no poison in moderator and symmetrical rod position at 44.536 cm withdrawn with respect to the core axial midplane) and fully poisoned moderator (1.35 grams of boron per liter in moderator and rods fully withdrawn) conditions. The observed deviations between the experimental and calculated values for both conditions were within the reported experimental uncertainties except for some foils located on the top and bottom edges of the fuel plates.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Historical global Earth System Model simulations with E3SM's dynamic root module

This data package contains global-scale Earth System Model simulations run with DOE's E3SM Model. The package includes one set of simulations run in the default configuration and a second set of simulations run with the dynamic root module enabled. These two datasets can be used to assess the impact that dynamic roots have on Land Surface fluxes such as carbon and water. The outputs are provided here on a 0.5 x 0.5 global grid at monthly resolution. The simulations were run with atmospheric forcing from the Global Soil Wetness Project. We provide specifically here Gross Primary Production, Transpiration, relative root fraction per soil layer as well as Air Temperature, Precipitation and all the parameterizations used for the simulations. Our purpose for generating and analyzing these simulations was to assess the role that root dynamics and foraging for water has in the recovery timescale of ecosystems following climate perturbation. All data provided are in the "netcdf" format using standard CF-1 (Climate and Forecast) Convention. The data are broken up into 20 year chunks to facilitate easier read-in. The data are all machine readable using various software platforms including Matlab, NCO, Panoply and GrADS.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of Shear Strength Threshold of Concern for Retrieval of Interim-Stored K-Basin Sludge in the Hanford Site

K-Basin sludge will be recovered into the Sludge Transport and Storage Containers (STSCs) and will be stored in the T Plant for interim storage (at least 10 years). Long-term sludge storage tests conducted by Pacific Northwest National Laboratory show that high uranium content K Basin sludge can self-cement and form a strong sludge with a bulk shear strength of up to 65 kPa. Some of this sludge has "paste" and "chunks" with shear strengths of approximately 3~5 kPa and 380 ~ 770 kPa, respectively. High uranium content sludge samples subjected to hydrothermal testing (e.g., 185°C, 10 h) have been observed to form agglomerates with a shear strength up to 170 kPa. After interim storage at T Plant, the sludge in the STSCs will be mobilized by water jets impinging the sludge. The objective of the evaluation was to determine the range of sludge shear strength for which there is high confidence that a water-jet retrieval system can mobilize stored K-Basin sludge from STSCs. The shear strength at which the sludge can be retrieved is defined as the "shear strength threshold of concern." If the sludge shear strength is greater than the value of the shear strength threshold of concern, a water-jet retrieval system will be unlikely to mobilize the sludge up to the container’s walls. The shear strength threshold of concern can be compared with the range of possible shear strengths of K-Basin stored sludge to determine if the current post interim-storage, water-jet retrieval method is adequate. Fourteen effective cleaning radius (ECR) models were reviewed, and their validity was examined by applying them to Hanford 241-SY-101 and 241-AZ-101 Tanks to reproduce the measured ECR produced by the mixer pumps. The validation test identified that the Powell-3 and Crowe-2 ECR models are more accurate than other ECR models reviewed. These ECR models were used to address a question as to whether the effective cleaning radius of a water jet is sufficient or if it can be readily expanded to cover the range of possible shear strengths. These results will assist CH2M HILL Plateau Remediation Company (CHPRC) to establish the technical basis of the feasibility of the sludge retrieval and storage plan and to develop an adequate water jet system to retrieve the stored K-Basin sludge in the STSCs. The STSCs are 2:1 elliptical-head vessels, 58 inches in diameter and 105 inches tall. Each STSC will contain 0.5 to 2.1 m3 of settled sludge with the specific loading dependent upon sludge type.

43 PARTICLE ACCELERATORS↗