SEARCH · Search NASA
Results for “data analysis”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Integrating HPC, AI, and Workflows for Scientific Data Analysis: Report from Dagstuhl Seminar 23352
The Dagstuhl Seminar 23352, titled “Integrating HPC, AI, and Workflows for Scientific Data Analysis,” held from August 27 to September 1, 2023, was a significant event focusing on the synergy between High-Performance Computing (HPC), Artificial Intelligence (AI), and scientific workflow technologies. The seminar recognized that modern Big Data analysis in science rests on three pillars: workflow technologies for reproducibility and steering, AI and Machine Learning (ML) for versatile analysis, and HPC for handling large data sets. These elements, while crucial, have traditionally been researched separately, leading to gaps in their integration. The seminar aimed to bridge these gaps, acknowledging the challenges and opportunities at the intersection of these technologies. The event highlighted the complex interplay between HPC, workflows, and ML, noting how ML has increasingly been integrated into scientific workflows, thereby enhancing resource demands and bringing new requirements to HPC architectures, like support for GPUs and iterative computations. The seminar also addressed the challenges in adapting HPC for large-scale ML tasks, including in areas like deep learning, and the need for workflow systems to evolve to leverage ML in data analysis fully. Moreover, the seminar explored how ML could optimize scientific workflow systems and HPC operations, such as through improved scheduling and fault tolerance. A key focus was on identifying prestigious use cases of ML in HPC and understanding their unique, unmet requirements. The stochastic nature of ML and its impact on the reproducibility of data analysis on HPC systems was also a topic of discussion.
ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage
© 2018 IEEE. Scientific data analysis typically involves reading massive amounts of data that was generated by simulations, experiments, and observations. Performance of reading such large volumes of data from disk-based file systems is often poor because of the slow and mechanical components in the disks. Recent supercomputing systems are adding non-volatile storage layers in a hierarchy to handle the performance gap between fast main memory and slow disk-based storage. Software libraries for managing this hierarchy not only need efficient reading of data but also reduce user-involvement for cross-layer data movement. Furthermore, these libraries need to support array data access patterns into hierarchical storage management as scientific data is often organized in array-based data structures. Existing software typically manage individual storage layers requiring significant manual process in moving data among them. In this paper, we introduce a new array caching in hierarchical storage (ARCHIE) to accelerate array data analysis in a seamless fashion. ARCHIE evaluates array access patterns and prefetches data with array semantics between storage layers. Our evaluation shows that ARCHIE outperforms state-of-the-art file systems, i.e., Lustre and DataWarp, on a production supercomputing system by up to 5.8× in accessing data by scientific analysis applications.
Xarray Climate Data Analysis Tools
xCDAT is an extension of xarray for climate data analysis on structured grids. It serves as a modern successor to the Community Data Analysis Tools (CDAT) library. Xarray is an "open source project and Python package that introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, which allows for more intuitive, more concise, and less error-prone user experience. Xarray includes a large and growing library of domain-agnostic functions for advanced analytics and visualization with these data structures" (source: https://xarray.dev/). The goal of xCDAT is to provide generalizable features and utilities for simple and robust analysis of climate data. xCDAT's design philosophy is focused on reducing the overhead required to accomplish certain tasks in xarray. Some key xCDAT features are inspired by or ported from the core CDAT library, while others leverage powerful libraries in the xarray ecosystem (e.g., xESMF and cf_xarray) to deliver robust APIs.
Validation of finite-element models using full-field experimental data: Levelling finite-element analysis data through a digital image correlation engine
Full-field data from digital image correlation (DIC) provide rich information for finite-element analysis (FEA) validation. However, there are several inherent inconsistencies between FEA and DIC data that must be rectified before meaningful, quantitative comparisons can be made, including strain formulations, coordinate systems, data locations, strain calculation algorithms, spatial resolutions and data filtering. As such, in this paper, we investigate two full-field validation approaches: (1) the direct interpolation approach, which addresses the first three inconsistencies by interpolating the quantity of interest from one mesh to the other, and (2) the proposed DIC-levelling approach, which addresses all six inconsistencies simultaneously by processing the FEA data through a stereo-DIC simulator to ‘level’ the FEA data to the DIC data in a regularisation sense. Synthetic ‘experimental’ DIC data were generated based on a reference FEA of an exemplar test specimen. The direct interpolation approach was applied, and significant strain errorswere computed, even though therewas no model form error, because the filtering effect of theDIC enginewas neglected. In contrast, the levelling approach provided accurate validation results, with no strain error when no model form error was present. Next, model form error was purposefully introduced via a mismatch of boundary conditions. With the direct interpolation approach, the mismatch in boundary conditions was completely obfuscated, while with the levelling approach, it was clearly observed. Finally, the ‘experimental’ DIC datawere purposefully misaligned slightly fromthe FEA data. Both validation techniques suffered from the misalignment, thus motivating continued efforts to develop a robust alignment process. In conclusion, direct interpolation is insufficient, and the proposed levelling approach is required to ensure that the FEA and the DIC data have the same spatial resolution and data filtering. Only after the FEA data have been ‘levelled’ to the DIC data can meaningful, quantitative error maps be computed.
Machine Learning Based Software Package for Reproducible Data Analysis for Electron Microscopy
Advances in hardware and computation technologies are enabling large hyperspectral imaging during Electron Microscopy (EM), resulting in datasets being produced at rapidly increasing spatial and temporal resolutions. This presents exciting opportunities for microstructure analysis, but also poses great challenges in data analysis. Under this STTR program, QuesTek Innovations LLC, a leader in the field of integrated computational materials engineering (ICME), collaborated with Lawrence Livermore National Laboratory and Argonne National Laboratory. The team developed an open-source machine learning (ML) powered tool for EM data analysis of multiple systems and data types, and collected EM data for ML model development using the cutting-edge PicoProbe equipment at Argonne National Laboratory.
Measurement Uncertainty in One-Of-A-Kind Event Data Analysis
A golden standard in science is to repeat an experiment a statistically significant number of times, recording data using the same set of detectors and the same data analysis methodology. In such case experimental error includes both the range of true values generated by repetitions of the experiment, and measurement uncertainty caused by the detector. They are independent. It is a huge and too frequently used simplification, to assume that one can measure multiple repetitions of an identical experiment, resulting in identical true experimental value. Repetitions, as similar is it is experimentally achievable, have unavoidable built-in differences resulting in a range of the true values rather than in a single value. When modern, very sensitive and well calibrated measurement systems are used, this range is not negligible, and sometimes dominates over the measurement uncertainty. Range of true values depends on built-in differences in physics of the experiment. Stochastic physical processes result typically in a broader range of true values than non-stochastic processes do. Measurement uncertainty depends on a measurement method (properties of the detector not of the experiment). Modern measurement methods, including digital ones, frequently make the measurement uncertainty very small. When data from one–of –a kind experiment are analyzed, only the measurement uncertainty is reported. It provides no information about the range of true experimental values, neither about reliability of a reported data point. Reliability of a data point is in general independent from its measurement uncertainty. However, in practice reliable measurement methods frequently have high measurement uncertainty, while low reliability methods are applied to limit measurement uncertainty. Comparison of reliable data with high measurement uncertainty to not so reliable data measured with low uncertainty is discussed – in different scenarios different data analysis methods are applicable. Methods for data analysis from an experiment repeated statistically significant number of times are very well developed. They do not require a detailed expertise in physics of an experiment, nor in the properties of the measurement system used, and meaning of the reported uncertainty is well understood in any scientific community. It all changes when data from one-of-a-kind experiment is analyzed. Analyst’s expertise is required both in the physics of the experiment and in all aspects of the measurement system, all possible malfunctions. Data users must remember that only measurement uncertainty is reported from any one-of-a-kind experiment. Theory with simulations may provide estimation of expected built-in differences in the experiment, and by this of expected range of true values for a given experiment; yet measurement uncertainty can never be used in place of the range of true experimental values.
Machine learning enabled electron microscopy data analysis tool
This software is for materials image and spectral data analysis. It consists machine learning models that can segment a large amount of image data in a high-throughput way, and an interface for image data and spectral data analysis.
Using integrated data analysis to extend measurement capability (invited)
The analysis approach called integrated data analysis (IDA) provides a means to exploit all information present in multiple streams of raw data to produce the best inference of a plasma parameter. This contrasts with the typical approach in which information (data) from a single diagnostic is used to measure a given parameter, e.g., visible bremsstrahlung → Z eff . Data from a given diagnostic usually contain information on many parameters. For example, a Thomson scattering diagnostic is sensitive to bremsstrahlung and line emission in addition to electron temperature. This background light is typically subtracted off and discarded but could be used to improve knowledge of Z eff . IDA encourages explicit awareness of such information and provides the quantitative framework to exploit it. This gives IDA the ability to increase spatial and temporal resolution, increase precision and accuracy of inferences, and measure plasma parameters that are difficult or impossible to measure using single diagnostic techniques. One example is the measurement of Z eff on Madison symmetric torus using IDA since no single diagnostic can provide a robust measurement. Thus, as we enter the burning plasma era, application of IDA will be critical to the measurement of certain parameters, as diagnostic access in the harsh fusion environment will be extremely limited.
Design and implementation of a data analysis pipeline for paper spray ionization mass spectrometry of UO 2 Cl 2
Ambient mass spectrometry is an emerging characterization method for nuclear materials analysis. SRNL has recently acquired a new JEOL AccuTOF™-DART® LP 4G with a Filter Spray™ front end sample injection system, to develop new methods for ambient mass spectrometry of uranium analytes. Preliminary measurements of uranium-containing samples have demonstrated a need to rapidly evaluate large datasets generated with tailored data analysis scripts. To address this gap a data analysis pipeline written in R within the R Studio environment has been developed. The pipeline comprises a series of processing steps where the data are ingested, analyte peaks of interest are extracted, subsequent ratio and normalization calculations are performed, and results are presented in a series of graphical plots. In detail, the data files containing up to several hundred chronological spectra of a single sample in netCDF format were first extracted into RStudio. Then, initial 3D plots of the data were created in respect to the time, intensity, and mass to charge domains. These graphics serve as an initial easily visualized survey of instrument data. Next, a .csv format file containing the names, masses, and abundances of isotopes of interest were ingested into RStudio and were used to automatically collect intensity values at the mass value. Uranium isotope ratios were plotted against gadolinium isotope ratios (gadolinium was used as an internal standard) and assessed for ion beam stability and intensity. The spectra were filtered by applying a threshold intensity for the 158GdO+ mass channel. A Monte Carlo experiment was performed by creating a theoretical distribution of the data in order to determine the ratio of the uranium isotopes. The results show that the developed data analysis pipeline successfully filters high quality data from overall poor quality runs. Our results also support the idea that paper spray ionization mass spectrometry can be used for both detection and quantification of trace amounts of uranium compounds.
Open Reproducible Electron Microscopy Data Analysis
Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.
Interlaboratory Reproducibility of Contour Method Data Analysis and Residual Stress Calculation
While the contour method for residual stress assessment has developed rapidly, no published study documents its interlaboratory reproducibility. Objective: Here we report an initial reproducibility experiment focused on contour method data analysis and residual stress calculation. The experiment uses surface topography data from a physical process simulation of elastic-plastic beam bending. The simulation provides surface topography, for input to the contour method data analysis, as well as a known residual stress field with 130 MPa peak magnitude. Additionally, to increase realism, noise and specific artifacts are added to the topography data. A group of participants received the topography data (without the known residual stress), independently analyzed the data, and submitted results as a two-dimensional residual stress field. Analysis of submissions provides a group average residual stress field and the spatial distribution of reproducibility standard deviation. The group average residual stress agrees with the known stress in magnitude and spatial trend. The reproducibility standard deviation ranges from 2 to 54 MPa over the measurement plane, with an average of 5.4 MPa. Reproducibility standard deviation is smaller in the cross-section interior (≤ 5 MPa), modest near local extrema in the stress field (5 to 10 MPa), and larger near the cross-section boundaries (10 to 30 MPa). Overall, the largest values of reproducibility standard deviation (up to 54 MPa) occur in limited areas where artifacts had been added to the topography data; while some participants identified and removed these artifacts, some did not, leading to systematic differences that elevated the standard deviation.
Aggregation and data analysis of corrosion studies in molten chloride and fluoride salts
Corrosion studies in molten fluoride and chloride salts were surveyed, and key data were aggregated into a single dataset. Studies were graphed by salt purity, temperature, sample material, container material, and experimental method (loop, capsule, crucible). To elucidate and quantify the factors that affect corrosion in molten salts, a data analysis using a newly-defined corrosion resistance score and a correlation analysis using techniques borrowed from machine learning were performed. It is shown that salt purity had the strongest correlation with corrosion rates in molten chlorides and fluorides. As a result, data analysis across varied works was inconsistent due to the lack of standardization among molten salt corrosion studies.
BatteryPro: A Python Toolkit for Battery Data Analysis and Machine Learning Predictions
Analyzing battery test data for research & development can be time-consuming since battery tests often run on the order of months to years, generating large volumes of data. BatteryPro is a comprehensive Python package and software designed to facilitate advanced analysis and performance predictions for battery test data. Developed for battery researchers, it supports data types from widely used battery testing instruments, including MACCOR and Biologic cycling systems. The software provides a variety of tools for extracting and plotting key battery parameters such as time, voltage, capacity, current, and pressure. In addition to its extensive data analysis capabilities, BatteryPro features a dedicated machine learning module that employs a Bayesian Gaussian Mixture Model (GMM) to predict battery performance and degradation. Users can generate synthetic capacity fade data, calculate fade metrics, and leverage predictive models to forecast long-term battery behavior. The software's graphical user interface (GUI) enhances usability, allowing researchers to upload, merge, and analyze multiple data files with full customizability. The GUI also supports machine learning predictions, enabling users to fit models and make predictions based on selected data and parameters. BatteryPro is built using QtDesigner, scikit-learn, matplotlib, and pandas, ensuring a high level of customization, flexibility, and accuracy in battery data analysis. This tool aims to empower researchers with the ability to perform detailed battery analysis and make informed predictions, ultimately advancing the field of battery research.
High Crimes and Misdemeanors in Data Analysis
Invited talk going over some of the various data analysis problems that have occurred over the years including the most extreme example of falsifying data.
Estimating the Diffusion Coefficient of Lithium in Graphite: Extremely Fast Charging and a Comparison of Data Analysis Techniques
Galvanostatic intermittent titration experiments were performed in three-electrode cells to characterize the effect of C/2, 2-C and 4-C charge rates on the observed lithium diffusion coefficient. As part of the data analysis process, we compared the classic Weppner-Huggins analysis of polarization data with a newer (Wang et al.) analysis method for depolarization data. At low values of x in Li x C 6 , both analysis methods showed the same general trend in the apparent lithium diffusion coefficient, 4-C > 2-C > C/2. The two techniques differed in the magnitude of the estimated diffusion coefficient by about a factor of 100. The observed increase in diffusion coefficient does not last over a large compositional range. Since the estimates from the method of Weppner and Huggins may contain artifacts due to the use of particulate electrodes and high charge rates, the method of Wang et al. may produce better values.
Topological data analysis of task-based fMRI data from experiments on schizophrenia
We use methods from computational algebraic topology to study functional brain networks, in which nodes represent brain regions and weighted edges represent similarity of fMRI time series from each region. With these tools, which allow one to characterize topological invariants such as loops in high-dimensional data, we are able to gain understanding into low-dimensional structures in networks in a way that complements traditional approaches based on pairwise interactions. In the present paper, we analyze networks constructed from task-based fMRI data from schizophrenia patients, healthy controls, and healthy siblings of schizophrenia patients using persistent homology, which allows us to explore the persistence of topological structures such as loops at different scales in the networks. We use persistence landscapes, persistence images, and Betti curves to create output summaries from our persistent-homology calculations, and we study the persistence landscapes and images using k-means clustering and community detection. Based on our analysis of persistence landscapes, we find that the members of the sibling cohort have topological features (specifically, their 1-dimensional loops) that are distinct from the other two cohorts. From the persistence images, we are able to distinguish all three subject groups and to determine the brain regions in the loops (with four or more edges) that allow us to make these distinctions.
Covariate Dependent Sparse Functional Data Analysis
This study proposes a method to incorporate covariate information into sparse functional data analysis. The method aims at cases where each subject has a limited number of longitudinal measurements and is associated with static covariates. This research is motivated by several use cases in practice. One representative example is void swelling, a nuclear-specific material degradation mechanism. Void swelling is affected by many covariates, including alloy composition and irradiation type. How to accurately model the complicated joint effects of such covariates on the swelling process is the key to mitigating the effect of swelling and ensuring safe operation. Unlike most of the existing methods, the proposed method can handle high-dimensional covariates with the informative covariate identification procedure and sparse and irregularly spaced measurements, that is, does not require complete or dense observations. The main innovation of the proposed method is that we model the variation coming from covariates and the variation left conditioned on covariates, such that the functional principal component analysis and Gaussian process can be conducted in a unified manner. Further, we also propose a systematic approach to identify important covariates in the hypothesis testing context. The methodology is demonstrated on applications in nuclear engineering and healthcare and simulation studies.