Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Intensive Science”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Employing artificial intelligence to steer exascale workflows with colmena

Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. In conclusion, our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing.

Workflows↗
DOE OSTI2025

Improving microstructures segmentation via pretraining with synthetic data

Image analysis of material microstructures through microscopy is an integral capability in the field of materials science. The topological and chemical information obtained through microscopy allow us to draw vital connections between material microstructures, properties, and processing. While scanning electron microscopy (SEM) is able to yield a considerable wealth of information interpretable by the intuition of experts, there has been considerable interest in using machine learning, convolutional neural networks (CNNs) in particular, for such image analysis task. Training CNNs for an image analysis task requires a large annotated dataset. However, in many materials science applications, obtaining a large annotated dataset is cost and labor intensive. In this work, we study the use of synthetic data to enlarge the available annotated experimental data of uranium oxide. We utilize a modified Potts model to simulate uranium oxide particles with morphologies similar to those observed experimentally. We then leverage an image-to-image translation model to synthesize the simulated particles as if they are acquired with SEM. Through this process, we obtain pairs of particle images and their corresponding SEM representations, which corresponds to pairs of annotations and images. Unlike previous works, we leverage synthetic data for pretraining a CNN model prior, and finetune that model further with experimental data. We experimentally demonstrate that using synthetic data as incremental learning process benefits the overall performance compared to training a model on combined synthetic and experimental data.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Automation is all you need: Faster Earth system models with AI/ML

Focal Area: Data acquisition and assimilation enabled by machine learning (ML), artificial intelligence (AI) and advanced methods. Science Challenge: Tropical cyclones can in- duce extreme water cycle events through dramatic precipitation and storm surge. More reliable models of intensity will translate into better prediction of the impact of extreme events in large scale Earth systems simulations. We demonstrate and describe AI/ML methodologies for rapid assimilation of new, in situ data products.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Multiple parameter dynamic photoresponse microscopy for data-intensive optoelectronic measurements of van der Waals heterostructures

Quantum devices made from van der Waals (vdW) heterostructures of two dimensional (2D) materials may herald a new frontier in designer materials that exhibit novel electronic properties and unusual electronic phases. However, due to the complexity of layered atomic structures and the physics that emerges, experimental realization of devices with tailored physical properties will require comprehensive measurements across a large domain of material and device parameters. Such multi-parameter measurements require new strategies that combine data-intensive techniques—often applied in astronomy and high energy physics—with the experimental tools of solid state physics and materials science. We discuss the challenges of comprehensive experimental science and present a technique, called Multi-Parameter Dynamic Photoresponse Microscopy (MPDPM), which utilizes ultrafast lasers, diffraction limited scanning beam optics, and hardware automation to characterize the photoresponse of 2D heterostructures in a time efficient manner. Using comprehensive methods on vdW heterostructures results in large and complicated data sets; in the case of MPDPM, we measure a large set of images requiring advanced image analysis to extract the underlying physics. We discuss how to approach such data sets in general and in the specific case of a graphene–boron nitride–graphite heterostructure photocell.

36 MATERIALS SCIENCE↗
DOE OSTI2025

PSTN-019: The LSST Science Pipelines Software: Optical Survey Pipeline Reduction and Analysis Environment

The NSF-DOE Vera C. Rubin Observatory is executing the Legacy Survey of Space and Time (LSST) as its prime mission, producing a series of data releases over the ten-year survey. The LSST Science Pipelines Software will be used to create these data releases and to perform the nightly prompt processing and alert production. This paper provides an overview of the LSST Science Pipelines Software, describing the components and their integration into pipelines that generate science-ready data products.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Brochure on the 2024 ASCR Workshop on Analog Computing for Science

Analog computing fundamentally differs from digital by representing data with fully continuous physical quantities, such as voltages, probabilities, chemical concentrations, or light intensities, rather than encoding values in discrete binary states. While digital computing has historically excelled in precision, scalability, and noise resistance, newer analog approaches are gaining interest for their potential to dramatically improve energy efficiency and processing speed. Analog systems can inherently solve mathematical problems through their physical behavior and offer distinct advantages in scenarios where continuous operations are more effective than Boolean logic.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Models and Methods for Sparse (Hyper)Network Science in Business, Industry, and Government

The authors are hosting an AMS sponsored Mathematics Research Community (MRC) focusing on two themes that have garnered intense attention in network models of complex relational data: (1) how to faithfully model multi-way relations in hypergraphs, rather than only pairwise interactions in graphs; and (2) challenges posed by modelling networks with extreme sparsity. Here we introduce and explore these two themes and their challenges. In this work, we hope to generate interest from researchers in pure and applied mathematics and computer science.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Uncertainty in inventories for life cycle assessment: State‐of‐the‐art, challenges, and new technologies

Uncertainty is a critical factor that can hinder the quality and potential applications of life cycle assessment (LCA) results. A prominent source of uncertainty stems from the life cycle inventory (LCI) data. Various methodologies exist to estimate the uncertainty associated with LCI data, primarily based on the widely used structured pedigree matrix approach or the computationally intensive Monte Carlo simulation. This perspective review explores how new technologies (e.g., computational algorithms and data collection methods) from data science and related fields can contribute to identifying, quantifying, and reducing uncertainty in LCI modeling. A brief overview of the sources of uncertainty in LCI modeling and how they are addressed in current LCA practice is provided. Additionally, several new technologies are identified, and the potential benefits of their implementation in reducing uncertainties in LCI modeling are discussed. This perspective review concludes by identifying potential areas that require further development for these technologies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2016

Mini-apps for high performance data analysis

Scaling-up scientific data analysis and machine learning algorithms for data-driven discovery is a grand challenge that we face today. Despite the growing need for analysis from science domains that are generating `Big Data' from instruments and simulations, building high-performance analytical workflows of data-intensive algorithms have been daunting because: (i) the `Big Data' hardware and software architecture landscape is constantly evolving, (ii) newer architectures impose new programming models, and (iii) data-parallel kernels of analysis algorithms and their performance facets on different architectures are poorly understood. To address these problems, we have: (i) identified scalable data-parallel kernels of popular data analysis algorithms, (ii) implemented `Mini-Apps' of those kernels using different programming models (e.g. Map Reduce, MPI, etc.), (iii) benchmarked and validated the performance of the kernels in diverse architectures. In this paper, we discuss two of those Mini-Apps and show the execution of principal component analysis built as a workflow of the Mini-Apps. We show that Mini-Apps enable scientists to (i) write domain-specific data analysis code that scales on most HPC hardware and (ii) and offers the ability (most times with over a 10x speed-up) to analyze data sizes 100 times the size of what off-the-shelf desktop/workstations of today can handle.

Sukumar, Sreenivas R.↗
DOE OSTI2016

On-demand data analytics in HPC environments at leadership computing facilities: Challenges and experiences

Scaling-up scientific data analysis and machine learning algorithms for data-driven discovery is a grand challenge that we face today. Despite the growing need for analysis from science domains that are generating `Big Data' from instruments and simulations, building high-performance analytical workflows of data-intensive algorithms have been daunting because: (i) the `Big Data' hardware and software architecture landscape is constantly evolving, (ii) newer architectures impose new programming models, and (iii) data-parallel kernels of analysis algorithms and their performance facets on different architectures are poorly understood. To address these problems, we have: (i) identified scalable data-parallel kernels of popular data analysis algorithms, (ii) implemented `Mini-Apps' of those kernels using different programming models (e.g. Map Reduce, MPI, etc.), (iii) benchmarked and validated the performance of the kernels in diverse architectures. In this paper, we discuss two of those Mini-Apps and show the execution of principal component analysis built as a workflow of the Mini-Apps. We show that Mini-Apps enable scientists to (i) write domain-specific data analysis code that scales on most HPC hardware and (ii) and offers the ability (most times with over a 10x speed-up) to analyze data sizes 100 times the size of what off-the-shelf desktop/workstations of today can handle.

Harney, John↗
DOE OSTI2020

Evaluation of the Four-Dimensional Ensemble-Variational Hybrid Data Assimilation with Self-Consistent Regional Background Error Covariance for Improved Hurricane Intensity Forecasts

The feasibility of a hurricane initialization framework based on the Gridpoint Statistical Interpolation (GSI)-based four-dimensional ensemble-variational (GSI-4DEnVar) hybrid data assimilation system for the Hurricane Weather Research and Forecasting model (HWRF) model is evaluated in this study. The system considers the temporal evolution of error covariances via the use of four-dimensional ensemble perturbations that are provided by high-resolution, self-consistent HWRF ensemble forecasts. It is different from the configuration of the GSI-based three-dimensional ensemble-variational (GSI-3DEnVar) hybrid data assimilation system, similar to that used in the operational HWRF, which employs background error covariances provided by coarser-resolution global ensembles from the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS) ensemble Kalman filtering data assimilation system. In addition, our proposed initialization framework discards the empirical intensity correction in the vortex initialization package that is employed by the GSI-3DEnVar initialization framework in operational HWRF. Data assimilation and numerical simulation experiments for Hurricanes Joaquin (2015), Patricia (2015), and Matthew (2016) are conducted during their intensity changes. The impacts of two initialization frameworks on the HWRF analyses and forecasts are compared. It is found that GSI-4DEnVar leads to a reduction in track, minimum sea level pressure (MSLP), and maximum surface wind (MSW) forecast errors in all of the HWRF simulations, compared with the GSI-3DEnVar initialization framework. With assimilating high-resolution observations within the hurricane inner-core region, GSI-4DEnVar can produce the initial hurricane intensity reasonably well without the empirical vortex intensity correction. Further diagnoses with Hurricane Joaquin indicate that GSI-4DEnVar can significantly alleviate the imbalances in the initial conditions and enhance the performance of the data assimilation and subsequent hurricane intensity and precipitation forecasts

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

ASC Data Science Infrastructure Project June 2023 [Slides]

Data Science Infrastructure (DSI) Project offers a standardized but flexible way of storing data and associated metadata to support AI/ML and data intensive scientific workflows: Establish requirements via use case interviews; Develop DSI framework to support flexible, secure, and accessible data stores tailored to researcher needs; Augment existing institutional frameworks for code management, performance testing and file indexing via automated DSI interfaces; Deploy DSI framework to support key use cases in collaboration with domain scientists.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

FRACTURE MECHANICS APPROACH TO ESTIMATE FATIGUE LIVES OF WELDED LAP-SHEAR SPECIMENS

A full range of stress intensity factor solutions for a kinked crack is developed as a function of weld width and the sheet thickness. When used with the associated main crack solutions (global stress intensity factors) in terms of the applied load and specimen geometry, the fatigue lives can be estimated for the laser-welded lap-shear specimens. The estimations are in good agreement with the experimental data. A classical solution for an infinitesimal kink is also employed in the approach. However, the life predictions tend to overestimate the actual fatigue lives. The traditional life estimations with the structural stress along with the experimental stress-fatigue life data (S-N curve) are also provided. In this case, the estimations only agree with the experimental data under higher load conditions.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Application of sulfur SAD to small crystals with a large asymmetric unit and anomalous substructure

The application of sulfur single-wavelength anomalous dispersion (S-SAD) to determine the crystal structures of macromolecules can be challenging if the asymmetric unit is large, the crystals are small, the size of the anomalously scattering sulfur structure is large and the resolution at which the anomalous signals can be accurately measured is modest. Here, as a study of such a case, approaches to the SAD phasing of orthorhombic Ric-8A crystals are described. The structure of Ric-8A was published with only a brief description of the phasing process [Zeng et al. (2019), Structure , 27 , 1137–1141]. Here, alternative approaches to determining the 40-atom sulfur substructure of the 103 kDa Ric-8A dimer that composes the asymmetric unit are explored. At the data-collection wavelength of 1.77 Å measured at the Frontier micro-focusing Macromolecular Crystallography (FMX) beamline at National Synchrotron Light Source II, the sulfur anomalous signal strength, |Δ ano |/σΔ ano ( d ′′/sig), approaches 1.4 at 3.4 Å resolution. The highly redundant, 11 000 000-reflection data set measured from 18 crystals was segmented into isomorphous clusters using BLEND in the CCP 4 program suite. Data sets within clusters or sets of clusters were scaled and merged using AIMLESS from CCP 4 or, alternatively, the phenix.scale_and_merge tool from the Phenix suite. The latter proved to be the more effective in extracting anomalous signals. The HySS tool in Phenix , SHELXC / D and PRASA as implemented in the CRANK 2 program suite were each employed to determine the sulfur substructure. All of these approaches were effective, although HySS , as a component of the phenix.autosol tool, required data from all crystals to find the positions of the sulfur atoms. Critical contributors in this case study to successful phase determination by SAD included (i) the high-flux FMX beamline, featuring helical-mode data collection and a helium-filled beam path, (ii) as recognized by many authors, a very highly redundant, multiple-crystal data set and (iii) the inclusion within that data set of data from crystals that were scanned over large ω ranges, yielding highly isomorphous and highly redundant intensity measurements.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

MIGHTEE: total intensity radio continuum imaging and the COSMOS/XMM-LSS Early Science fields

ABSTRACT MIGHTEE is a galaxy evolution survey using simultaneous radio continuum, spectropolarimetry, and spectral line observations from the South African MeerKAT telescope. When complete, the survey will image ∼20 deg2 over the COSMOS, E-CDFS, ELAIS-S1, and XMM-Newton Large Scale Structure field (XMM-LSS) extragalactic deep fields with a central frequency of 1284 MHz. These were selected based on the extensive multiwavelength data sets from numerous existing and forthcoming observational campaigns. Here, we describe and validate the data processing strategy for the total intensity continuum aspect of MIGHTEE, using a single deep pointing in COSMOS (1.6 deg2) and a three-pointing mosaic in XMM-LSS (3.5 deg2). The processing includes the correction of direction-dependent effects, and results in thermal noise levels below 2 $\mathrm{\mu }$Jy beam−1 in both fields, limited in the central regions by classical confusion at ∼8 arcsec angular resolution, and meeting the survey specifications. We also produce images at ∼5 arcsec resolution that are ∼3 times shallower. The resulting image products form the basis of the Early Science continuum data release for MIGHTEE. From these images we extract catalogues containing 9896 and 20 274 radio components in COSMOS and XMM-LSS, respectively. We also process a close-packed mosaic of 14 additional pointings in COSMOS and use these in conjunction with the Early Science pointing to investigate methods for primary beam correction of broad-band radio images, an analysis that is of relevance to all full-band MeerKAT continuum observations, and wide-field interferometric imaging in general. A public release of the MIGHTEE Early Science continuum data products accompanies this article.

Heywood, I. (ORCID:0000000168645057)↗
DOE OSTI2021

High-Accuracy Module Emulators from Physically-Constrained AI Algorithms

Focal Area(s): How do we use AI tools to integrate observations, simulated data and physical and chemical fundamentals (Focal Area 3) into model components (Focal Area 2) that have high accuracy and stability and low computational burden to improve Earth System Predictability? Science Challenge: Earth system modeling of the hydrological cycle involves compute-intensive modules representing complex chemical and physical process. Recently, AI tools that are far less compute intensive have been developed that emulate these modules, but many of these efforts are not yet sufficiently accurate or even stable. We know a lot about the physics and chemistry of earth system processes. The Science Challenge is developing AI tools that not only incorporate observations and simulated data, but also incorporate the physics and chemistry of the process, while still maintaining the compute efficiency.

54 ENVIRONMENTAL SCIENCES↗