Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Real-time plasma monitoring framework for advanced plasma control and ML-research in DIII-D

Real-time and adaptive plasma control is crucial for robust tokamak operation, requiring sensitivity and tolerance measurements of the plasma state. This paper presents the implementation of an integrated real-time plasma monitoring framework on the DIII-D tokamak to support advanced control approaches, including machine-learning (ML) methods. The system is built on the SHIELD framework, a high-performance modular architecture that provides a unified pipeline for integrating diverse diagnostics. The framework leverages high-bandwidth digitizers, fast numerical processing, and deterministic, low-latency interconnects to stream high-fidelity data from diagnostics such as electron cyclotron emission (ECE), beam emission spectroscopy (BES), CO interferometers, and a visible tangential divertor camera (TangTV). The system’s validity is demonstrated through direct comparisons of real-time and offline data. Furthermore, we present two key applications of the developed plasma monitoring system with ML-based plasma control strategies, including real-time divertor detachment and active Alfvén Eigenmode control. As a result, this work presents a robust and scalable approach for integrating high-frequency, multidimensional diagnostics into advanced control algorithms for future fusion devices.

AI/ML↗
DOE OSTI2025

Subject-specific modeling framework for particle deposition using computational fluid dynamics

Quantifying particle deposition and dose in the respiratory tract requires a physiologically realistic representation and reproducible computational workflows. However, existing modeling frameworks, such as the International Commission on Radiological Protection (ICRP) compartmental models and the Multiple Path Particle Dosimetry (MPPD) tool, lack detailed deposition profiles and subject-specific capabilities. The combination of advances in computer vision algorithms applied to the respiratory tract and Computational Fluid and Particle Dynamics (CFPD) allows high-fidelity simulations of particle behavior in anatomically accurate geometries derived from individual CT scans. The segmentation, preprocessing, and file preparation task for a CFPD simulation was often time-consuming, and no prior studies to-date have yet presented a fully automated framework. This work presents a fully automated workflow to obtain individualized particle deposition profiles in the human respiratory tract. The pipeline starts with segmenting upper and lower airway geometries using morphological and deep learning-based methods, generating three-dimensional (3D) models from CT imaging data. Next, a series of algorithms are presented to quality check and prepare the 3D geometry for a CFD or CFPD simulation. The preprocessing step includes correcting geometric artifacts, enforcing a physically consistent mesh, and automatically identifying and capping multiple outlets, which is required for CFD/CFPD simulations. These processed models are then input into open-source (OpenFOAM) or commercial (StarCCM+) CFD solvers, where flow and transient particle transport equations — including turbulence and particle–wall interactions are solved under realistic breathing conditions. Finally, the resulting particle deposition profiles can be integrated with Monte Carlo radiation transport codes and state-of-the-art computational phantoms to assess organ-specific absorbed doses in scenarios of radioactive aerosol inhalation. The presented work streamlines respiratory tract segmentation, preprocessing for CFD/CFPD simulations, and integration with dose assessment workflows, reducing manual intervention and improving access to high-fidelity, subject-specific modeling. The high precision in predicted particle deposition and dose distributions can improve personalized treatment strategies in respiratory medicine and refine dose estimates for radiation protection.

AI↗
DOE OSTI2024

VIBES: a workflow for annotating and visualizing viral sequences integrated into bacterial genomes

Abstract Bacteriophages are viruses that infect bacteria. Many bacteriophages integrate their genomes into the bacterial chromosome and become prophages. Prophages may substantially burden or benefit host bacteria fitness, acting in some cases as parasites and in others as mutualists. Some prophages have been demonstrated to increase host virulence. The increasing ease of bacterial genome sequencing provides an opportunity to deeply explore prophage prevalence and insertion sites. Here we present VIBES (Viral Integrations in Bacterial genomES), a workflow intended to automate prophage annotation in complete bacterial genome sequences. VIBES provides additional context to prophage annotations by annotating bacterial genes and viral proteins in user-provided bacterial and viral genomes. The VIBES pipeline is implemented as a Nextflow-driven workflow, providing a simple, unified interface for execution on local, cluster and cloud computing environments. For each step of the pipeline, a container including all necessary software dependencies is provided. VIBES produces results in simple tab-separated format and generates intuitive and interactive visualizations for data exploration. Despite VIBES’s primary emphasis on prophage annotation, its generic alignment-based design allows it to be deployed as a general-purpose sequence similarity search manager. We demonstrate the utility of the VIBES prophage annotation workflow by searching for 178 Pf phage genomes across 1072 Pseudomonas spp. genomes.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

OmniXAS: A universal deep-learning framework for materials x-ray absorption spectra

X-ray absorption spectroscopy (XAS) is a powerful characterization technique for probing the local chemical environment of absorbing atoms. However, analyzing XAS data presents significant challenges, often requiring extensive, computationally intensive simulations, as well as significant domain expertise. These limitations hinder the development of fast, robust XAS analysis pipelines that are essential in high-throughput studies and for autonomous experimentation. Here, we address these challenges with OmniXAS, a framework that contains a suite of transfer learning approaches for XAS prediction, each uniquely contributing to improved accuracy and efficiency, as demonstrated on the K-edge spectra database covering eight 3⁢d transition metals (Ti–Cu). The OmniXAS framework is built upon three distinct strategies. First, we use M3GNet [Nat. Comput. Sci. 2, 718 (2022)] to derive latent representations of the local chemical environment of absorption sites as input for XAS prediction, achieving significant improvements over conventional featurization techniques. Second, we employ a hierarchical transfer learning strategy, training a universal multitask model across elements before fine-tuning for element-specific predictions. Models based on this cascaded approach after elementwise fine-tuning outperform element-specific models by up to 69%. Third, we implement cross-fidelity transfer learning, adapting a universal model to predict spectra generated by simulation of a different fidelity with a much higher computational cost. This approach improves prediction accuracy by up to 11% over models trained on the target fidelity alone. Our approach significantly boosts the throughput of XAS modeling by orders of magnitude as compared to first-principles simulations and is extendable to XAS prediction for a broader range of elements. The proposed transfer learning framework is generalizable to enhance deep-learning models that target other properties in materials research.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Path Integral Monte Carlo Simulations of Iron Plasmas (Final Technical Report)

This documents is the final technical report for our grant entitled "Path Integral Monte Carlo Simulations of Iron Plasmas" that focused on developing path integral Monte Carlo (PIMC) computer simulations. This techniques will be developed to study plasmas composed of heavier elements including iron and other third row elements. Equations of state (EOS) and transport properties will be derived in the regime of warm dense matter (WDM) and dense plasmas where existing first-principles methods cannot be applied. While standard density functional theory (DFT) has been used to accurately predict the structure of many solids and liquids up to temperatures on the order of 100,000 K, this method is not applicable at much higher temperature because the number of partially occupied electronic orbitals reaches intractably large numbers or the use of finite-temperature free energy functionals in orbital-free DFT introduces an uncontrolled approximation. Here we focus on PIMC methods that become more and more efficient with increasing temperatures and still include all electronic correlation effects. In this approach, electronic excitations increase the efficiency rather than reduce it. While it had commonly been assumed this method could only be applied to elements without core electrons, we showed that PIMC with free-particle nodes works well for first-row elements (PRL 108 (2012) 115502). Most recently, we extended the applicability range of all-electron PIMC to second-row elements by adopting localized nodal surfaces (PRL 115 (2015) 176403). To simulate third-row elements efficiently under WDM conditions, we propose a new method to remove core electrons by introducing pseudo-nodes. We explain our approach step by step and present preliminary results. We focus our method development on getting PIMC simulations of iron to work because of its fundamental importance for WDM and astrophysics. Then we move on to krypton and copper-doped beryllium, a ICF ablator material. We plan to continue working on key second-row material such as Na, Mg, MgO, Al, silica, and silicon-doped plastic ablators. Our collaborators at LLNL, will use our PIMC EOS data both as comparisons to existing semi-empirical, EOS-generating schemes, and as input for continuum radiation hydrodynamics simulations. We will establish an efficient pipeline from PIMC to macroscopic continuum studies of materials response. An emphasis will be placed on benchmarking such methods for plasmas of heavy elements at the very high temperatures (~100 eV) and low densities that are generated when Hohlraum radiation heats the ablator material in indirect drive laser experiments. Results from changes to the EOS will be of immeasurable importance to the designers at the National Ignition Facility (NIF) and at other facilities. Starting with our EOS of Cu-doped Be, our second collaborator at LLE, will perform real-time simulations of laser fusion experiments at the Omega laser and at the NIF to determine how sensitive the compression path depends on the ablator EOS. Since our collaborator also has experience in performing orbital-free DFT calculations, we propose to compare predictions from this method with PIMC results. In joint publications, we plan to analyze the accuracy of different free-energy functionals in order to understand why existing orbital-free DFT calculations do not predict compression peaks along the shock Hugoniot curve that we see with PIMC. The peaks are caused by the ionization of various electron shells. Their accurate characterization is important to compare with experimental results. We will break new ground by developing PIMC techniques that can simulate iron and all other third row elements in the plasma and WDM regimes. We introduce the concept of pseudo-nodes for the efficient treatment of the core-electrons. The EOS and transport properties will be derived and published online in the form of a new WDM database. Our PIMC EOS calculations will benchmark and possibly replace semi-analytical EOS tables like QEOS or SESAME, which will impact the hydrocode simulation community and will affect the design of NIF targets. Our PIMC results will help to improve the accuracy of orbital-free DFT simulations.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

A Geodatabase Designed to Inform and Support Safe CO2 Transport-Route Planning

The National Energy Technology Laboratory developed the Carbon Capture and Storage (CCS) Pipeline Route Planning Database to inform safe and sustainable CO2 transport planning in support of decarbonization efforts. This comprehensive Esri Geodatabase contains over 90 GBs of data and 60+ spatial layers representing key considerations including natural hazards, infrastructure, energy, and social justice. Leveraging ArcGIS Pro, nationwide raster, and vector datasets containing millions of features were processed to be easily digestible for users and complex modeling software.

Romeo, Lucy↗
DOE OSTI2021

Artificial Judgement Assistance from teXt (AJAX): Applying Open Domain Question Answering to Nuclear Non-proliferation Analysis

Nuclear non-proliferation analysis is complex and subjective, as the data is sparse, and examples are rare and diverse. While analysing non-proliferation data, it is often desired that the findings be completely auditable such that any claim or assertion can be sourced directly to the reference material from which it was derived. Currently this is accomplished by analysts thoroughly documenting underlying assumptions and clearly referencing details to source documents. This is a labour-intensive and time-consuming process that can be difficult to scale with geometrically increasing quantities of data. In this work, we describe an approach to leverage bi-directional language models for nuclear non-proliferation analysis. It has been shown recently that these models not only capture language syntax but also some of the relational knowledge present in the training data. We have devised a unique Salt and Pepper strategy for testing the knowledge present in the language models, while also introducing auditability function in our pipeline. We demonstrate that fine-tuning the bi-directional language models on domain specific corpus improves their ability to answer domain-specific factoid questions. Our hope is that the results presented in this paper will further the natural language processing (NLP) field by introducing the ability to audit the answers provided by the language models to bring forward the source of said knowledge.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2023

Propagating Uncertainties in the SALT3 Model-training Process to Cosmological Constraints

Type Ia supernovae (SNe Ia) are standardizable candles that must be modeled empirically to yield cosmological constraints. To understand the robustness of this modeling to variations in the model-training procedure, we build an end-to-end pipeline to test the recently developed SALT3 model. We explore the consequences of removing pre-2000s low-z or poorly calibrated U-band data, adjusting the amount and fidelity of SN Ia spectra, and using a model-independent framework to simulate the training data. We find that the SALT3 model surfaces are improved by having additional spectra and U-band data, and can be shifted by ~5% if host-galaxy contamination is not sufficiently removed from SN spectra. We find that resulting measurements of w are consistent to within 2.5% for all of the training variants explored in this work, with the largest shifts coming from variants that add color-dependent calibration offsets or host-galaxy contamination to the training spectra and those that remove pre-2000s low-z data. These results demonstrate that the SALT3 model-training procedure is largely robust to reasonable variations in the training data, but that additional attention must be paid to the treatment of spectroscopic data in the training process. We also find that the training procedure is sensitive to the color distributions of the input data—the resulting w measurement can be biased by ~2% if the color distribution is not sufficiently wide. Future low-z data, particularly u-band observations and high signal-to-noise ratio SN Ia spectra, will help to significantly improve SN Ia modeling in the coming years.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

PROTOCALC, a W -band Polarized Calibrator for Cosmic Microwave Background Telescopes: Application to Simons Observatory and CLASS

Current- and next-generation cosmic microwave background (CMB) experiments will measure polarization anisotropies with unprecedented sensitivities. The need for high precision in these measurements underscores the importance of gaining a comprehensive understanding of instrument properties, with a particular emphasis on the study of the beam properties, and especially their polarization characteristics and the measurement of the polarization angle. In this context, a major challenge lies in the scarcity of millimeter polarized astrophysical sources with sufficient brightness and calibration knowledge to meet the stringent accuracy requirements of future CMB missions. This led to the development of a drone-borne calibration source designed for the frequency band centered on approximately 90 GHz, matching a commonly used channel in ground-based CMB measurements. The Prototype Calibrator for Cosmology, PROTOCALC, has undergone thorough in-lab testing, and its properties have been subsequently modeled through simulation software integrated into the standard Simons Observatory analysis pipeline. Moreover, the PROTOCALC system has been tested in the field, having been deployed twice on calibration campaigns with CMB telescopes in the Atacama Desert. The data collected constrain the roll angle of the source with a statistical accuracy of 0$^°_•$045.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

The Geothermal Data Repository: Ten Years of Supporting the Geothermal Industry with Open Access to Geothermal Data: Preprint

The Department of Energy's (DOE) Geothermal Data Repository (GDR) is celebrating its tenth anniversary! Over the last decade it has grown from the simple idea of storing public data in a centralized location to a valuable tool at the center of the US geothermal scientific community and an integral part of the DOE Geothermal Technologies Office (DOE GTO) project management strategy. Researchers funded by the DOE GTO have contributed over 1,300 data submissions to the GDR. These data have been used to further advancements in geothermal science, economic analysis, exploration, research, development, and operational efficiency. The adoption of open data methodologies and a data management strategy that prioritizes universal open access and standardized, interoperable data have further increased the value of GDR data, making them available across a distributed network of data sharing partners and improving their utility to other industries and related fields, including material science and space exploration. Incorporating feedback from users has been critical to the GDRs success, allowing it to grow over the years to meet the evolving needs of the geothermal community. This paper will explore some of many changes that occurred throughout the GDRs tenure and the lessons learned along the way, as well as highlight some of the new features and recent improvements that been implemented to support innovation, reduce duplication of effort, and advance the geothermal industry as a whole.

accessibility↗
DOE OSTI2023

Automatic detection of ship-induced cloud features in satellite imagery

Ships crossing the ocean are known to produce long, curvilinear features called ship tracks visible in satellite imagery via the Twomey effect; however, there has been little exploitation of satellite imagery for broad atmospheric studies or global monitoring of ship emissions due to the difficulty of automated ship track detection. Prior studies are either proof-of-concept, qualitatively assessed, or restricted to a certain time of day. We propose a statistical method for the automated identification of ship tracks and demonstrate using GOES-West ABI data. We first present a human-assisted segmentation method, which we use to generate a ground truth data set of 529 annotated ship tracks in GOES-West ABI products. We then describe a two-stage automated approach comprising a detection stage to generate ship track proposals and a classification stage to reduce false positives. For detection, we present a novel pipeline based around a z-score filtering technique, and for classification, we demonstrate several classifiers from literature. In a final experiment, we quantitatively tune the detection parameters and train the classifier using the ground truth dataset, then test on a sequestered set of images; the detect-then-classify system had an overall Pd of 0.68 and 0.80 for daytime and nighttime data, respectively, and the classifier reduced false positive detections by 67% and 75%.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

DESI DR1 Ly α 1D power spectrum: Validation of estimators

The Data Release 1 (DR1) of the Dark Energy Spectroscopic Instrument (DESI) is the largest sample to date for small-scale Lyα forest cosmology, accessed through its one-dimensional power spectrum (P 1D ). The Lyα forest P 1D is extracted from quasar spectra that are highly inhomogeneous (both in wavelength and between quasars) in noise properties due to intrinsic properties of the quasar, atmospheric and astrophysical contamination, and also sensitive to low-level details of the spectral extraction pipeline. We employ two estimators in DR1 analysis to measure P 1D : the optimal estimator and the fast Fourier transform (FFT) estimator. To ensure robustness of our DR1 measurements, we validate these two power spectrum and covariance matrix estimation methodologies against the challenging aspects of the data. First, using a set of 20 synthetic 1D realizations of DR1, we derive the masking bias corrections needed for the FFT estimator and the continuum fitting bias needed for both estimators. We demonstrate that both estimators, including their covariances, are unbiased with these corrections using the Kolmogorov-Smirnov test. Second, we substantially extend our previous suite of CCD image simulations to include 675,000 quasars, allowing us to accurately quantify the pipeline's performance. This set of simulations reveals biases at the highest k values, corresponding to a resolution error of a few percent. We base the resolution systematics error budget of DR1 P 1D on these values, but do not derive corrections from them since the simulation fidelity is insufficient for precise corrections.

Lyman alpha forest↗
DOE OSTI2024

Optimization of foreground moment deprojection for semi-blind CMB polarization reconstruction

Abstract Upcoming Cosmic Microwave Background (CMB) experiments, aimed at measuring primordial CMB polarization B-modes, require exquisite control of instrumental systematics and Galactic foreground contamination. Blind minimum-variance techniques, like the Needlet Internal Linear Combination (NILC), have proven effective in reconstructing the CMB polarization signal and mitigating foregrounds and systematics across diverse sky models without suffering from foreground mismodelling errors. Still, residual foreground contamination from NILC may bias the recovered CMB polarization at large angular scales when confronted with the most complex foreground scenarios.By adding constraints to NILC to deproject statistical moments of the Galactic emission, the Constrained Moment ILC (cMILC) method has been demonstrated to further enhance foreground subtraction, albeit with an associated increase in overall noise variance. Faced with this trade-off between foreground bias reduction and overall variance minimization, there is still no recipe on which moments to deproject and which are better suited for blind variance minimization. To address this, we introduce the optimized cMILC (ocMILC) pipeline, which performs full automated optimization of the required number and set of foreground moments to deproject, pivot parameter values, and deprojection coefficients across the sky and angular scales, depending on the actual sky complexity, available frequency coverage, and experiment sensitivity. The optimal number of moments for deprojection, before paying significant noise penalty, is determined through a data diagnosis inspired by the Generalized NILC (GNILC) method.Validated on B-mode simulations of thePICOspace mission concept with four challenging foreground models, ocMILC exhibits lower Galactic foreground contamination compared to NILC and cMILC at all angular scales, with limited noise penalty. This multi-layer optimization enables the ocMILC pipeline to achieve unbiased posteriors of the tensor-to-scalar ratio, regardless of foreground complexity.

Astronomy & Astrophysics↗
DOE OSTI2020

Reducing Communication in Graph Neural Network Training

Graph Neural Networks (GNNs) are powerful and flexible neural networks that use the naturally sparse connectivity information of the data. GNNs represent this connectivity as sparse matrices, which have lower arithmetic intensity and thus higher communication costs compared to dense matrices, making GNNs harder to scale to high concurrencies than convolutional or fully-connected neural networks. Here, we introduce a family of parallel algorithms for training GNNs and show that they can asymptotically reduce communication compared to previous parallel GNN training methods. We implement these algorithms, which are based on 1D, 1. 5D, 2D, and 3D sparse-dense matrix multiplication, using torch.distributed on GPU-equipped clusters. Our algorithms optimize communication across the full GNN training pipeline. We train GNNs on over a hundred GPUs on multiple datasets, including a protein network with over a billion edges.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Expanding the Domain of Applicability of Machine Learning Models with Limited Data for Drug Property Prediction

Accurate machine learning models for predicting small molecule interactions with biological targets are essential for therapeutic discovery, biothreat response, and computational drug design, but their performance is often limited for understudied targets with sparse experimental data. To address this challenge, we developed and evaluated methods to improve molecular property prediction under low-data conditions, using the NimA-related kinase (NEK) family as a proof-of-concept. This work focused on two complementary goals within the ATOM Modeling PipeLine (AMPL) and the Generative Molecular Design (GMD) loop: expanding model applicability through transfer learning, representation learning, feature scaling, sampling strategies, and active-learning-inspired compound selection; and enabling efficient virtual screening to prioritize compounds that balance predicted activity, design objectives, and synthetic accessibility.

organic↗
DOE OSTI2022

Shear Measurement with Poorly Resolved Images

Weak lensing studies typically require excellent seeing conditions for the purpose of maximizing the number density of well-resolved galaxy images. It is interesting to ask to what extent the seeing size limits the usefulness of the astronomical images in weak lensing. In this work, we study this issue with the data of the DECam Legacy Survey, which is a part of the target selection program for the Dark Energy Spectroscopic Instrument. Using the Fourier_Quad shear measurement pipeline, we demonstrate that images with relatively poor seeing conditions (~1$^"_.$5) can still yield accurate shear estimators. We do not find any correlation between systematic shear error and the image resolution.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Gas-mapping 3D imager measurement techniques and method of data processing

Measurement approaches and data analysis methods are disclosed for combining 3D topographic data with spatially-registered gas concentration data to increase the efficiency of gas monitoring and leak detection tasks. Here, the metric for efficiency is defined as reducing the measurement time required to achieve the detection, or non-detection, of a gas leak with a desired confidence level. Methods are presented for localizing and quantifying detected gas leaks. Particular attention is paid to the combination of 3D spatial data with path-integrated gas concentration measurements acquired using remote gas sensing technologies, as this data can be used to determine the path-averaged gas concentration between the sensor and points in the measurement scene. Path-averaged gas concentration data is useful for finding and quantifying localized regions of elevated (or anomalous) gas concentration making it ideal for a variety of applications including: oil and gas pipeline monitoring, facility leak and emissions monitoring, and environmental monitoring.

Thorpe, Michael↗