Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,009 records · Page 56
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

XRIPL

XRIPL is a library of tools for processing x-ray radiographs and extracting contours of interest within the image using computer vision techniques. Features include spatial calibration, denoising, background and attenuation correction through pseudo-flatfielding, watershed segmentation, contour processing, and visualization. The details of the XRIPL analysis pipeline are published in the Proceedings of the 23rd Topical Conference on High-Temperature Plasma Diagnostics Proceedings (LA-UR-20-30133) [1]. [1] P. M. Kozlowski, Y. Kim, B. M. Haines, H. F. Robey, T. J. Murphy, H. M. Johns, and T. S. Perry. Use of Computer Vision for analysis of image data sets from high temperature plasma experiments. Review of Scientific Instruments 92, 033532 (2021) https://doi.org/10.1063/5.0040285

Kozlowski, Pawel↗
DOE OSTI2025

Firm Synthesizer and Supply-chain Simulator (SynthFirm) v2.0

SynthFirm is a national-scale agent-based freight demand model which generates a complete synthetic population of firms in the U.S. and the business-to-business commodity flows between them. Using publicly available data sources as inputs, SynthFirm simulates detailed firm and fleet characteristics, commodity production and consumption, formation of supply chains, and selection of shipping modes, all of which are essential drivers of commodity flow at a disaggregate level. The SynthFirm 2.0 version includes national commercial vehicle fleet generation, international trade simulation and automized model validation pipeline, which allows seemless deployment across the nation and build a comprehensive freight inventories at national scale or for selected region.

Yang, Hung-Chia [Lawrence Berkeley National Labora↗
DOE OSTI2022

PARETO 0.3.0 Release

PARETO 0.3.0 Release. Highlights: - Fixing Warning in get_data - Clean up of old main function which has been replaced by the run files - Adding plot_scatter() function - Support node capacity - Add Gurobi troubleshooting files to gitignore by @melody-shellman - Set slacks to zero - Water treatment case study - Fix defaultvalue treatement - Fix bug for incorrect number of input arguments in water quality - Pipeline configs documentation - Enable rendering Unicode characters in ReadTheDocs PDF builds - Adding support for units - Add discrete water quality to the model - Updating documentation for strategic model, and corresponding fixes to strategic model - Rework getting started docs build section and more - Enable Codecov reports in CI builds - Correct error in CompletionsPadSupplyBalanceRule

PARETO,PSE,Process Systems Engineering,Produced wa↗
DOE OSTI2023

Detecting and Characterizing Mg ii Absorption in DESI Survey Validation Quasar Spectra

Abstract We present findings of the detection of Magnesium II (Mg ii , λ = 2796, 2803 Å) absorbers from the early data release of the Dark Energy Spectroscopic Instrument (DESI). DESI is projected to obtain spectroscopy of approximately 3 million quasars (QSOs), of which over 99% are anticipated to be at redshifts greater than z > 0.3, such that DESI would be able to observe an associated or intervening Mg ii absorber illuminated by the background QSO. We have developed an autonomous supplementary spectral pipeline that detects these systems through an initial line-fitting process and then confirms the line properties using a Markov Chain Monte Carlo sampler. Based upon a visual inspection of the resulting systems, we estimate that this sample has a purity greater than 99%. We have also investigated the completeness of our sample in regard to both the signal-to-noise properties of the input spectra and the rest-frame equivalent width ( W 0 ) of the absorber systems. From a parent catalog containing 83,207 quasars, we detect a total of 23,921 Mg ii absorption systems following a series of quality cuts. Extrapolating from this occurrence rate of 28.8% implies a catalog at the completion of the five-year DESI survey that will contain over eight hundred thousand Mg ii absorbers. The cataloging of these systems will enable significant further research because they carry information regarding circumgalactic medium environments, the distribution of intervening galaxies, and the growth of metallicity across the redshift range 0.3 ≤ z < 2.5.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Learning dynamical systems from data: An introduction to physics-guided deep learning

Modeling complex physical dynamics is a fundamental task in science and engineering. Traditional physics-based models are first-principled, explainable, and sample-efficient. However, they often rely on strong modeling assumptions and expensive numerical integration, requiring significant computational resources and domain expertise. While deep learning (DL) provides efficient alternatives for modeling complex dynamics, they require a large amount of labeled training data. Furthermore, its predictions may disobey the governing physical laws and are difficult to interpret. Physics-guided DL aims to integrate first-principled physical knowledge into data-driven methods. It has the best of both worlds and is well equipped to better solve scientific problems. Recently, this field has gained great progress and has drawn considerable interest across discipline Here, we introduce the framework of physics-guided DL with a special emphasis on learning dynamical systems. We describe the learning pipeline and categorize state-of-the-art methods under this framework. We also offer our perspectives on the open challenges and emerging opportunities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Comparative Study of Large Language Model Architectures on Frontier

Large language models (LLMs) have garnered significant attention in both the AI community and beyond. Among these, the Generative Pre-trained Transformer (GPT) has emerged as the dominant architecture, spawning numerous variants. However, these variants have undergone pre-training under diverse conditions, including variations in input data, data preprocessing, and training methodologies, resulting in a lack of controlled comparative studies. Here we meticulously examine two prominent open-sourced GPT architectures, GPT-NeoX and LLaMA, leveraging the computational power of Frontier, the world’s first Exascale supercomputer. Employing the same materials science text corpus and a comprehensive end-to-end pipeline, we conduct a comparative analysis of their training and downstream performance. Our efforts culminate in achieving state-of-the-art performance on a challenging materials science benchmark. Furthermore, we investigate the computation and energy efficiency, and propose a computationally efficient method for architecture design. To our knowledge, these pre-trained models represent the largest available for materials science. Our findings provide practical guidance for building LLMs on HPC platforms.

Yin, Junqi↗
DOE OSTI2026

lumicap v0.1

Automated HDR luminance imaging system designed for daylighting research and building science. It controls a fisheye-lens camera to capture time-lapse bracket sequences, merges them into calibrated HDR images, and runs a full post-processing pipeline — all unattended. Features: - Scheduled LDR bracket capture via gphoto2 - HDR merging with vignetting, ND filter, and fisheye projection corrections - Illuminance and luminance meter integration (Konica Minolta T-10A, LS-100/150) - Daylight glare probability (DGP) and solar position computation - Automated false-color rendering, JPEG thumbnails, and daily time-lapse video - CSV data logging per timestep Uses: - Long-term monitoring of daylight conditions in buildings - Glare analysis for occupant comfort research - Solar irradiance and sky luminance studies Advantages: - End-to-end automation — capture, calibration, analysis, and archiving run without manual intervention - Built on the proven Radiance toolchain, ensuring photometrically accurate HDR output - Hardware-agnostic meter support via serial auto-detection - Lightweight — no GUI overhead, deployable on a headless Raspberry Pi or similar embedded system

Wang, Taoning [Lawrence Berkeley National Laborato↗
DOE OSTI2021

Modeling Study of Reduced Tubing Size Effects on Flow in Depleted Reservoir Natural Gas Storage Wells

This report explores the effects of tubing size reductions on natural gas flow from representative depleted reservoir underground storage wells and fields using basic models for coupled reservoir and pipe flow. This work was motivated by interest at the U.S. Department of Transportation, Pipeline and Hazardous Materials Safety Administration, in evaluating the effects of tubing and packer as a potential safety upgrade to implement double-barrier systems to existing underground natural gas storage wells. Reservoir and well flow models were developed from widely accepted industry equations, verified against a commercial process simulator model, and validated against field data. The study utilized U.S. operator survey data to provide context and assure that modeling parameters including aver age deliverability rates for wells and fields, operating pressures, well depths, and storage capacities were all carefully considered to keep the modeling relevant to the known range of U.S. operations. The models generally found that wells and fields with inherently low deliverability were relatively insensitive to reductions in tubing diameter, primarily because the hydraulics in those cases were controlled by reservoir properties. For the high-producing wells and fields, the models found that reducing tubing diameter could produce significant reductions in deliverability, both at the field- and well-level. When put into context with occurrence data regarding average deliverability of fields and wells, it appears that most wells and most fields across the U.S. would experience deliverability reductions on the low end of what was simulated here, generally below 20%. For fields with moderate to high deliverability, reductions were generally larger, and could reach as high as 60% for the highest-producing wells and fields.

03 NATURAL GAS↗
DOE OSTI2026

Bayesian Component Separation for DESI LAE Automated Spectroscopic Redshifts and Photometric Targeting

Lyα emitters (LAEs) are valuable high-redshift cosmological probes traditionally identified using specialized narrowband photometric surveys. In ground-based spectroscopy, it can be difficult to distinguish the sharp LAE peak from residual sky emission lines using automated methods, leading to misclassified redshifts. We present a Bayesian spectral component separation technique to automatically determine spectroscopic redshifts for LAEs while marginalizing over sky residuals. We use visually inspected spectra of LAEs obtained using the Dark Energy Spectroscopic Instrument (DESI) to create a data-driven prior and can determine redshift by jointly inferring sky residual, LAE, and residual components for each individual spectrum. We demonstrate this method on 881 spectroscopically observed z = 2–4 DESI LAE candidate spectra and determine their redshifts with >90% accuracy when validated against visually inspected redshifts. Using the Δχ$^{2}$ value from our pipeline as a proxy for detection confidence, we then explore potential survey design choices and implications for targeting LAEs with medium-band photometry. This method allows for scalability and accuracy in determining redshifts from DESI spectra, and the results provide recommendations for LAE targeting in anticipation of future high-redshift spectroscopic surveys.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

GenomeDepot v1.0

GenomeDepot is a web-based platform for annotation, management, and comparative analysis of microbial genomic sequences and associated data including ortholog families, protein domains, operons, regulatory interactions, strain taxonomy, and sample metadata. GenomeDepot supports rapid creation of web-sites for user-defined genome collections that include bioinformatic tools for interactive genome browsing, BLAST search, annotation search, comparative genomic neighborhood visualization, and sequence download. Gene function annotations are generated by a customizable annotation pipeline. The pipeline runs annotation tools in Conda environments and can be easily extended with additional user-specified tools.

Kazakov, Alexey [Lawrence Berkeley National Labora↗
DOE OSTI2021

A Search for L4 Earth Trojan Asteroids Using a Novel Track-before-detect Multiepoch Pipeline

We report Earth Trojan asteroids are an important but elusive population that co-orbit with Earth at the L4 and L5 Lagrange points. There is only one known, but a large population is theoretically stable and could provide insight into our solar system’s past and present, as well as planetary defense. In this paper, we present the results of an Earth Trojan survey that uses a novel shift-and-stack detection method on 2 nights of data from the Dark Energy Camera. We find no new Earth Trojan asteroids. We calculate an upper limit on the population that is consistent with previous searches despite much less sky coverage. Additionally, we elaborate on previous upper-limit calculations using current asteroid population statistics and an extensive asteroid simulation to provide the most up-to-date population constraints. We find an L4 Earth Trojan population of N ET < 1 for H = 13.93, N ET < 10 for H = 16, and N ET < 938 for H = 22.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Archetype-based Redshift Estimation for the Dark Energy Spectroscopic Instrument Survey

We present a computationally efficient galaxy archetype-based redshift estimation and spectral classification method for the Dark Energy Survey Instrument (DESI) survey. The DESI survey currently relies on a redshift fitter and spectral classifier using a linear combination of principal component analysis–derived templates, which is very efficient in processing large volumes of DESI spectra within a short time frame. However, this method occasionally yields unphysical model fits for galaxies and fails to adequately absorb calibration errors that may still be occasionally visible in the reduced spectra. Our proposed approach improves upon this existing method by refitting the spectra with carefully generated physical galaxy archetypes combined with additional terms designed to absorb data reduction defects and provide more physical models to the DESI spectra. We test our method on an extensive data set derived from the survey validation (SV) and Year 1 (Y1) data of DESI. Our findings indicate that the new method delivers marginally better redshift success for SV tiles while reducing catastrophic redshift failure by 10%–30%. At the same time, results from millions of targets from the main survey show that our model has relatively higher redshift success and purity rates (0.5%–0.8% higher) for galaxy targets while having similar success for QSOs. These improvements also demonstrate that the main DESI redshift pipeline is generally robust. Additionally, it reduces the false-positive redshift estimation by 5%–40% for sky fibers. We also discuss the generic nature of our method and how it can be extended to other large spectroscopic surveys, along with possible future improvements.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

A goldilocks computational protocol for inhibitor discovery targeting DNA damage responses including replication-repair functions

While many researchers can design knockdown and knockout methodologies to remove a gene product, this is mainly untrue for new chemical inhibitor designs that empower multifunctional DNA Damage Response (DDR) networks. Here, we present a robust Goldilocks (GL) computational discovery protocol to efficiently innovate inhibitor tools and preclinical drug candidates for cellular and structural biologists without requiring extensive virtual screen (VS) and chemical synthesis expertise. By computationally targeting DDR replication and repair proteins, we exemplify the identification of DDR target sites and compounds to probe cancer biology. Our GL pipeline integrates experimental and predicted structures to efficiently discover leads, allowing early-structure and early-testing (ESET) experiments by many laboratories. By employing an efficient VS protocol to examine protein-protein interfaces (PPIs) and allosteric interactions, we identify ligand binding sites beyond active sites, leveraging in silico advances for molecular docking and modeling to screen PPIs and multiple targets. A diverse 3,174 compound ESET library combines Diamond Light Source DSI-poised, Protein Data Bank fragments, and FDA-approved drugs to span relevant chemotypes and facilitate downstream hit evaluation efficiency for academic laboratories. Two VS per library and multiple ranked ligand binding poses enable target testing for several DDR targets. This GL library and protocol can thus strategically probe multiple DDR network targets and identify readily available compounds for early structural and activity testing to overcome bottlenecks that can limit timely breakthrough drug discoveries. By testing accessible compounds to dissect multi-functional DDRs and suggesting inhibitor mechanisms from initial docking, the GL approach may enable more groups to help accelerate discovery, suggest new sites and compounds for challenging targets including emerging biothreats and advance cancer biology for future precision medicine clinical trials.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

H-2-SALT: Storing Fossil Energy as Hydrogen in Salt Caverns

Hydrogen storage in subsurface caverns is known as perhaps largest and longest-term energy storage system. While few such systems are currently in service (e.g., Texas; Teesside, UK), they are envisioned as being a pillar of the current energy transition that will allow intermittent power production from renewable sources (e.g., wind, solar) to be balanced with demand at grid-scale. In addition, hydrogen energy storage can allow existing fossil power plants to run more economically either by minimizing startups/shutdowns or by allowing them to take advantage of arbitrage. Finally, stored hydrogen can feed a variety of non-power users such as pipeline gas, heating, transportation, and manufacturing. The H-2-SALT paper study assessed the feasibility of a power-to-hydrogen system utilizing salt cavern storage of hydrogen in bedded salt in central Kansas, where over 750 such caverns have been constructed to date, of which approximately 350 are still in service. Legacy well and salt cavern data were collected to develop a regional geological database for two sites in Central Kansas and adjacent areas for use in more detailed geological and geomechanical site characterization efforts. This study found commercial viability and competitiveness of large-scale, electrolytic H 2 production and storage ($1.78 per kg H 2 ) that can be used for both electrical power supply to the grid during high-priced electricity periods (including co-firing with NG and fuel cell power production) and sale of H 2 for various industries, such as petrochemicals or transportation. The commercial potential of the H-2-SALT system is also relatively high because each of its components operate commercially today. It is recommended that a stratigraphic test well be drilled to collect log data and core samples that can reduce uncertainty surrounding geological and geomechanical properties of the salt beds that will be required for further cavern and energy storage system design studies.

03 NATURAL GAS↗
DOE OSTI2022

Design-Space Exploration for Inverse-Design of Wind Turbine Blades Using Data-Driven Methods

The state-of-the-practice aerodynamic design methods for wind turbine blades is typically based on Blade-Element Momentum (BEM) theory using a pre-designed frozen family of airfoils. The airfoils are themselves typically designed using panel methods. The design of the next-generation of large flexible rotors will need to capture the non-linear aerodynamics and three-dimensional flow to reduce the levelized cost of wind energy. Data-driven methods for aerodynamic design using data generated by computational fluid dynamics offer an attractive alternative to BEM-based methods that captures the non-linear aerodynamics of the component airfoils as well as the root and tip sections. In this work, we develop and demonstrate a framework to "smartly" explore the relevant design space in combination with an appropriate automated CFD pipeline to evaluate the aerodynamics of each design. The design-space exploration framework uses appropriate perturbations to the airfoil shape and induction profile from a baseline shape in combination with the inverse-design using BEM. The resulting designs are evaluated using an automated CFD pipeline using the in-house CFD solver framework "Mercury". We perform verification and validation to establish the capability of the Mercury framework to predict the aerodynamic performance of wind turbines. The CFD simulation of the perturbed blade shapes are optimized to restart from the converged baseline simulation to reduce the computational time. Finally, we demonstrate the design-space exploration technique for the design of the outboard section and the full rotor using perturbations to the shape and operating conditions of the NREL 5-MW turbine.

aerodynamic design methods↗
DOE OSTI2026

Developing a complete AI-accelerated workflow for superconductor discovery

The quest to identify new superconducting materials with enhanced properties is hindered by the prohibitive cost of computing electron-phonon spectral functions, severely limiting the materials space that can be explored. Here, we introduce a Bootstrapped Ensemble of Equivariant Graph Neural Networks (BEE-NET), a machine-learning model trained to predict the Eliashberg spectral function and superconducting critical temperature with a mean-absolute-error of 0.87 K relative to DFT-based Allen-Dynes calculations. Intriguingly, BEE-NET achieves a true-negative-rate of 99.4%, enabling highly efficient screening for the rare property of superconductivity. Integrated into a multi-stage, AI-accelerated discovery pipeline that incorporates elemental-substitution strategies and machine-learned interatomic potentials, our workflow reduced over 1.3 million candidate structures to 741 dynamically and thermodynamically stable compounds with DFT-confirmed T c > 5 K. We report the successful synthesis and experimental confirmation of superconductivity in two of these previously unreported compounds. This study establishes a data-driven framework that integrates machine learning, quantum calculations, and experiments to systematically accelerate superconductor discovery.

Gibson, Jason B. [Quantum Formatics, Cambridge, MA↗
DOE OSTI2024

Force Field X: A computational microscope to study genetic variation and organic crystals using theory and experiment

Force Field X (FFX) is an open-source software package for atomic resolution modeling of genetic variants and organic crystals that leverages advanced potential energy functions and experimental data. FFX currently consists of nine modular packages with novel algorithms that include global optimization via a many-body expansion, acid–base chemistry using polarizable constant-pH molecular dynamics, estimation of free energy differences, generalized Kirkwood implicit solvent models, and many more. Applications of FFX focus on the use and development of a crystal structure prediction pipeline, biomolecular structure refinement against experimental datasets, and estimation of the thermodynamic effects of genetic variants on both proteins and nucleic acids. The use of Parallel Java and OpenMM combines to offer shared memory, message passing, and graphics processing unit parallelization for high performance simulations. Overall, the FFX platform serves as a computational microscope to study systems ranging from organic crystals to solvated biomolecular systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗