SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

All sources NASA NTRS DOE OSTI

DOE OSTI2024

PyPop: a mature open-source software pipeline for population genomics

Python for Population Genomics (PyPop) is a software package that processes genotype and allele data and performs large-scale population genetic analyses on highly polymorphic multi-locus genotype data. In particular, PyPop tests data conformity to Hardy-Weinberg equilibrium expectations, performs Ewens-Watterson tests for selection, estimates haplotype frequencies, measures linkage disequilibrium, and tests significance. Standardized means of performing these tests is key for contemporary studies of evolutionary biology and population genetics, and these tests are central to genetic studies of disease association as well. Here, we present PyPop 1.0.0, a new major release of the package, which implements new features using the more robust infrastructure of GitHub, and is distributed via the industry-standard Python Package Index. New features include implementation of the asymmetric linkage disequilibrium measures and, of particular interest to the immunogenetics research communities, support for modern nomenclature, including colon-delimited allele names, and improvements to meta-analysis features for aggregating outputs for multiple populations.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Fragment-based determination of a proteinase K structure from MicroED data using ARCIMBOLDO_SHREDDER

Structure determination of novel biological macromolecules by X-ray crystallography can be facilitated by the use of small structural fragments, some of only a few residues in length, as effective search models for molecular replacement to overcome the phase problem. Independence from the need for a complete pre-existing model with sequence similarity to the crystallized molecule is the primary appeal of ARCIMBOLDO , a suite of programs which employs this ab initio algorithm for phase determination. Here, the use of ARCIMBOLDO is investigated to overcome the phase problem with the electron cryomicroscopy (cryoEM) method known as microcrystal electron diffraction (MicroED). The results support the use of the ARCIMBOLDO_SHREDDER pipeline to provide phasing solutions for a structure of proteinase K from 1.6 Å resolution data using model fragments derived from the structures of proteins sharing a sequence identity of as low as 20%. ARCIMBOLDO_SHREDDER identified the most accurate polyalanine fragments from a set of distantly related sequence homologues. Alternatively, such templates were extracted in spherical volumes and given internal degrees of freedom to refine towards the target structure. Both modes relied on the rotation function in Phaser to identify or refine fragment models and its translation function to place them. Model completion from the placed fragments proceeded through phase combination of partial solutions and/or density modification and main-chain autotracing using SHELXE . The combined set of fragments was sufficient to arrive at a solution that resembled that determined by conventional molecular replacement using the known target structure as a search model. This approach obviates the need for a single, complete and highly accurate search model when phasing MicroED data, and permits the evaluation of large fragment libraries for this purpose.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Processing Full-Scale Square Kilometre Array Data on the Summit Supercomputer

This work presents a workflow for simulating and processing the full-scale low-frequency telescope data of the Square Kilometre Array (SKA) Phase 1. The SKA project will enter the construction phase soon, and once completed, it will be the world’s largest radio telescope and one of the world’s largest data generators. The authors used Summit to mimic an endto-end SKA workflow, simulating a dataset of a typical 6 hour observation and then processing that dataset with an imaging pipeline. This workflow was deployed and run on 4,560 compute nodes, and used 27,360 GPUs to generate 2.6 PB of data. This was the first time that radio astronomical data were processed at this scale. Results show that the workflow has the capability to process one of the key SKA science cases, an Epoch of Reionization observation. This analysis also helps reveal critical design factors for the next-generation radio telescopes and the required dedicated processing facilities.

Wang, Ruonan↗

DOE OSTI2023

Plan for a Data Base Approach to Evaluating LWR Plant Integration with Hydrogen and Industrial Facilities

The purpose of this report is to provide an outline and scope of a database tool that could be used to identify the most beneficial locations to site a hydrogen, chemical facility or other industrial facility that would be powered by nuclear energy. This energy may be in the form of electricity produced by the plant, heat in the form of steam or both. In addition, this siting tool should consider factors such as economics, geography, industries, residences, geological hydrogen storage, and key infrastructure. The infrastructure to be considered could include pipelines or other storage and transportation facilities to carry the created products from the production facility to the end user. Also considered should be infrastructure that could deliver needed chemicals or gasses to allow the production of desired products such as ammonia or biofuels. This data base tool should allow a screening study of all U.S. nuclear plants to identify opportunities and barriers to creating hydrogen hubs around nuclear power plants. For this effort, it has been decided that the Siting Tool for Advanced Nuclear Development (STAND) would be used to conduct this screening study which is a follow-on research project to this present effort. Currently, plans are being developed to modify the STAND tool to provide the capability necessary to perform screening of each US commercial nuclear plant to identify the best locations to site hydrogen or chemical production facilities. Many features of the current STAND tool will be employed to meet this objective, but additional infrastructure data (such as the location of pipelines) may need to be added to allow these locations to be fully vetted. In addition, new analysis capabilities may need to be added so economic considerations can be understood

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2023

Using low-coverage whole genome sequencing (genome skimming) to delineate three introgressed species of buffalofish ( Ictiobus )

Consumption of buffalofish has been sporadically associated with Haff disease-like illnesses involving sudden onset muscle pain and weakness due to skeletal muscle rhabdomyolysis, but determination of precisely which species are associated with these illnesses has been impeded by a lack of species-specific DNA-based markers. Here, three closely related species of buffalofish native to the Mississippi River Basin (Ictiobus bubalus, Ictiobus cyprinellus and Ictiobus niger) that have previously proven genetically indistinguishable using both mitochondrial and nuclear single-locus sequencing were reliably discriminated using low-coverage whole genome sequencing (‘genome skimming’). Using 44 specimens representing the three species collected from the mid/upper (Missouri) and lower (Louisiana) regions of the species’ native ranges, the SISRS (Site Identification from Short Read Sequences) bioinformatics pipeline was adapted to (1) identify over 620Mbp of putatively homologous nuclear sequence data and (2) isolate over 140,000 single-nucleotide polymorphisms (SNPs) that supported accurate species delimitation, all without the use of a reference genome or annotation data. These sites were used to classify Ictiobus spp. samples with genome-skim data, along with a larger set (n = 67) where ultraconserved elements (UCEs) were sequenced. Analyses of whole mitochondrial data revealed more limited signal. Nearly all samples matched their purported species based on morphologic identification, but two Missouri samples morphologically identified as I. niger grouped with samples of I. bubalus, albeit with significant enrichment of I. niger SNPs. To our knowledge this is the first report of a DNA-based tool to reliably discriminate these three morphologically distinct species.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Disturbed, diffuse, or just missing? A global study of the H I content of Hickson compact groups

Hickson compact groups (HCGs) are dense configurations of four to ten galaxies, whose H I morphology appears to follow an evolutionary sequence of three phases, with gas initially confined to galaxies, then significant amounts spread throughout the intra-group medium, and finally with almost no gas remaining in the galaxies themselves. It has also been suggested that several groups may harbour a diffuse H I component that is resolved out by interferometric observations. The H I deficiency of HCGs is expected to increase as the H I morphological phase progresses along the evolutionary sequence. If this is the case, H I deficiency would be a rough proxy for the age and evolutionary state of a HCG. We aim to test this hypothesis for the first time using a large sample of HCGs and to investigate the evidence for diffuse H I in HCGs. We performed a uniform reduction of all publicly available VLA H I observations (38 HCGs) with a purpose-built pipeline that also maximises the reproducibility of this study. The resulting H I data cubes were then analysed with the latest software tools to perform a manual separation of emission features into those belonging to galaxies and those extending into the intra-group medium. We thereby classified the H I morphological phase of each group as well as quantified their H I deficiency compared to galaxies in isolation. We find little evidence that H I deficiency can be used as a proxy for the evolutionary phase of a compact group in either of the first two phases, with the distribution of H I deficiency being consistent in both. However, for the final phase, the distribution clearly shifts to high H I deficiencies, with more than 90% of the expected H I content typically missing. Across all HCGs studied, we identify a few cases where there is strong evidence for a diffuse gas component in the intra-group medium, which might be detectable with improved observations. We also classify a new sub-phase where groups contain a lone H I -bearing galaxy, but are otherwise devoid of gas. The new morphological phase we have identified is likely the result of an evolved, gas-poor group acquiring a new, gas-rich member. The large spread of H I deficiencies in the first two morphological phases suggests that there is a broad range of initial H I content in HCGs, which is perhaps influenced by large-scale environment, and that the timescale for morphological changes is, in general, considerably shorter than the timescale for the destruction or consumption of neutral gas in these systems.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

Fusion of Experiments and Simulations for Real-Time Identification of Pipeline Defects

In this study, we explored fusion of experiments and simulations for real time identification of pipeline defects across physical and non-physical domains. The challenges associated to data processing were addressed and a combined classification models was presented via CNN models. In addition, regression model based on XGBOOST is built to determine the defect location and defect dimension from data-driven features of guided wave signals captured by SMS fiber optic sensor.

deep learning↗

DOE OSTI2020

APOGEE Data and Spectral Analysis from SDSS Data Release 16: Seven Years of Observations Including First Results from APOGEE-South

The spectral analysis and data products in Data Release 16 (DR16; 2019 December) from the high-resolution near-infrared Apache Point Observatory Galactic Evolution Experiment (APOGEE)-2/Sloan Digital Sky Survey (SDSS)-IV survey are described. Compared to the previous APOGEE data release (DR14; 2017 July), APOGEE DR16 includes about 200,000 new stellar spectra, of which 100,000 are from a new southern APOGEE instrument mounted on the 2.5 m du Pont telescope at Las Campanas Observatory in Chile. DR16 includes all data taken up to 2018 August, including data released in previous data releases. All of the data have been re-reduced and re-analyzed using the latest pipelines, resulting in a total of 473,307 spectra of 437,445 stars. Changes to the analysis methods for this release include, but are not limited to, the use of MARCS model atmospheres for calculation of the entire main grid of synthetic spectra used in the analysis, a new method for filling “holes” in the grids due to unconverged model atmospheres, and a new scheme for continuum normalization. Abundances of the neutron-capture element Ce are included for the first time. A new scheme for estimating uncertainties of the derived quantities using stars with multiple observations has been applied, and calibrated values of surface gravities for dwarf stars are now supplied. Compared to DR14, the radial velocities derived for this release more closely match those in the Gaia DR2 database, and a clear improvement in the spectral analysis of the coolest giants can be seen. The reduced spectra as well as the result of the analysis can be downloaded using links provided on the SDSS DR16 web page.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

Long-term measurements of ice nucleating particles at Atmospheric Radiation Measurement (ARM) sites worldwide

Ice nucleating particles (INPs) play a critical role in cloud microphysics and precipitation formation, yet long-term, spatially extensive observational datasets remain limited. Here, we present one of the most comprehensive publicly available datasets of immersion-mode INP concentrations using a single analytical method, generated through the U.S. Department of Energy's (DOE) Atmospheric Radiation Measurement (ARM) user facility. INP filter samples have been collected across a broad range of environments – including agricultural plains, Arctic coastlines, high-elevation mountain sites, marine regions, and urban areas – via fixed observatories, mobile facility deployments, and vertically-resolved tethered balloon system operations. We describe the standardized processing and quality assurance pipeline, from filter collection and processing using the Ice Nucleation Spectrometer to final data products archived on the ARM Data Discovery portal. The dataset includes both total INP concentrations and selectively treated samples, allowing for classification of biological, organic, and inorganic INP types. It features a continuous 5-year record of INP measurements from a central U.S. site, with data collection still ongoing. Seasonal and site-specific differences in INP concentrations are illustrated through intercomparisons at −10 and −20 °C, revealing distinct regional sources and atmospheric drivers. We also outline mechanisms for researchers to access existing data, request additional sample analyses, and propose future field campaigns involving ARM INP measurements. This dataset supports a wide range of scientific applications, from observational and mechanistic studies to model development, and provides critical constraints on aerosol-cloud interactions across diverse atmospheric regimes (Creamean et al., 2024, 2020b; https://doi.org/10.5439/1770816).

Creamean, Jessie M. [Colorado State Univ., Fort Co↗

DOE OSTI2020

Track Seeding and Labelling with Embedded-space Graph Neural Networks

To address the unprecedented scale of HL-LHC data, the Exa.TrkX project is investigating a variety of machine learning approaches to particle track reconstruction. The most promising of these solutions, graph neural networks (GNN), process the event as a graph that connects track measurements (detector hits corresponding to nodes) with candidate line segments between the hits (corresponding to edges). Detector information can be associated with nodes and edges, enabling a GNN to propagate the embedded parameters around the graph and predict node-, edge- and graph-level observables. Previously, message-passing GNNs have shown success in predicting doublet likelihood, and we here report updates on the state-of-the-art architectures for this task. In addition, the Exa.TrkX project has investigated innovations in both graph construction, and embedded representations, in an effort to achieve fully learned end-to-end track finding. Hence, we present a suite of extensions to the original model, with encouraging results for hitgraph classification. In addition, we explore increased performance by constructing graphs from learned representations which contain non-linear metric structure, allowing for efficient clustering and neighborhood queries of data points. We demonstrate how this framework fits in with both traditional clustering pipelines, and GNN approaches. The embedded graphs feed into high-accuracy doublet and triplet classifiers, or can be used as an end-to-end track classifier by clustering in an embedded space. A set of post-processing methods improve performance with knowledge of the detector physics. Finally, we present numerical results on the TrackML particle tracking challenge dataset, where our framework shows favorable results in both seeding and track finding.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2025

Fusion of Experiments and Simulations for Real-Time Identification of Pipeline Defects

deep learning↗

DOE OSTI2025

HEPTAPOD: Orchestrating High Energy Physics Workflows Towards Autonomous Agency

Many workflows in high-energy-physics (HEP) stand to benefit from recent advances in transformer-based large language models (LLMs). While early applications of LLMs focused on text generation and code completion, modern LLMs now support orchestrated agency: the coordinated execution of complex, multi-step tasks through tool use, structured context, and iterative reasoning. We introduce the HEP Toolkit for Agentic Planning, Orchestration, and Deployment (HEPTAPOD), an orchestration framework designed to bring this emerging paradigm to HEP pipelines. The framework enables LLMs to interface with domain-specific tools, construct and manage simulation workflows, and assist in common utility and data analysis tasks through schema-validated operations and run-card-driven configuration. To demonstrate these capabilities, we consider a representative Beyond the Standard Model (BSM) Monte Carlo validation pipeline that spans model generation, event simulation, and downstream analysis within a unified, reproducible workflow. HEPTAPOD provides a structured and auditable layer between human researchers, LLMs, and computational infrastructure, establishing a foundation for transparent, human-in-the-loop systems.

Menzo, Tony [Alabama U.; Fermilab] (ORCID:00000002↗

DOE OSTI2020

The Sixth Data Release of the Radial Velocity Experiment (Rave). II. Stellar Atmospheric Parameters, Chemical Abundances, and Distances

We present part 2 of the sixth and final Data Release (DR6) of the Radial Velocity Experiment (Rave), a magnitude-limited (9<I<12) spectroscopic survey of Galactic stars randomly selected in Earth’s southern hemisphere. The Rave medium-resolution spectra (R ∼ 7500) cover the Ca triplet region (8410–8795 Å) and span the complete time frame from the start of Rave observations on 2003 April 12 to their completion on 2013 April 4. In the second of two publications, we present the data products derived from 518,387 observations of 451,783 unique stars using a suite of advanced reduction pipelines focusing on stellar atmospheric parameters, in particular purely spectroscopically derived stellar atmospheric parameters (T{sub eff}, logg, and the overall metallicity), enhanced stellar atmospheric parameters inferred via a Bayesian pipeline using Gaia DR2 astrometric priors, and asteroseismically calibrated stellar atmospheric parameters for giant stars based on asteroseismic observations for 699 K2 stars. In addition, we provide abundances of the elements Fe, Al, and Ni, as well as an overall [α/Fe] ratio obtained using a new pipeline based on the GAUGUIN optimization method that is able to deal with variable signal-to-noise ratios. The Rave DR6 catalogs are cross-matched with relevant astrometric and photometric catalogs, and are complemented by orbital parameters and effective temperatures based on the infrared flux method. The data can be accessed via the Rave website (http://rave-survey.org) or the Vizier database.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

Toward Drilling the Perfect Geothermal Well: An International Research Coordination Network for Geothermal Drilling Optimization Supported by Deep Machine Learning and Cloud Based Data Aggregation

The EDGE project, supported by the U.S. Department of Energy Geothermal Technologies Office under award DE-EE0008793, established a data-driven framework for improving the efficiency, cost-effectiveness, and reliability of geothermal well drilling. The project focused on developing scalable data infrastructure, advanced machine learning and probabilistic models, and integrated analytics tools to support continuous drilling optimization. A central objective was to reduce geothermal drilling costs by up to seventy percent while minimizing the risk of well failure through predictive diagnostics and adaptive planning. Over the project period, a comprehensive data repository was designed and deployed, incorporating records from over one hundred geothermal wells across varied geological settings. This repository supported both structured and unstructured data and adhered to FAIR data principles, enabling provenance tracking, quality control, and standardized metadata. The project introduced automated ingestion pipelines and a cloud-hosted platform that facilitated access to raw, processed, and derived datasets. This infrastructure served as the foundation for model development and analysis. Machine learning workflows were developed to predict key drilling metrics including rate of penetration, non-productive time, and total drilling costs. Self-organizing maps and dimensionality reduction methods were used to uncover operational patterns and outliers, while supervised learning algorithms such as random forests and deep neural networks were applied to forecast performance outcomes. The models were validated on heterogeneous datasets from both U.S. and Icelandic fields, demonstrating variable but significant predictive accuracy. The results indicated that finer temporal resolution, inclusion of lithological data, and consistency in operational annotations could substantially improve model performance. The project also implemented process mining techniques to reconstruct state-transition models from drilling event logs. These models enabled the identification of deviations from optimal workflows and provided insights into recurring failure modes. Analysis of non-productive time highlighted the impact of equipment failures, geological challenges, and human factors, offering opportunities for targeted mitigation strategies. The EDGE Dashboard was developed as a web-based expert system integrating data visualization, model outputs, and user-driven queries. It provided an accessible interface for operators to explore historical data, evaluate predicted outcomes, and compare drilling scenarios. Initial feedback from project partners suggested that the dashboard could serve as a foundation for more advanced advisory and optimization tools. Overall, the EDGE project demonstrated the feasibility and value of applying modern data science techniques to geothermal drilling. It delivered a set of interoperable tools and models that can support more efficient, lower-risk well development. The findings point toward a viable path for transitioning from advisory analytics to semi-autonomous drilling systems, contingent on continued collaboration, expanded datasets, and field validation. The project results have immediate relevance for drilling operations, data management practices, and future geothermal R&D efforts aimed at achieving reliable, cost-competitive geothermal energy at scale.

15 GEOTHERMAL ENERGY↗

DOE OSTI2026

NREL ASSIST Rhode Island / Thermodynamic retrievals TROPoe

This dataset contains daily files with thermodynamic profiles retrieved with the optimal estimation physical retrieval TROPoe (Turner and Löhnert 2014; Turner and Blumberg 2019; Turner and Löhnert 2021). The profiles are retrieved every 10 minutes from instantaneous observations from the NREL ASSIST-II (SN 11) infrared spectrometer. Observations are noise-filtered but not averaged in time to minimize errors due to non-uniform clouds. Additional input data in TROPoe are cloud base height from a collocated NREL Upgraded Galion lidar #1212-60 and temperature, RH, and pressure from collocated PNNL met station. The lidar beta is used to estimate the cloud base height through a modified algorithm by Newsom et al., 2019 used at ARM SGP. The full pipeline for running the retrieval is available at https://github.com/StefanoWind/TROPoe_processor. In addition to these temporally resolved input data, TROPoe requires an a priori dataset (prior) which provides mean climatological estimates of thermodynamic profiles and specifies how temperature and humidity covary with height as an input (for details see e.g. Djalalova et al. 2022). The prior is a key component of the retrieval and provides a constraint on the ill-posed inversion problem. A monthly prior was computed from operational radiosonde launches at Upton, NY.

17 WIND ENERGY↗

DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Failure diagnosis and trend‐based performance losses routines for the detection and classification of incidents in large‐scale photovoltaic systems

Abstract Fault detection and classification in photovoltaic (PV) systems through real‐time monitoring is a fundamental task that ensures quality of operation and significantly improves the performance and reliability of operating systems. Different statistical and comparative approaches have already been proposed in the literature for fault detection; however, accurate classification of fault and loss incidents based on PV performance time series remains a key challenge. Failure diagnosis and trend‐based performance loss routines were developed in this work for detecting PV underperformance and accurately identifying the different fault types and loss mechanisms. The proposed routines focus mainly on the differentiation of failures (e.g., inverter faults) from irreversible (e.g., degradation) and reversible (e.g., snow and soiling) performance loss factors based on statistical analysis. The proposed routines were benchmarked using historical inverter data obtained from a 1.8 MWp PV power plant. The results demonstrated the effectiveness of the routines for detecting failures and loss mechanisms and the capability of the pipeline for distinguishing underperformance issues using anomaly detection and change‐point (CP) models. Finally, a CP model was used to extract significant changes in time series data, to detect soiling and cleaning events and to estimate both the performance loss and degradation rates of fielded PV systems.

14 SOLAR ENERGY↗

DOE OSTI2025

Revealing the evolution of order in materials microstructures using multi-modal computer vision

The development of high-performance materials for microelectronics, energy storage, and extreme environments depends on our ability to describe and direct property-defining microstructural order. Our present understanding is typically derived from laborious manual analysis of imaging and spectroscopy data, which is difficult to scale, challenging to reproduce, and lacks the ability to reveal latent associations needed for mechanistic models. Here, we demonstrate a multi-modal machine learning (ML) approach to describe order from electron microscopy analysis of the complex oxide La 1−x Sr x FeO 3 . We construct a hybrid pipeline based on fully and semi-supervised classification, allowing us to evaluate both the characteristics of each data modality and the value each modality adds to the ensemble. We observe distinct differences in the performance of uni- and multi-modal models, from which we draw general lessons in describing crystal order using computer vision.

36 MATERIALS SCIENCE↗