Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Study of a Proposed fissionTPC Measurement of 233 U(n,f)/ 235 U(n,f) Cross Section Ratio

A review of the available 233 U( n, f )/ 235 U( n, f ) cross section ratio data compared to the ENDF evaluation in the fast neutron energy range, 0.1–20 MeV, shows that discrepancies exist between some of the various measurements and the evaluation. The discrepancies vary and a function of incident neutron energy and range between 1–5%. In this document we review two potential options for performing new measurements of the cross section ratio to potentially resolve these discrepancies. The two options are compared and contrasted in terms of measurement accuracy and cost. The first option would utilize the fissionTPC, which was designed specifically to achieve measurement uncertainties of better than 1% to resolve a similar data–evaluation discrepancy in the 239 Pu( n, f ) cross section. While this would seem to be an ideal option in this case also, there are challenges. The fissionTPC is a complex instrument and the cost of maintenance and operations are significant. Furthermore, some of the fissionTPC equipment is nearing the end of its useful life which presents a risk to completion of a measurement campaign. The second option is to use a dual-fission chamber, which is the more traditional approach to these types of measurements. The cost is estimated to be about half that of a fissionTPC measurement, while the much simpler design of the detector reduces overall risk. The trade-off is the measurement uncertainty, which would be on order of 1–3% with a careful measurement and analysis. This uncertainty however may not be good enough to fully resolve the discrepancies. The objective of a new measurement would be the improvement in the ENDF evaluation with the end goal of improving the accuracy of models and simulations at the end of the nuclear data pipeline. An understanding of the sensitivity of these models to the data uncertainties would help guide the choice of measurement techniques.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Advanced Infrastructure Integrity Modeling (AIIM) Onshore Pipeline Database

The Advanced Infrastructure Integrity Modeling (AIIM) Onshore Pipeline Database is an interoperable spatial resource containing critical environmental, operational, and reported stressors tied to publicly available oil and gas pipeline locations across the contiguous U.S. and Alaska. This database contains two layers: 1. Pipeline point locations (‘pipeline_points’) – More than 500,000 points (at every kilometer along pipelines, and end points) to which more than 350 stress-related variables have been appended. 2. Merged pipelines (‘merged_pipelines’) – The original, publicly available pipeline data (see table below) merged together into one feature class.

Carbon Transport↗
DOE OSTI2023

ERSAP: Toward Better NP Data-Stream Analytics With Flow-Based Programming

This paper presents an reactive, actor-model and FBP paradigm based framework that we develop to design data-stream processing applications for HEP and NP. This framework encourages a functional decomposition of the overall data processing application into small mono-functional artifacts. Artifacts that are easy to understand, develop, deploy and debug. The fact that these artifacts (actors) are programmatically independent they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages of this approach is fault tolerance where independent actors can come and go on the data-stream without forcing the entire application to crash. Furthermore, it also makes it is easy to locate the faulty actor in the data pipeline. Due the fact that the actors are loosely coupled, and that the data (inevitably) carries the context, they can run on heterogeneous environments, utilizing different accelerators. This paper describes the main design concepts of the framework and presents a ?proof of concept? application design and deployment results obtain processing on-beam calorimeter streaming data.

Gyurjyan, V.↗
DOE OSTI2025

Nuclear data sensitivity and uncertainty study of copper-reflected integral experiments [Slides]

This presentation touches on reducing uncertainties in intermediate-energy actinide nuclear data and this continues to be a high priority for many applications. The goal of PARADIGM (PARallel Approach of Differential and InteGral Measurements): accelerate efforts to reduce biases and uncertainties in nuclear data through improvements to the nuclear data pipeline. This presentation includes integral experiments and a summarization of existing copper nuclear data.

Cu63↗
DOE OSTI2021

SDSS-IV MaNGA: Modeling the Spectral Line-spread Function to Subpercent Accuracy

The Sloan Digital Sky Survey IV Mapping Nearby Galaxies at APO (MaNGA) program has been operating from 2014 to 2020, and has now observed a sample of 9269 galaxies in the low redshift universe (z ∼ 0.05) with integral-field spectroscopy. With rest-optical (λλ0.36–1.0 μm) spectral resolution R ∼ 2000 the instrumental spectral line-spread function (LSF) typically has 1σ width of about 70 km s{sup −1}, which poses a challenge for the study of the typically 20–30 km s{sup −1} velocity dispersion of the ionized gas in present-day disk galaxies. In this contribution, we present a major revision of the MaNGA data pipeline architecture, focusing particularly on a variety of factors impacting the effective LSF (e.g., under-sampling, spectral rectification, and data cube construction). Through comparison with external assessments of the MaNGA data provided by substantially higher-resolution R ∼ 10,000 instruments, we demonstrate that the revised MPL-10 pipeline measures the instrumental LSF sufficiently accurately (≤0.6% systematic, 2% random around the wavelength of Hα) that it enables reliable measurements of astrophysical velocity dispersions σ {sub Hα} ∼ 20 km s{sup −1} for spaxels with emission lines detected at signal-to-noise ratio > 50. Velocity dispersions derived from [O II], Hβ, [O III], [N II], and [S II] are consistent with those derived from Hα to within about 2% at σ {sub Hα} > 30 km s{sup −1}. Although the impact of these changes to the estimated LSF will be minimal at velocity dispersions greater than about 100 km s{sup −1}, scientific results from previous data releases that are based on dispersions far below the instrumental resolution should be reevaluated.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Leveraging AI and Spatial Data to Unlock Pipeline Integrity Insights: NETL’s Advanced Infrastructure Integrity Model (AIIM)

Maintaining the integrity of natural gas infrastructure plays a critical role in ensuring energy security. Robust, data-driven foundational AI models for pipeline integrity can help address risk management and mitigation issues. Trusted foundational models can help with industry adoption and accelerate innovation by enhancing integrity predictions, reduce costs, and informing infrastructure build-out. The AIIM dashboard was released in 2022 and utilizes multi-ML models for ensemble-type insights. It was expanded to include analytics on reported incidents. It was developed as an ESRI Dashboard to support data visualization & interrogation and contains pipeline data and model results.

Advanced Infrastructure Integrity Model (AIIM)↗
DOE OSTI2023

Digital Twin for Optimizing Real-time Economy of the Integrated Energy Systems

Economic and safe operation of integrated energy systems (IES) requires real-time optimization (RTO) of the control and actions conducted on each system component. In this regard, digital twins (DTs), which consist of a physical system, a virtual system, and the data communication that occurs between the two, are essential for effective RTO. Through the data warehouse, the virtual system is constantly updated with real-time data from the physical system, and functions as the model in the optimization framework. The reduced-order model of the dynamic process model in the virtual system is used in the optimization framework. The optimization results are then returned, via the data warehouse, as control actions to the physical system. This work demonstrates the software capabilities of DT assets for an IES in the context of preparing a DT for an experimental system comprised of Idaho National Laboratory (INL)’s Thermal Energy Delivery System and battery system. For the virtual demonstration, the DTs encompass (1) a physical system, including the Modelica models of the Thermal Energy Delivery System and the battery system; (2) virtual optimization via the Optimization of Real-Time Capacity Allocation (ORCA) platform; and (3) the open-source data warehouse software DeepLynx. This work assesses the performance of ORCA, which utilizes a reduced-order model built using the Risk Analysis Virtual Environment (RAVEN) and trained on the Modelica models and real-time data pipeline through the graph database hosted in DeepLynx. The proposed optimization workflow will be an RTO model based on DTs and the data they generate.

25 ENERGY STORAGE↗
DOE OSTI2021

Expanding Benchmarks for Nuclear Data Validation [Slides]

The presentation discusses benchmarking in comparison to experimental truth, the integral experiments that were performed, and that benchmarks are evaluated experiments. It also discusses validation testing, and that the ultimate goal of the validation end product is to improve evaluated nuclear data for applications It does state that validation highlights errors in the Nuclear Data Pipeline that affect applications, for example: Missing Cd Capture Gammas in ENDF/B-VII.1. It acknowledges that additional types of experiments are needed to test data used in applications, and that validation experiments do not have to be complicated and expensive. The presentation also discusses different types of measurements including activation foil and fission chamber measurements, reactor kinetics measurements, and subcritical measurements. The presentation concludes with an experiment summary and states that all applications need validation.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Navigating Exascale Operational Data Analytics: From Inundation to Insight

In this paper, we address the challenges in achieving sustainable data-driven efficiency by providing a detailed exploration of the end-to-end operational data analytics (ODA) framework that evolved through two generations of supercomputer systems at the Oak Ridge Leadership Computing Facility (OLCF). This framework addresses large data streams ingested from heavily instrumented HPC environment that accumulates multi-terabytes per day. We outline the multifaceted data life cycle across HPC procurement, operations, and research & development, identifying key obstacles and design decisions that shape effective strategies in building and supporting data pipelines end-to-end. By sharing key insights and lessons learned from our experience, we offer recommendations for the HPC community on enabling sustainable operational data analytics and beyond. Our contributions aim to bridge the gap between potential and real benefits of operational data, guiding future efforts towards integrated and sustainable operational intelligence in high-performance computing environments.

Shin, Woong↗
DOE OSTI2023

Report on ISR-1 High-Altitude Balloon Flight

To test small technologies at lower cost for space science applications, LANL has developed a small high altitude balloon payload that could, in the future, be regularly and inexpensively launched from LANL. A neutron detector, NEMO, was integrated to evaluate its performance in a space-like mixed-radiation environment and collect neutron data in the atmosphere. In collaboration with EES-14, a high-altitude balloon payload was launched from LANL Technical Area 51 on February 27, 2023 and April 17, 2023. For real-time geolocation, a SAM-M8Q M8 GNSS module was used to get position and time, and an Iridium RockBLOCK 9603 was used to communicate with the ground using the Iridium satellite fleet. These modules were all controlled using an Iteaduino Mega microcontroller board. Finally, a High Altitude Science Eagle Flight Computer with a temperature pressure sensor ran independently, writing data to an SD card. All of these modules were powered by a 5 mAh lithium polymer battery. The battery was attached to the bottom of the payload while the remaining electronics were embedded in the underside of the top of the payload. These modules were wired as seen in Figure 1-2. The Iteaduino Mega microcontroller board was programmed to use the RockBLOCK to send a message once every 10 minutes containing neutron and GPS data read off the NEMO and SAM M8Q, respectively. Once the message send attempt finished, the RockBLOCK would be slept for the rest of the 10 minute interval. The Eagle Flight Computer ran continuously throughout the flight, taking data every 6 seconds. The RockBLOCK message data was set up to be delivered from the Iridium satellite fleet to a website, where it was stored and parsed to create live maps and plots for analysis and balloon retrieval. The RockBLOCK message data was additionally configured to be sent to an email as a fail-safe. The payload was ground-tested successfully for over 50 hours, with multiple revisions occurring to best prepare for conditions at altitude and improve the software and firmware to fix any issues that cropped up with the data pipeline. Additional to the balloon payload, the flight had an attached iMet-4 radiosonde and Garmin T5 GPS Dog Collar. The radiosonde provided GPS and meteorological data. The T5 dog collar is used along with a Garmin Astro 430 to track the balloon at a range of up to 9 miles for retrieval. The balloon itself was initially a 1600 g meteorological balloon with an attached High Altitude Science parachute, both of which can be seen in Figure 1-3. After the first flight, the EES team swapped to a Rocketman parachute.

42 ENGINEERING↗
DOE OSTI2024

Online Detection of Power Grid Anomalies via Federated Learning

Data from sensors is critical for advanced applica- tions that support efficient, reliable, and resilient electric grid operations. Historically, data from phasor measurement units (PMU) has been utilized to develop a wide variety of wide area control and protection applications suitable for power grid control centers. However, until now, most of these could not be deployed for automated operations due to a set of data corruption challenges and uncertainty in the incoming data pipeline. In this paper, we address the problem of detecting different variety of anomalies that are evident in different high- speed power grid measurements. The paper discusses a workflow for handling problems with data acquisition and highlights some of the key findings suitable for anomaly detection in a centralized and distributed environment. The effectiveness of the proposed method was demonstrated with results utilizing realistic PMU datasets

Shinkle, Matthew W.↗
DOE OSTI2024

Automated pipeline processing X-ray diffraction data from dynamic compression experiments on the Extreme Conditions Beamline of PETRA III

Presented and discussed here is the implementation of a software solution that provides prompt X-ray diffraction data analysis during fast dynamic compression experiments conducted within the dynamic diamond anvil cell technique. It includes efficient data collection, streaming of data and metadata to a high-performance cluster (HPC), fast azimuthal data integration on the cluster, and tools for controlling the data processing steps and visualizing the data using the DIOPTAS software package. This data processing pipeline is invaluable for a great number of studies. The potential of the pipeline is illustrated with two examples of data collected on ammonia–water mixtures and multiphase mineral assemblies under high pressure. The pipeline is designed to be generic in nature and could be readily adapted to provide rapid feedback for many other X-ray diffraction techniques, e.g. large-volume press studies, in situ stress/strain studies, phase transformation studies, chemical reactions studied with high-resolution diffraction etc.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

The DECADE cosmic shear project III: validation of analysis pipeline using spatially inhomogeneous data

We present the pipeline for the cosmic shear analysis of the Dark Energy Camera All Data Everywhere (DECADE) weak lensing dataset: a catalog consisting of 107 million galaxies observed by the Dark Energy Camera (DECam) in the northern Galactic cap. The catalog derives from a large number of disparate observing programs and is therefore more inhomogeneous across the sky compared to existing lensing surveys. First, we use simulated data-vectors to show the sensitivity of our constraints to different analysis choices in our inference pipeline, including sensitivity to residual systematics. Next we use simulations to validate our covariance modeling for inhomogeneous datasets. Finally, we show that our choices in the end-to-end cosmic shear pipeline are robust against inhomogeneities in the survey, by extracting relative shifts in the cosmology constraints across different subsets of the footprint/catalog and showing they are all consistent within 1σ to 2σ. This is done for forty-six subsets of the data and is carried out in a fully consistent manner: for each subset of the data, we re-derive the photometric redshift estimates, shear calibrations, survey transfer functions, the data vector, measurement covariance, and finally, the cosmological constraints. Our results show that existing analysis methods for weak lensing cosmology can be fairly resilient towards inhomogeneous datasets. This also motivates exploring a wider range of image data for pursuing such cosmological constraints.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Tsdat: An Open-Source Data Standardization Framework for Marine Energy and Beyond: Preprint

Many organizations are tasked with the collection and processing of large quantities of data from various measurement devices. Data reported from these sources are often not interoperable with datasets and software used by analysts and other organizations in the same field, introducing barriers for collaboration on large-scale projects. This poses a particular problem for cross-device comparisons and machine learning applications. To address these challenges, the open source Time-Series Data Pipelines (Tsdat) software was developed by a joint collaboration between Pacific Northwest National Laboratory, the National Renewable Energy Laboratory, and Sandia National Laboratories to facilitate collaboration and accelerate advancements in the Marine Energy domain through the development of an open-source ecosystem of tools. This paper will describe the Tsdat software and the data standards within which the framework operates. A beta version of the framework has been released and is currently being used by several projects in marine energy, wind energy, and building energy systems.

big data↗
DOE OSTI2023

Phonon-informed Neural Thermal Scattering (NeTS) Optimization for Crystalline Graphite and Beryllium Metal

Fast neutrons born from fission lose energy through scattering interactions in the process of slowing-down. As neutrons thermalize to the order of $k$ $b$ $T$ (where $k$ $b$ is the Boltzmann constant, and $T$ is the temperature of the medium), their de Broglie wavelength and energy approaches the order of inter-atomic spacing and quantized lattice vibrations, i.e., phonons. At thermal energies, the thermal scattering law (TSL), i.e., $S$($α, β$), captures crystal binding contributions to the total reaction rate, or cross section. This dimensionless material property describes the energy ($β$) and momentum ($α$) exchanges available in a medium. Currently, $S$($α, β$) is evaluated in the Full Law Analysis Scattering System Hub (FLASSH) code for discrete inputs and stored as ENDF/B File 7 for 0-phonon elastic (MT 2) and n-phonon inelastic (MT 4) processes. Further processing recasts $S$($α, β$) into cumulative distribution functions for sampling post-collision scattering kinematics. In practice, interpolation schemes are employed to access data between tabulated values. An improvement to this juncture of the nuclear data pipeline is supplying cross sections on-the-fly (OTF), as has been developed for the un-resolved resonance region to minimize non-physical interpolation errors. This capability may improve simulation accuracy for accident and transient analyses, where rapidly varying changes in temperature and pressure are difficult to predict beforehand. To do so, deep artificial neural networks (ANNs) can be employed which collapse non-linear, complex data into a lightweight dictionary of neural weights and biases. This has been successfully demonstrated for the hydrogen in light water $S$($α, β$) dataset in the form of a Neural Thermal Scattering (NeTS) module. In this work, the NeTS framework is extended to consider the impact of material-dependent dynamical features on optimal neural pre-processing and architecture design decisions, such as number of neurons per hidden layer, residual skip connections and neural depth. New NeTS modules for crystalline graphite and beryllium metal illuminate a novel correlation between dynamical nonlinearity and optimal neural parametrization when deploying $S$($α, β$) on-the-fly.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Embedded FPGA developments in 130 nm and 28 nm CMOS for machine learning in particle detector readout

Embedded field programmable gate array (eFPGA) technology allows the implementation of reconfigurable logic within the design of an application-specific integrated circuit (ASIC). This approach offers the low power and efficiency of an ASIC along with the ease of FPGA configuration, particularly beneficial for the use case of machine learning in the data pipeline of next-generation collider experiments. An open-source framework called "FABulous" was used to design eFPGAs using 130 nm and 28 nm CMOS technology nodes, which were subsequently fabricated and verified through testing. The capability of an eFPGA to act as a front-end readout chip was assessed using simulation of high energy particles passing through a silicon pixel sensor. A machine learning-based classifier, designed for reduction of sensor data at the source, was synthesized and configured onto the eFPGA. A successful proof-of-concept was demonstrated through reproduction of the expected algorithm result on the eFPGA with perfect accuracy. Finally, further development of the eFPGA technology and its application to collider detector readout is discussed.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

LDM-151: Data Management Science Pipelines Design

The LSST Science Requirements Document (the LSST SRD) specifies a set of data product guidelines, designed to support science goals envisioned to be enabled by the LSST observing program. Following these guidelines, the details of these data products have been described in the LSST Data Products Definition Document (DPDD), and captured in a formal flow-down from the SRD via the LSST System Requirements (LSR), Observatory System Specifications (OSS), to the Data Management System Requirements (DMSR). The LSST Data Management subsystem's responsibilities include the design, implementation, deployment and execution of software pipelines necessary to generate these data products. This document describes the design of the scientific aspects of those pipelines.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Importance of Higher Fidelity Model Geometries during Optimization of Critical Experiments

PARADIGM, PARallel Approach of Differential and InteGral Measurements, is a cross-collaborative effort at Los Alamos National Laboratory between nuclear data theorists, differential and integral experimenters, as well as machine learning statisticians to tackle uncertainties in the intermediate region of 239 Pu. In essence, the idea behind PARADIGM is to remove the linear conceptualization of the nuclear data pipeline, shown in Figure 1, and replace it with a far more parallelized approach. The novel approach leverages machine learning to guide which differential measurements and integral experiments will result in the largest decrease in uncertain ties for a nuclide reaction pair in a given energy range. The concept builds off earlier work, EUCLID, which focused on the fast region of 239 Pu. The practical benefit of having evaluation, differential measurement, and integral experiment personnel in collaboration with machine learning is to represent the entire nuclear data in one snapshot. This enable large reduction in the time to deliver improved nuclear data, which using the PARADIGM approach could be done in 3 years. A general outline of PARADIGM and specific topics are available in other papers. The discussion here will pertain directly to the integral experiment design. More specifically, the process of taking a rough design and transforming it into a finalized neutronic model will be discussed.

97 MATHEMATICS AND COMPUTING↗