Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Rossi-alpha Analysis of CURIE Experiment

Critical assembly measurement and operations are crucial to the development of benchmark data to support research into criticality safety, radiation-detection development, and the overall application of nuclear technologies. Accurate nuclear data are needed for accurate predictive simulations, and validation using critical experiments is an important part of the nuclear data pipeline. The accuracy of nuclear data are improved using more robust and targeted measurements. In particular, the intermediate energy range of uranium is of great interest. LANL has successfully performed the Zeus series of experiments on the Comet assembly to investigate the intermediate energy range for HEU with various moderators. A successor to these experiments is the benchmark for the Critical Unresolved Region Integral Experiment (CURIE), designed to be sensitive to the unresolved resonance region (URR). This experiment is designed using polytetrafluoroethylene, more commonly known as Teflon, moderators. The CURIE experiment was successfully conducted at the National Criticality Experiments Research Center (NCERC). The Rossi-alpha method was used to evaluate the propensity of the CURIE configurations to sustain fission chains by estimating the prompt neutron decay constant α. This work evaluates the α at delayed critical using Rossi-alpha for different Teflon moderator thicknesses in the CURIE experiment. These results will help improve understanding of the CURIE benchmark experiments.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Data Fusion to Enhance Quality Control and Analysis with Instruments at the Marine and Coastal Research Laboratory

Deploying environmental monitoring instruments in the marine environment can be challenging, facing challenges around device survivability, biofouling and corrosion, and consistent data collection. This project explores the use of data fusion – the process of integrating multiple data sources to produce more consistent, accurate, and useful information – to build a consistent long-term monitoring system at the Marine and Coastal Research Laboratory (MCRL) in Sequim, Washington. Unused instruments that had been acquired from past projects were inventoried and deployments planned on the MCRL pier and floating dock. A total of 8 instruments were deployed including a tide gauge, hydrophone, acoustic Doppler current profiler (ADCP), photosynthetically active radiation (PAR) sensors, meteorological station, and three water quality sensors. Deployments were planned to be well-protected around the pier structure and a maintenance schedule was created for cleaning and recalibration. An automated data pipeline was created to aggregate data on edge computers that push data to Amazon Web Services (AWS) cloud storage every 15 minutes, performing automated quality control and data transformations using the Time Series Data Analytical Toolkit (TSDAT). Continued efforts are underway to maintain this system into the future, take a data-driven approach to maintenance scheduling, improve the reliability of the system, and share the data with a variety of end-users.

54 ENVIRONMENTAL SCIENCES↗

AI-Ready Data Pilot Project Report

The proliferation of artificial intelligence in scientific research has created an urgent need to define "AI-ready data" for researchers and, more importantly, provide resources to help them produce AI-ready data. At Pacific Northwest National Laboratory, we conducted a pilot study with three data scientists evaluating three CSV datasets from different scientific domains, followed by semi-structured interviews capturing assessment practices. Our findings reveal that AI-readiness evaluation is intuition-based, with practitioners asking "How fast can I go from raw data to my machine learning pipeline?" Data scientists consistently prioritized workflow efficiency, human interpretability, and quality stewardship signals. From these insights, we developed a practical evaluation framework comprising data requirements, metadata standards, and validation tests that provides actionable criteria for producing and curating AI-ready datasets, addressing the gap between theoretical understanding and practical implementation.

97 MATHEMATICS AND COMPUTING↗

AUTONOMIE VID

Autonomie Vehicle Information Database (VID) offers a comprehensive list of vehicle specifications since 1990. The database details more than 65,000 vehicles with hundreds of attributes. The database is the result of the development of a general automated data collection framework as well as the development of building blocks for processing, cleaning, integrating and analyzing complex data. The data has undergone several layers of outlier detections processes, machine learning based imputations methods have been used to deal with missing data problems, and new fields have been created according to the rules of feature engineering. Thanks to this streamlined data pipelines, the resulting processed aggregated data should deliver a unique level of information to the user in which the content can be efficiently maintained and updated.

Moswd, Ayman↗

Dynamic Black-Level Correction and Artifact Flagging for Kepler Pixel Time Series

Methods applied to the calibration stage of Kepler pipeline data processing [1] (CAL) do not currently use all of the information available to identify and correct several instrument-induced artifacts. These include time-varying crosstalk from the fine guidance sensor (FGS) clock signals, and manifestations of drifting moire pattern as locally correlated nonstationary noise, and rolling bands in the images which find their way into the time series [2], [3]. As the Kepler Mission continues to improve the fidelity of its science data products, we are evaluating the benefits of adding pipeline steps to more completely model and dynamically correct the FGS crosstalk, then use the residuals from these model fits to detect and flag spatial regions and time intervals of strong time-varying black-level which may complicate later processing or lead to misinterpretation of instrument behavior as stellar activity.

Kolodziejczak, J. J.↗

Data about data – when, why and how metadata can support the digital plant

A structured approach for recording data quality and contextual information about how and why a signal exists – i.e. metadata – is central to interpret and use sensor data correctly. This is becoming increasingly important with the global trend with data-driven applications such as digital twins and AI-models. But a structured metadata collection and organization of sensor data is not routine in most plants, which can result in lost information and missed opportunities to make use of the investments made in the data collection. Therefore, the IWA task group on Metadata Collection and Organization in wastewater resource recovery systems (MetaCO) was initiated in 2020 and recently delivered the IWA scientific and technical report number 31. The report gives and in-depth description about metadata in water resources recovery facilities (WRRFs) and is available as open access at IWA publishing. The report is the outcome of the collaboration between more than 80 water professionals with the intention to serve WRRF data users with a guide on how to structure and make use of metadata throughout the data pipeline in order to maximize the value of sensor data.

Alferes, Janelcy [VITO, Belgium]↗

Principal Components as a Data Reduction and Noise Reduction Technique

The potential of principal components as a pipeline data reduction technique for thematic mapper data was assessed and principal components analysis and its transformation as a noise reduction technique was examined. Two primary factors were considered: (1) how might data reduction and noise reduction using the principal components transformation affect the extraction of accurate spectral classifications; and (2) what are the real savings in terms of computer processing and storage costs of using reduced data over the full 7-band TM complement. An area in central Pennsylvania was chosen for a study area. The image data for the project were collected using the Earth Resources Laboratory's thematic mapper simulator (TMS) instrument.

Imhoff, M. L.↗

Product Operations Status Summary Metrics

The Product Operations Status Summary Metrics (POSSUM) computer program provides a readable view into the state of the Phoenix Operations Product Generation Subsystem (OPGS) data pipeline. POSSUM provides a user interface that can search the data store, collect product metadata, and display the results in an easily-readable layout. It was designed with flexibility in mind for support in future missions. Flexibility over various data store hierarchies is provided through the disk-searching facilities of Marsviewer. This is a proven program that has been in operational use since the first day of the Phoenix mission.

Takagi, Atsuya↗

2.3.3.404 - National Lab and University Collaboration for MHK Instrumentation and Data Processing Tools

Field and laboratory validation, testing, demonstration, and operation are critical steps for increasing the technology readiness level of marine energy (ME) converters because they provide high-quality testing and performance data that are critical information used to feed all aspects of technology development. This project, in partnership with industry, enables the marine and hydrokinetic energy (MHK) community to reliably and efficiently collect, process, manage, and share quality data by facilitating access to and development of instrumentation, guidelines and data processing/QA tools. Under this project, open-source data processing code (MHKiT) and tools (ME Data Pipeline, MRE Code Hub, PRIMRE Code Catalog), instrumentation (loads measurements), data acquisition systems (miniDAQ), and measurement guidance tools (Telesto, high EMI guidance) were developed to facilitate the collection and processing of quality laboratory and field data. Overall, this project is intended to improve the quality of the data collected during laboratory and field demonstration projects by standardizing the collection and processing techniques, as well as by improving access to instrumentation, code, and measurement guidance. Quality data will, in turn, lead to improved knowledge capture following ME device testing.

data processing↗

Study of a Proposed fissionTPC Measurement of 233 U(n,f)/ 235 U(n,f) Cross Section Ratio

A review of the available 233 U( n, f )/ 235 U( n, f ) cross section ratio data compared to the ENDF evaluation in the fast neutron energy range, 0.1–20 MeV, shows that discrepancies exist between some of the various measurements and the evaluation. The discrepancies vary and a function of incident neutron energy and range between 1–5%. In this document we review two potential options for performing new measurements of the cross section ratio to potentially resolve these discrepancies. The two options are compared and contrasted in terms of measurement accuracy and cost. The first option would utilize the fissionTPC, which was designed specifically to achieve measurement uncertainties of better than 1% to resolve a similar data–evaluation discrepancy in the 239 Pu( n, f ) cross section. While this would seem to be an ideal option in this case also, there are challenges. The fissionTPC is a complex instrument and the cost of maintenance and operations are significant. Furthermore, some of the fissionTPC equipment is nearing the end of its useful life which presents a risk to completion of a measurement campaign. The second option is to use a dual-fission chamber, which is the more traditional approach to these types of measurements. The cost is estimated to be about half that of a fissionTPC measurement, while the much simpler design of the detector reduces overall risk. The trade-off is the measurement uncertainty, which would be on order of 1–3% with a careful measurement and analysis. This uncertainty however may not be good enough to fully resolve the discrepancies. The objective of a new measurement would be the improvement in the ENDF evaluation with the end goal of improving the accuracy of models and simulations at the end of the nuclear data pipeline. An understanding of the sensitivity of these models to the data uncertainties would help guide the choice of measurement techniques.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Development of the Science Data System for the International Space Station Cold Atom Lab

Cold Atom Laboratory (CAL) is a facility that will enable scientists to study ultra-cold quantum gases in a microgravity environment on the International Space Station (ISS) beginning in 2016. The primary science data for each experiment consists of two images taken in quick succession. The first image is of the trapped cold atoms and the second image is of the background. The two images are subtracted to obtain optical density. These raw Level 0 atom and background images are processed into the Level 1 optical density data product, and then into the Level 2 data products: atom number, Magneto-Optical Trap (MOT) lifetime, magnetic chip-trap atom lifetime, and condensate fraction. These products can also be used as diagnostics of the instrument health. With experiments being conducted for 8 hours every day, the amount of data being generated poses many technical challenges, such as downlinking and managing the required data volume. A parallel processing design is described, implemented, and benchmarked. In addition to optimizing the data pipeline, accuracy and speed in producing the Level 1 and 2 data products is key. Algorithms for feature recognition are explored, facilitating image cropping and accurate atom number calculations.

bose einstein condensate↗

Advanced Infrastructure Integrity Modeling (AIIM) Onshore Pipeline Database

The Advanced Infrastructure Integrity Modeling (AIIM) Onshore Pipeline Database is an interoperable spatial resource containing critical environmental, operational, and reported stressors tied to publicly available oil and gas pipeline locations across the contiguous U.S. and Alaska. This database contains two layers: 1. Pipeline point locations (‘pipeline_points’) – More than 500,000 points (at every kilometer along pipelines, and end points) to which more than 350 stress-related variables have been appended. 2. Merged pipelines (‘merged_pipelines’) – The original, publicly available pipeline data (see table below) merged together into one feature class.

Carbon Transport↗

ERSAP: Toward Better NP Data-Stream Analytics With Flow-Based Programming

This paper presents an reactive, actor-model and FBP paradigm based framework that we develop to design data-stream processing applications for HEP and NP. This framework encourages a functional decomposition of the overall data processing application into small mono-functional artifacts. Artifacts that are easy to understand, develop, deploy and debug. The fact that these artifacts (actors) are programmatically independent they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages of this approach is fault tolerance where independent actors can come and go on the data-stream without forcing the entire application to crash. Furthermore, it also makes it is easy to locate the faulty actor in the data pipeline. Due the fact that the actors are loosely coupled, and that the data (inevitably) carries the context, they can run on heterogeneous environments, utilizing different accelerators. This paper describes the main design concepts of the framework and presents a ?proof of concept? application design and deployment results obtain processing on-beam calorimeter streaming data.

Gyurjyan, V.↗

Nuclear data sensitivity and uncertainty study of copper-reflected integral experiments [Slides]

This presentation touches on reducing uncertainties in intermediate-energy actinide nuclear data and this continues to be a high priority for many applications. The goal of PARADIGM (PARallel Approach of Differential and InteGral Measurements): accelerate efforts to reduce biases and uncertainties in nuclear data through improvements to the nuclear data pipeline. This presentation includes integral experiments and a summarization of existing copper nuclear data.

Cu63↗

SDSS-IV MaNGA: Modeling the Spectral Line-spread Function to Subpercent Accuracy

The Sloan Digital Sky Survey IV Mapping Nearby Galaxies at APO (MaNGA) program has been operating from 2014 to 2020, and has now observed a sample of 9269 galaxies in the low redshift universe (z ∼ 0.05) with integral-field spectroscopy. With rest-optical (λλ0.36–1.0 μm) spectral resolution R ∼ 2000 the instrumental spectral line-spread function (LSF) typically has 1σ width of about 70 km s{sup −1}, which poses a challenge for the study of the typically 20–30 km s{sup −1} velocity dispersion of the ionized gas in present-day disk galaxies. In this contribution, we present a major revision of the MaNGA data pipeline architecture, focusing particularly on a variety of factors impacting the effective LSF (e.g., under-sampling, spectral rectification, and data cube construction). Through comparison with external assessments of the MaNGA data provided by substantially higher-resolution R ∼ 10,000 instruments, we demonstrate that the revised MPL-10 pipeline measures the instrumental LSF sufficiently accurately (≤0.6% systematic, 2% random around the wavelength of Hα) that it enables reliable measurements of astrophysical velocity dispersions σ {sub Hα} ∼ 20 km s{sup −1} for spaxels with emission lines detected at signal-to-noise ratio > 50. Velocity dispersions derived from [O II], Hβ, [O III], [N II], and [S II] are consistent with those derived from Hα to within about 2% at σ {sub Hα} > 30 km s{sup −1}. Although the impact of these changes to the estimated LSF will be minimal at velocity dispersions greater than about 100 km s{sup −1}, scientific results from previous data releases that are based on dispersions far below the instrumental resolution should be reevaluated.

47 OTHER INSTRUMENTATION↗

Leveraging AI and Spatial Data to Unlock Pipeline Integrity Insights: NETL’s Advanced Infrastructure Integrity Model (AIIM)

Maintaining the integrity of natural gas infrastructure plays a critical role in ensuring energy security. Robust, data-driven foundational AI models for pipeline integrity can help address risk management and mitigation issues. Trusted foundational models can help with industry adoption and accelerate innovation by enhancing integrity predictions, reduce costs, and informing infrastructure build-out. The AIIM dashboard was released in 2022 and utilizes multi-ML models for ensemble-type insights. It was expanded to include analytics on reported incidents. It was developed as an ESRI Dashboard to support data visualization & interrogation and contains pipeline data and model results.

Advanced Infrastructure Integrity Model (AIIM)↗

FloodPlanet: High-Resolution Commercial Imagery for Training and Validation of Deep Learning-Based Models of Inundation Extent

Flooding events are becoming increasingly frequent worldwide and are known to cause extensive damage. Public optical and radar satellite imagery can be used to detect large areas of inundation in rural areas, however, long revisit times and coarse spatial resolution limit applications for short-lived events and urban areas. Commercial constellations such as those operated by Planet offer increased spatial and temporal resolution and can supplement mapping efforts to provide more information to disaster response, relief, and mitigation efforts. Deep learning requires high quality labeled data for training across coincident sensors. The FloodPlanet dataset presented here contains labeled surface water for 18 events across the world based on Planetscope imagery with coincident Harmonized Landsat Sentinel-2 ( HLS) or Sentinel-1 and builds upon the previously existing Sen1Floods11, xBD, and NASA Sentinel-1 datasets. Sen1Floods11 includes 4,831 512x512 pixel overlapping tiles of coincident Sentinel-1 and Sentinel-2 data observing 11 flood events across the world from 2017-2019. The dataset contains a combination of automated and hand-labeled surface water for use in training and validation of inundation modeling efforts. The xBD dataset identifies flood-damaged buildings and indicates the scale of damage to each (none, minor, moderate, and major) from four flood events which occurred in the United States, India, Nepal, and Bangladesh from the same time period. The NASA dataset contains hand-labeled water bodies observed in Sentinel-1 imagery during five flood events within the 2017-2019 period. The effort presented here utilizes observations from these previously investigated flood events to generate labels of surface water at the 3-5m spatial resolution provided by Planetscope and facilitate the comparison between public and commercial data. A data pipeline was built which uses clustering algorithms to pick the most suitable overlapping chips between the public data and PlanetScope data for manual labeling. Labels were created manually using NASA’s ImageLabeler tool and include areas of high- and low-confidence water. The high confidence designation is reserved for areas of open, unobstructed water while low confidence is used for areas of suspected water beneath vegetation, clouds, or cloud shadows. Expected to be released in late 2022, the FloodPlanet dataset will include tiled imagery with a unique ID for each 1024x1024 pixel tile, 7 bands of HLS data, and high- and low-confidence flood labels in both shapefile and tiff formats. The authors will follow Spatial Temporal Access Catalog (STAC) guidelines to release FloodPlanet on the Radiant Earth ML hub, which hosts public datasets for machine learning.

Alexander Melancon↗