Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
NASA NTRS2020

TESS Data Release Notes:Reprocessing of Sectors 14–19, DR30 & DR33

TESS data release 30 (DR30) provides reprocessed data products of Sector 14 to 19. The updated data products were generated using version 4.0 of the science processing pipeline and conform to the final set of data anomaly flags defined over the last two years of TESS data analysis and pipeline development. Data release 33 (DR33) corresponds to a multisector search for transiting planets in the same reprocessed data. A detailed description of the changes in the data products in DR30 and DR33 is discussed in§2, and a brief list of changes is summarized here: The timestamps for 2 minute cadence and FFI data are more accurate. The differences between reprocessed data and previous data releases are less than 2.0 seconds in all cases. Photometric apertures were increased in size for targets with T mag<11. Three new Data Anomaly Flags were added to mitigate the effects of scattered light:–Cadences with strong scattered light signals or saturation effects that corrupt the calibration data are flagged and removed from analysis (bit 15, value 16384, “Bad Calibration Exclude”).–Scattered light data anomaly flags are customized for each target, and flagged automatically based on the local background level (bit 13, value 4096, ”Scattered light flag”).–Cadences with insufficient targets to derive cotrending basis vectors are flagged and the PDCSAP FLUX light curves are set to NULL at these times (bit 16, value 32768, “Insufficient Targets for Error Correction Exclude”). The planet search of the reprocessed light curves produced a different set of TCEs from the original processed data. Although there is a high degree of overlap between the original and reprocessed data (∼83% of targets produced TCEs in common), new TCEs were produced in DR30 and not every TCE from previous data releases was recovered. The same is true of the multisector search results from DR33 compared to DR28

TESS'↗
DOE OSTI2025

A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data

High-resolution environmental monitoring is necessary to record, understand, and predict biogeochemical and ecological changes particularly in coastal systems but brings significant challenges in processing and making rapidly available the resulting data. The COMPASS-FME project established a network of coastal observational sites across the Chesapeake Bay and western Lake Erie regions extensively instrumented with soil, vegetation, and weather sensors logging data every 15 min. Our data processing framework, written in R and completely open source, prioritizes rapid model-experiment iteration and makes biogeochemical data rapidly available for quality assurance/quality control, analysis, and model ingestion. This pipeline is distinguished by a standardized and modular approach to data curation, extensive metadata and documentation, and its high performance. These attributes combine to make biogeochemical data rapidly accessible across COMPASS-FME and the broader community. Flexible, powerful, and reproducible approaches to handling high-volume environmental data are crucial for accelerating biogeosciences research.

Pennington, Stephanie C. [Pacific Northwest Nation↗
DOE OSTI2023

Open‐source photovoltaic model pipeline validation against well‐characterized system data

Abstract All freely available plane‐of‐array (POA) transposition models and photovoltaic (PV) temperature and performance models in pvlib‐python and pvpltools‐python were examined against multiyear field data from Albuquerque, New Mexico. The data include different PV systems composed of crystalline silicon modules that vary in cell type, module construction, and materials. These systems have been characterized via IEC 61853‐1 and 61853‐2 testing, and the input data for each model were sourced from these system‐specific test results, rather than considering any generic input data (e.g., manufacturer's specification [spec] sheets or generic Panneau Solaire [PAN] files). Six POA transposition models, 7 temperature models, and 12 performance models are included in this comparative analysis. These freely available models were proven effective across many different types of technologies. The POA transposition models exhibited average normalized mean bias errors (NMBEs) within ±3%. Most PV temperature models underestimated temperature exhibiting mean and median residuals ranging from −6.5°C to 2.7°C; all temperature models saw a reduction in root mean square error when using transient assumptions over steady state. The performance models demonstrated similar behavior with a first and third interquartile NMBEs within ±4.2% and an overall average NMBE within ±2.3%. Although differences among models were observed at different times of the day/year, this study shows that the availability of system‐specific input data is more important than model selection. For example, using spec sheet or generic PAN file data with a complex PV performance model does not guarantee a better accuracy than a simpler PV performance model that uses system‐specific data.

14 SOLAR ENERGY↗
DOE OSTI2025

DoCeph: DPU-Offloaded Messaging in Ceph for Reduced Host CPU Utilization

Ceph is a widely used distributed object store, but its messenger layer imposes substantial CPU overhead on the host. To address this limitation, we propose DoCeph, a DPU-offloaded storage architecture for Ceph that disaggregates the system by offloading the communication-intensive messaging component to the DPU while retaining the storage backend on the host. The DPU efficiently manages communication, using lightweight RPC for metadata operations and DMA for data transfer. Moreover, DoCeph introduces a pipelining technique that overlaps data transmission with buffer preparation, mitigating hardware-imposed transfer size limitations. We implemented DoCeph on a Ceph cluster with NVIDIA BlueField-3 DPUs. Evaluation results indicate that DoCeph cuts host CPU usage by up to 92% while sustaining stable throughput and providing larger performance benefits for object writes over 1 MB.

Park, Kuri [Sogang University]↗
NASA NTRS2003

Infrared Spectroscopy of Star Formation in Galactic and Extragalactic Regions

In this program we proposed to perform a series of spectroscopic studies, including data analysis and modeling, of star formation regions using an ensemble of archival space-based data from the Infrared Space Observatory's Long Wavelength Spectrometer and Short Wavelength Spectrometer, and to take advantage of other spectroscopic databases including the first results from SIRTF. Our emphasis has been on star formation in external, bright IR galaxies, but other areas of research have included young, low or high mass pre-main sequence stars in star formation regions, and the galactic center. The OH lines in the far infrared were proposed as one key focus of this inquiry, because the Principal Investigator (H. Smith) had a full set of OH IR lines from IS0 observations. It was planned that during the proposed 2-1/2 year timeframe of the proposal other data (including perhaps from SIRTF) would become available, and we intended to be responsive to these and other such spectroscopic data sets. The program has the following goals: 1) Refine the data analysis of IS0 observations to obtain deeper and better SNR results on selected sources. The IS0 data itself underwent pipeline 10 reductions in early 2001, and the more 'hands-on data reduction packages' have been released. The IS0 Fabry-Perot database is particularly sensitive to noise and can have slight calibration errors, and improvements are anticipated. We plan to build on these deep analysis tools and contribute to their development. Model the atomic and molecular line shapes, in particular the OH lines, using revised montecarlo techniques developed by the Submillimeter Wave Astronomy Satellite (SWAS) team at the Center for Astrophysics. 2) 3) Use newly acquired space-based SIRTF or SOFIA spectroscopic data as they become available, and contribute to these observing programs as appropriate. 4) Attend scientific meetings and workshops. 5) E&PO activities, especially as related to infrared astrophysics and/or spectroscopy.

Smith, Howard A.↗
NASA NTRS2012

A Detection Pipeline for Galactic Binaries in LISA Data

The Galaxy is suspected to contain hundreds of millions of binary white dwarf systems, a large fraction of which will have sufficiently small orbital period to emit gravitational radiation in band for space-based gravitational wave detectors such as the Laser Interferometer Space Antenna (LISA). LISA's main science goal is the detection of cosmological events (supermassive black hole mergers) etc.) however the gravitational signal from the galaxy will be the dominant contribution to the data - including instrumental noise - over approximately two decades in frequency. The catalogue of detectable binary systems will serve as an unparalleled means of studying the Galaxy. Furthermore, to maximize the scientific return from the mission, the data must be "cleansed" of the galactic foreground. We will present an algorithm that can accurately resolve and subtract greater than or equal to 10000 of these sources from simulated data supplied by the Mock LISA Data Challenge Task Force. Using the time evolution of the gravitational wave frequency, we will reconstruct the position of the recovered binaries and show how LISA will sample the entire compact binary population in the Galaxy.

Littenberg, Tyson B.↗
NASA NTRS2021

Search for anisotropic gravitational-wave backgrounds using data from Advanced LIGO and Advanced Virgo's first three observing runs

We report results from searches for anisotropic stochastic gravitational-wave backgrounds using data from the first three observing runs of the Advanced LIGO and Advanced Virgo detectors. For the first time, we include Virgo data in our analysis and run our search with a new efficient pipeline called PyStochon data folded over one sidereal day. We use gravitational-wave radiometry(broadband and narrow band) to produce sky maps of stochastic gravitational-wave backgrounds and to search for gravitational waves from point sources. A spherical harmonic decomposition method is employed to look for gravitational-wave emission from spatially-extended sources. Neither technique found evidence of gravitational-wave signals. Hence we derive 95% confidence-level upper limit sky maps on the gravitational-wave energy flux from broadband point sources, ranging from F(α,Θ) < (0.013−7.6)×10^(−8)erg/sq. cm s Hz,and on the (normalized) gravitational-wave energy density spectrum from extended sources, ranging from Ω(α,Θ) < (0.57−9.3)×10^(−9) per sr, depending on direction (Θ) and spectral index (α). These limits improve upon previous limits by factors of 2.9−3.5. We also set 95% confidence level upper limits on the frequency-dependent strain amplitudes of quasimonochromatic gravitational waves coming from three interesting targets, Scorpius X-1, SN1987A and the Galactic Center, with best upper limits range fromh(0) < (1.7−2.1)×10^(−25), a factor of ≥ 2.0 improvement compared to previous stochastic radiometer searches.

R. Abbott↗
NASA NTRS2021

Automated Data Accountability for Missions in Mars Rover Data

As the Mars Curiosity Rover transmits data to the JPL Ground Data System (GDS), it frequently observes data loss and corruption, requiring re-transmits from the rover and Ground Data System Analysts (GDSA) to monitor the downlink process. As new missions are launched, the GDSA team redistributes analysts to these new missions, causing shortages in previous missions. The GDSA team can significantly benefit from the automation and optimization of the downlink process of telemetry data. In fact, there is a need for a better understanding of why the data is corrupted, so that the GDSA team can best determine the root cause of the issues in the GDS. This paper presents machine learning and deep learning based approaches to automate and optimize the detection of data loss. We first created a pipeline to automatically accumulate data from the telemetry databases (MAROS, Telemetry Data Storage, and GDS Elastic Search Database) in the downlink process. With our newly created datasets, we perform feature selection to supplement the GDSA understanding of the downlink process and provide supplemental analysis on the importance of different features. We implement various machine learning and deep learning based models, including support vector machines, ensemble methods, and deep neural networks and evaluate their accuracies in identifying whether a downlink process is complete or incomplete. We utilize fast hyperparameter optimization methods that allow our models to quickly be re-trained, allowing them to quickly be tuned and optimized on daily incoming data in real time. This hyperparameter optimization also allows our methods to be quickly integrated into other JPL missions. Our results show that our best-performing machine learning and deep learning based models outperform the existing GDSA detection software by 6 accuracy points and can aid analysts by providing insights into the data accountability problem. Since these various machine learning and deep learning approaches vary significantly in interpretability, we provide a discussion on the tradeoffs between their performance and trustworthiness in helping detect issues in data transmission.

Divsalar, Dariush↗
NASA NTRSDate not supplied

Enabling Model Organism and Commercial Astronaut Data Access Through the NASA Open Science Data Repository

NASA’s Open Science Data Repository (OSDR) brings together omics data from NASA’s GeneLab project and non-omics data, including physiological, phenotypic, imaging, and behavioral data from NASA’s Ames Life Sciences Data Archive (ALSDA) collected from decades of space biology research, providing open and FAIR (findable, accessible, interoperable, and reusable) access of these precious data to scientists world-wide. This rich source of meticulously curated metadata and data from spaceflight and analog studies has been mined by the scientific community resulting in dozens of high impact scientific publications that reveals a complex network of molecular and physiological effects of spaceflight across living systems, from microbes to plants, to mammals. Understanding how these effects translate to the human condition is critical as we move deeper into the era of commercial space travel. However, the integration of data, specifically omics data, from astronauts is particularly challenging due to their sensitive nature. OSDR has risen to this challenge by developing a mechanism to control access to identifiable levels of omics data, such as raw sequence data, while enabling public access to processed, unidentifiable, data and associated metadata that will allow the scientific community to interrogate human astronaut data alongside data from model organisms to begin answering these critical questions. The 2021 SpaceX Inspiration4 (I4) mission collected a comprehensive atlas of biological measurements from four civilian astronauts, providing a wealth of data to characterize the effects of spaceflight on the human body. These data include both non-omics and omics assays such as direct RNA sequencing (RNA-seq), single nuclei ATAC-seq and RNA-seq, metagenomics, proteomics, and comprehensive metabolic and cytokine panels, all of which have been integrated into the OSDR system across no less than 9 studies. Each study has been carefully curated using community-backed OSDR standards for sample and assay level metadata ensuring these data are findable and accessible. In addition to hosting both raw and processed data from the principal investigator team for each assay type, the GeneLab team plans to re-process the I4 omics data using GeneLab’s standard processing pipelines. The GeneLab processed data outputs will allow for comparisons across studies on OSDR and enable visualization of these data through the OSDR data visualization platform thereby enabling data reusability and interoperability. Here we describe the robust privacy and security protocols implemented by OSDR to safeguard sensitive health data from astronauts while facilitating metadata and processed data sharing for research purposes. We further provide a road map for navigating the vast amount of data provided for each I4 study on the OSDR, including experimental design, associated experiments, payloads, and missions, data generation and analysis protocols, and associated scientific articles. Additionally, we illustrate how to interrogate the standardized metadata provided in the sample and assay tables as well as various means to download and access the data including programmatically through the GeneLab Open API (GLOpenAPI). The open access of datasets in NASA’s OSDR provides a unique opportunity for the scientific community, as well as citizen scientists and students, to continue using OSDR resources to further unlock profound insights into the consequences of space travel on the human body. Through implementation of security measures to protect sensitive human data, the OSDR seeks to strengthen the science exchange between the Biological and Physical Sciences Program and the Human Research Program, per recommendation 4-1 of the 2023-2032 Decadal Survey, and encourage further sharing and dissemination of astronaut data to provide the scientific community with the resources needed to lay the groundwork for developing targeted mitigation strategies to help withstand the rigors of long-duration spaceflight.

Amanda Marie Saravia-butler↗
NASA NTRSDate not supplied

Enabling Model Organism and Commercial Astronaut Data Access Through the NASA Open Science Data Repository

NASA’s Open Science Data Repository (OSDR) brings together omics data from NASA’s GeneLab project and non-omics data, including physiological, phenotypic, imaging, and behavioral data from NASA’s Ames Life Sciences Data Archive (ALSDA) collected from decades of space biology research, providing open and FAIR (findable, accessible, interoperable, and reusable) access of these precious data to scientists world-wide. This rich source of meticulously curated metadata and data from spaceflight and analog studies has been mined by the scientific community resulting in dozens of high impact scientific publications that reveals a complex network of molecular and physiological effects of spaceflight across living systems, from microbes to plants, to mammals. Understanding how these effects translate to the human condition is critical as we move deeper into the era of commercial space travel. However, the integration of data, specifically omics data, from astronauts is particularly challenging due to their sensitive nature. OSDR has risen to this challenge by developing a mechanism to control access to identifiable levels of omics data, such as raw sequence data, while enabling public access to processed, unidentifiable, data and associated metadata that will allow the scientific community to interrogate human astronaut data alongside data from model organisms to begin answering these critical questions. The 2021 SpaceX Inspiration4 (I4) mission collected a comprehensive atlas of biological measurements from four civilian astronauts, providing a wealth of data to characterize the effects of spaceflight on the human body. These data include both non-omics and omics assays such as direct RNA sequencing (RNA-seq), single nuclei ATAC-seq and RNA-seq, metagenomics, proteomics, and comprehensive metabolic and cytokine panels, all of which have been integrated into the OSDR system across no less than 9 studies. Each study has been carefully curated using community-backed OSDR standards for sample and assay level metadata ensuring these data are findable and accessible. In addition to hosting both raw and processed data from the principal investigator team for each assay type, the GeneLab team plans to re-process the I4 omics data using GeneLab’s standard processing pipelines. The GeneLab processed data outputs will allow for comparisons across studies on OSDR and enable visualization of these data through the OSDR data visualization platform thereby enabling data reusability and interoperability. Here we describe the robust privacy and security protocols implemented by OSDR to safeguard sensitive health data from astronauts while facilitating metadata and processed data sharing for research purposes. We further provide a road map for navigating the vast amount of data provided for each I4 study on the OSDR, including experimental design, associated experiments, payloads, and missions, data generation and analysis protocols, and associated scientific articles. Additionally, we illustrate how to interrogate the standardized metadata provided in the sample and assay tables as well as instructions for how to download and access the data. The I4 datasets described here re present the first ever comprehensive collection of commercial astronaut data.

Amanda M Saravia-Butler↗
DOE OSTI2025

The Dynamic Networks Experiments: Virtual Experiments to Quantify Gains in Nuclear Explosion Monitoring

We describe an ongoing series of virtual experiments conducted collaboratively by four United States National Laboratories: Sandia National Laboratories, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and Pacific Northwest National Laboratory. These Dynamic Network Experiments (DNEs) provide an experimental framework to evaluate the potential impact of new research tools on nuclear explosion monitoring. The second DNE (DNE2), completed in 2024, exploited waveform data (seismic, infrasound, and electromagnetic) that was recorded by multi-modal sensors within and near the Nevada National Security Site and synthetic radionuclide signatures over multiple time periods. During the execution of DNE2, we processed and analyzed data through a multi-stage event processing pipeline that ingested raw data, performed quality control, detected signals, built events from these signals, located these events, and characterized the events’ source types and sizes. For each stage and over the entire event processing pipeline, we evaluated performance changes by comparing the performance of new data processing methods, models, and algorithms against a baseline. We also performed an additional execution phase to assess event processing pipeline function, speed, and efficiency against that of an expert analyst, including computational and manual efforts. Finally, we assessed the impact and effort of modern computing infrastructure on the monitoring pipeline. This paper describes key elements of the DNEs, from formulation through execution, as demonstrated in DNE2. The DNEs introduce several novel concepts to quantitatively measure the potential impact of new methods on explosion monitoring, including the collaborative design of multi-modal datasets, performance and logistical metrics, and integrated analyses.

42 ENGINEERING↗
NASA NTRSDate not supplied

Open Science for Life in Space: Data Sharing and Tools for Knowledge Discovery

Molecular-omics, physiological-phenotypic-behavioral, and environmental-radiation telemetry data from spaceflight biological and health studies are increasingly being made findable, accessible, interoperable, and reusable for the scientific public. These data, as well as space science-relevant biospecimens, are available through NASA’s Open Science Data Repository (OSDR), which is the new umbrella grouping of NASA GeneLab, the Ames Life Sciences Data Archive (ALSDA), and the NASA Biological Institutional Scientific Collection (NBISC). The quality of data is underpinned by datasets having rich metadata (determined through Analysis Working Group members), processing pipelines to enable data reuse standards, and ontologies specifying terminology semantics (e.g., the Radiation Biology Ontology).

space biology↗
DOE OSTI2024

Benefits and Limits of Phasing Alleles for Network Inference of Allopolyploid Complexes

Abstract Accurately reconstructing the reticulate histories of polyploids remains a central challenge for understanding plant evolution. Although phylogenetic networks can provide insights into relationships among polyploid lineages, inferring networks may be hindered by the complexities of homology determination in polyploid taxa. We use simulations to show that phasing alleles from allopolyploid individuals can improve phylogenetic network inference under the multispecies coalescent by obtaining the true network with fewer loci compared with haplotype consensus sequences or sequences with heterozygous bases represented as ambiguity codes. Phased allelic data can also improve divergence time estimates for networks, which is helpful for evaluating allopolyploid speciation hypotheses and proposing mechanisms of speciation. To achieve these outcomes in empirical data, we present a novel pipeline that leverages a recently developed phasing algorithm to reliably phase alleles from polyploids. This pipeline is especially appropriate for target enrichment data, where the depth of coverage is typically high enough to phase entire loci. We provide an empirical example in the North American Dryopteris fern complex that demonstrates insights from phased data as well as the challenges of network inference. We establish that our pipeline (PATÉ: Phased Alleles from Target Enrichment data) is capable of recovering a high proportion of phased loci from both diploids and polyploids. These data may improve network estimates compared with using haplotype consensus assemblies by accurately inferring the direction of gene flow, but statistical nonidentifiability of phylogenetic networks poses a barrier to inferring the evolutionary history of reticulate complexes.

Evolutionary Biology↗
DOE OSTI2023

Agilent CRADA (Abstract)

The CRADA between Agilent Technologies Inc. and Battelle will focus on five software components as listed below: Prototype 4D Feature Finding functionality with a particular focus on recovering low level features and extending the bottom end dynamic range of IM-MS technology. Compare and contrast developments to current 4D Feature Finding capabilities. Highlight important algorithmic aspects employed. Implement the PNNL saturation correction algorithm. Agilent will give PNNL the needed data file access API and assistance in understanding it implementation and any needed instrumental aspects. Supported high resolution products to include Agilent’s TOF, QTOF and IM-QTOF mass spectrometers. PNNL will then work with Agilent to benchmark performance. Implementation of the PNNL Hadamard de-multiplexing algorithm. Agilent will give provide PNNL the needed date file access API access and as needed assistance in understanding the current Agilent multiplexed IM offering. PNNL will then work with Agilent on benchmark performance. Add ion mobility collision cross sections to existing and new metabolomic libraries for data analysis with Agilent’s informatics program MPP/ID Browser. PNNL will work with Agilent to create a software pipeline that takes data from chemical and metabolic standards and properly formats it for inclusion in MPP accessible libraries, using the collision cross section as a new separation dimension. Improvements of MPP multidimensional matching to identify metabolomic features using multiple characteristics beyond retention time and accurate mass. Most significantly matching will include analyte collision cross section with proposed support for sample fraction or RapidFire cartridge and fragmentation spectra. PNNL will work with Agilent to modify and improve the current MPP analysis pipeline to allow for creating, aligning, and identifying MS features defined by accurate mass, collision cross section and chromatographic retention time. As additional criteria such as fraction or RapidFire cartridge type are supported in the identification process, then they also will become part of the automation workflow. This includes the automation of said system to work with command line program (i.e. not a GUI) sufficient for programmatic execution in a pipeline.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Comparison of Ring Diagrams Based on the Doppler Shifts of Synthetic Data Obtained with Bisector Method and SDO/HMI Pipeline

Ring diagrams are cross-sections of three-dimensional spatiotemporal power spectra of solar oscillations. The rings reveal information about sub-surface flows and represent an important tool for helioseismology. Ring diagrams can be constructed using Doppler velocity or intensity maps of spectral lines. How the velocities are computed is an important factor for accuracy of information we can retrieve on subsurface flows. In our work, the ring diagrams are generated from Doppler shift data of synthesized Fe I 6173 Å line. We compare ring diagrams computed by two methods–HMI line-of-sight pipeline and the bisector of Fe line. Fe I line is synthesized for StellarBox 3D Radiative hydrodynamic simulations under LTE assumption. We aim to answer the following questions: 1.How do power spectra obtained from velocities computed with the HMI pipeline and bisector of the Fe I 6173Å compare? 2.How do the power spectral density retrieved with each method vary with heliocentric angle? 3.What is the effect of changing resolution on the power spectral density in ring diagrams obtained with the two methods?

SMD↗
NASA NTRS2019

Numerical Simulation on Liquid Hydrogen Chill-Down Process of Vertical Pipeline

In order to improve the cryogenic propellant management technologies for a liquid hydrogen rocket with high specific impulse, JAXA, the University of Tokyo, and the NASA Glenn Research Center have jointly organized a multi-agency model validation collaboration project. As part of this project, JAXA's boiling simulation was validated with NASA's experimental data on vertical pipeline chill-down. Simulation results were in good agreement with the experimental data obtained using an improved boiling model to reproduce the spray flow. This activity achieved liquid hydrogen turbo-pump simulation at JAXA for grasping the boiling flow phenomenon from engine cut-off to re-ignition. This joint research resulted in an international cooperative relationship for discussing the cryogenic propellant management technologies necessary to develop next-generation liquid rockets.

Umemura, Yutaka↗