Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
NASA NTRS1988

Real time pipelined system for forming the sum of products in the processing of video data

A 3-by-3 convolver utilizes 9 binary arithmetic units connected in cascade for multiplying 12-bit binary pixel values P sub i which are positive or two's complement binary numbers by 5-bit magnitide (plus sign) weights W sub i which may be positive or negative. The weights are stored in registers including the sign bits. For a negative weight, the one's complement of the pixel value to be multiplied is formed at each unit by a bank of 17 exclusive or gates G sub i under control of the sign of the corresponding weight W sub i, and a correction is made by adding the sum of the absolute values of all the negative weights for each 3-by-3 kernel. Since this correction value remains constant as long as the weights are constant, it can be precomputed and stored in a register as a value to be added to the product PW of the first arithmetic unit.

Wilcox, Brian↗
NASA NTRS2017

Status of the TESS Science Processing Operations Center

The Transiting Exoplanet Survey Satellite (TESS) science pipeline is being developed by the Science Processing Operations Center (SPOC) at NASA Ames Research Center based on the highly successful Kepler Mission science pipeline. Like the Kepler pipeline, the TESS science pipeline will provide calibrated pixels, simple and systematic error-corrected aperture photometry, and centroid locations for all 200,000+ target stars, observed over the 2-year mission, along with associated uncertainties. The pixel and light curve products are modeled on the Kepler archive products and will be archived to the Mikulski Archive for Space Telescopes (MAST). In addition to the nominal science data, the 30-minute Full Frame Images (FFIs) simultaneously collected by TESS will also be calibrated by the SPOC and archived at MAST. The TESS pipeline will search through all light curves for evidence of transits that occur when a planet crosses the disk of its host star. The Data Validation pipeline will generate a suite of diagnostic metrics for each transit-like signature discovered, and extract planetary parameters by fitting a limb-darkened transit model to each potential planetary signature. The results of the transit search will be modeled on the Kepler transit search products (tabulated numerical results, time series products, and pdf reports) all of which will be archived to MAST.

high performance computing↗
NASA NTRS2017

Kepler Planet Detection Metrics: Per-Target Detection Contours for Data Release 25

A necessary input to planet occurrence calculations is an accurate model for the pipeline completeness (Burke et al., 2015). This document describes the use of the Kepler planet occurrence rate products in order to calculate a per-target detection contour for the measured Data Release 25 (DR25) pipeline performance. A per-target detection contour measures for a given combination of orbital period, Porb, and planet radius, Rp, what fraction of transit signals are recoverable by the Kepler pipeline (Twicken et al., 2016; Jenkins et al., 2017). The steps for calculating a detection contour follow the procedure outlined in Burke et al. (2015), but have been updated to provide improved accuracy enabled by the substantially larger database of transit injection and recovery tests that were performed on the final version (i.e., SOC 9.3) of the Kepler pipeline (Christiansen, 2017; Burke Catanzarite, 2017a). In the following sections, we describe the main inputs to the per-target detection contour and provide a worked example of the python software released with this document (Kepler Planet Occurrence Rate Tools KeplerPORTs)1 that illustrates the generation of a detection contour in practice. As background material for this document and its nomenclature, we recommend the reader be familiar with the previous method of calculating a detection contour (Section 2 of Burke et al.,2015), input parameters relevant for describing the data quantity and quality of Kepler targets (Burke Catanzarite, 2017b), and the extensive new transit injection and recovery tests of the Kepler pipeline (Christiansen et al., 2016; Burke Catanzarite, 2017a; Christiansen, 2017).

Planet Detection Metrics↗
DOE OSTI2025

Explainable Machine Learning for Functional Data

Black-box machine learning models are recognized as useful tools for prediction applications, but the algorithmic complexity of some models causes interpretation challenges. Explainability methods have been proposed to provide insight into these models, but there is little research focused on supervised modeling with functional data inputs. We argue that, especially in applications of high consequence, it is important to explicitly model the functional dependence in a black-box analysis to not obscure or misrepresent patterns in explanations. As such, we propose the V ariable importance E xplainable E lastic S hape A nalysis (VEESA) pipeline for training supervised machine learning models with functional inputs. The pipeline is an analysis process that includes the data preprocessing, modeling, and post-hoc explanations. The preprocessing is done using elastic functional principal components analysis, which accounts for vertical and horizontal variability in functional data and, ultimately, allows for explanations in the original data space that identify the important functional variability without bias due to correlated variables. Here, we demonstrate the pipeline on two high-consequence applications: explosives classification for national security and inkjet printer identification in forensic science. The applications exhibit the VEESA pipeline’s ability to provide an understanding of the characteristics of the functional data useful for prediction. Code for implementing the pipeline is available in the veesa R package (and supplemental python code).

Elastic Shape Analysis↗
DOE OSTI2023

SPLENDAQ: A Detector-Agnostic Data Acquisition System for Small-Scale Physics Experiments

Many scientific applications from rare-event searches to condensed matter system characterization to high-rate nuclear experiments require time-domain triggering on a raw stream of data, where the triggering is generally threshold-based or randomly acquired. When carrying out detector R &D, there is a need for a general data acquisition (DAQ) system to quickly and efficiently process such data. In the SPLENDOR collaboration, we are developing the Python-based SPLENDAQ package for this exact purpose—it offers two main features for offline analysis of continuous data: a threshold triggering algorithm based on the time-domain optimal filter formalism and an algorithm for randomly choosing nonoverlapping segments for noise measurements. Further, combined with the commercially available Moku platform, developed by Liquid Instruments, we have a full pipeline of event building off raw data with minimal setup. Here, we review the underlying principles of this detector-agnostic DAQ package and give concrete examples of its utility in various applications.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2021

Outlook for artificial intelligence and machine learning at the NSLS-II

Abstract We describe the current and future plans for using artificial intelligence and machine learning (AI/ML) methods at the National Synchrotron Light Source II (NSLS-II), a scientific user facility at the Brookhaven National Laboratory. We discuss the opportunity for using the AI/ML tools and techniques developed in the data and computational science areas to greatly improve the scientific output of large scale experimental user facilities. We describe our current and future plans in areas including from detecting and recovering from faults, optimizing the source and instrument configurations, streamlining the pipeline from measurement to insight, through data acquisition, processing, analysis. The overall strategy and direction of the NSLS-II facility in relation to AI/ML is presented.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A Task Based Approach for Co-Scheduling Ensemble Workloads on Heterogeneous Nodes

Scientific workflows consist of multiple, connected applications, with data and results flowing from one to another in a pipeline. Traditionally, such workflows are executed in sequential order, storing intermediate data in storage disks. Co-scheduling application workflows concurrently on the same compute nodes would greatly reduce the cost of moving data to/from storage and allow real-time analysis of intermediate results. Nevertheless, most parallel programming runtimes do not allow seamless integration of various applications in a scientific workflow, in part due to the complexity of managing data and resources. The situation is even more complicated for heterogeneous systems. In this work we extend the Minos Computing Library (MCL) runtime to accelerate pipe-lined and parallel workloads where multiple applications are running in the same system. MCL’s asynchronous task library and runtime dynamically manages resources to allow co-scheduling of multiple processes sharing heterogeneous resources. In addition, we design a custom ex- tension of the Open Compute Language (OpenCL) to enable multiple processes to share device memory. We enable MCL to coordinate these shared buffers to allow for easy, fast data sharing between applications. Using malleable micro-benchmarks and two application workflows that combine scientific simulation and AI-based analysis, we show that our method outperforms traditional approaches.

Index Terms—Parallel systems, Scheduling and Task ↗
DOE OSTI2023

A review on recent machine learning applications for imaging mass spectrometry studies

Imaging mass spectrometry (IMS) is a powerful analytical technique widely used in biology, chemistry, and materials science fields that continue to expand. IMS provides a qualitative compositional analysis and spatial mapping with high chemical specificity. The spatial mapping information can be 2D or 3D depending on the analysis technique employed. Due to the combination of complex mass spectra coupled with spatial information, large high-dimensional datasets (hyperspectral) are often produced. Therefore, the use of automated computational methods for an exploratory analysis is highly beneficial. The fast-paced development of artificial intelligence (AI) and machine learning (ML) tools has received significant attention in recent years. These tools, in principle, can enable the unification of data collection and analysis into a single pipeline to make sampling and analysis decisions on the go. There are various ML approaches that have been applied to IMS data over the last decade. Here, in this review, we discuss recent examples of the common unsupervised (principal component analysis, non-negative matrix factorization, k-means clustering, uniform manifold approximation and projection), supervised (random forest, logistic regression, XGboost, support vector machine), and other methods applied to various IMS datasets in the past five years. The information from this review will be useful for specialists from both IMS and ML fields since it summarizes current and representative studies of computational ML-based exploratory methods for IMS.

47 OTHER INSTRUMENTATION↗
NASA NTRS2002

Thermal Performance of Cryogenic Piping Multilayer Insulation in Actual Field Installations

A standardized way of comparing the thermal performance of different pipelines in different sizes is needed. Vendor data for vacuum-insulated piping are typically given in heat leak rate per unit length (W/m) for a specific diameter pipeline. An overall k-value for actual field installations (k(sub oafi)) is therefore proposed as a more generalized measure for thermal performance comparison and design calculation. The k(sub oafi) provides a direct correspondence to the k-values reported for insulation materials and illustrates the large difference between ideal multilayer insulation (MLI) and actual MLI performance. In this experimental research study, a section of insulated piping was tested under cryogenic vacuum conditions, including simulated spacers and bending. Several different insulation systems were tested using a 1-meter-long cylindrical cryostat test apparatus. The simulated spacers tests showed significant degradation in the thermal performance of a given insulation system. An 18-meter-long pipeline test apparatus is now in operation at the Cryogenics Test Laboratory, NASA Kennedy Space Center, for conducting liquid nitrogen thermal performance tests.

Fesmire, J.↗
DOE OSTI2025

A DATA EFFICIENT SPARSE MODELING FRAMEWORK FOR POWER ESTIMATION IN WATER TREATMENT SENSING OPERATIONS

With increasing freshwater scarcity, advanced process design mechanisms such as Closed-Circuit Reverse Osmosis (CCRO) and Digital/Physical Twin systems are gaining traction in water treatment and reuse operations. While digital and physical twin models enable improved system insight and control, their development is often expensive and computationally intensive, requiring large volumes of synthetic or experimental data to characterize underlying process dynamics. This work introduces a sparse surrogate modeling framework to estimate power consumption from measured flow and pressure variables, along with their nonlinear polynomial and interaction expansions. To ensure model reliability and reduce overfitting, a two-stage pipeline is proposed. First, a dynamic data filtering algorithm is employed to remove uninformative observations and transient operational states. Second, a sparse penalized regression technique is applied to select a minimal set of parsimonious features. The proposed model achieves high sparsity, retaining only 7 out of 34 candidate features (≈79.41% sparsity) while delivering a root mean square error (RMSE) of 0.072 on the test dataset.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗
DOE OSTI2024

An empirical DNA-based identification of morphologically similar snappers (Lutjanus campechanus, Lutjanus purpureus) using a versatile bioinformatics workflow for the discovery and analysis of informative single-nucleotide polymorphisms

The commercially important speciesLutjanus campechanus(Northern/Gulf red snapper) andLutjanus purpureus(Southern/Caribbean red snapper) are the protagonists of a decade’s long taxonomic debate over their species delimitation, due in part to partial habitat overlap, extensive morphological similarity, and the lack of resolution when applying canonically reliable DNA barcoding approaches. In this study, we leveraged publicly available RAD-Seq data forL. campechanusandL. purpureusto identify species-informative single‐nucleotide polymorphisms (SNPs) at the genome scale that were successful in distinguishing the Northern and Southern red snappers, while also detecting individuals exhibiting introgression. This 4-step empirical approach demonstrates the value of applying novel bioinformatics pipelines to existing genome-scale data to maximize the distillation of informative subsets. Our results facilitate economically relevant species identification in addition to confirming or challenging species identifications for specimens with data in public databases. These findings and their applications will benefit future sustainability strategies and broader research questions surrounding these overfished and evolutionarily entangled snapper species.

Environmental Sciences & Ecology↗
NASA NTRS2000

A Complete UV Atlas of Standard Stars

The general objective of this project is to provide a comprehensive ultraviolet spectral atlas of stars based on the data in the IUE Final Archive. The data have been uniformly processed by the IUE NEWSIPS pipeline system. The NEWSIPS gives an improved signal to noise ratio for the data that will be especially beneficial to low signal portions of the spectra; and it also gives more accurate fluxes for stars by adopting the latest absolute calibration and measured instrument parameters. Our new atlas, therefore, provides more uniform and accurate data than the version previously published. In addition, considerable efforts were devoted to obtain reliable spectral types, V,B-V, and E(B-V). Our goal is to provide the information which can be used with reasonable confidence by scientists.

Wu, Chi-Chao↗
NASA NTRS2003

Probing the Reverse Shock in an Oxygen-Rich Supernova Remnant

The aim of this proposal was to examine the O VI emission at three positions around the X-ray bright ring of the remnant in order to investigate the relation between the O VI emission, the X-ray O VII and O VIII emission, and the optical [O III] emission, and how these vary around the rim of the remnant. So far, only one Far Ultraviolet Spectroscopic Explorer (FUSE) pointing at the remnant has been performed, together with the background pointing needed to subtract the diffuse scattered light from nearby O stars. The data were recalibrated using the CalFUSE calibration pipeline, but the emission is faint enough that hand processing will be needed for this pointing to optimize the extraction. Rereduction of the data is underway. A postdoctoral fellow has also been hired, and in part she will assist with the analysis.

Sonneborn, George↗
DOE OSTI2024

BioSiting Tool (BioSiting) v2

The BioSiting Tool provides a geospatial interface for analyzing bioeconomy resources and infrastructure across the continental U.S. The tool integrates empirical and modeled data from a broad range of sources. Bioeconomy resources mapped in the tool include agricultural residues, forest residues, municipal solid waste streams, food waste, manure, fats, oils and greases and potential yields of energy crops. Infrastructure mapped in the tool includes biorefineries, material recovery facilities, anaerobic digesters, wastewater treatment plants, combustion plants, district energy systems, crude oil pipelines, petroleum pipelines, natural gas pipelines, railways and freight terminals. Additional data layers include environmental justice indicators at the census tract level and carbon dioxide geologic storage potential. Users can select a location on the map, define a buffer radius in kilometers and generate an inventory of all bioecomony resources within the buffer zone. Data from the tool can be downloaded from individual buffer zones, or at the state or national level.

Huntington, Tyler↗
DOE OSTI2024

The Phenix‐AlphaFold webservice: Enabling AlphaFold predictions for use in Phenix

Abstract Advances in machine learning have enabled sufficiently accurate predictions of protein structure to be used in macromolecular structure determination with crystallography and cryo‐electron microscopy data. The Phenix software suite has AlphaFold predictions integrated into an automated pipeline that can start with an amino acid sequence and data, and automatically perform model‐building and refinement to return a protein model fitted into the data. Due to the steep technical requirements of running AlphaFold efficiently, we have implemented a Phenix‐AlphaFold webservice that enables all Phenix users to run AlphaFold predictions remotely from the Phenix GUI starting with the official 1.21 release. This webservice will be improved based on how it is used by the research community and the future research directions for Phenix.

Poon, Billy K.↗
DOE OSTI2026

PV Fleet Performance Data Initiative 2026 Update

We provide an update on the PV Fleet Performance Data Initiative at the 2026 PV Reliability Workshop. Our latest runs incorporate additional data sources and an integrated analysis pipeline run on our Kestrel HPC cluster. Initial degradation findings suggest that single-axis tracked PV systems exhibit higher performance loss rates than fixed-tilt systems, an increase of 0.5 %/yr, almost double. We discuss multiple methods for identifying stuck tracker rows, which are suspected to be a contributor to the enhanced degradation. Through satellite image detection and data-driven approaches we address the topic of identifying when stuck trackers are occuring and to what extent the problem exists. Preliminary results suggest that the increased performance loss detected for the tracked systems would be consistent with stuck tracker rows affecting on the order of 5% - 10% of the system.

14 SOLAR ENERGY↗
NASA NTRS2019

Citizen Science Twitter Data Management for Earth Science Applications

Social media data can provide useful real-time and historical information relating to the natural world, but managing this data poses challenges. Scientists at GES DISC are exploring the potential of Twitter data to augment precipitation data from the Global Precipitation Measurement (GPM) mission. However, the format of Twitter data is unconventional in the context of NASA data centers, resulting in frustration for scientists who need to work with the data. This study investigated procedures and standards needed to properly manage Twitter data to make them compatible with these data centers. After comparing databases, the study found that the MongoDB database was best suited for the storage of raw Twitter data due to its flexibility, ability to be accessed by multiple users, and querying functionality. The study used the Python package Zarr to transform processed Twitter data into a gridded format similar to that of satellite data. Each Tweet was mapped onto a time-space grid; each grid location contained information about Tweet attributes and precipitation. The study developed a pipeline for downloading, storing, and gridding Twitter data and transformed Twitter data into an understandable format for users of NASA satellite data.

Li, Rachel↗
NASA NTRS1993

The initial data products from the EUVE software - A photon's journey through the End-to-End System

The End-to-End System (EES) is a unique collection of software modules created for use at the Center for EUV Astrophysics. The 'pipeline' is a shell script which executes selected EES modules and creates initial data products: skymaps, data sets for individual sources (called 'pigeonholes') and catalogs of sources. This article emphasizes the data from the all-sky survey, conducted between July 22, 1992 and January 21, 1993. A description of each of the major data products will be given and, as an example of how the pipeline works, the reader will follow a photon's path through the software pipeline into a pigeonhole. These data products are the primary goal of the EUVE all-sky survey mission, and so their relative importance for the follow-up science will also be discussed.

Antia, Behram↗