SEARCH · Search NASA

Results for “Science Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

NASA NTRS2018

The Zwicky Transient Facility: Data Processing, Products, and Archive

The Zwicky Transient Facility (ZTF) is a new robotic time-domain survey currently in progress using the Palomar 48-inch Schmidt Telescope. ZTF uses a 47 square degree field with a 600 megapixel camera to scan the entire northern visible sky at rates of ∼3760 square degrees/hour to median depths of g ~ 20.8 and r ~ 20.6 mag (AB, 5σ in 30 sec). We describe the Science Data System that is housed at IPAC, Caltech. This comprises the data-processing pipelines, alert production system, data archive, and user interfaces for accessing and analyzing the products. The real-time pipeline employs a novel image-differencing algorithm, optimized for the detection of point-source transient events. These events are vetted for reliability using a machine-learned classifier and combined with contextual information to generate data-rich alert packets. The packets become available for distribution typically within 13 minutes (95th percentile) of observation. Detected events are also linked to generate candidate moving-object tracks using a novel algorithm. Objects that move fast enough to streak in the individual exposures are also extracted and vetted. We present some preliminary results of the calibration performance delivered by the real-time pipeline. The reconstructed astrometric accuracy per science image with respect to Gaia DR1 is typically 45 to 85 milliarcsec. This is the RMS per-axis on the sky for sources extracted with photometric S/N ≥10 and hence corresponds to the typical astrometric uncertainty down to this limit. The derived photometric precision (repeatability) at bright unsaturated fluxes varies between 8 and 25 millimag. The high end of these ranges corresponds to an airmass approaching ∼2—the limit of the public survey. Photometric calibration accuracy with respect to Pan-STARRS1 is generally better than 2%. The products support a broad range of scientific applications: fast and young supernovae; rare flux transients; variable stars; eclipsing binaries; variability from active galactic nuclei; counterparts to gravitational wave sources; a more complete census of Type Ia supernovae; and solar-system objects.

Frank J. Masci↗

DOE OSTI2024

A Science Gateway for the Repeatable Analysis of Machine Learning Predicted Gravity Anomalies

In recent years, deep learning has become an increasingly popular alternative for modeling in geoscience applications due to its scalability and efficiency. However, the interpretability, compute, data volume, and hyperparameter tuning requirements of deep learning models make development and monitoring difficult. Furthermore, model explainability and communicating results obtained by these models to users or domain experts is a challenge, as domain experts in geoscience also need to have a deep understanding of how those models function in order to support their scientific works. Here, we describe a science gateway and machine learning pipeline for predicting gravity anomalies from geophysical data. The gateway, built on open-source technologies, provides a holistic view of the pipeline through interactive visualizations aimed at enabling efficient exploratory data analysis. The repeatability, reproducibility, and monitoring capabilities of this overall system allow us to iterate and analyze at scale. Using this pipeline and gateway, we can repeatedly produce accurate high-resolution gravity anomaly datasets. By describing the underlying technologies, implementation, and results, here we provide a foundation for the broader adoption of science gateways into cross-cutting geoscience and machine learning research projects as a means to improve the scientific discovery and collaboration in the geophysics and computational sciences community.

58 GEOSCIENCES↗

DOE OSTI2025

Explainable Machine Learning for Functional Data

Black-box machine learning models are recognized as useful tools for prediction applications, but the algorithmic complexity of some models causes interpretation challenges. Explainability methods have been proposed to provide insight into these models, but there is little research focused on supervised modeling with functional data inputs. We argue that, especially in applications of high consequence, it is important to explicitly model the functional dependence in a black-box analysis to not obscure or misrepresent patterns in explanations. As such, we propose the V ariable importance E xplainable E lastic S hape A nalysis (VEESA) pipeline for training supervised machine learning models with functional inputs. The pipeline is an analysis process that includes the data preprocessing, modeling, and post-hoc explanations. The preprocessing is done using elastic functional principal components analysis, which accounts for vertical and horizontal variability in functional data and, ultimately, allows for explanations in the original data space that identify the important functional variability without bias due to correlated variables. Here, we demonstrate the pipeline on two high-consequence applications: explosives classification for national security and inkjet printer identification in forensic science. The applications exhibit the VEESA pipeline’s ability to provide an understanding of the characteristics of the functional data useful for prediction. Code for implementing the pipeline is available in the veesa R package (and supplemental python code).

Elastic Shape Analysis↗

NASA NTRS2020

Gamma-Ray Urgent Archiver for Novel Opportunities (GUANO): Swift/BAT Event Data Dumps on Demand to Enable Sensitive Subthreshold GRB Searches

We introduce a new capability of the Neil Gehrels Swift Observatory to provide event-level data from the Burst Alert Telescope (BAT) on demand in response to transients detected by other instruments. We show that the availability of these data can effectively increase the rate of detections and arcminute localizations of gamma-ray bursts (GRB) like GRB 170817 by >400%. We describe an autonomous spacecraft-commanding pipeline purpose built to enable this science; to our knowledge, this is the first fully autonomous extremely low-latency commanding of a space telescope for scientific purposes. This pipeline has been successfully run in its complete form since 2020 January, and has resulted in the recovery of BAT event data for >800 externally triggered events to date (gravitational waves, GWs; neutrinos; GRBs triggered by other facilities; fast radio bursts; and very high-energy detections), now running with a success rate of ∼90%. We exemplify the utility of this new capability by using the resultant data to (1) set the most sensitive upper limits on prompt 1 s duration short GRB-like emission within ±15 s around the unmodeled GW burst candidate S200114f, and (2) provide an arcminute localization for short GRB 200325A and other bursts. We also show that using data from GUANO to localize GRBs discovered by other instruments, we can increase the net rate of arcminute-localized GRBs by 10%–20% per year. Along with the scientific yield of more sensitive searches for subthreshold GRBs, the new capabilities designed for this project will serve as the foundation for further automation and rapid target of opportunity capabilities for the Swift mission, and have implications for the design of future rapid-response space telescopes.

Aaron Tohuvavohu↗

NASA NTRS2016

EVEREST: Pixel Level Decorrelation of K2 Light Curves

We present EPIC Variability Extraction and Removal for Exoplanet Science Targets (EVEREST), an open-source pipeline for removing instrumental noise from K2 light curves. EVEREST employs a variant of pixel level decorrelation to remove systematics introduced by the spacecraft’s pointing error and a Gaussian process to capture astrophysical variability. We apply EVEREST to all K2 targets in campaigns 0–7, yielding light curves with precision comparable to that of the original Kepler mission for stars brighter than K(sub p) ≈ 13, and within a factor of two of the Kepler precision for fainter targets. We perform cross-validation and transit injection and recovery tests to validate the pipeline and compare our light curves to the other de-trended light curves available for download at the MAST High Level Science Products archive. We find that EVEREST achieves the highest average precision of any of these pipelines for unsaturated K2 stars. The improved precision of these light curves will aid in exoplanet detection and characterization, investigations of stellar variability, asteroseismology, and other photometric studies. The EVEREST pipeline can also easily be applied to future surveys, such as the TESS mission, to correct for instrumental systematics and enable the detection of low signal-to-noise transiting exoplanets. The EVEREST light curves and the source code used to generate them are freely available online.

Catalogs↗

DOE OSTI2024

Probabilistic Forward Modeling of Galaxy Catalogs with Normalizing Flows

Abstract Evaluating the accuracy and calibration of the redshift posteriors produced by photometric redshift (photo- z ) estimators is vital for enabling precision cosmology and extragalactic astrophysics with modern wide-field photometric surveys. Evaluating photo- z posteriors on a per-galaxy basis is difficult, however, as real galaxies have a true redshift but not a true redshift posterior. We introduce PZFlow, a Python package for the probabilistic forward modeling of galaxy catalogs with normalizing flows. For catalogs simulated with PZFlow, there is a natural notion of “true” redshift posteriors that can be used for photo- z validation. We use PZFlow to simulate a photometric galaxy catalog where each galaxy has a redshift, noisy photometry, shape information, and a true redshift posterior. We also demonstrate the use of an ensemble of normalizing flows for photo- z estimation. We discuss how PZFlow will be used to validate the photo- z estimation pipeline of the Dark Energy Science Collaboration, and the wider applicability of PZFlow for statistical modeling of any tabular data.

Astronomy & Astrophysics↗

NASA NTRSDate not supplied

ASDC’s Python-Based Metadata Extraction Pipeline for Suborbital Campaigns

The FAIRness of data products, especially findability and accessibility depend on rich metadata which, when extracted, can allow for proper curation. Over the past few years, the Atmospheric Science Data Center (ASDC) suborbital science support team has developed a metadata extraction pipeline to ensure the required metadata can be retrieved systematically, effectively, and efficiently to ensure the data can be used by a broad community. The development of a pipeline has presented many, but necessary, challenges to support archival and distribution of ASDC’s 30+ suborbital missions. Though sufficient metadata is provided by instrument scientists, the metadata may not be readily machine actionable due to different formats and templates. Further complicating metadata extraction, our team has found that the nature of metadata can be quite diverse given the difference in measurement types, instruments, and measurement platforms. A metadata extraction pipeline has been developed to provide an efficient, plugin-in based, method for adding new parsers, a configuration system that lets non-developers customize how files are processed, and a system for identifying and logging metadata quality issues to ensure they are readily found and addressed. The metadata extraction pipeline identifies critical pieces of metadata that are needed to promote data FAIRness, including location, file revision, measurement start/end datetime and can be easily modified to extract further information (such as variables). Given the wide-ranging datasets, the pipeline has been modified to accommodate multiple file formats, including multiple versions of ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), HDF (Hierarchical Data Format), netCDF (network Common Data Form), and multiple versions of the Ames File Format. The pipeline also supports building metadata for file formats that cannot have metadata easily extracted from them, such as PDF (Portable Document Format) and GIF (Graphics Interchange Format). The pipeline has allowed our team to maintain a consistent flow of data and metadata to archival and distribution services, ensuring the ASDC meets the needs of the suborbital science community. This presentation will highlight the ASDC’s suborbital metadata extraction pipeline, its development, how it’s been modified to support data FAIRness, and plans for maintaining the pipeline and adding new features.

Abraham Porter↗

DOE OSTI2020

E-PiPEline: Face Shields and Face Coverings Using Commonly Available Materials

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS2004

Caltrans Keeps the Spitzer Pipelines Moving

The computer pipelines used to process digital infrared astronomical images from NASA's Spitzer Space Telescope require various input calibration-data files for characterizing the attributes and behaviors of the onboard focal-plane-arrays and their detector pixels, such as operability, dark-current offset, linearity, non- uniformity, muxbleed, droop, and point-response functions. The telescope has three very different science instruments, each with three or four spectral-band-pass channels, depending on the instrument. Moreover, each instrument has various operating modes (e-g., full array or sub-array in one case) and parameters (e.g., integration time). Calibration data that depend on these considerations are needed by pipelines for generating both science products (production pipelines) and higher-level calibration products (calibration pipelines). The calibration files are created in various formats either 'off-line' or by the aforementioned calibration pipelines, depending on the above configuration details. Also, the calibration files are generally applicable to a certain time period and therefore must be selected accordingly for a given raw input image to be correctly processed. All of this complexity in selecting and retrieving calibration files for pipeline processing is handled by a procedural software-program called 'caltrans' . This software, which is implemented in C and interacts with an Informix database, was developed at the Spitzer Science Center (SSC) and is now deployed in SSC daily operations. The software is rule-based, very flexible, and, for efficiency, capable of retrieving multiple calibration files with a single software-execution command.

Spitzer↗

NASA NTRS2012

Enabling Earth Science Through Cloud Computing

Cloud Computing holds tremendous potential for missions across the National Aeronautics and Space Administration. Several flight missions are already benefiting from an investment in cloud computing for mission critical pipelines and services through faster processing time, higher availability, and drastically lower costs available on cloud systems. However, these processes do not currently extend to general scientific algorithms relevant to earth science missions. The members of the Airborne Cloud Computing Environment task at the Jet Propulsion Laboratory have worked closely with the Carbon in Arctic Reservoirs Vulnerability Experiment (CARVE) mission to integrate cloud computing into their science data processing pipeline. This paper details the efforts involved in deploying a science data system for the CARVE mission, evaluating and integrating cloud computing solutions with the system and porting their science algorithms for execution in a cloud environment.

science data system↗

NASA NTRS1997

Hubble Space Telescope Astrometry Science Team

Topics considered include: Calibration and Astrometry Pipeline; Astrometry Anomaly Analysis; The STAT pipeline and software analysis tools; GTO Program; Parallaxes of Astrophysically Interesting Objects; HST- Hipparcos Extragalactic Link; Ground-Based Observations of Radial Velocities; GTO Globular Cluster Internal Motions; and GTO Hyades Cluster Member FGS Parallaxes.

Benedict, George F.↗

NASA NTRS2010

A Framework for Propagation of Uncertainties in the Kepler Data Analysis Pipeline

The Kepler space telescope is designed to detect Earth-like planets around Sun-like stars using transit photometry by simultaneously observing 100,000 stellar targets nearly continuously over a three and a half year period. The 96-megapixel focal plane consists of 42 charge-coupled devices (CCD) each containing two 1024 x 1100 pixel arrays. Cross-correlations between calibrated pixels are introduced by common calibrations performed on each CCD requiring downstream data products access to the calibrated pixel covariance matrix in order to properly estimate uncertainties. The prohibitively large covariance matrices corresponding to the ~75,000 calibrated pixels per CCD preclude calculating and storing the covariance in standard lock-step fashion. We present a novel framework used to implement standard propagation of uncertainties (POU) in the Kepler Science Operations Center (SOC) data processing pipeline. The POU framework captures the variance of the raw pixel data and the kernel of each subsequent calibration transformation allowing the full covariance matrix of any subset of calibrated pixels to be recalled on-the-fly at any step in the calibration process. Singular value decomposition (SVD) is used to compress and low-pass filter the raw uncertainty data as well as any data dependent kernels. The combination of POU framework and SVD compression provide downstream consumers of the calibrated pixel data access to the full covariance matrix of any subset of the calibrated pixels traceable to pixel level measurement uncertainties without having to store, retrieve and operate on prohibitively large covariance matrices. We describe the POU Framework and SVD compression scheme and its implementation in the Kepler SOC pipeline.

Clarke, Bruce D.↗

NASA NTRS1990

Pipeline issues

The declining pool of graduates, the lack of rigorous preparation in science and mathematics, and the declining interest in science and engineering careers at the precollege level promises a shortage of technically educated personnel at the college level for industry, government, and the universities in the next several decades. The educational process, which starts out with a large number of students at the elementary level, but with an ever smaller number preparing for science and engineering at each more advanced educational level, is in a state of crisis. These pipeline issues, so called because the educational process is likened to a series of ever smaller constrictions in a pipe, were examined in a workshop at the Space Grant Conference and a summary of the presentations and the results of the discussion, and the conclusions of the workshop participants are reported.

Eisley, Joe T.↗

NASA NTRSDate not supplied

Earth Science Data Processing With Nextflow

Earth science data processing tasks present many challenges. These tasks often process large input datasets and require scores of CPU-hours to generate results. All but the simplest tasks will be decomposed into a series of computational or data manipulation steps, also known as a scientific workflow. In order to reduce the burden of orchestrating and running the dependent processing steps, a workflow execution engine is required. This poster describes the lessons learned by the CLARREO Pathfinder (CPF) team while developing multiple scientific workflows and utilizing the open-source Nextflow engine to execute them in a cloud computing environment. The Nextflow engine is designed with the following stated goals: first, the engine does not dictate how individual steps in the task are implemented (i.e. it is language and interface agnostic); second, the engine supports easy configuration and modularity at the workflow level so that others can easily execute our workflows to reproduce results; lastly, the engine eases development by transparently scaling execution from local to remote environments. Nextflow was developed for the bioinformatics domain but is a good fit for other scientific workflows where the overall task is well-described by a dataflow diagram. The CPF team has developed Nextflow pipelines (i.e. scientific workflows) to simulate CLARREO radiance, generate large look-up tables for inter-calibration algorithms, and generate L4 intercalibration data products. These pipelines consume from single-digits to hundreds of thousands of CPU-hours. In the development and evolution of these pipelines we have discovered many design patterns, pitfalls, and solutions to common problems. Our goal is to demonstrate important aspects of how to design, implement, run, and ultimately share Nextflow pipelines in the domain of Earth science.

Aron D Bartle↗

DOE OSTI2022

High energy X-ray diffraction and small-angle scattering measurements of hydrogen fatigue damage in AISI 4130 steel

Accurate lifetime predictions are critical for repurposing existing pipelines for hydrogen transmission as well as for developing novel steels which are minimally susceptible to lifetime degradation by hydrogen. Ultimately, lifetime prediction models assess the amount of damage a material undergoes during a typical service cycle and the cumulative damage a material can withstand prior to failure. However, not all damage processes are equal, and neither is the manner in which mechanical loading translates to damage the same when materials are in inert environments compared to in hydrogen environments. For example, in the three leading proposed mechanisms of hydrogen embrittlement (Hydrogen-Enhanced Decohesion (HEDE), the Hydrogen-Enhanced Localized Plasticity (HELP), and the Nano-Void Coalescence (NVC)), hydrogen is proposed to enhance the manifestation of grain separation, dislocation generation/movement, and void coalescence, respectively. A full understanding of the damage modes requires a measurement capable of probing all three mechanisms at once. Here we present simultaneous High Energy X-ray Diffraction (HEXRD) and Small-Angle X-ray Scattering (SAXS) during fatiguing of steel in hydrogen. HEXRD measurements probe strain and dislocation density; SAXS measurements probe nano-pore generation and coalescence. We will discuss the differences in damage modes between steels fatigued in air and in hydrogen and the role these difference play in lifetime predictions.

36 MATERIALS SCIENCE↗

DOE OSTI2020

E-PiPEline: Quick to Market Face Coverings and Face Shields Using Commonly Available Materials

The Center for Disease Control has recommended that the public should wear cloth face coverings in public settings. Face coverings and face shields can be made by using Commonly Available Materials (CAMs). As part of the Sandia COVID-19 LDRD effort (funded under the Materials Science Investment Area), the Sandia E-PiPEline task evaluated design options for face coverings and face shields considering their effectiveness, durability, build difficulty, build cost, and comfort. Observations from this investigation are presented here to provide guidelines for home construction of face coverings and face shields. This executive summary includes a brief roadmap of the analysis methodology, two one-page handouts geared to be distributed to the public at large (one for face coverings and one for face shields), and additional observations regarding potential solutions for face coverings and face shields included to further support the one-page handouts.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2025

A hybrid neural architecture: Online attosecond x-ray characterization

The emergence of high-repetition-rate x-ray free-electron lasers (XFELs), such as SLAC’s LCLS-II, serves as our canonical example for autonomous controls that necessitate high-throughput diagnostics paired with streaming computational pipelines capable of single-shot analysis with extremely low latency. We present the deterministic characterization with an integrated parallelizable hybrid resolver architecture, a hybrid machine learning framework designed for fast, accurate analysis of XFEL diagnostics using angular streaking-based sinogram images. This architecture integrates convolutional neural networks and bidirectional long short-term memory models to denoise input, identify x-ray sub-spike features, and extract sub-spike relative delays with sub-30 attosecond temporal resolution. Deployed on low-latency hardware, it achieves over 10 kHz throughput with 168.3 μs inference latency, indicating scalability to 14 kHz with field-programmable gate array integration. By transforming regression tasks into classification problems and leveraging optimized error encoding, we achieve high precision with low-latency performance that is critical for real-time streaming event selection and experimental control feedback signals. This represents a key development in real-time control pipelines for next-generation autonomous science, generally, and high repetition-rate x-ray experiments in particular.

Accelerator Physics (physics.acc-ph)↗

DOE OSTI2025

Iron-starvation induces photosystem I antenna remodeling in green algae

Dunaliella salina and Dunaliella tertiolecta are extremophile, marine algae that can survive in very low Fe conditions. In this study, we used TMT-proteomics to compare the Fe starvation responses to the Fe replete responses. Samples were digested with trypsin, labeled with TMT 10-Plex, then analyzed by LC-MS/MS. Data was searched with MS-GF+ using PNNL's DMS Processing pipeline.

59 BASIC BIOLOGICAL SCIENCES↗