Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 685 records · Page 38

Supercomputing in the Age of Discovering Superearths, Earths and Exoplanet Systems

With the Kepler Mission, exoplanet science has entered the era of Big Data and has benefited tremendously from the availability of the NAS (NASA Advanced Supercomputing Division) Pleiades supercomputer. This presentation provides an overview of the history of exoplanet discoveries and highlights the key enabling technologies for Kepler. I provide an overview of the Kepler science pipeline and the algorithms used to analyze the data and detect transiting planets, all of which run routinely on the NAS Pleiades system. The presentation also provides a description of the Blender analysis program used to validate Kepler planet discoveries statistically, and the effort to discover exomoons lurking in Kepler data, both of which also run on Pleiades. Finally, I describe NASA's TESS (Transiting Exoplanet Survey Satellite) Mission, which leverages Kepler heritage to provide a science pipeline similar to that for Kepler and that will also run routinely on the NAS Pleiades supercomputer.

big data↗

Geonex: A NASA-NOAA Collaboration for Producing Land Surface Products from Geostationary Sensors Using Cloud Computing

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement.Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. We estimate the cost of deploying GEONEX to be $400 - 750 a month for processing data (every 30 minutes) and producing products over the conterminous US. For products such as Fire, latency can be as little as 10 minutes from the time of data acquisition.

Geostationary↗

Caltrans Keeps the Spitzer Pipelines Moving

The computer pipelines used to process digital infrared astronomical images from NASA's Spitzer Space Telescope require various input calibration-data files for characterizing the attributes and behaviors of the onboard focal-plane-arrays and their detector pixels, such as operability, dark-current offset, linearity, non- uniformity, muxbleed, droop, and point-response functions. The telescope has three very different science instruments, each with three or four spectral-band-pass channels, depending on the instrument. Moreover, each instrument has various operating modes (e-g., full array or sub-array in one case) and parameters (e.g., integration time). Calibration data that depend on these considerations are needed by pipelines for generating both science products (production pipelines) and higher-level calibration products (calibration pipelines). The calibration files are created in various formats either 'off-line' or by the aforementioned calibration pipelines, depending on the above configuration details. Also, the calibration files are generally applicable to a certain time period and therefore must be selected accordingly for a given raw input image to be correctly processed. All of this complexity in selecting and retrieving calibration files for pipeline processing is handled by a procedural software-program called 'caltrans' . This software, which is implemented in C and interacts with an Informix database, was developed at the Spitzer Science Center (SSC) and is now deployed in SSC daily operations. The software is rule-based, very flexible, and, for efficiency, capable of retrieving multiple calibration files with a single software-execution command.

Spitzer↗

Machine Learning Digital Twin for Lithium Ion Battery State of Health Predictions

A digital twin system has been established to model the long term degradation of the state of health of lithium ion batteries. Two data streams result from the computational model of the system and from the physical experiment measurements. A machine learning pipeline has been developed at the nexus of these data streams. Leveraging the unique data sources in multiple transfer learning approaches has lead to the development of multiple cell specific machine learning digital twins. Our discussion on this first of its kind technology will cover challenges in data ingestion, scalability, architecture and orchestration, and research findings.

25 - ENERGY STORAGE↗

OpenPATH and Itinerum Integration: A Case Study of Merging Open-Source Projects: Preprint

Throughout the transportation sector, multiple mobility tracking platforms have been developed independently, in parallel, with similar use cases and infrastructure. Maintaining such platforms on an individual level can result in duplicated effort. Innovation within the field could be slowed by the limited resource of people working on extremely similar projects. We provide a case study in how such platforms can be combined with low effort and provide an example in the mobility domain of what the resulting integration points, modules and ownership may look like. OpenPATH (formerly e-mission) and Itinerum are two examples of similar independently developed human mobility tracking platforms that extend beyond standard traffic data collection. Due to the open source nature of the projects, members from both teams decided to merge the functionality, thereby only hosting a single data collection process and limiting maintenance of separate intake platforms. On the front-end, specific questionnaire functionality from Itinerum, mainly the ability to display and return questions and results focused on sociodemographic attributes, was integrated into the OpenPATH user interface. On the back-end, an export pipeline stage was created to transport collected data back to the Itinerum database. Upon completion, the OpenPATH application retained its original functionality while adding the ability to export data for future data sharing and creating an avenue to incorporate Itinerum hosted questions and answers. Itinerum benefits from the rich feature set and infrastructure of OpenPATH without losing needed elements such as the questionnaire component. This case study, within the transportation sector, illustrates the various configurations in which open source software can be developed, maintained and deployed.

ADVANCED PROPULSION SYSTEMS↗

Geonex: Land Surface Monitoring from a New Generation of Geostationary Sensors

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

Nemani, Ramakrishna R.↗

Earth Observations from Geostationary Satellites

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GeoNEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GeoNEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GeoNEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

Earth↗

GeoNEX: Land Monitoring from a New Generation of Geostationary Sensors

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

GeoNEX↗

Kepler Data Validation Time Series File: Description of File Format and Content

The Kepler space mission searches its time series data for periodic, transit-like signatures. The ephemerides of these events, called Threshold Crossing Events (TCEs), are reported in the TCE tables at the NASA Exoplanet Archive (NExScI). Those TCEs are then further evaluated to create planet candidates and populate the Kepler Objects of Interest (KOI) table, also hosted at the Exoplanet Archive. The search, evaluation and export of TCEs is performed by two pipeline modules, TPS (Transit Planet Search) and DV (Data Validation). TPS searches for the strongest, believable signal and then sends that information to DV to fit a transit model, compute various statistics, and remove the transit events so that the light curve can be searched for other TCEs. More on how this search is done and on the creation of the TCE table can be found in Tenenbaum et al. (2012), Seader et al. (2015), Jenkins (2002). For each star with at least one TCE, the pipeline exports a file that contains the light curves used by TPS and DV to find and evaluate the TCE(s). This document describes the content of these DV time series files, and this introduction provides a bit of context for how the data in these files are used by the pipeline.

DV Time Series↗

Predicting metabolic modules in incomplete bacterial genomes with MetaPathPredict

The reconstruction of complete microbial metabolic pathways using ‘omics data from environmental samples remains challenging. Computational pipelines for pathway reconstruction that utilize machine learning methods to predict the presence or absence of KEGG modules in incomplete genomes are lacking. Here, we present MetaPathPredict, a software tool that incorporates machine learning models to predict the presence of complete KEGG modules within bacterial genomic datasets. Using gene annotation data and information from the KEGG module database, MetaPathPredict employs deep learning models to predict the presence of KEGG modules in a genome. MetaPathPredict can be used as a command line tool or as a Python module, and both options are designed to be run locally or on a compute cluster. Benchmarks show that MetaPathPredict makes robust predictions of KEGG module presence within highly incomplete genomes.

59 BASIC BIOLOGICAL SCIENCES↗

A Complete UV Atlas of Standard Stars

This program is an Astrophysics Data Program (ADP) contract, together with a follow-on contract (S- 57791 -Z), intended to produce a comprehensive ultraviolet spectral atlas of stars based on the data in the IUE Final Archive. The data in the Final Archive were uniformly processed by the NEWSIPS pipeline system. The NEWSIPS gives an improved signal-to-noise ratio for the data that will be especially beneficial to low signal portions of the spectra; and it also gives more accurate fluxes for stars by adopting the latest absolute calibration and measured instrument parameters. Therefore, this atlas provides more uniform and accurate data than the version previously published.

Wu, Chi-Chao↗

VA-Environmental Determinants of Health’s Software Pipeline Framework

In order to characterize US communities in terms of social and environmental determinants of health (EDH) that affect veterans, the Social and Environmental Determinants of Health (SEDH) project, a collaboration between the Veterans Affairs (VA) Social Department and the Oak Ridge National Laboratory (ORNL), aims to identify, extract, and prepare datasets that include community variables known to be associated with health outcomes in veterans. The EDH project's responsibilities include investigating SEDH data, providing technical documentation for SEDH datasets, and automating the download, extraction, and preparation of the data through the creation of a software pipeline framework.

54 ENVIRONMENTAL SCIENCES↗

Packaging data products using data grid middleware for Deep Space Mission Systems

We are investigating a distributed computing infrastructure for releasing science products out of the mission pipeline. We are creating a DSMS Product Service and DSMS Data Packaging Service based on the OODT middleware, a Data Grid middleware technology that has proven successful in providing access to heterogeneous, disparate data, and federating that data using a common data model.

Deep Space Mission Systems↗

DynPaC: Coarse-Grained, Dynamic, and Partially Reconfigurable Array for Streaming Applications

Coarse-grained reconfigurable arrays (CGRAs) provide higher flexibility than application-specific integrated circuits (ASICs) and higher efficiency than fine-grained reconfigurable devices such as Field Programmable Gate Arrays (FPGAs). However, CGRAs are generally designed to support offloading of a single kernel. While their design, based on communicating functional units, appears to naturally suit streaming applications composed of multiple cooperating kernels, current approaches only statically partition the resources across kernels. However, streaming applications often are data-dependent, leading to variable kernel execution times depending on the input data and impacting the throughput of the entire pipeline if resources are statically allocated. Therefore, in this paper, we discuss the design of DynPaC — a coarse-grained, dynamically, and partially reconfigurable array for data-dependent streaming applications. We discuss the required software and hardware components to manage partial dynamic reconfiguration. We demonstrate that by supporting partial dynamic reconfiguration, we can obtain an average speedup of 1.44X for a representative set of applications w.r.t. static partitioning, with a limited area overhead (6.4% of the entire chip).

Tan, Cheng↗

Automated and High-Throughput Phase Separation Control for Supramolecular Polymer Blends Enabled by Machine Learning

Supramolecular polymer blends (SPBs) offer tunable morphologies that dictate their macroscopic properties, yet their rational design is limited by the absence of predictive structure−morphology models. Here, we introduce a data-driven highthroughput workflow that integrates modular polymer synthesis, robotic formulation, automated morphology characterization, and machine learning (ML) for accelerated SPB discovery. Using a plug-and-play synthetic strategy, 33 hydrogen-bonding endfunctional homopolymers were prepared and orthogonally combined to generate 260 SPBs in 1 day. A fully automated atomic force microscopy (AFM) pipeline enabled systematic imaging, producing 2340 morphology data sets with minimal human intervention. Domain spacings were extracted through complementary imageprocessing methods and used to train ML models. A support vector regression (SVR) model accurately predicted target phase-separation sizes (50, 100, and 150 nm), which were experimentally validated. This work demonstrates the power of coupling high-throughput experimentation with ML to accelerate morphology discovery and provides one of the first large-scale experimental data sets for supramolecular polymer systems.

ML-guided polymer design↗

Implementation of an ICS Ransomware Testbed: Scenarios, Variants, and Evaluation Methods

Ransomware attacks on Industrial Control Systems (ICS) have emerged as a formidable threat to the United States’ critical infrastructure, eliciting grave concerns regarding national security. In March 2023, the FBI Internet Crime Complaint Center (IC3) unveiled its 2022 Internet Crime Report, highlighting a concerning 870 complaints related to ransomware impacting U.S. critical infrastructure. Of the country's 16 critical infrastructure sectors, 14 encountered at least one ransomware attack. Notably, while the Healthcare and Public Health sector suffered the most, reporting 210 attacks, sectors pivotal to ICS networks and governmental organizations were also targeted: the Defense Industrial Base reported 1 attack, Water and Wastewater Systems 3, Chemical 19, Energy 15, Government Facilities 115, and Critical Manufacturing 157. For instance, a ransomware attack on a major chemical company could jeopardize not only its production but also pose environmental risks should systems controlling hazardous materials be compromised. In 2022, three ransomware variants predominantly targeted U.S. critical infrastructure: HIVE, with 87 attacks; ALPHV/BlackCat, with 114; and LOCKBIT, with 149. Several cyber-attacks, such as the MOVEit data breach in May 2023 and the Colonial Pipeline ransomware attack in May 2021, have been so impactful that they commanded national attention. The DarkSide hacking group's assault on the Colonial Pipeline, initiated on May 6th, 2021, stands as one of the most substantial and publicly acknowledged cyber-attacks against U.S. critical infrastructure. The group exploited an exposed Virtual Private Network (VPN) password, paving the way for initial intrusion and subsequent data theft. A mere day later, DarkSide unleashed a ransomware attack that compromised vital accounting and billing systems, prompting an immediate shutdown of the pipeline to mitigate further ransomware proliferation across its network. This crisis spurred a robust response from the U.S. president and regulators, culminating in a national emergency declaration related to the pipeline shutdown on May 9th, 2021. This incident mirrors the 2017 NotPetya ransomware attack that significantly impacted the shipping giant Maersk, highlighting an urgent need for fortified cybersecurity across various industries. Future incidents, akin to the Colonial Pipeline attack, could potentially be mitigated—or entirely averted—should government agencies and private entities scrutinize system vulnerabilities, exploring various ransomware types and entry points. Proactive measures, such as conducting experiments on VPN accounts or auditing passwords to pinpoint duplicate usage across diverse systems and software, might illuminate feasible entry points and vulnerability zones within an organization's systems.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Laue-DIALS: Open-source software for polychromatic x-ray diffraction data

Most x-ray sources are inherently polychromatic. Polychromatic (“pink”) x-rays provide an efficient way to conduct diffraction experiments as many more photons can be used and large regions of reciprocal space can be probed without sample rotation during exposure—ideal conditions for time-resolved applications. Analysis of such data is complicated, however, causing most x-ray facilities to discard >99% of x-ray photons to obtain monochromatic data. Key challenges in analyzing polychromatic diffraction data include lattice searching, indexing and wavelength assignment, correction of measured intensities for wavelength-dependent effects, and deconvolution of harmonics. We recently described an algorithm, Careless, that can perform harmonic deconvolution and correct measured intensities for variation in wavelength when presented with integrated diffraction intensities and assigned wavelengths. Here, we present Laue-DIALS, an open-source software pipeline that indexes and integrates polychromatic diffraction data. Laue-DIALS is based on the dxtbx toolbox, which supports the DIALS software commonly used to process monochromatic data. As such, Laue-DIALS provides many of the same advantages: an open-source, modular, and extensible architecture, providing a robust basis for future development. We present benchmark results showing that Laue-DIALS, together with Careless, provides a suitable approach to the analysis of polychromatic diffraction data, including for time-resolved applications.

97 MATHEMATICS AND COMPUTING↗

Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.

Yang, Wei [SLAC]↗