SEARCH · Search NASA

Results for “pipeline data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

All sources NASA NTRS DOE OSTI

NASA NTRS1994

Analysis and Optimization of Parallel Software Pipeline Performance

Pipelining is a common strategy for extracting parallelism from a collection of independent computational tasks, each of which is spread among a number of processors and has an implied data dependence. When implemented on MIMD parallel computers with finite process interrupt times, pipeline algorithms suffer from slowdown--in addition to the expected pipeline fill time--due to a wave-like propagation of delays. This phenomenon, which has been observed experimentally using the performance monitoring system AIMS, is investigated analytically, and an optimal correction is derived to eliminate the wave. Efficiency increase through the correction is verified experimentally.

VanderWijngaart, Rob F.↗

NASA NTRS1995

The Effect of Interrupts on Software Pipeline Execution on Message-Passing Architectures

VanderWijngaart, Rob F.↗

NASA NTRS2010

The Kepler DB, a Database Management System for Arrays, Sparse Arrays and Binary Data

The Kepler Science Operations Center stores pixel values on approximately six million pixels collected every 30-minutes, as well as data products that are generated as a result of running the Kepler science processing pipeline. The Kepler Database (Kepler DB) management system was created to act as the repository of this information. After one year of ight usage, Kepler DB is managing 3 TiB of data and is expected to grow to over 10 TiB over the course of the mission. Kepler DB is a non-relational, transactional database where data are represented as one dimensional arrays, sparse arrays or binary large objects. We will discuss Kepler DB's APIs, implementation, usage and deployment at the Kepler Science Operations Center.

McCauliff, Sean↗

NASA NTRS2018

Land Surface Reflectances from Geostationary Sensors

GEONEX is a processing pipeline that produces a suite of satellite land surface products using data streams from the latest geostationary (GEO) sensors including the GOES016/ABI and the Himawari-8/AHI. The suite, created collaboratively by scientists from NASA and NOAA, includes top-of-atmosphere (TOA) reflectances, land surface reflectances (LSRs), vegetation indices, LAI/fPAR, and other downstream products. As a key component of the GEONEX product processing, we have adapted the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to produce LSRs from the TOA data. Because the algorithm depends on building "stacks" of images, we first run internal geo-registration checks to ensure geo-spatial accuracy and consistency of the input (L1B) data before transferring them from the geostationary projection into a tile system in geographic grids. Scan-time is inferred from metadata and applied to calculate the sun-sensor angles for each grid cell. The MAIAC algorithm is run to detect clouds/shadows, estimate aerosol optical thickness (AOT), perform atmospheric corrections, and generate LSRs. We have processed 18-months (from 2016/04 onward) of AHI data over East Asia and Oceania at a 10-minute time step and 10-months (from 2018/01 onward) of ABI data over North and South Americas at a 15-minute time step. As a verification measure, we compare the GEONEX (AHI/ABI) surface reflectances with the standard MODIS products (MOD09GA) and the MODIS MAIAC products over pixels that have similar sun-view geometries. The results indicate general linear relationships between GEONEX and corresponding MODIS LSRs. In particular, the RMSEs between GEONEX and MOD09 data are comparable to those between MOD09 and MODIS MAIAC products, suggesting that the uncertainties of GEONEX LSRs fall into an acceptable range. However, direct comparisons of LSRs over pixels with different sun-view angles are not as straightforward and require more modeling efforts to correct the directional effects. Evaluation of such angular influences on the downstream products (e.g., vegetation indices) is also under investigation.

Geostationary satellite; Remote Sensing; Atmospher↗

NASA NTRS2000

An ASCA Study of the Composite Supernova Remnant G18.95-1.1

This is the final report on the work done on Supernova Remnant (SNR) G18-95-1.1. The data were taken on April, 2. 1998 and delivered a couple of months later to the Principal Investigator (PI: Dr. Ilana Harrus). We received a CD-ROM containing the results of the standard processing pipeline and all the files needed for the analysis. We have analyzed the data and presented a poster on this object at the 194th American Astronomical Society Meeting in Chicago (June 1999). A copy of the poster is appended to this report. The poster presentation triggered several discussions and we are summarizing the analysis results and those discussions in a paper to be submitted soon to the Astrophysical Journal. We have appended the draft of the paper to this report. It must be noted that the paper is still in its early stages. In particular more work is needed in the physical implications of the results of the spectral analysis and in the comparison with theoretical models to understand the curious morphology of the remnant. The project should be completed within the next two months. Attachment: "ASCA study of the centrally-peaked thermal supernova remnant: G18.95-1.1".

Harrus, Ilana↗

NASA NTRSDate not supplied

Developing a Cloud-Based ArcGIS Image Service for TROPOMI Level 2 Data: Preprocessing, Transformation, and Publication

TROPOMI, the Tropospheric Monitoring Instrument aboard the Sentinel-5 Precursor satellite, provides high spatiotemporal resolution atmospheric measurements. It is essential for monitoring air quality, greenhouse gases, and other trace gases. Integrating TROPOMI Level 2 data into an ArcGIS Image Service marks a significant advancement in the accessibility and utility of satellite-derived environmental information for GIS applications. This paper details the methodology for building an ArcGIS Image Service tailored to handle TROPOMI Level 2 data, with a focus on preprocessing, transformation, and cloud-based publication. The workflow utilizes Python and ArcPy for data reformatting, reprojection, and updating, ensuring efficient processing and cloud-based notifications. By establishing a robust pipeline, the ArcGIS Image Service provides real-time access to TROPOMI Level 2 data, enabling users to visualize, analyze, and interpret atmospheric phenomena effectively.

Level 2 Data, ArcGIS, Image Service, ArcPy, OGC↗

NASA NTRS2017

Kepler Planet Detection Metrics: Window and One-Sigma Depth Functions for Data Release 25

This document describes the window and one-sigma depth functions relevant to the Transiting Planet Search (TPS) algorithm in the Kepler pipeline (Jenkins 2002; Jenkins et al. 2017). The window function specifies the fraction of unique orbital ephemeris epochs over which three transits are observable as a function of orbital period. In this context, the epoch and orbital period, together, comprise the ephemeris of an orbiting companion, and ephemerides with the same period are considered equivalent if their epochs differ by an integer multiple of the period. The one-sigma depth function specifies the depth of a signal (in ppm) for a given light curve that results in a one-sigma detection of a transit signature as a function of orbital period when averaged over all unique orbital ephemerides. These planet detection metrics quantify the ability of TPS to detect a transiting planet signature on a star-by-star basis. They are uniquely applicable to a specific Kepler data release, since they are dependent on the details of the light curves searched and the functionality of the TPS algorithm used to perform the search. This document describes the window and one-sigma depth functions relevant to Kepler Data Release 25 (DR25), where the data were processed (Thompson et al. 2016) and searched (Twicken et al. 2016) with the SOC 9.3 pipeline. In Section 4, we describe significant differences from those reported in Kepler Data Release 24 (Burke Seader 2016) and document our verification method.

Planet Detection Metrics↗

NASA NTRS2018

NASA Pilot-Engaged Expert Response Using IBM Watson Technology: Prototype Evaluation of Knowledge Retrieval System

NASA Langley Research Center and IBM have been investigating the use of IBM Watson technology in aerospace research and development. One application of Watson technology is the Pilot-Engaged Expert Response (PEER) use case. The PEER system is envisioned as an in-cockpit advisor that will act as a source of situationally-relevant information for pilots and other flight crew members to assist in decision making about real-time events and situations that arise in the course of aircraft operations. PEER will make available vast stores of knowledge and information quickly and directly, putting important informational resources where they are needed most. IBM has worked with NASA to develop an architecture and articulate a roadmap for the development of the PEER system. That vision is built around Watson Discovery Advisor (WDA) software solution, derived from IBM's Jeopardy!-winning automatic question answering system. PEER makes use of WDA's sophisticated question-answering capabilities as its core, adding important User Interface components and other customizations for the cockpit environment, including communication with flight systems and other external data sources. The development plan for PEER includes four development stages, with the current project constituting the first phase. In this project, a prototype instance of PEER was successfully adapted to the aviation domain, enabling users to ask questions about aviation topics and receive useful and accurate answers to these questions. Major tasks accomplished include the development of procedures for domain adaptation through automatic lexicon extraction from domain glossaries; generation of question-answer training data which was used to train the system; and assessment of the effectiveness of domain adaptation, which showed a dramatic improvement in the ability of the PEER system to answer domain-relevant questions. In addition, the vision for the PEER system was pushed forward by the articulation of a plan for the automatic enhancement of question-answering with contextual information. This initial phase focused on two main goals: 1) the targeted domain adaptation of the underlying WDA system to the aviation domain; and, 2) the design of the software systems needed to leverage flight-contextual data. Domain adaptation of the WDA system proceeds via three main activities: Domain data ingestion, lexical customization and model training. A textual corpus consisting of 1,147 individual documents with more than 7.5 million words of text was ingested into the system and this served as the basis of all further development. A domain lexicon of over 3,500 aviation-domain terms was semi-automatically generated from domain documents and used to train the system. In addition, a set of over 500 question-answer (QA) pairs relevant to the PEER use case was developed; these were used to train and assess the system. These important first steps established the basis for the PEER system. In addition, steps were taken towards the integration of the PEER system into the cockpit environment with the development of a functional design for the Contextual Data Augmentation (CDA) subsystem. This subsystem brings to bear contextual data to improve system responses. It has three main submodules: the Contextual Data Collection module, the Contextual Data Selection module, and the Contextual QA Augmentation module. These modules form a processing pipeline that addresses the problems associated with automatically integrating information from external resources into the knowledge-retrieval mechanism.

Machine learning↗

NASA NTRSDate not supplied

Development and Demonstration of a Digital NDE Pipeline for Streamlined Analysis of Ultrasonic Data

Currently when nondestructive evaluation (NDE) is performed on composite structures, the results, although recorded digitally are often manually interpreted and indicated (drawn) on the part being inspected by hand. Following this, a determination must be made on how to disposition that part. This decision could be based on engineering guidelines and best practices, rule-of-thumb, expert opinion or finite-element analysis of the part with some approximate representation of the damage. In order to streamline this process for ultrasonic inspection an effort was undertaken as part of NASA Advanced Composites Project to develop the Digital NDE Pipeline. The Digital NDE Pipeline is an integrated tool suite and associated framework that streamlines the inspection and defect disposition process through model-assisted inspection optimization, automated defect analysis of NDE data, and mapping of NDE data into finite element analysis software. This paper will provide an overview of the Digital NDE Pipeline and provide details of the individual tools developed along with the results of applying these tools to a demonstration test case.

Composites↗

NASA NTRS2010

Kepler Mission's Focal Plane Characterization Models Implementation

The Kepler Mission photometer is an unusually complex array of CCDs. A large number of time-varying instrumental and systemic effects must be modeled and removed from the Kepler pixel data to produce light curves of sufficiently high quality for the mission to be successful in its planet-finding objective. After the launch of the spacecraft, many of these effects are difficult to remeasure frequently, and various interpolations over a small number of sample measurements must be used to determine the correct value of a given effect at different points in time. A library of software modules, called Focal Plane Characterization (FC) Models, is the element of the Kepler Science Data Pipeline (hereafter "pipeline") that handles this. FC, or products generated by FC, are used by nearly every element of the SOC processing chain. FC includes Java components: database persistence classes, operations classes, model classes, and data importers; and MATLAB code: model classes, interpolation methods, and wrapper functions. These classes, their interactions, and the database tables they represent, are discussed. This paper describes how these data and the FC software work together to provide the pipeline with the correct values to remove non-photometric effects caused by the photometer and its electronics from the Kepler light curves. The interpolation mathematics is reviewed, as well as the special case of the sky-to-pixel,pixel-to-sky coordinate transformation code, which incorporates a compound model that is unique in the SOC software.

mission↗

NASA NTRS2019

Geonex: Land Surface Monitoring from a New Generation of Geostationary Sensors

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

Nemani, Ramakrishna R.↗

NASA NTRS2019

Earth Observations from Geostationary Satellites

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GeoNEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GeoNEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement. Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GeoNEX pipeline including choice CPUs, storage media, and automation. By making the GEONEX pipeline available on the cloud, we hope to engage a broad community of Earth scientists from around the world in utilizing this new source of data for Earth monitoring.

Earth↗

NASA NTRS2019

GeoNEX: Land Monitoring from a New Generation of Geostationary Sensors

GeoNEX↗

NASA NTRS2021

Mass Inferencing Model Creation And Deployment To Lunar Excavation Robot, RASSOR

NASA’s Regolith Advanced Surface Systems Operations Robot (RASSOR) Excavator is a teleoperated mobile robotic platform with a unique space regolith excavation capability. This research project developed functionality for inferencing regolith mass ingested during RASSOR operation, enhancing RASSOR’s ability to successfully complete ISRU missions. Radio wave propagation time to the Moon and back is ~2.56 seconds. Though teleoperation is possible with this delay, autonomous capability that enables RASSOR to plan and execute excavation missions intelligently and efficiently is preferred. To teleoperate or run autonomously, it is crucial for the quantity of regolith mass ingested by RASSOR to be available as a system state for efficient operation (e.g. knowledge of whether drums are full informs the task of highest priority, whether it be continuing to dig, or returning to a processing plant to offload regolith). A configurable data reduction and analysis pipeline was created to allow for straightforward incorporation of new data, such as that from lunar excavation, to improve model performance in new environments. Four distinct modeling approaches were employed in developing a mass inferencing approach that could work on RASSOR. All four models take in system states and output a mass prediction for each set of the robot’s bucket drums. Initial results from deployment to RASSOR and testing in a simulated lunar environment show that the models have <10% mean error during robot operation. Future work includes refinement of a model that estimates regolith mass in real-time during excavation as well as further testing of the developed models on the hardware.

ROS↗

NASA NTRSDate not supplied

Improving Sim-to-Real Transfer in Vision-Based Robot Navigation Via Instance-Level GAN-Based Data Augmentation

Achieving robust vision-based robotic tasks requires large amounts of data, which are often difficult to obtain in real-world scenarios. Simulators and synthetic data offer a cost-effective alternative, but the visual gap between simulation and reality hinders the performance of models when deployed in real-world environments. In this paper, we present a data augmentation pipeline that integrates a foundation model (Segment Anything Model) with an unsupervised image-to-image translation model (CycleGAN) for instance-level domain transfer from simulation to reality. This pipeline enables the generation of realistic labeled data from synthetic images for training supervised machine learning models in vision-based navigation tasks. We evaluate our approach on real-world data for ego-vehicle pose estimation, a critical autonomous navigation task involving the prediction of cross-track position and heading angle relative to road center line markings. The results of our tests show that our GAN-based data augmentation pipeline significantly outperforms models trained solely on simulation data or on data processed with standard image augmentation methods for sim-to-real transfer, enhancing model robustness and generalizability in real-world scenarios. Our method provides a scalable and flexible data augmentation tool for leveraging large synthetic datasets to enhance vision-based robotic navigation tasks.

artificial intelligence↗

NASA NTRSDate not supplied

Open Science for Life in Space: Data Sharing and Tools for Knowledge Discovery

The fast-growing array of space biological data, which in the past was simply archived after minimal analysis, holds great potential if it can be reorganized and formatted for Open Science. Organizing the data for such analysis is a challenge because of its diverse nature (molecular, cellular, tissue, whole organism, behavior; tabular, imagery). Open Science is the concept that the more people have access to scientifically curated data, the more knowledge will be gained. This led NASA to start the development of GeneLab in 2015. GeneLab houses spaceflight and space-analog multi-omics datasets from plant, rodent, small animal, and microbial experiments. The success and knowledge gained from GeneLab led to a new alliance of NASA “Open Science Data Repositories” (OSDR), which include the Ames Life Sciences Data Archive (ALSDA) and the NASA Biological Institutional Scientific Collection (NBISC). Both are adopting the GeneLab data system, so data are more findable, accessible, interoperable, and reusable (FAIR). OSDR systems provide users the ability to upload, download, search, share, analyze, and visualize. Open Science also needs strong confidence in the data, which is gained through building science communities. With ~400 current members, GeneLab and ALSDA formed Analysis Working Groups (AWGs) to provide feedback on processing pipelines, metadata curation standards (for ‘omics and phenotypic-physiological-behavioral assays), and to collaborate in effectively reusing data. The AWG also led to the development of the Radiation Biology Ontology (RBO), ensuring radiation metadata are efficiently captured, connected, and interoperable. Feedback from the AWG provided design input toward the new single point-of-entry data submission portal for all investigators to submit, curate, and share their research data. Space biological data is now maximally open access, collected-curated with rich metadata, and formatted for interoperability to enable systems biology, meta-analysis, knowledge graphs, machine learning, modeling, and other reuse approaches. With potential for further federation of OSDR for data mining with traditional biological and medical databases (NIH, NCI, EBI, etc.), a new era for space biology has begun to support the knowledge discovery necessary for Lunar and Martian missions.

Ryan T Scott↗

NASA NTRS2017

Kepler Planet Detection Metrics: Per-Target Flux-Level Transit Injection Tests of TPS for Data Release 25

Quantifying the ability of a transiting planet survey to recover transit signals has commonly been accomplished through Monte-Carlo injection of transit signals into the observed data and subsequent running of the signal search algorithm (Gilliland et al., 2000; Weldrake et al., 2005; Burke et al., 2006). In order to characterize the performance of the Kepler pipeline (Twicken et al., 2016; Jenkins et al., 2017) on a sample of over 200,000 stars, two complementary injection and recovery tests are utilized:1. Injection of a single transit signal per target into the image or pixel-level data, hereafter referred to as pixel-level transit injection (PLTI), with subsequent processing through the Photometric Analysis (PA), Presearch Data Conditioning (PDC), Transiting Planet Search (TPS), and Data Validation (DV) modules of the Kepler pipeline. The PLTI quantification of the Kepler pipeline's completeness has been described previously by Christiansen et al. (2015, 2016); the completeness of the final SOC 9.3 Kepler pipeline acting on the Data Release 25 (DR25) light curves is described by Christiansen (2017).2. Injection of multiple transit signals per target into the normalized flux time series data with a subsequent transit search using a stream-lined version of the Transiting Planet Search (TPS) module. This test, hereafter referred to as flux-level transit injection (FLTI), is the subject of this document. By running a heavily modified version of TPS, FLTI is able to perform many injections on selected targets and determine in some detail which injected signals are recoverable. Significant numerical efficiency gains are enabled by precomputing the data conditioning steps at the onset of TPS and limiting the search parameter space (i.e., orbital period, transit duration, and ephemeris zero-point) to a small region around each injected transit signal.The PLTI test has the advantage that it follows transit signals through all processing steps of the Kepler pipeline, and the recovered signals can be further classified as planet candidates or false positives in the exact same manner as detections from the nominal (i.e., observed) pipeline run (Twicken et al., 2016, Thompson et al., in preparation). To date, the PLTI test has been the standard means of measuring pipeline completeness averaged over large samples of targets (Christiansen et al., 2015, 2016; Christiansen, 2017). However, since the PLTI test uses only one injection per target, it does not elucidate individual-target variations in pipeline completeness due to differences in stellar properties or astrophysical variability. Thus, we developed the FLTI test to provide a numerically efficient way to fully map individual targets and explore the performance of the pipeline in greater detail. The FLTI tests thereby allow a thorough validation of the pipeline completeness models (such as window function (Burke and Catanzarite, 2017a), detection efficiency (Burke Catanzarite, 2017b), etc.) across the spectrum of Kepler targets (i.e., various astrophysical phenomena and differences in instrumental noise). Tests during development of the FLTI capability revealed that there are significant target-to-target variations in the detection efficiency.

DR25↗

NASA NTRS2017

GEONEX: Land Monitoring From a New Generation of Geostationary Satellite Sensors

The latest generation of geostationary satellites carry sensors such as ABI (Advanced Baseline Imager on GOES-16) and the AHI (Advanced Himawari Imager on Himawari) that closely mimic the spatial and spectral characteristics of Earth Observing System flagship MODIS for monitoring land surface conditions. More importantly they provide observations at 5-15 minute intervals. Such high frequency data offer exciting possibilities for producing robust estimates of land surface conditions by overcoming cloud cover, enabling studies of diurnally varying local-to-regional biosphere-atmosphere interactions, and operational decision-making in agriculture, forestry and disaster management. But the data come with challenges that need special attention. For instance, geostationary data feature changing sun angle at constant view for each pixel, which is reciprocal to sun-synchronous observations, and thus require careful adaptation of EOS algorithms. Our goal is to produce a set of land surface products from geostationary sensors by leveraging NASA's investments in EOS algorithms and in the data/compute facility NEX. The land surface variables of interest include atmospherically corrected surface reflectances, snow cover, vegetation indices and leaf area index (LAI)/fraction of photosynthetically absorbed radiation (FPAR), as well as land surface temperature and fires. In order to get ready to produce operational products over the US from GOES-16 starting 2018, we have utilized 18 months of data from Himawari AHI over Australia to test the production pipeline and the performance of various algorithms for our initial tests. The end-to-end processing pipeline consists of a suite of modules to (a) perform calibration and automatic georeference correction of the AHI L1b data, (b) adopt the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to produce surface spectral reflectances along with compositing schemes and QA, and (c) modify relevant EOS retrieval algorithms (e.g., LAI and FPAR, GPP, etc.) for subsequent science product generation. Initial evaluation of Himawari AHI products against standard MODIS products indicate general agreement, suggesting that data from geostationary sensors can augment low earth orbit (LEO) satellite observations.

geostationary↗