Search NASA⌕ Search

SEARCH · Search NASA

Results for “pipeline data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

GL4U: Using Space Biology Omics Data to Provide Bioinformatics Training for Students and Educators

NASA’s GeneLab project provides researchers open access to space-relevant multi-omics data via the Open Science Data Repository (OSDR) that can be mined to understand the effects of spaceflight on biological systems. To maximize the number of scientists who understand and utilize GeneLab data and data processing pipelines, GeneLab created GeneLab for Colleges and Universities (GL4U). GL4U provides space biology-relevant training in bioinformatics to the next generation of scientists through direct (training students) and indirect (training educators) approaches. The GL4U pilot programs were conducted in June 2021 (direct training) and 2022 (indirect training). During the pilots, students and educators at Historically Black Colleges and Universities (HBCUs) and Minority Serving Institutions (MSIs) participated in a week-long (direct training) or two-week-long (indirect training) bootcamp consisting of space biology-specific lectures and hands-on instruction using Jupyter Notebooks to analyze space biology RNA sequencing data from OSDR. During the educator pilot, participants received materials, training, and the necessary compute resources to enable them to run the bootcamp at their home institutions, thereby extending the reach of this initiative. In July 2023, GeneLab is partnering with JPL to expand GL4U to include amplicon sequencing (Amp-Seq) analysis training. During the GL4U Amp-Seq bootcamp, student and educator participants will receive training on how to analyze and interpret Amp-Seq data using the NASA GeneLab data processing pipeline. All bootcamp material, including instructions for requesting compute resources, will be made publicly available on GitHub for educators to teach the GL4U content in subsequent semesters. GL4U provides undergraduate students from underrepresented groups the opportunity to learn about NASA and Space Biology, and to enhance their career prospects by gaining hands-on experience analyzing omics data, a skillset that is highly applicable and marketable in the life sciences. We present results from pre- and post-training surveys completed by all participants of the Amp-Seq bootcamp.

Amanda M Saravia-Butler↗
NASA NTRS2018

The Zwicky Transient Facility: Data Processing, Products, and Archive

The Zwicky Transient Facility (ZTF) is a new robotic time-domain survey currently in progress using the Palomar 48-inch Schmidt Telescope. ZTF uses a 47 square degree field with a 600 megapixel camera to scan the entire northern visible sky at rates of ∼3760 square degrees/hour to median depths of g ~ 20.8 and r ~ 20.6 mag (AB, 5σ in 30 sec). We describe the Science Data System that is housed at IPAC, Caltech. This comprises the data-processing pipelines, alert production system, data archive, and user interfaces for accessing and analyzing the products. The real-time pipeline employs a novel image-differencing algorithm, optimized for the detection of point-source transient events. These events are vetted for reliability using a machine-learned classifier and combined with contextual information to generate data-rich alert packets. The packets become available for distribution typically within 13 minutes (95th percentile) of observation. Detected events are also linked to generate candidate moving-object tracks using a novel algorithm. Objects that move fast enough to streak in the individual exposures are also extracted and vetted. We present some preliminary results of the calibration performance delivered by the real-time pipeline. The reconstructed astrometric accuracy per science image with respect to Gaia DR1 is typically 45 to 85 milliarcsec. This is the RMS per-axis on the sky for sources extracted with photometric S/N ≥10 and hence corresponds to the typical astrometric uncertainty down to this limit. The derived photometric precision (repeatability) at bright unsaturated fluxes varies between 8 and 25 millimag. The high end of these ranges corresponds to an airmass approaching ∼2—the limit of the public survey. Photometric calibration accuracy with respect to Pan-STARRS1 is generally better than 2%. The products support a broad range of scientific applications: fast and young supernovae; rare flux transients; variable stars; eclipsing binaries; variability from active galactic nuclei; counterparts to gravitational wave sources; a more complete census of Type Ia supernovae; and solar-system objects.

Frank J. Masci↗
NASA NTRS2010

Kepler Data Release 3 Notes

This describes the collection of data and the processing done on it so when researchers around the world get the Kepler data sets (which are a set of pixels from the telescope of a particular target (star, galaxy or whatever) over a 3 month period) they can adjust their algorithms fro things that were done (like subtracting all of one particular wavelength for example). This is used to calibrate their own algorithms so that they know what it is they are starting with. It is posted so that whoever is accessing the publicly available data (not all of it is made public) can understand it .. (most of the Kepler data is under restriction for 1 - 4 years and is not available, but the handbook is for everyone (public and restricted) The Data Analysis Working Group have released long and short cadence materials, including FFls and Dropped Targets for the Public. The Kepler Science Office considers Data Release 3 to provide "browse quality" data. These notes have been prepared to give Kepler users of the Multimission Archive at STScl (MAST) a summary of how the data were collected and prepared, and how well the data processing pipeline is functioning on flight data. They will be updated for each release of data to the public archive and placed on MAST along with other Kepler documentation, at http:// archive.stsci.edu/kepler/documents.html .Data release 3 is meant to give users the opportunity to examine the data for possibly interesting science and to involve the users in improving the pipeline for future data releases. To perform the latter service, users are encouraged to notice and document artifacts, either in the raw or processed data, and report them to the Science Office.

Cleve, Jeffrey E.↗
NASA NTRS2010

Kepler Data Release 4 Notes

The Data Analysis Working Group have released long and short cadence materials, including FFIs and Dropped Targets for the Public. The Kepler Science Office considers Data Release 4 to provide "browse quality" data. These notes have been prepared to give Kepler users of the Multimission Archive at STScl (MAST) a summary of how the data were collected and prepared, and how well the data processing pipeline is functioning on flight data. They will be updated for each release of data to the public archive and placed on MAST along with other Kepler documentation, at http://archive.stsci.edu/kepler/documents.html. Data release 3 is meant to give users the opportunity to examine the data for possibly interesting science and to involve the users in improving the pipeline for future data releases. To perform the latter service, users are encouraged to notice and document artifacts, either in the raw or processed data, and report them to the Science Office.

Van Cleve, Jeffrey↗
NASA NTRS2012

The Kepler End-to-End Data Pipeline: From Photons to Far Away Worlds

The Kepler mission is described in overview and the Kepler technique for discovering exoplanets is discussed. The design and implementation of the Kepler spacecraft, tracing the data path from photons entering the telescope aperture through raw observation data transmitted to the ground operations team is described. The technical challenges of operating a large aperture photometer with an unprecedented 95 million pixel detector are addressed as well as the onboard technique for processing and reducing the large volume of data produced by the Kepler photometer. The technique and challenge of day-to-day mission operations that result in a very high percentage of time on target is discussed. This includes the day to day process for monitoring and managing the health of the spacecraft, the annual process for maintaining sun on the solar arrays while still keeping the telescope pointed at the fixed science target, the process for safely but rapidly returning to science operations after a spacecraft initiated safing event and the long term anomaly resolution process.The ground data processing pipeline, from the point that science data is received on the ground to the presentation of preliminary planetary candidates and supporting data to the science team for further evaluation is discussed. Ground management, control, exchange and storage of Kepler's large and growing data set is discussed as well as the process and techniques for removing noise sources and applying calibrations to intermediate data products.

data archiving↗
NASA NTRS2019

Kepler Data Validation II–Transit Model Fitting and Multiple-Planet Search

This paper discusses the transit model-fitting and multiple-planet search algorithms and performance of the Kepler Science Data Processing Pipeline, developed by the Kepler Science Operations Center (SOC). Threshold crossing events (TCEs), which are transit candidate events, are generated by the Transiting Planet Search (TPS) component of the pipeline and subsequently processed in the data validation (DV) component. The transit model is used in DV to fit TCEs to characterize planetary candidates and to derive parameters that are used in various diagnostic tests to classify them. After the signature associated with the TCE is removed from the light curve of the target star, the residual light curve goes through TPS again to search for additional TCEs. The iterative process of transit model fitting and multiple-planet search continues until no TCE is generated from the residual light curve or an upper limit is reached. The transit model-fitting and multiple-planet search performance of the final release (9.3, 2016January) of the pipeline is demonstrated with the results of the processing of four years (17 quarters) of flight data from the primary Kepler Mission. The transit model-fitting results are accessible from the NASA Exoplanet Archive. The final version of the SOC codebase is available through GitHub.

Threshold crossing events (TCEs↗
NASA NTRS2016

Kepler: A Search for Terrestrial Planets - Kepler Data Characterization Handbook

The Kepler Data Characteristics Handbook (KDCH) provides a description of all phenomena identified in the Kepler data throughout the mission, and an explanation for how these characteristics are handled by the final version of the Kepler Data Processing Pipeline (SOC 9.3).The KDCH complements the Kepler Data Release Notes (KDRNs), which document phenomena and processing unique to a data release. The original motivation for this separation into static, explanatory text and a more journalistic set of figures and tables in the KDRN was for the user to become familiar with the Data Characteristics Handbook, then peruse the short Notes for a new quarter, referring back to the Handbook when necessary. With the completion of the Kepler mission and the final Data Release 25, both the KDCH and the DRN encompass the entire Kepler mission, so the distinction between them is in the level of exposition, not the extent of the time interval discussed.

Kepler↗
NASA NTRS2016

Description of the TCERT Vetting Reports for Data Release 25

This document, the Kepler Instrument Handbook (KIH), is for Kepler and K2 observers, which includes the Kepler Science Team, Guest Observers (GOs), and astronomers doing archival research on Kepler and K2 data in NASAs Astrophysics Data Analysis Program (ADAP). The KIH provides information about the design, performance, and operational constraints of the Kepler flight hardware and software, and an overview of the pixel data sets available. The KIH is meant to be read with these companion documents:1. Kepler Data Processing Handbook (KSCI-19081) or KDPH (Jenkins et al., 2016). The KDPH describes how pixels downlinked from the spacecraft are converted by the Kepler Data Processing Pipeline (henceforth just the pipeline) into the data products delivered to the MAST archive. 2. Kepler Archive Manual (KDMC-10008) or KAM (Thompson et al., 2016). The KAM describes the format and content of the data products, and how to search for them.3. Kepler Data Characteristics Handbook (KSCI-19040) or KDCH (Christiansen et al., 2016). The KDCH describes recurring non-astrophysical features of the Kepler data due to instrument signatures, spacecraft events, or solar activity, and explains how these characteristics are handled by the pipeline.4. Kepler Data Release Notes 25 (KSCI-19065) or DRN 25 (Thompson et al., 2015). DRN 25 describes signatures and events peculiar to individual quarters, and the pipeline software changes between a data release and the one preceding it.Together, these documents supply the information necessary for obtaining and understanding Kepler results, given the real properties of the hardware and the data analysis methods used, and for an independent evaluation of the methods used if so desired.

Instrument↗
NASA NTRSDate not supplied

The Nasa SRA Process as It Relates to Open-Source Workflows Developed for GeneLab Data Processing

To release open, standards-compliant processed data sets in the Open Science Data Repository (OSDR), the GeneLab Data Processing team works with the scientific community through the OSDR Analysis Working Groups to design and build open-source data processing pipelines. Once baselined internally, these pipelines are wrapped into workflows and published on the NASA GeneLab Data Processing public GitHub repository along with detailed instructions for installation and use. Each workflow must be approved through NASA's Software Release Authorization (SRA) process prior to publishing. However, the SRA process lacks sufficient documentation and clarity regarding which forms are applicable for new open-source software that utilizes publicly available 3rd party tools, and the SRA process can take several months to complete, making sharing software outside of NASA cumbersome and in contradiction with the concept of Open Science. Furthermore, the SRA process was designed as a one-size fits all approach and thus many of the questions asked are not applicable to our open-source workflows. Here we describe the software provided on the NASA GeneLab Data Processing GitHub repository, summarize our experiences with the SRA process to release these software, and propose a more stream-lined approach for review of open-source projects.

Software Release Authorization↗
NASA NTRSDate not supplied

The NASA SRA Process as it Relates to Open-Source Workflows Developed for GeneLab Data Processing

To release open, standards-compliant processed data sets in the Open Science Data Repository (OSDR), the GeneLab Data Processing team works with the scientific community through the OSDR Analysis Working Groups to design and build open-source data processing pipelines. Once baselined internally, these pipelines are wrapped into workflows and published on the NASA GeneLab Data Processing public GitHub repository along with detailed instructions for installation and use. Each workflow must be approved through NASA's Software Release Authorization (SRA) process prior to publishing. However, the SRA process lacks sufficient documentation and clarity regarding which forms are applicable for new open-source software that utilizes publicly available 3rd party tools, and the SRA process can take several months to complete, making sharing software outside of NASA cumbersome and in contradiction with the concept of Open Science. Furthermore, the SRA process was designed as a one-size fits all approach and thus many of the questions asked are not applicable to our open-source workflows. Here we describe the software provided on the NASA GeneLab Data Processing GitHub repository, summarize our experiences with the SRA process to release these software, and propose a more stream-lined approach for review of open-source projects.

Software Release Authorization↗
DOE OSTI2024

RectifHydPlus Data Pipeline

The RectifHydPlus Data Pipeline is an open source and fully reproducible data processing pipeline for creating RectifHydPlus—a dataset of historical monthly net electricity generation for all US hydropower plants (>10MW). The pipeline is coded in R, applying tidyverse libraries and code principles, and using the targets data pipeline framework. All data inputs to the RectifHydPlus Data Pipeline are available from public sources. References to all data inputs, as well as instructions for running the RectifHydPlus Data Pipeline, are available on the GitLab code repository: https://code.ornl.gov/turnersw/rectifhydplus

Turner, SeanWilliam Donald [Oak Ridge National Lab↗
DOE OSTI2025

RectifHydPlus Data Pipeline v1.1.0

The RectifHydPlus Data Pipeline is an open source and fully reproducible data processing pipeline for creating RectifHydPlus—a dataset of historical monthly net electricity generation for all US hydropower plants (>10MW). The pipeline is coded in R, applying tidyverse libraries and code principles, and using the targets data pipeline framework. All data inputs to the RectifHydPlus Data Pipeline are available from public sources. References to all data inputs, as well as instructions for running the RectifHydPlus Data Pipeline, are available on the GitLab code repository: https://code.ornl.gov/turnersw/rectifhydplus

Turner, SeanWilliam Donald [Oak Ridge National Lab↗
NASA NTRS2010

A Framework for Propagation of Uncertainties in the Kepler Data Analysis Pipeline

The Kepler space telescope is designed to detect Earth-like planets around Sun-like stars using transit photometry by simultaneously observing 100,000 stellar targets nearly continuously over a three and a half year period. The 96-megapixel focal plane consists of 42 charge-coupled devices (CCD) each containing two 1024 x 1100 pixel arrays. Cross-correlations between calibrated pixels are introduced by common calibrations performed on each CCD requiring downstream data products access to the calibrated pixel covariance matrix in order to properly estimate uncertainties. The prohibitively large covariance matrices corresponding to the ~75,000 calibrated pixels per CCD preclude calculating and storing the covariance in standard lock-step fashion. We present a novel framework used to implement standard propagation of uncertainties (POU) in the Kepler Science Operations Center (SOC) data processing pipeline. The POU framework captures the variance of the raw pixel data and the kernel of each subsequent calibration transformation allowing the full covariance matrix of any subset of calibrated pixels to be recalled on-the-fly at any step in the calibration process. Singular value decomposition (SVD) is used to compress and low-pass filter the raw uncertainty data as well as any data dependent kernels. The combination of POU framework and SVD compression provide downstream consumers of the calibrated pixel data access to the full covariance matrix of any subset of the calibrated pixels traceable to pixel level measurement uncertainties without having to store, retrieve and operate on prohibitively large covariance matrices. We describe the POU Framework and SVD compression scheme and its implementation in the Kepler SOC pipeline.

Clarke, Bruce D.↗
NASA NTRS1999

SIRTF Science Operations System Design

SIRTF Science Operations System Design William B. Green Manager, SIRTF Science Center California Institute of Technology M/S 310-6 1200 E. California Blvd., Pasadena CA 91125 (626) 395 8572 Fax (626) 568 0673 bgreen@ipac.caltech.edu. The Space Infrared Telescope Facility (SIRTF) will be launched in December 2001, and perform an extended series of science observations at wavelengths ranging from 20 to 160 microns for five years or more. The California Institute of Technology has been selected as the home for the SIRTF Science Center (SSC). The SSC will be responsible for evaluating and selecting observation proposals, providing technical support to the science community, performing mission planning and science observation scheduling activities, instrument calibration during operations and instrument health monitoring, production of archival quality data products, and management of science research grants. The science payload consists of three instruments delivered by instrument Principal Investigators located at University of Arizona, Cornell, and Harvard Smithsonian Astrophysical Observatory. The SSC is responsible for design, development, and operation of the Science Operations System (SOS) which will support the functions assigned to the SSC by NASA. The SIRTF spacecraft, mission profile, and science instrument design have undergone almost ten years of refinement. SIRTF development and operations activities are highly cost constrained. The cost constraints have impacted the design of the SOS in several ways. The Science Operations System has been designed to incorporate a set of efficient, easy to use tools which will make it possible for scientists to propose observation sequences in a rapid and automated manner. The use of highly automated tools for requesting observations will simplify the long range observatory scheduling process, and the short term scheduling of science observations. Pipeline data processing will be highly automated and data-driven, utilizing a variety of tools developed at JPL, the instrument development teams, and Space Telescope Science Institute to automate processing. An incremental ground data system development approach has been adopted, featuring periodic deliveries that are validated with the flight hardware throughout the various phases of system level development and testing. This approach minimizes development time and decreases operations risk. This paper will describe the top level architecture of the SOS and the basic design concepts. A summary of the incremental development approach will be presented. Examples of the unique science user tools now under final development prior to the first proposal call scheduled for mid-2000 will be shown.

Green, William↗
NASA NTRS2016

Astro-H Data Analysis, Processing and Archive

Astro-H (Hitomi) is an X-ray Gamma-ray mission led by Japan with international participation, launched on February 17, 2016. The payload consists of four different instruments (SXS, SXI, HXI and SGD) that operate simultaneously to cover the energy range from 0.3 keV up to 600 keV. This paper presents the analysis software and the data processing pipeline created to calibrate and analyze the Hitomi science data along with the plan for the archive and user support.These activities have been a collaborative effort shared between scientists and software engineers working in several institutes in Japan and USA.

Archive↗
NASA NTRS2017

Kepler: A Search for Terrestrial Planets - SOC 9.3 DR25 Pipeline Parameter Configuration Reports

This document describes the manner in which the pipeline and algorithm parameters for the Kepler Science Operations Center (SOC) science data processing pipeline were managed. This document is intended for scientists and software developers who wish to better understand the software design for the final Kepler codebase (SOC 9.3) and the effect of the software parameters on the Data Release (DR) 25 archival products.

exoplanet↗
NASA NTRS2018

Astro-H/Hitomi Data Analysis, Processing, and Archive

Astro-H is the x-ray/gamma-ray mission led by Japan with international participation, launched on February 17, 2016. Soon after launch, Astro-H was renamed Hitomi. The payload consists of four different instruments (SXS, SXI, HXI, and SGD) that operate simultaneously to cover the energy range from 0.3 keV up to 600 keV. On March 27, 2016, JAXA lost contact with the satellite and, on April 28, they announced the cessation of the efforts to restore mission operations. Hitomi collected about one months worth of data with its instruments. This paper presents the analysis software and the data processing pipeline created to calibrate and analyze the Hitomi science data, along with the plan for the archive. These activities have been a collaborative effort shared between scientists and software engineers working in several institutes in Japan and United States.

Angelini, Lorella↗
NASA NTRS2012

Enabling Earth Science Through Cloud Computing

Cloud Computing holds tremendous potential for missions across the National Aeronautics and Space Administration. Several flight missions are already benefiting from an investment in cloud computing for mission critical pipelines and services through faster processing time, higher availability, and drastically lower costs available on cloud systems. However, these processes do not currently extend to general scientific algorithms relevant to earth science missions. The members of the Airborne Cloud Computing Environment task at the Jet Propulsion Laboratory have worked closely with the Carbon in Arctic Reservoirs Vulnerability Experiment (CARVE) mission to integrate cloud computing into their science data processing pipeline. This paper details the efforts involved in deploying a science data system for the CARVE mission, evaluating and integrating cloud computing solutions with the system and porting their science algorithms for execution in a cloud environment.

science data system↗