SEARCH · Search NASA
Results for “science data processing”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Onboard science data processing: ST6 autonomous science experiment
Explore the source record for details and available documents.
Earth Science Data Processing With Nextflow
Earth science data processing tasks present many challenges. These tasks often process large input datasets and require scores of CPU-hours to generate results. All but the simplest tasks will be decomposed into a series of computational or data manipulation steps, also known as a scientific workflow. In order to reduce the burden of orchestrating and running the dependent processing steps, a workflow execution engine is required. This poster describes the lessons learned by the CLARREO Pathfinder (CPF) team while developing multiple scientific workflows and utilizing the open-source Nextflow engine to execute them in a cloud computing environment. The Nextflow engine is designed with the following stated goals: first, the engine does not dictate how individual steps in the task are implemented (i.e. it is language and interface agnostic); second, the engine supports easy configuration and modularity at the workflow level so that others can easily execute our workflows to reproduce results; lastly, the engine eases development by transparently scaling execution from local to remote environments. Nextflow was developed for the bioinformatics domain but is a good fit for other scientific workflows where the overall task is well-described by a dataflow diagram. The CPF team has developed Nextflow pipelines (i.e. scientific workflows) to simulate CLARREO radiance, generate large look-up tables for inter-calibration algorithms, and generate L4 intercalibration data products. These pipelines consume from single-digits to hundreds of thousands of CPU-hours. In the development and evolution of these pipelines we have discovered many design patterns, pitfalls, and solutions to common problems. Our goal is to demonstrate important aspects of how to design, implement, run, and ultimately share Nextflow pipelines in the domain of Earth science.
Post-launch lessons learned from the AIRS Science Data Processing System
The AIRS Science Data Processing System, responsible for processing data from the 4-instrument AIRS suite, includes 14 separate executable programs and produces dozens of products.
Landsat 7 Science Data Processing: An Overview
The Landsat 7 Science Data Processing System, developed by NASA for the Landsat 7 Project, provides the science data handling infrastructure used at the Earth Resources Observation Systems (EROS) Data Center (EDC) Landsat Data Handling Facility (DHF) of the United States Department of Interior, United States Geological Survey (USGS) located in Sioux Falls, South Dakota. This paper presents an overview of the Landsat 7 Science Data Processing System and details of the design, architecture, concept of operation, and management aspects of systems used in the processing of the Landsat 7 Science Data.
EOS MLS Science Data Processing System: A Description of Architecture and Capabilities
This paper describes the architecture and capabilities of the Science Data Processing System (SDPS) for the EOS MLS. The SDPS consists of two major components--the Science Computing Facility and the Science Investigator-led Processing System. The Science Computing Facility provides the facilities for the EOS MLS Science Team to perform the functions of scientific algorithm development, processing software development, quality control of data products, and scientific analyses. The Science Investigator-led Processing System processes and reprocesses the science data for the entire mission and delivers the data products to the Science Computing Facility and to the Goddard Space Flight Center Earth Science Distributed Active Archive Center, which archives and distributes the standard science products.
Use of a multimission system for cost effective support of planetary science data processing
JPL's Multimission Operations Systems Office (MOSO) provides a multimission facility at JPL for processing science instrument data from NASA's planetary missions. This facility, the Multimission Image Processing System (MIPS), is developed and maintained by MOSO to meet requirements that span the NASA family of planetary missions. Although the word 'image' appears in the title, MIPS is used to process instrument data from a variety of science instruments. This paper describes the design of a new system architecture now being implemented within the MIPS to support future planetary mission activities at significantly reduced operations and maintenance cost.
Landsat 7 Science Data Processing: A System's Overview
The Landsat Science Data Processing System, developed by NASA for the Landsat 7 Project provides science data handling infrastructure used at the EROS Data Center Landsat 7 Data Handling Facility of the USGS Department of Interior. This paper presents an overview the designs, architectures, and details of the various systems used in the processing of the Landsat 7 Science Data.
HypsIRI On-Board Science Data Processing
Topics include On-board science data processing, on-board image processing, software upset mitigation, on-board data reduction, on-board 'VSWIR" products, HyspIRI demonstration testbed, and processor comparison.
Processing and Analysis of Mars Pathfinder Science Data at JPL's Science Data Processing Section
The Mars Pathfinder mission required new capabilities and adaptation of existing capabilities in order to support science analysis and flight operations requirements imposed by the in-situ nature of the mission.
Science data processing for the Multi-angle Imaging SpectroRadiometer
Ground processing of data from the Multi-angle Imaging SpectroRadiometer (MISR) instrument, part of NASA's Earth Observing System (EOS), uses new and unique science algorithms because MISR's multi-camera observing method has not previously been used anywhere.
Use of Multimission System for Cost Effective Support of Planetary Science Data Processing
JPL's Multimission Operations System Office (MOSO) provides a multimission facility at JPL for processing science instrument data from NASA's planetary missions.
Use of a Multimission System for Cost Effective Support of Planetary Science Data Processing
JPL's Multimission Operatins Systems Office (MOSO) provides a multimissin facility at JPL for processing science instrument data from NASA's planetary missions.
Science data processing
Processing and storage of space science data from Mariner IV space probe
EP/TOMS Science Data Processing Programmer's Guide. Version 1.9
This EP/TOMS Programmer's Guide provides detailed information concerning the data, software, operations, and maintenance of the EP/TOMS Science Data Processing System. This system was developed for the NASA Goddard Space Flight Center (GSFC), Laboratory for Atmospheres, Atmospheric Chemistry Branch (Code 916) by SSAI. The description of the EP/TOMS Science Data Processing System starts with an overall system perspective (Section 3). A description of all data files ingested, stored, and/or produced follows in Section 4. Detailed descriptions of each major software subsystem are in Section 5. Section 6 presents processing procedures. Section 7 presents maintenance procedures. Procedures for updating this document are included in Section 7.
Photometer Performance Assessment in Kepler Science Data Processing
This paper describes the algorithms of the Photometer Performance Assessment (PPA) software component in the science data processing pipeline of the Kepler mission. The PPA performs two tasks: One is to analyze the health and performance of the Kepler photometer based on the long cadence science data down-linked via Ka band approximately every 30 days. The second is to determine the attitude of the Kepler spacecraft with high precision at each long cadence. The PPA component is demonstrated to work effectively with the Kepler flight data.
Anomaly Detection for the Roman Space Telescope Wide Field Instrument’s Science Data Processing Pipeline
The Roman Space Telescope (RST) Wide Field Instrument (WFI) will be utilizing a preliminary Science Data Processing (SDP) pipeline during its Integration and Test, and to some extent during Operations, to track basic statistics and identify known features such as cosmic rays, snowballs as well as possible anomalies in raw detector data. In our detectors, these anomalies appear as jumps in the ramp of a readout and are classified as cosmic rays if they appear as a streak or snowballs if they’re more circular. The WFI employs an array of 18 H4RG-10 detectors that collect image samples. Each set of raw frames within a non-destructive exposure is packaged by the SDP pipeline into image cubes for each detector. Each cube is a time series of 4096 × 4096 accumulating pixel frames. The preliminary analysis pipeline is used to locate anomalies in these time-series accumulation frames and identify the type of anomaly, either natural phenomena or detector characteristic. To compare different methods, we’ve implemented both heuristic-based and data-driven methods to identify anomalies. For the heuristic-based approach, we identify snowballs and cosmic rays by the size and shape of outlier pixel clusters between consecutive frames. For data driven methods, we evaluated a Convolutional Neural Network (CNN) model, and more traditional methods like Principal Component Analysis (PCA). CNN is a supervised learning/classification method. Thus, we used a labeled dataset of anomalies to perform segmentation of the image and identify anomalies. We used previously identified cosmic rays and snowballs to measure the accuracy and efficiency of the mentioned approaches. In evaluating these methods, we aim to pick the best fit for the SDP pipeline’s anomaly detection in terms of both performance and runtime.
Sizing the science data processing requirements for EOS
The methodology used in the compilation and synthesis of baseline science requirements associated with the 30 + EOS (Earth Observing System) instruments and over 2,400 EOS data products (both output and required input) proposed by EOS investigators is discussed. A brief background on EOS and the EOS Data and Information System (EOSDIS) is presented, and the approach is outlined in terms of a multilayer model. The methodology used to compile, synthesize, and tabulate requirements within the model is described. The principal benefit of this approach is the reduction of effort needed to update the analysis and maintain the accuracy of the science data processing requirements in response to changes in EOS platforms, instruments, data products, processing center allocations, or other model input parameters. The spreadsheets used in the model provide a compact representation, thereby facilitating review and presentation of the information content.