Search NASA⌕ Search

SEARCH · Search NASA

Results for “data processing and archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
NASA NTRS1993

EOSDIS - Its role in the EOS program and its importance to the scientific community

In order to manage the global data sets required to understand the earth as a system, the EOS Data and Information System (EOSDIS) will collect and store satellite, aircraft, and in situ measurements and their resultant data products, and will distribute the data conveniently. EOSDIS will also provide product generation and science computing facilities to support the development, processing, and validation of standard EOS science data products. The overall architecture of EOSDIS, and how the Distributed Active Archive Centers fit into that structure, are shown. EOSDIS will enable users to query data bases nationally, make use of keywords and other mnemonic identifiers, and see graphic images of subsets of available data prior to ordering full (or selected pieces of) data sets for use in their 'home' environment.

Price, Robert D.↗
NASA NTRS1999

Tropical Rainfall Measuring Mission

Tropical rainfall affects the lives and economics of a majority of the Earth's population. Tropical rain systems, such as hurricanes, typhoons, and monsoons, are crucial to sustaining the livelihoods of those living in the tropics. Excess rainfall can cause floods and great property and crop damage, whereas too little rainfall can cause drought and crop failure. The latent heat release during the process of precipitation is a major source of energy that drives the atmospheric circulation. This latent heat can intensify weather systems, affecting weather thousands of kilometers away, thus making tropical rainfall an important indicator of atmospheric circulation and short-term climate change. Tropical forests and the underlying soils are major sources of many of the atmosphere's trace constituents. Together, the forests and the atmosphere act as a water-energy regulating system. Most of the rainfall is returned to the atmosphere through evaporation and transpiration, and the atmospheric trace constituents take part in the recycling process. Hence, the hydrological cycle provides a direct link between tropical rainfall and the global cycles of carbon, nitrogen, and sulfur, all important trace materials for the Earth's system. Because rainfall is such an important component in the interactions between the ocean, atmosphere, land, and the biosphere, accurate measurements of rainfall are crucial to understanding the workings of the Earth-atmosphere system. The large spatial and temporal variability of rainfall systems, however, poses a major challenge to estimating global rainfall. So far, there has been a lack of rain gauge networks, especially over the oceans, which points to satellite measurement as the only means by which global observation of rainfall can be made. The Tropical Rainfall Measuring Mission (TRMM), jointly sponsored by the National Aeronautics and Space Administration (NASA) of the United States and the National Space Development Agency (NASDA) of Japan, provides visible, infrared, and microwave observations of tropical and subtropical rain systems.The satellite observations are complemented by ground radar and rain gauge measurements to validate satellite rain estimation techniques. Goddard Space Flight Center's involvement includes the observatory, four instruments, integration and testing of the observatory, data processing and distribution, and satellite operations. TRMM has a design lifetime of three years. Data generated from TRMM and archived at the GDAAC are useful not only for hydrologists, atmospheric scientists, and climatologists, but also for the health community studying infectious diseases, the ocean research community, and the agricultural community.

Source record↗
NASA NTRS2008

The NPOESS Preparatory Project (NPP) Science Data Segment (SDS) Data Depository and Distribution Element (SD3E) System Architecture

The National Polar-orbiting Operational Environmental Satellite System (NPOESS), the U.S. Government's future low-Earth orbiting satellite system, will monitor global weather and environmental conditions. Serving as a risk reduction for NPOESS, the NPOESS Preparatory Project (NPP) will provide remotely sensed atmospheric, land, ocean, ozone, and sounder data that will serve the meteorological and global climate change scientific communities. The National Aeronautics and Space Administration (NASA) NPP Science Data Segment's (SDS) primary role is to independently assess the quality of the NPP science and environmental data records for their ability to support climate research. The SDS is composed of nine elements; an input element that receives data from the operational agencies and acts as a buffer, a calibration analysis element, five elements devoted to measurement based quality assessment, an element used to test algorithmic improvements, and an element that provides overall science direction. Each element requires a set of sensor specific science data products for their evaluation. There are four NPP sensors that will be flown on the NPP observatory. They are the Visible Infrared Imagining Radiometer Suite (VIIRS), the Advanced Technology Microwave Sounder (ATMS), the Cross-Track Infrared Sounder (CrIS), and the Ozone Mapper/Profiler Suite (OMPS). It is estimated that these four sensors combined will make daily data requests for approximately six terabytes of NPP science products from the operational data providers. As a result, issues associated with duplicate data requests, data transfers of large volumes of diverse products, and data transfer failures raised concerns with respect to the network traffic and bandwidth consumption. Therefore, a central data broker system for receiving and buffering data requests and data products for the SDS was developed. The data element for this system is called the SDS Data Depository and Distribution Element (SD3E). It supports science mission data assessment by assuring the timely and validated acquisition and subsequent transfer of the NPP Science Mission data to the SDS Elements and NPP Science Team. The six science elements that interface with the SD3E span across the NASA Goddard Space Flight Center (GSFC), the NASA Jet Propulsion Laboratory (JPL), and the University of Wisconsin. As the primary communication vehicle for the science elements and science team, the SD3E has an interface to the operational data providers: National Environment Satellite, Data, and Information Service (NESDIS) Interface Data Processing System (IDPS) and the National Oceanic Atmospheric Administration's (NOAA) Comprehensive Large Array-data Stewardship system (CLASS) Archive Data System (ADS), that are responsible for product generation and archive and distribution respectively. The SD3E is designed to be a semi-customizable and semi-automated system. This system is designed to provide flexibility and ease of use for the science users in accessing the latest data products by creating a rolling data cache that temporarily stores the products locally before transferring the data to the SDS Measurement based elements for the land, ocean, atmosphere, sounder, and ozone. This paper describes the design and architecture of one of the nine SDS elements, the SD3E, and how this system has provided a mechanism for efficient data exchange, how it has helped in alleviating some of the network traffic and usage, and how it has contributed to reducing operational costs.

Ho, Evelyn L.↗
NASA NTRSDate not supplied

TPSAS-NF1676L-31179-DND

This presentation will brief the GEWEX Radiation Panel on the progress and results of the GEWEX SRB project during the last year toward production and assessment of the Release 4.0 data set. First, an update is presented regarding SRB Release 4.0 "Integrated Product" that produces consistent with GEWEX projects estimating land and ocean latent and sensible heat fluxes. Changes from the Release 3/3.1 data products to Release 4 are reviewed both for key inputs (such as the ISCCP HXS, the nn-HIRS, Land and Sea surface temperatures) and key algorithm changes. Differences between the old and new input properties are assessed and flux sensitivities to those inputs are summarized. Sample radiative flux data products are presented and compared against the previous version for span of the production run, currently from 1998 through 2012. These fluxes are also compared to surface measurements (i.e., BSRN) and CERES (Clouds and Earth's Radiant Energy System) data products both in terms of bulk statistics and also time series variability. Lastly, the plans for continued processing, archival, assessment and research and will be presented to the panel for discussion and feedback.

Paul W Stackhouse, Jr↗
NASA NTRS1990

Evolution of the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) flight and ground data processing system

The modifications made to both the AVIRIS instrument and the data processing facility since the instrument made its initial flight in the summer of 1987 are described. Historical development of the data system is discussed and attention is given to enhancements to instrument stability and noise performance. AVIRIS data facility objectives include rapid and automated decommutation and archiving of data, the ability to assess the quality of the data and health of the instrument, and the provision of an automated procedure for applying radiometric corrections to the data and providing responsive processing of data requests from investigators. The modifications described have resulted in an overall radiometric stability of better than 10 percent, with stability of only a few percent during a single flight.

Porter, Wallace M.↗
NASA NTRS1992

AVIRIS ground data processing system

During the last year and a half, Feb. 1991 to Jun. 1992, a major upgrade of the Airborne Visible/Infrared Imaging Spectrometers (AVIRIS) ground data processing system took place. Both the hardware and software components were changed significantly to improve the processing capacity and performance and to structure a data facility capable of handling the projected work load into the near future. A summary report of these changes and some projections for the future are provided. The objectives of the AVIRIS data facility are to decommutate and archive AVIRIS data and to provide raw or radiometrically calibrated data products to the science investigator. These primary objectives have not changed from the initial concepts. The upgrade effort has greatly improved the processing system. These objectives can now be accomplished in a more timely fashion at a reasonable cost and there is sufficient capacity to manage the current processing load and provide for future growth. The method of implementation added the flexibility to provide better service to the investigator and allow for future changes.

Hansen, Earl G.↗
NASA NTRS1993

Interim report on Landsat national archive activities

The Department of the Interior (DOI) has the responsibility to preserve and to distribute most Landsat Thematic Mapper (TM) and Multispectral Scanner (MSS) data that have been acquired by the five Landsat satellites operational since July 1972. Data that are still covered by exclusive marketing rights, which were granted by the U.S. Government to the commercial Landsat operator, cannot be distributed by the DOI. As the designated national archive for Landsat data, the U.S. Geological Survey's EROS Data Center (EDC) has initiated two new programs to protect and make available any of the 625,000 MSS scenes currently archived and the 200,000 TM scenes to be archived at EDC by 1995. A specially configured system has begun converting Landsat MSS data from obsolete high density tapes (HDT's) to more dense digital cassette tapes. After transcription, continuous satellite swaths are (1) divided into standard scenes defined by a world reference system, (2) geographically located by latitude and longitude, and (3) assessed for overall quality. Digital browse images are created by subsampling the full-resolution swaths. Conversion of the TM HDT's will begin in the fourth quarter of 1992 and will be conducted concurrently with MSS conversion. Although the TM archive is three times larger than the entire MSS archive, conversion of data from both sensor systems and consolidation of the entire Landsat archive at EDC will be completed by the end of 1994. Some MSS HDT's have deteriorated, primarily as a result of hydrolysis of the pigment binder. Based on a small sample of the 11 terabytes of post-1978 MSS data and the 41 terabytes of TM data to be converted, it appears that to date, less than 2 percent of the data have been lost. The data loss occurs within small portions of some scenes; few scenes are lost entirely. Approximately 10,000 pre-1979 MSS HDT's have deteriorated to such an extent, as a result of hydrolysis, that the data cannot be recovered without special treatment of the tapes. An independent consulting division of a major tape manufacturer has analyzed affected tapes and is confident that restorative procedures can be applied to the HDT's to permit one pass to reproduce the data on another recording media. A system to distribute minimally processed Landsat data will be procured in 1992 and will be operational by mid-1994. Any TM or MSS data in the national archive that are not restricted by exclusive marketing rights will be reproduced directly from the archive media onto user specified computer-compatible media. TM data will be produced either at a raw level (radiometrically and geometrically uncorrected) or at an intermediate level (radiometrically corrected and geometrically indexed). MSS data will be produced to an intermediate level or to a fully corrected level (radiometrically corrected and geometrically transformed to an Oblique Mercator projection). The system will be capable of providing ordered scenes within 48 hours of receipt of order.

Boyd, John E.↗
NASA NTRS2009

Characterizing Space Environments with Long-Term Space Plasma Archive Resources

A significant scientific benefit of establishing and maintaining long-term space plasma data archives is the ready access the archives afford to resources required for characterizing spacecraft design environments. Space systems must be capable of operating in the mean environments driven by climatology as well as the extremes that occur during individual space weather events. Long- term time series are necessary to obtain quantitative information on environment variability and extremes that characterize the mean and worst case environments that may be encountered during a mission. In addition, analysis of large data sets are important to scientific studies of flux limiting processes that provide a basis for establishing upper limits to environment specifications used in radiation or charging analyses. We present applications using data from existing archives and highlight their contributions to space environment models developed at Marshall Space Flight Center including the Chandra Radiation Model, ionospheric plasma variability models, and plasma models of the L2 space environment.

Minow, Joseph I.↗
NASA NTRS1988

The GONG Data Reduction and Analysis System

Each of the six GONG observing stations will produce three, 16-bit, 256X256 images of the Sun every 60 sec of sunlight. These data will be transferred from the observing sites to the GONG Data Management and Analysis Center (DMAC), in Tucson, on high-density tapes at a combined rate of over 1 gibabyte per day. The contemporaneous processing of these data will produce several standard data products and will require a sustained throughput in excess of 7 megaflops. Peak rates may exceed 50 megaflops. Archives will accumulate at the rate of approximately 1 terabyte per year, reaching nearly 3 terabytes in 3 yr of observing. Researchers will access the data products with a machine-independent GONG Reduction and Analysis Software Package (GRASP). Based on the Image Reduction and Analysis Facility, this package will include database facilities and helioseismic analysis tools. Users may access the data as visitors in Tucson, or may access DMAC remotely through networks, or may process subsets of the data at their local institutions using GRASP or other systems of their choice. Elements of the system will reach the prototype stage by the end of 1988. Full operation is expected in 1992 when data acquisition begins.

Pintar, James A.↗
DOE OSTI2025

A Curated Dataset of Regional Meteor Events with Simultaneous Optical and Infrasound Observations (2006–2011)

We present a curated, openly accessible dataset of 71 regional meteor events simultaneously recorded by optical and infrasound instrumentation between 2006 and 2011. These events were captured during an observational campaign using the all-sky cameras of the Southern Ontario Meteor Network and the co-located Elginfield Infrasound Array. Each entry provides optical trajectory measurements, infrasound waveforms, and atmospheric specification profiles. The integration of optical and acoustic data enables robust linkage between observed acoustic signals and specific points along meteor trajectories, offering new opportunities to examine shock wave generation, propagation, and energy deposition processes. This release fills a critical observational gap by providing the first validated, openly accessible archive of simultaneous optical–infrasound meteor observations that supports trajectory reconstruction, acoustic propagation modeling, and energy deposition analyses. By making these data openly available in a structured format, this work establishes a durable reference resource that advances reproducibility, fosters cross-disciplinary research, and underpins future developments in meteor physics, atmospheric acoustics, and planetary defense.

astrometry↗
NASA NTRS1992

The NSO FTS database program and archive (FTSDBM)

Data from the NSO Fourier transform spectrometer is being re-archived from half inch tape onto write-once compact disk. In the process, information about each spectrum and a low resolution copy of each spectrum is being saved into an on-line database. FTSDBM is a simple database management program in the NSO external package for IRAF. A command language allows the FTSDBM user to add entries to the database, delete entries, select subsets from the database based on keyword values including ranges of values, create new database files based on these subsets, make keyword lists, examine low resolution spectra graphically, and make disk number/file number lists. Once the archive is complete, FTSDBM will allow the database to be efficiently searched for data of interest to the user and the compact disk format will allow random access to that data.

Lytle, D. M.↗
DOE OSTI2024

Open Source Synergy: Developing and Validating PMU Data Analysis Techniques Using Open Source Tools and Datasets

This paper presents an exploration into the development and validation of data analysis approaches for Phasor Measurement Units (PMUs) using open-source datasets and tools. Various methods for event detection, event classification, frequency response, and oscillation analysis were tested. We leverage the capabilities of Archive Walker (AW), the Frequency Response Analysis Tool (FRAT), and the Oscillation Baselining and Analysis Tool (OBAT), all open-source tools, for efficient processing and analysis of synchrophasor data. The open-source Transmission Signature Library (TSL) dataset was employed as a dataset for a comprehensive evaluation to assess the performance and reliability of the proposed methods.

PMU, event analysis, oscillation, Frequency Respon↗
NASA NTRS2019

Collecting and Processing Earth Science Data Metrics at NASA ESDIS

Since the launch of Terra satellite in 1999, the number of Earth Science remote sensing data products created and distributed by NASA's Earth Observing System (EOS) Data and Information System (EOSDIS) has increased from a few hundred to nearly ten thousand. NASA's Earth Science Data and Information System (ESDIS) Metrics System (EMS) collects metrics on data ingest, archive, and distribution by its Distributed Active Archive Centers (DAACs) and the Science Investigator-led Systems (SIPS), known as Data Providers. These metrics are critical in helping NASA management as well as data producers in resource planning and gaining a wide range of knowledge of data users and data usage.EMS receives flat files, or log files of data archive, ingest, and distribution either in their raw format, such as Apache web logs, or text files of log records formatted by the Data Providers. Tens of millions of records are processed each day to extract metrics on data products, user information, distribution protocols and services, and so on. The metrics are then made available to designated parties.This presentation provides an overview of the EMS processing workflow and improvement efforts made in recent years to handle ever-increasing number of data records and new metrics requirements, discusses several key steps including mapping log records to data products and identifying user communities along with geo-distribution, and demonstrates typical metrics capabilities produced by the EMS system. Challenges and potential approaches to improve the system are also discussed.

Pan, Jianfu↗
NASA NTRS2001

The NOAA-NASA CZCS Reanalysis Effort

Satellite observations of global ocean chlorophyll span over two decades. However, incompatibilities between processing algorithms prevent us from quantifying natural variability. We applied a comprehensive reanalysis to the Coastal Zone Color Scanner (CZCS) archive, called the NOAA-NASA CZCS Reanalysis (NCR) Effort. NCR consisted of 1) algorithm improvement (AI), where CZCS processing algorithms were improved using modernized atmospheric correction and bio-optical algorithms, and 2) blending, where in situ data were incorporated into the CZCS AI to minimize residual errors. The results indicated major improvement over the previously available CZCS archive. Global spatial and seasonal patterns of NCR chlorophyll indicated remarkable correspondence with modern sensors, suggesting compatibility. The NCR permits quantitative analyses of interannual and interdecadal trends in global ocean chlorophyll.

Gregg, Watson W.↗
NASA NTRS1985

Planetary image conversion task

The Planetary Image Conversion Task group processed 12,500 magnetic tapes containing raw imaging data from JPL planetary missions and produced an image data base in consistent format on 1200 fully packed 6250-bpi tapes. The output tapes will remain at JPL. A copy of the entire tape set was delivered to US Geological Survey, Flagstaff, Ariz. A secondary task converted computer datalogs, which had been stored in project specific MARK IV File Management System data types and structures, to flat-file, text format that is processable on any modern computer system. The conversion processing took place at JPL's Image Processing Laboratory on an IBM 370-158 with existing software modified slightly to meet the needs of the conversion task. More than 99% of the original digital image data was successfully recovered by the conversion task. However, processing data tapes recorded before 1975 was destructive. This discovery is of critical importance to facilities responsible for maintaining digital archives since normal periodic random sampling techniques would be unlikely to detect this phenomenon, and entire data sets could be wiped out in the act of generating seemingly positive sampling results. Reccomended follow-on activities are also included.

Martin, M. D.↗
NASA NTRS2017

Limiting Data Friction by Reducing Data Download Using Spatiotemporally Aligned Data Organization Through STARE

Current data processing practice limits the volume and variety of relevant geoscience data that can practically be applied to important problems. File archives in centralized data centers are the principal means by which Earth Science data are accessed. This approach, however, requires laborious search, retrieval, and eventual customization/adaptation for the data to be used. Such fractionation makes it even more difficult to share outcomes, i.e. research artifacts and data products, hampering reusability and repeatability, since end users generally have their own research agenda and preferences as well as scarce resources. Thus, while finding and downloading data files from central data centers are already costly for end users working in their own field, using data products from other disciplines rapidly becomes prohibitive. This curtails scientific productivity, limits avenues of study, and endangers quality and reproducibility. The Spatio-Temporal Adaptive Resolution Encoding (STARE) is a unifying scheme that facilitates the indexing, access, and fusion of diverse Earth Science data. STARE implements an innovative encoding of geo-spatiotemporal information, originally developed for aligning datasets with diverse spatiotemporal characteristics in an array database. The spatial component of STARE recursively quadfurcates a root polyhedron, producing a hierarchical scheme for addressing geographic locations and regions. The temporal component of STARE uses conventional date-time units as an indexing hierarchy. The additional encoding of spatial and temporal resolution information in STARE enables comparisons and conditional selections across diverse datasets. Moreover, spatiotemporal set-operations, e.g. union and intersection, are mapped to efficient integer operations with STARE. Applied to existing data models (point, grid, spacecraft swath) and corresponding granules, STARE indexes provide a streamlined description usable as geo-spatiotemporal metadata. When coupled with large scale, distributed hardware and software, STARE-based data access reduces pre-analysis data preparation costs by offering a convenient means to align different datasets spatiotemporally without specialized effort in parallel computing or distributed data management.

Kuo, Kwo-Sen↗
NASA NTRS2019

TESS Science Processing Operations Center Pipeline and Data Products

TESS (Transiting Exoplanet Survey Satellite) launched on 18-4-2018 to conduct a two-year, near all-sky survey for at least 50 nearby exoplanets for which masses can be obtained. TESS just completed surveying the southern hemisphere, identifying hundreds of candidate exoplanet systems and unveiling a plethora of exciting non-exoplanet astrophysics results, such as asteroseismology, asteroids, and supernova. The TESS Science Processing Operations Center (SPOC) at NASA Ames Research Center processes the image data downlinked from TESS every two weeks to generate a variety of data products hosted at the Mikulski Archive for Space Telescopes (MAST). For each approximately 1-month sector, the SPOC calibrates the image data for both 30-minute Full Frame Images (FFIs) and up to 20,000 pre-selected 2-minute target star postage stamps. Simple aperture photometry and systematic error-corrected flux-time series are generated for the 2-minute data. The data products also include co-trending basis vectors (CBVs) and calibration files, such as the Pixel Response Functions (PRF). The archival files are modeled after Kepler's for ease of use, and include Target Pixel Files (TPFs) containing original and calibrated 2-minute image data, Light Curve files (LCs) containing the photometric time series for each 2-minute target, as well as the Data Validation products. New products derived from the FFIs include light curves for the 2-minute targets and CBVs. The TESS Mission is funded by NASA's Science Mission Directorate as an Astrophysics Explorer Mission.

Science Pipeline↗
NASA NTRS2019

TESS Science Processing Operations Center Pipeline and Data Products

TESS launched 18-4-2018 to conduct a two-year, near all-sky survey for at least 50 nearby exoplanets for which masses can be. TESS just completed surveying the southern hemisphere, identifying hundreds of candidate exoplanet systems and unveiling a plethora of exciting non-exoplanet astrophysics results, such as asteroseismology, asteroids, and supernova. The TESS Science Processing Operations Center (SPOC) at NASA Ames Research Center processes the image data downlinked from TESS every two weeks to generate a variety of data products hosted at the Mikulski Archive for Space Telescopes (MAST). For each ~1 month sector, the SPOC calibrates the image data for both 30-min Full Frame Images (FFIs) and up to 20,000 pre-selected 2-min target star postage stamps. Simple aperture photometry and systematic error-corrected flux time series are generated for the 2-min data. The data products also include co-trending basis vectors (CBVs) and calibration files, such as the Pixel Response Functions (PRF). The archival files are modeled after Kepler's for ease of use, and include Target Pixel Files (TPFs) containing original and calibrated 2-min image data, Light Curve files (LCs) containing the photometric time series for each 2-min target, as well as the Data Validation products. New products derived from the FFIs include light curves for the 2-min targets and CBVs. The TESS Mission is funded by NASA's Science Mission Directorate as an Astrophysics Explorer Mission.

Jenkins, Jon M.↗