Search NASA⌕ Search

SEARCH · Search NASA

Results for “data processing and archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Use NASA Earthdata in the Cloud

With the impending arrival of new, high-data-volume Earth observation missions, NASA’s ability to effectively ingest, process, and archive large amounts of data requires the most cost-effective, flexible, and scalable data-management architectures and technologies. To meet these demands, NASA's Earth Science Data Systems (ESDS) Program is implementing a strategic vision to develop and operate multiple components of NASA's Earth Observing System Data and Information System (EOSDIS) in a commercial cloud environment. As more earth observing data is moved into the cloud, NASA EOSDIS wants to share our reasoning for moving data into the commercial cloud, as well as demonstrate how end-users can take advantage of both data in the cloud and cloud-deployed services. This presentation explains the enterprise reasons for moving data into the cloud, specifically the scalability of cloud systems, the flexibility to pivot to the most optimal language/system/architecture, and the reduced redundancy of a common architecture. Then we primarily focus on the user-driven reasons for moving to the cloud. Equal access to high performance computing helps all parties, especially users that don't have access to their own on-premise computing infrastructure. Rapid deployment allows users to spin up their own systems without needing the extensive platform previously required for large scale development. Cost effectiveness allows users a wide breadth of computing options, from super computer ability to small scale development, which can help underserved communities. Finally we link to internal resources that users can utilize to begin their cloud based development journey without having to be an expert in the field.

Nicholas Doty↗
NASA NTRS2006

A-Train Data Depot: Integrating, Visualizing, and Extracting Cloudsat, CALIPSO, MODIS, and AIRS Atmospheric Measurements Along the A-Train Tracks

The succession of US and international Earth observing satellites that follow each other, seconds to minutes apart, across the local afternoon equator crossing is called the ATrain. The A-Train consists of the following satellites, in order of equator crossing: OCO, EOS Aqua, CloudSat, CALIPSO, PARASOL, and EOS Aura. Flying in such formation increases the number of observations, validates observations, and enables coordination between science observations, resulting in a more complete virtual science platform (Kelly, 2000). The goal of this project is to create the first ever A-Train virtual data portal/center, the A-Train Data Depot (ATDD), to process, archive, access, visualize, analyze and correlate distributed atmosphere measurements from various A-Train instruments along A-Train tracks. The ATDD will enable the free movement of remotely located A-Train data so that they are combined to create a consolidated vertical view of the Earth's Atmosphere along the A-Train tracks. Once the infrastructure of the ATDD is in place, it will be easily evolved to serve data from all A-Train data measurements: one stop shopping. The innovative approach of analyzing and visualizing atmospheric profiles along the platforms track (i.e., time) will be accommodated by reusing the GSFC Atmospheric Composition Data and Information Services Center (ACDISC) visualization and analysis tool, GIOVANNI, existing data reduction tools, on-line archiving for fast data access, access to remote data without unnecessary data transfers, and data retrieval by users finding data desirable for further study. Initial measurements utilized include CALIPSO lidar backscatter, CloudSat radar reflectivity, clear air relative humidity, water vapor and temperature from AIRS, and cloud properties and aerosols from both MODIS. This will be foilowed by associated measurements from TVILS, =MI, HIRDLS, sad TES. Given the independent nature of instrumentlplatform development, the ATDD project has been met with many interesting challenges that, once resolved, will provide a much greater understanding of the relative flight dynamics and data co-registration of the suite of A-Train instruments, thus greatly increasing the accuracy of A-Train data analysis. Some of these challenges will be illustrated and discussed. The project's early visualizations and analysis efforts illustrate the importance of managing data so that measurements from various missions can be combined to enhance the understanding of the atmosphere. A-Train data management coordination, as performed here, is extremely significant in facilitating the A-Train science of clouds, precipitation, aerosol and chemistry.

Kempler, Steven↗
NASA NTRS2000

BOREAS Level-0 ER-2 Navigation Data

The BOREAS Staff Science effort covered those activities that were BOREAS community-level activities or required uniform data collection procedures across sites and time. These activities included the acquisition, processing, and archiving of aircraft navigation/attitude data to complement the digital image data. The level-0 ER-2 navigation data files contain aircraft attitude and position information acquired during the digital image and photographic data collection missions. Temporally, the data were acquired from April to September 1994. Data were recorded at intervals of 5 seconds. The data are stored in tabular ASCII files.

Strub, Richard↗
NASA NTRS2016

NASA SNPP SIPS - Following in the Path of EOS

NASA's Earth Science Data Information System (ESDIS) Project has been operating NASA's Suomi National Polar-Orbiting Partnership (SNPP) Science Data Segment (SDS) since the launch in October 2011. At launch, the SDS focused primarily on the evaluation of Sensor Data Records (SDRs) and Environmental Data Records (EDRs) produced by the Joint Polar Satellite System (JPSS), a National Oceanic and Atmosphere Administration (NOAA) Program, as to their suitability for Earth system science. During the summer of 2014, NASA transitioned to the production of standard Earth Observing System (EOS)-like science products for all instruments aboard Suomi NPP. The five Science Investigator-led Processing Systems (SIPS): Land, Ocean, Atmosphere, Ozone, and Sounder were established to produce the NASA SNPP standard Level 1, Level 2, and global Level 3 products developed by the SNPP Science Teams and to provide the products to NASA's Distributed Active Archive Centers (DAACs) for archive and distribution to the user community. The processing, archiving and distribution of data from NASA's Clouds and the Earth's Radiant Energy System (CERES) and Ozone Mapper/Profiler Suite (OMPS) Limb instruments will continue. With the implementation of the JPSS Block 2 architecture and the launch of JPSS-1, the SDS will receive SNPP data in near real-time via the JPSS Stored Mission Data Hub (JSH), as well as JPSS-1 and future JPSS-2 data. The SNPP SIPS will ingest EOS compatible Level 0 data from the EOS Data Operations System (EDOS) element for their data processing, enabling the continuous EOS-SNPP-JPSS Satellite Data Record.

near real time↗
NASA NTRS1985

User data management

The primary objective is to identify, develop, and demonstrate key data management technologies to support user access to Space Station data. To accomplish this objective, there are several technical challenges which must be addressed. The first is how to provide routine customer access to high volume, dynamic and distributed data bases. This access will encompass the functions of mission and payload planning and operations, data processing and analysis, and data archive and distribution. Secondly, there must be some analysis of architectures for handling high volume data streams like those expected from the Space Station. This analysis will examine the use of packetized versus non-packetized data formats, modular expansion capabilities, real-time versus non-real-time data processing, and the interfaces and architecture required to support telescience operations. The task will also determine benchmarks of performance capabilities for technology operations, such as varied data base structures, data access procedures, distributed data base design, and data base machines. Information is provided here in outline form.

Source record↗
NASA NTRS2011

Forty-Year Calibrated Record of Earth-Surface Reflected Radiance from Landsat: A Review

Sensors on Landsat satellites have been collecting images of the Earth's surface for nearly 40 years. These images have been invaluable for characterizing and detecting changes in the land cover and land use of the world. Although initially conceived as primarily picture generating sensors, even the early sensors were radiometrically calibrated and spectrally characterized prior to launch and incorporated some capabilities to monitor their radiometric calibration once on orbit. Recently, as the focus of studies has shifted to monitoring Earth surface parameters over significant periods of time, serious attention has been focused toward bringing the data from all these sensors onto a common radiometric scale over this 40-year period. This effort started with the most recent systems and then was extended back in time. Landsat-7 ETM+, the best-characterized sensor of the series prior to launch and once on orbit, and the most stable system to date, was chosen to serve as the reference. The Landsat-7 project was the first of the series to build an image assessment system into its ground system, allowing systematic characterization of its sensors and data. Second, the Landsat-5 TM (still operating at the time of the Landsat-7 launch and continues to operate) calibration history was reconstructed based on its internal calibrator, vicarious calibrations, pseudo-invariant sites and a tie to Landsat-7 ETM+ at the time of the commissioning of Landsat-7. This process was performed in two iterations: the earlier one relied primarily on the TM internal calibrator. When this was found to have some deficiencies, a revised calibration was based more on pseudo-invariant sites, though the internal calibrator was still used to establish the short-term variations in response due to icing build up on the cold focal plane. As time progressed, a capability to monitor the Landsat-5 TM was added to the image assessment system. The Landsat-4 TM, which operated from 1982-1992, was the third system to which the radiometric scale was extended. The limited and broken use of the Landsat-4 TM made this analysis more difficult. Eight-day separated image pairs from Landsat-5 combined with analysis of pseudo invariant sites established this history. The fourth and most challenging effort was making the Landsat-1 to -5 MSS sensors' data internally radiometrically consistent. This effort was particularly complicated by the age of the MSS data, varying formats and processing levels in the archive, limited datasets, and limited documentation available. Ultimately, pseudo-invariant sites were identified in North America and used for this effort. Note that most of the Landsat-MSS archived data had already been calibrated using the MSS internal calibrators, so this processing was imbedded in the result. The final effort was developing an absolute scale for Landsat MSS similar to what was already established for the "TM" sensors. This was accomplished by using simultaneous data from Landsat-5 MSS and Landsat-5 TM, accounting for spectral differences between the sensors using EO-1 Hyperion data. The recalibrated history of the Landsat data and implications to users are discussed. The key result from this work is a consistently calibrated Landsat data archive that spans nearly 40 years with total uncertainties on the order of 10% or less for most sensors and bands.

Markham, Brian↗
NASA NTRS2011

Determining the Completeness of the Nimbus Meteorological Data Archive

NASA launched the Nimbus series of meteorological satellites in the 1960s and 70s. These satellites carried instruments for making observations of the Earth in the visible, infrared, ultraviolet, and microwave wavelengths. The original data archive consisted of a combination of digital data written to 7-track computer tapes and on various film media. Many of these data sets are now being migrated from the old media to the GES DISC modern online archive. The process involves recovering the digital data files from tape as well as scanning images of the data from film strips. Some of the challenges of archiving the Nimbus data include the lack of any metadata from these old data sets. Metadata standards and self-describing data files did not exist at that time, and files were written on now obsolete hardware systems and outdated file formats. This requires creating metadata by reading the contents of the old data files. Some digital data files were corrupted over time, or were possibly improperly copied at the time of creation. Thus there are data gaps in the collections. The film strips were stored in boxes and are now being scanned as JPEG-2000 images. The only information describing these images is what was written on them when they were originally created, and sometimes this information is incomplete or missing. We have the ability to cross-reference the scanned images against the digital data files to determine which of these best represents the data set from the various missions, or to see how complete the data sets are. In this presentation we compared data files and scanned images from the Nimbus-2 High-Resolution Infrared Radiometer (HRIR) for September 1966 to determine whether the data and images are properly archived with correct metadata.

Johnson, James↗
NASA NTRS1992

Mission to Planet Earth

Mission to Planet Earth (MTPE) is NASA's concept for an international science program to produce the understanding needed to predict changes in the Earth's environment. NASA and its interagency and international partners will place satellites carrying advanced sensors in strategic Earth orbits to gather multidisciplinary data. A sophisticated data system will process and archive an unprecedented amount of information about the Earth and how it works as a system. Increased understanding of the Earth system is a basic human responsibility, a prerequisite to informed management of the planet's resources and to the preservation of the global environment. An overview of the MTPE, flight programs, data and information systems, interdisciplinary research efforts, and international coordination, is presented.

Wilson, Gregory S.↗
NASA NTRS1995

A visiting scientist program for the burst and transient source experiment

During this project, Universities Space Research Association provided program management and the administration for overseeing the performance of the total contractual effort. The program director and administrative staff provided the expertise and experience needed to efficiently manage the program.USRA provided a program coordinator and v visiting scientists to perform scientific research with Burst and Transient Source Experiment (BATSE) data. This research was associated with the primary scientific objectives of BATSE and with the various BATSE collaborations which were formed in response to the Compton Gamma Ray Observatory Guest Investigator Program. USRA provided administration for workshops, colloquia, the preparation of scientific documentation, etc. and also provided flexible program support in order to meet the on-going needs of MSFC's BATSE program. USRA performed tasks associated with the recovery, archiving, and processing of scientific data from BATSE. A bibliography of research in the astrophysics discipline is attached as Appendix 1. Visiting Scientists and Research Associates performed activities on this project, and their technical reports are attached as Appendix 2.

Kerr, Frank J.↗
NASA NTRS2016

NASA's Earth Science Flight Program Meets the Challenges of Today and Tomorrow

NASA's Earth science flight program is a dynamic undertaking that consists of a large fleet of operating satellites, an array of satellite and instrument projects in various stages of development, a robust airborne science program, and a massive data archiving and distribution system. Each element of the flight program is complex and present unique challenges. NASA builds upon its successes and learns from its setbacks to manage this evolving portfolio to meet NASA's Earth science objectives. NASA fleet of 16 operating missions provide a wide range of scientific measurements made from dedicated Earth science satellites and from instruments mounted to the International Space Station. For operational missions, the program must address issues such as an aging satellites operating well beyond their prime mission, constellation flying, and collision avoidance with other spacecraft and orbital debris. Projects in development are divided into two broad categories: systematic missions and pathfinders. The Earth Systematic Missions (ESM) include a broad range of multi-disciplinary Earth-observing research satellite missions aimed at understanding the Earth system and its response to natural and human-induced forces and changes. Understanding these forces will help determine how to predict future changes, and how to mitigate or adapt to these changes. The Earth System Science Pathfinder (ESSP) program provides frequent, regular, competitively selected Earth science research opportunities that accommodate new and emerging scientific priorities and measurement capabilities. This results in a series of relatively low-cost, small-sized investigations and missions. Principal investigators whose scientific objectives support a variety of studies lead these missions, including studies of the atmosphere, oceans, land surface, polar ice regions, or solid Earth. This portfolio of missions and investigations provides opportunity for investment in innovative Earth science that enhances NASA's capability for better understanding the current state of the Earth system. ESM and ESSP projects often involve partnerships with other US agencies and/or international organizations. This adds to the complexity of mission development, but allows for a greater scientific return on NASA's investments. The Earth Science Airborne Science Program provides manned and unmanned aircraft systems that further science and advance the use of satellite data. NASA uses these assets worldwide in campaigns to investigate extreme weather events, observe Earth system processes, obtain data for Earth science modeling activities, and calibrate instruments flying aboard Earth science spacecraft. The Airborne Science Program has six dedicated aircraft and access to many other platforms. The Earth Science Multi-Mission Operations program acquires, preserves, and distributes observational data from operating spacecraft to support Earth Science research focus areas. The Earth Observing System Data and Information System (EOSDIS), which has been in operations since 1994, primarily accomplishes this. EOSDIS acquires, processes, archives, and distributes Earth Science data and information products. The archiving of NASA Earth Science information happens at eight Distributed Active Archive Centers (DAACs) and four disciplinary data centers located across the United States. The DAACs specialize by topic area, and make their data available to researchers around the world. The DAACs currently house over 9 petabytes of data, growing at a rate of 6.4 terabytes per day. NASA's current Earth Science portfolio is responsive to the National Research Council (NRC) 2007 Earth Science Decadal Survey and well as the 2010 NASA Response to President Obama's Climate Plan. As the program evolves into the future it will leverage the lessons learned from the current missions in operations and development, and plan for adjustments to future objectives in response to the anticipated 2017 NRC Decadal Survey.

Flight Program↗
NASA NTRS2017

Restoration of the Apollo 15 Heat Flow Experiment Data from 1975 to 1977

The Apollo 15 Heat Flow Experiment (HFE) was conducted from July 1971 through January 1977. Two heat flow probes were deployed roughly 8.5 meters apart. Probe 1 and Probe 2 penetrated to 1.4-meters and 1-meter depths into the lunar regolith, respectively. Temperatures at different depths and the surface were logged with 7.25-minute intervals and transmitted to Earth. At the conclusion of the experiment, only data obtained from July 1971 through December 1974 were processed and archived at the National Space Science Data Center (NSSDC) by the principal investigator of the experiment, Marcus Langseth of Columbia University. Langseth died in 1997. It is not known what happened to the HFE data tapes he used. Current researchers have strong interests in re-examining the HFE data for the full duration of the experiment. We have recovered and processed large portions of the Apollo 15 HFE data from 1975 through 1977 by assembling data and metadata from various sources.

HFE↗
NASA NTRS1993

ACE: A distributed system to manage large data archives

Competitive pressures in the oil and gas industry are requiring a much tighter integration of technical data into E and P business processes. The development of new systems to accommodate this business need must comprehend the significant numbers of large, complex data objects which the industry generates. The life cycle of the data objects is a four phase progression from data acquisition, to data processing, through data interpretation, and ending finally with data archival. In order to implement a cost effect system which provides an efficient conversion from data to information and allows effective use of this information, an organization must consider the technical data management requirements in all four phases. A set of technical issues which may differ in each phase must be addressed to insure an overall successful development strategy. The technical issues include standardized data formats and media for data acquisition, data management during processing, plus networks, applications software, and GUI's for interpretation of the processed data. Mass storage hardware and software is required to provide cost effective storage and retrieval during the latter three stages as well as long term archival. Mobil Oil Corporation's Exploration and Producing Technical Center (MEPTEC) has addressed the technical and cost issues of designing, building, and implementing an Advanced Computing Environment (ACE) to support the petroleum E and P function, which is critical to the corporation's continued success. Mobile views ACE as a cost effective solution which can give Mobile a competitive edge as well as a viable technical solution.

Daily, Mike I.↗
NASA NTRS2017

GES DISC Data Recipes in Jupyter Notebooks

The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.

discoverability↗
NASA NTRS1984

Industrial use of land observation satellite systems

The principal industrial users of land observation satellite systems are the geological industries; oil/gas, mining, and engineering/environmental companies. The primary system used is LANDSAT/MSS. Currently, use is also being made of the limited amounts of SKYLAB photography, SEASAT and SIR-A radar, and the new LANDSAT/TM data available. Although considered experimental, LANDSAT data is now used operationally by several hundred exploration and engineering companies worldwide as a vastly improved geological mapping tool to help direct more expensive geophysical and drilling phases, leading to more efficient decision-making and results. Future needs include global LANDSAT/TM; higher spatial resolution; stereo and radar; improved data handling, processing distribution and archiving systems, and integrated geographical information systems (GIS). For a promising future, governments must provide overall continuity (government and/or private sector) of such systems, insure continued government R and D, and commit to operating internationally under the civil Open Skies policy.

Henderson, F. B., III↗
NASA NTRS1988

MODIS information, data and control system (MIDACS) operations concepts

The MODIS Information, Data, and Control System (MIDACS) Operations Concepts Document provides a basis for the mutual understanding between the users and the designers of the MIDACS, including the requirements, operating environment, external interfaces, and development plan. In defining the concepts and scope of the system, how the MIDACS will operate as an element of the Earth Observing System (EOS) within the EosDIS environment is described. This version follows an earlier release of a preliminary draft version. The individual operations concepts for planning and scheduling, control and monitoring, data acquisition and processing, calibration and validation, data archive and distribution, and user access do not yet fully represent the requirements of the data system needed to achieve the scientific objectives of the MODIS instruments and science teams. The teams are not yet formed; however, it is possible to develop the operations concepts based on the present concept of EosDIS, the level 1 and level 2 Functional Requirements Documents, and through interviews and meetings with key members of the scientific community. The operations concepts were exercised through the application of representative scenarios.

Han, D.↗
NASA NTRS1992

Mission to Planet Earth

Mission to Planet Earth (MTPE) is NASA's concept for an international science program to produce the understanding needed to predict changes in the earth's environment. NASA and its interagency and international partners will place satellites carrying advanced sensors in strategic earth orbits to gather multidisciplinary data. A sophisticated data system will process and archive an unprecedented amount of information about the earth and how it works as a system. Increased understanding of the earth system is a basic human responsibility, a prerequisite to informed management of the planet's resources and to the preservation of the global environment.

Wilson, Gregory S.↗
NASA NTRS1990

Science operations for future space astrophysics missions

Plans for astrophysics science operations during the decade of the nineties are described from the point of view of a scientist who wishes to make a space-borne astronomical observation or to use archival astronomical data. In the process of preparing a proposal making an observation, and carrying out data processing, analysis, and dissemination of results, the scientist will be able to use a variety of services and infrastructure, including the 'Astrophysics Data System'. The current status and plans for these science operations services are described.

Riegler, Guenther H.↗
NASA NTRS1993

SpaceOps 1992: Proceedings of the Second International Symposium on Ground Data Systems for Space Mission Operations

The Second International Symposium featured 135 oral presentations in these 12 categories: Future Missions and Operations; System-Level Architectures; Mission-Specific Systems; Mission and Science Planning and Sequencing; Mission Control; Operations Automation and Emerging Technologies; Data Acquisition; Navigation; Operations Support Services; Engineering Data Analysis of Space Vehicle and Ground Systems; Telemetry Processing, Mission Data Management, and Data Archiving; and Operations Management. Topics focused on improvements in the productivity, effectiveness, efficiency, and quality of mission operations, ground systems, and data acquisition. Also emphasized were accomplishments in management of human factors; use of information systems to improve data retrieval, reporting, and archiving; design and implementation of logistics support for mission operations; and the use of telescience and teleoperations.

Source record↗