Search NASA⌕ Search

SEARCH · Search NASA

Results for “data processing and archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Microphone Phased Array NetCDF/HDF5 Archival Files: Application Program Interface Reference

An application program interface (API) has been developed for the creation and access of structured data files generated by microphone phased arrays utilized in aeroacoustics research. Two structured binary file formats are supported, namely NetCDF (Network Common Data Form) and HDF5 (Hierarchical Data Format) files. The API consists of a library of routines callable from C, Fortran or Matlab, with native versions of the API provided for each language. The libraries are divided into categories for file handling, file definition and initialization, data writing, data recovery, and error handling. The API is intended to provide a mechanism for generating self-describing binary files for long-term archiving of raw and processed data generated by phased array systems.

Humphreys, William M., Jr.↗
NASA NTRS2014

NASA Plan for Increasing Access to the Results of Scientific Research

This plan is issued in response to the Executive Office of the President's February 22, 2013, Office of Science and Technology Policy (OSTP) Memorandum for the Heads of Executive Departments and Agencies, "Increasing Access to the Results of Federally Funded Scientific Research." Through this memorandum, OSTP directed all agencies with more than $100 million in annual research and development expenditures to prepare a plan for improving the public's access to the results of federally funded research. The National Aeronautics and Space Administration (NASA) invests on the order of $3 billion annually in fundamental and applied research and technology development1 across a broad range of topics, including space and Earth sciences, life and physical sciences, human health, aeronautics, and technology. Promoting the full and open sharing of data with research communities, private industry, academia, and the general public is one of NASA's longstanding core values. For example, NASA's space and suborbital mission personnel routinely process, archive, and distribute their data to researchers around the globe. This plan expands the breadth of NASA's open-access culture to include data and publications for all of the scientific research that the Agency sponsors.

Source record↗
NASA NTRS1996

Durable High-Density Data Storage

The focus ion beam (FIB) micromilling process for data storage provides a new non-magnetic storage method for archiving large amounts of data. The process stores data on robust materials such as steel, silicon, and gold coated silicon. The storage process was developed to provide a method to insure the long term storage life of data. We estimate that the useful life of data written on silicon or gold-coated silicon to be on the order of a few thousand years without the need to rewrite the data every few years. The process uses an ion beam to carve material from the surface, much like stone cutters in ancient civilizations removed material from stone. The deeper the information is carved into the media, the longer the expected life of the information. The process can record information in three formats: (1) binary at densities of 23 Gbits/square inch, (2) alphanumeric at optical or non-optical density, and (3) graphical at optical and non-optical density. The formats can be mixed on the same media; and thus, it is possible to record, in a human-viewable format, instructions that can be read using an optical microscope. These instructions provide guidance on reading the remaining higher density information.

Lamartine, Bruce C.↗
NASA NTRSDate not supplied

Use NASA Earthdata in the Cloud

With the impending arrival of new, high-data-volume Earth observation missions, NASA’s ability to effectively ingest, process, and archive large amounts of data requires the most cost-effective, flexible, and scalable data-management architectures and technologies. To meet these demands, NASA's Earth Science Data Systems (ESDS) Program is implementing a strategic vision to develop and operate multiple components of NASA's Earth Observing System Data and Information System (EOSDIS) in a commercial cloud environment. As more earth observing data is moved into the cloud, NASA EOSDIS wants to share our reasoning for moving data into the commercial cloud, as well as demonstrate how end-users can take advantage of both data in the cloud and cloud-deployed services. This presentation explains the enterprise reasons for moving data into the cloud, specifically the scalability of cloud systems, the flexibility to pivot to the most optimal language/system/architecture, and the reduced redundancy of a common architecture. Then we primarily focus on the user-driven reasons for moving to the cloud. Equal access to high performance computing helps all parties, especially users that don't have access to their own on-premise computing infrastructure. Rapid deployment allows users to spin up their own systems without needing the extensive platform previously required for large scale development. Cost effectiveness allows users a wide breadth of computing options, from super computer ability to small scale development, which can help underserved communities. Finally we link to internal resources that users can utilize to begin their cloud based development journey without having to be an expert in the field.

Nicholas Doty↗
NASA NTRS2006

A-Train Data Depot: Integrating, Visualizing, and Extracting Cloudsat, CALIPSO, MODIS, and AIRS Atmospheric Measurements Along the A-Train Tracks

The succession of US and international Earth observing satellites that follow each other, seconds to minutes apart, across the local afternoon equator crossing is called the ATrain. The A-Train consists of the following satellites, in order of equator crossing: OCO, EOS Aqua, CloudSat, CALIPSO, PARASOL, and EOS Aura. Flying in such formation increases the number of observations, validates observations, and enables coordination between science observations, resulting in a more complete virtual science platform (Kelly, 2000). The goal of this project is to create the first ever A-Train virtual data portal/center, the A-Train Data Depot (ATDD), to process, archive, access, visualize, analyze and correlate distributed atmosphere measurements from various A-Train instruments along A-Train tracks. The ATDD will enable the free movement of remotely located A-Train data so that they are combined to create a consolidated vertical view of the Earth's Atmosphere along the A-Train tracks. Once the infrastructure of the ATDD is in place, it will be easily evolved to serve data from all A-Train data measurements: one stop shopping. The innovative approach of analyzing and visualizing atmospheric profiles along the platforms track (i.e., time) will be accommodated by reusing the GSFC Atmospheric Composition Data and Information Services Center (ACDISC) visualization and analysis tool, GIOVANNI, existing data reduction tools, on-line archiving for fast data access, access to remote data without unnecessary data transfers, and data retrieval by users finding data desirable for further study. Initial measurements utilized include CALIPSO lidar backscatter, CloudSat radar reflectivity, clear air relative humidity, water vapor and temperature from AIRS, and cloud properties and aerosols from both MODIS. This will be foilowed by associated measurements from TVILS, =MI, HIRDLS, sad TES. Given the independent nature of instrumentlplatform development, the ATDD project has been met with many interesting challenges that, once resolved, will provide a much greater understanding of the relative flight dynamics and data co-registration of the suite of A-Train instruments, thus greatly increasing the accuracy of A-Train data analysis. Some of these challenges will be illustrated and discussed. The project's early visualizations and analysis efforts illustrate the importance of managing data so that measurements from various missions can be combined to enhance the understanding of the atmosphere. A-Train data management coordination, as performed here, is extremely significant in facilitating the A-Train science of clouds, precipitation, aerosol and chemistry.

Kempler, Steven↗
NASA NTRS2000

BOREAS Level-0 ER-2 Navigation Data

The BOREAS Staff Science effort covered those activities that were BOREAS community-level activities or required uniform data collection procedures across sites and time. These activities included the acquisition, processing, and archiving of aircraft navigation/attitude data to complement the digital image data. The level-0 ER-2 navigation data files contain aircraft attitude and position information acquired during the digital image and photographic data collection missions. Temporally, the data were acquired from April to September 1994. Data were recorded at intervals of 5 seconds. The data are stored in tabular ASCII files.

Strub, Richard↗
NASA NTRS2016

NASA SNPP SIPS - Following in the Path of EOS

NASA's Earth Science Data Information System (ESDIS) Project has been operating NASA's Suomi National Polar-Orbiting Partnership (SNPP) Science Data Segment (SDS) since the launch in October 2011. At launch, the SDS focused primarily on the evaluation of Sensor Data Records (SDRs) and Environmental Data Records (EDRs) produced by the Joint Polar Satellite System (JPSS), a National Oceanic and Atmosphere Administration (NOAA) Program, as to their suitability for Earth system science. During the summer of 2014, NASA transitioned to the production of standard Earth Observing System (EOS)-like science products for all instruments aboard Suomi NPP. The five Science Investigator-led Processing Systems (SIPS): Land, Ocean, Atmosphere, Ozone, and Sounder were established to produce the NASA SNPP standard Level 1, Level 2, and global Level 3 products developed by the SNPP Science Teams and to provide the products to NASA's Distributed Active Archive Centers (DAACs) for archive and distribution to the user community. The processing, archiving and distribution of data from NASA's Clouds and the Earth's Radiant Energy System (CERES) and Ozone Mapper/Profiler Suite (OMPS) Limb instruments will continue. With the implementation of the JPSS Block 2 architecture and the launch of JPSS-1, the SDS will receive SNPP data in near real-time via the JPSS Stored Mission Data Hub (JSH), as well as JPSS-1 and future JPSS-2 data. The SNPP SIPS will ingest EOS compatible Level 0 data from the EOS Data Operations System (EDOS) element for their data processing, enabling the continuous EOS-SNPP-JPSS Satellite Data Record.

near real time↗
NASA NTRS1985

User data management

The primary objective is to identify, develop, and demonstrate key data management technologies to support user access to Space Station data. To accomplish this objective, there are several technical challenges which must be addressed. The first is how to provide routine customer access to high volume, dynamic and distributed data bases. This access will encompass the functions of mission and payload planning and operations, data processing and analysis, and data archive and distribution. Secondly, there must be some analysis of architectures for handling high volume data streams like those expected from the Space Station. This analysis will examine the use of packetized versus non-packetized data formats, modular expansion capabilities, real-time versus non-real-time data processing, and the interfaces and architecture required to support telescience operations. The task will also determine benchmarks of performance capabilities for technology operations, such as varied data base structures, data access procedures, distributed data base design, and data base machines. Information is provided here in outline form.

Source record↗
NASA NTRS2011

Forty-Year Calibrated Record of Earth-Surface Reflected Radiance from Landsat: A Review

Sensors on Landsat satellites have been collecting images of the Earth's surface for nearly 40 years. These images have been invaluable for characterizing and detecting changes in the land cover and land use of the world. Although initially conceived as primarily picture generating sensors, even the early sensors were radiometrically calibrated and spectrally characterized prior to launch and incorporated some capabilities to monitor their radiometric calibration once on orbit. Recently, as the focus of studies has shifted to monitoring Earth surface parameters over significant periods of time, serious attention has been focused toward bringing the data from all these sensors onto a common radiometric scale over this 40-year period. This effort started with the most recent systems and then was extended back in time. Landsat-7 ETM+, the best-characterized sensor of the series prior to launch and once on orbit, and the most stable system to date, was chosen to serve as the reference. The Landsat-7 project was the first of the series to build an image assessment system into its ground system, allowing systematic characterization of its sensors and data. Second, the Landsat-5 TM (still operating at the time of the Landsat-7 launch and continues to operate) calibration history was reconstructed based on its internal calibrator, vicarious calibrations, pseudo-invariant sites and a tie to Landsat-7 ETM+ at the time of the commissioning of Landsat-7. This process was performed in two iterations: the earlier one relied primarily on the TM internal calibrator. When this was found to have some deficiencies, a revised calibration was based more on pseudo-invariant sites, though the internal calibrator was still used to establish the short-term variations in response due to icing build up on the cold focal plane. As time progressed, a capability to monitor the Landsat-5 TM was added to the image assessment system. The Landsat-4 TM, which operated from 1982-1992, was the third system to which the radiometric scale was extended. The limited and broken use of the Landsat-4 TM made this analysis more difficult. Eight-day separated image pairs from Landsat-5 combined with analysis of pseudo invariant sites established this history. The fourth and most challenging effort was making the Landsat-1 to -5 MSS sensors' data internally radiometrically consistent. This effort was particularly complicated by the age of the MSS data, varying formats and processing levels in the archive, limited datasets, and limited documentation available. Ultimately, pseudo-invariant sites were identified in North America and used for this effort. Note that most of the Landsat-MSS archived data had already been calibrated using the MSS internal calibrators, so this processing was imbedded in the result. The final effort was developing an absolute scale for Landsat MSS similar to what was already established for the "TM" sensors. This was accomplished by using simultaneous data from Landsat-5 MSS and Landsat-5 TM, accounting for spectral differences between the sensors using EO-1 Hyperion data. The recalibrated history of the Landsat data and implications to users are discussed. The key result from this work is a consistently calibrated Landsat data archive that spans nearly 40 years with total uncertainties on the order of 10% or less for most sensors and bands.

Markham, Brian↗
NASA NTRS2011

Determining the Completeness of the Nimbus Meteorological Data Archive

NASA launched the Nimbus series of meteorological satellites in the 1960s and 70s. These satellites carried instruments for making observations of the Earth in the visible, infrared, ultraviolet, and microwave wavelengths. The original data archive consisted of a combination of digital data written to 7-track computer tapes and on various film media. Many of these data sets are now being migrated from the old media to the GES DISC modern online archive. The process involves recovering the digital data files from tape as well as scanning images of the data from film strips. Some of the challenges of archiving the Nimbus data include the lack of any metadata from these old data sets. Metadata standards and self-describing data files did not exist at that time, and files were written on now obsolete hardware systems and outdated file formats. This requires creating metadata by reading the contents of the old data files. Some digital data files were corrupted over time, or were possibly improperly copied at the time of creation. Thus there are data gaps in the collections. The film strips were stored in boxes and are now being scanned as JPEG-2000 images. The only information describing these images is what was written on them when they were originally created, and sometimes this information is incomplete or missing. We have the ability to cross-reference the scanned images against the digital data files to determine which of these best represents the data set from the various missions, or to see how complete the data sets are. In this presentation we compared data files and scanned images from the Nimbus-2 High-Resolution Infrared Radiometer (HRIR) for September 1966 to determine whether the data and images are properly archived with correct metadata.

Johnson, James↗
NASA NTRS1992

Mission to Planet Earth

Mission to Planet Earth (MTPE) is NASA's concept for an international science program to produce the understanding needed to predict changes in the Earth's environment. NASA and its interagency and international partners will place satellites carrying advanced sensors in strategic Earth orbits to gather multidisciplinary data. A sophisticated data system will process and archive an unprecedented amount of information about the Earth and how it works as a system. Increased understanding of the Earth system is a basic human responsibility, a prerequisite to informed management of the planet's resources and to the preservation of the global environment. An overview of the MTPE, flight programs, data and information systems, interdisciplinary research efforts, and international coordination, is presented.

Wilson, Gregory S.↗
NASA NTRS1995

A visiting scientist program for the burst and transient source experiment

During this project, Universities Space Research Association provided program management and the administration for overseeing the performance of the total contractual effort. The program director and administrative staff provided the expertise and experience needed to efficiently manage the program.USRA provided a program coordinator and v visiting scientists to perform scientific research with Burst and Transient Source Experiment (BATSE) data. This research was associated with the primary scientific objectives of BATSE and with the various BATSE collaborations which were formed in response to the Compton Gamma Ray Observatory Guest Investigator Program. USRA provided administration for workshops, colloquia, the preparation of scientific documentation, etc. and also provided flexible program support in order to meet the on-going needs of MSFC's BATSE program. USRA performed tasks associated with the recovery, archiving, and processing of scientific data from BATSE. A bibliography of research in the astrophysics discipline is attached as Appendix 1. Visiting Scientists and Research Associates performed activities on this project, and their technical reports are attached as Appendix 2.

Kerr, Frank J.↗
NASA NTRS2016

NASA's Earth Science Flight Program Meets the Challenges of Today and Tomorrow

NASA's Earth science flight program is a dynamic undertaking that consists of a large fleet of operating satellites, an array of satellite and instrument projects in various stages of development, a robust airborne science program, and a massive data archiving and distribution system. Each element of the flight program is complex and present unique challenges. NASA builds upon its successes and learns from its setbacks to manage this evolving portfolio to meet NASA's Earth science objectives. NASA fleet of 16 operating missions provide a wide range of scientific measurements made from dedicated Earth science satellites and from instruments mounted to the International Space Station. For operational missions, the program must address issues such as an aging satellites operating well beyond their prime mission, constellation flying, and collision avoidance with other spacecraft and orbital debris. Projects in development are divided into two broad categories: systematic missions and pathfinders. The Earth Systematic Missions (ESM) include a broad range of multi-disciplinary Earth-observing research satellite missions aimed at understanding the Earth system and its response to natural and human-induced forces and changes. Understanding these forces will help determine how to predict future changes, and how to mitigate or adapt to these changes. The Earth System Science Pathfinder (ESSP) program provides frequent, regular, competitively selected Earth science research opportunities that accommodate new and emerging scientific priorities and measurement capabilities. This results in a series of relatively low-cost, small-sized investigations and missions. Principal investigators whose scientific objectives support a variety of studies lead these missions, including studies of the atmosphere, oceans, land surface, polar ice regions, or solid Earth. This portfolio of missions and investigations provides opportunity for investment in innovative Earth science that enhances NASA's capability for better understanding the current state of the Earth system. ESM and ESSP projects often involve partnerships with other US agencies and/or international organizations. This adds to the complexity of mission development, but allows for a greater scientific return on NASA's investments. The Earth Science Airborne Science Program provides manned and unmanned aircraft systems that further science and advance the use of satellite data. NASA uses these assets worldwide in campaigns to investigate extreme weather events, observe Earth system processes, obtain data for Earth science modeling activities, and calibrate instruments flying aboard Earth science spacecraft. The Airborne Science Program has six dedicated aircraft and access to many other platforms. The Earth Science Multi-Mission Operations program acquires, preserves, and distributes observational data from operating spacecraft to support Earth Science research focus areas. The Earth Observing System Data and Information System (EOSDIS), which has been in operations since 1994, primarily accomplishes this. EOSDIS acquires, processes, archives, and distributes Earth Science data and information products. The archiving of NASA Earth Science information happens at eight Distributed Active Archive Centers (DAACs) and four disciplinary data centers located across the United States. The DAACs specialize by topic area, and make their data available to researchers around the world. The DAACs currently house over 9 petabytes of data, growing at a rate of 6.4 terabytes per day. NASA's current Earth Science portfolio is responsive to the National Research Council (NRC) 2007 Earth Science Decadal Survey and well as the 2010 NASA Response to President Obama's Climate Plan. As the program evolves into the future it will leverage the lessons learned from the current missions in operations and development, and plan for adjustments to future objectives in response to the anticipated 2017 NRC Decadal Survey.

Flight Program↗
NASA NTRS2017

Restoration of the Apollo 15 Heat Flow Experiment Data from 1975 to 1977

The Apollo 15 Heat Flow Experiment (HFE) was conducted from July 1971 through January 1977. Two heat flow probes were deployed roughly 8.5 meters apart. Probe 1 and Probe 2 penetrated to 1.4-meters and 1-meter depths into the lunar regolith, respectively. Temperatures at different depths and the surface were logged with 7.25-minute intervals and transmitted to Earth. At the conclusion of the experiment, only data obtained from July 1971 through December 1974 were processed and archived at the National Space Science Data Center (NSSDC) by the principal investigator of the experiment, Marcus Langseth of Columbia University. Langseth died in 1997. It is not known what happened to the HFE data tapes he used. Current researchers have strong interests in re-examining the HFE data for the full duration of the experiment. We have recovered and processed large portions of the Apollo 15 HFE data from 1975 through 1977 by assembling data and metadata from various sources.

HFE↗
NASA NTRS1993

ACE: A distributed system to manage large data archives

Competitive pressures in the oil and gas industry are requiring a much tighter integration of technical data into E and P business processes. The development of new systems to accommodate this business need must comprehend the significant numbers of large, complex data objects which the industry generates. The life cycle of the data objects is a four phase progression from data acquisition, to data processing, through data interpretation, and ending finally with data archival. In order to implement a cost effect system which provides an efficient conversion from data to information and allows effective use of this information, an organization must consider the technical data management requirements in all four phases. A set of technical issues which may differ in each phase must be addressed to insure an overall successful development strategy. The technical issues include standardized data formats and media for data acquisition, data management during processing, plus networks, applications software, and GUI's for interpretation of the processed data. Mass storage hardware and software is required to provide cost effective storage and retrieval during the latter three stages as well as long term archival. Mobil Oil Corporation's Exploration and Producing Technical Center (MEPTEC) has addressed the technical and cost issues of designing, building, and implementing an Advanced Computing Environment (ACE) to support the petroleum E and P function, which is critical to the corporation's continued success. Mobile views ACE as a cost effective solution which can give Mobile a competitive edge as well as a viable technical solution.

Daily, Mike I.↗
NASA NTRS2017

GES DISC Data Recipes in Jupyter Notebooks

The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.

discoverability↗
NASA NTRS1984

Industrial use of land observation satellite systems

The principal industrial users of land observation satellite systems are the geological industries; oil/gas, mining, and engineering/environmental companies. The primary system used is LANDSAT/MSS. Currently, use is also being made of the limited amounts of SKYLAB photography, SEASAT and SIR-A radar, and the new LANDSAT/TM data available. Although considered experimental, LANDSAT data is now used operationally by several hundred exploration and engineering companies worldwide as a vastly improved geological mapping tool to help direct more expensive geophysical and drilling phases, leading to more efficient decision-making and results. Future needs include global LANDSAT/TM; higher spatial resolution; stereo and radar; improved data handling, processing distribution and archiving systems, and integrated geographical information systems (GIS). For a promising future, governments must provide overall continuity (government and/or private sector) of such systems, insure continued government R and D, and commit to operating internationally under the civil Open Skies policy.

Henderson, F. B., III↗