Search NASA⌕ Search

SEARCH · Search NASA

Results for “subsetting archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

TPSAS-NF1676L-17867-DND

In response to the exponential growth in science data analysis and visualization capabilities, data centers have been developing new processes to package and deliver large volumes of aggregated subsets of archived data. New standards are evolving to help data providers and application programmers manage the growing needs of the science community. These standards evolve from the best practices gleaned from new products and capabilities. The NASA Atmospheric Sciences Data Center (ASDC) has developed and deployed production provider-specific search and subset web applications for the CALIPSO, CERES, TES, and MOPITT missions. This presentation explores a CERES CCCM (CALIPSO, CloudSat, CERES, MODIS) data validation use case that leverages aggregated subset results from CERES CCCM (Level2), CERES SSF (Level2), and CALIPSO LIDAR (Level) datasets. Additionally, it examines the standards and formats that ASDC developers have applied to the delivered files as well as the implementation strategies for subsetting and processing the aggregated products.

Walter E Baskin↗
NASA NTRS1992

Remote Sensing Data from CLARET: A Prototype Cart Data Set

A data set containing radiation, meteorological, and cloud sensor observations is documented. It was prepared for use by the Department of Energy's Atmospheric Radiation Measurement (ARM) program and other interested scientists. These data are a precursor of the types of data that ARM Cloud And Radiation Testbed (CART) sites will provide. The data are from the Cloud Lidar And Radar Exploratory Test (CLARET) conducted by the Wave Propagation Laboratory during autumn 1989 in the Denver-Boulder area of Colorado primarily for the purpose of developing new cloud-sensing techniques on cirrus. After becoming aware of this experiment, ARM scientists requested archival of subsets or the data to assist in the developing ARM program. Five CLARET cases were selected: two with cirrus, one with stratus, one with mixed-phase clouds, and one with clear skies. The cases range from 2 to 9.5 h in length. A pyranometer, pyrgeometer, pyrheliometer, and an infrared radiometer constituted the ensemble of instruments that provided surface radiation data. A lidar, radar, and ceilometer observed the cloud geometrical structure, and visual reports and all-sky camera observations were assimilated to provide cloud cover data. Radiosondes, wind profiler, RASS (profiling virtual temperature), microwave radiometers (observing column integrated liquid water and water vapor), and standard surface measurements provided meteorological data. Satellite data from the stratus case and one cirrus case were analyzed for statistics on cloud cover and top height. The main body of the selected data are available on diskette from the Wave Propagation Laboratory or Los Alamos National Laboratory. In addition to documenting the data set, this report describes CLARET and gives a bibliography of publications associated with the project. Some preliminary results of CLARET' research are also summarized. Simultaneous CO 2 lidar and radar backscatter measurements were shown to provide estimates of the effective radius of ice particles. Simultaneous radar and infrared radiometer data appear useful for estimating column-integrated numbers and average sizes of ice cloud particles. Ice water content obtained with this method compared favorably with values from another empirical technique using radar data alone. Depolarization of the CO 2 lidar signal from ice clouds was surprisingly small, suggesting that calculation of backscatter from nonspherical particles for this lidar is a tractable problem. Examples are also cited of CO 2 lidar measurements of the effective radius of water cloud drop size distributions and of inference of the size of pristine ice crystals that assume a particular orientation in the air. These parameters are all important to radiative transfer through clouds.

Clouds (Meteorology)↗
NASA NTRS2012

Globally Gridded Satellite (GridSat) Observations for Climate Studies

Geostationary satellites have provided routine, high temporal resolution Earth observations since the 1970s. Despite the long period of record, use of these data in climate studies has been limited for numerous reasons, among them: there is no central archive of geostationary data for all international satellites, full temporal and spatial resolution data are voluminous, and diverse calibration and navigation formats encumber the uniform processing needed for multi-satellite climate studies. The International Satellite Cloud Climatology Project set the stage for overcoming these issues by archiving a subset of the full resolution geostationary data at approx.10 km resolution at 3 hourly intervals since 1983. Recent efforts at NOAA s National Climatic Data Center to provide convenient access to these data include remapping the data to a standard map projection, recalibrating the data to optimize temporal homogeneity, extending the record of observations back to 1980, and reformatting the data for broad public distribution. The Gridded Satellite (GridSat) dataset includes observations from the visible, infrared window, and infrared water vapor channels. Data are stored in the netCDF format using standards that permit a wide variety of tools and libraries to quickly and easily process the data. A novel data layering approach, together with appropriate satellite and file metadata, allows users to access GridSat data at varying levels of complexity based on their needs. The result is a climate data record already in use by the meteorological community. Examples include reanalysis of tropical cyclones, studies of global precipitation, and detection and tracking of the intertropical convergence zone.

Knapp, Kenneth R.↗
NASA NTRS1988

Orbiter Flying Qualities (OFQ) Workstation user's guide

This project was devoted to the development of a software package, called the Orbiter Flying Qualities (OFQ) Workstation, for working with the OFQ Archives which are specially selected sets of space shuttle entry flight data relevant to flight control and flying qualities. The basic approach to creation of the workstation software was to federate and extend commercial software products to create a low cost package that operates on personal computers. Provision was made to link the workstation to large computers, but the OFQ Archive files were also converted to personal computer diskettes and can be stored on workstation hard disk drives. The primary element of the workstation developed in the project is the Interactive Data Handler (IDH) which allows the user to select data subsets from the archives and pass them to specialized analysis programs. The IDH was developed as an application in a relational database management system product. The specialized analysis programs linked to the workstation include a spreadsheet program, FREDA for spectral analysis, MFP for frequency domain system identification, and NIPIP for pilot-vehicle system parameter identification. The workstation also includes capability for ensemble analysis over groups of missions.

Myers, Thomas T.↗
NASA NTRSDate not supplied

Hydrological Data at the NASA GES DISC: Current Capabilities and New Opportunities

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of twelve NASA Earth science data centers that document, process, archive and distribute data from Earth observation missions and projects. GES DISC maintains an archive of several hydrology datasets, including the Land Data Assimilation Systems (LDAS) and the Gravity Recovery and Climate Experiment (GRACE) Data Assimilation for Drought Monitoring (GRACE-DA-DM) data products. These datasets include model output of heat fluxes, rain, snow, soil temperature, soil moisture, and runoff; and observational forcing data, including surface pressure, temperature, precipitation, downward shortwave and longwave radiation, humidity, and wind. The temporal resolution of the hydrology data at GES DISC ranges from hourly to monthly, and spatial resolutions range from 0.1° to 1.0°. The GES DISC provides services which enable users to aggregate, temporally and spatially subset, regrid, and visualize archived data including the GES DISC Subsetter, Hydrology Data Rods, and the Geospatial Interactive Online Visualization and Analysis Infrastructure (GIOVANNI). The Hydrology Data Rods service optimally reorganizes large hydrological data sets as extended time series, providing more efficient access for the hydrological community. The time series data (aka “data rods”) were integrated into hydrology community tools, such as the Data Rods Explorer on HydroShare. Furthermore, the GES DISC is in the process of migrating its data and services to the cloud. Hydrological data available at the GES DISC are now available in the Amazon Web Services (AWS) cloud (us-west-2 region) providing users Direct S3 data access and the capability for cloud computing operations. In this presentation, the hydrology data products and services currently available at the GES DISC will be summarized. Also discussed are the migration to the cloud, user support through this transition, and the status of migrating the data rods service to the cloud.

Ashley Heath↗
NASA NTRS1992

The EXOSAT database and archive

The EXOSAT database provides on-line access to the results and data products (spectra, images, and lightcurves) from the EXOSAT mission as well as access to data and logs from a number of other missions (such as EINSTEIN, COS-B, ROSAT, and IRAS). In addition, a number of familiar optical, infrared, and x ray catalogs, including the Hubble Space Telescope (HST) guide star catalog are available. The complete database is located at the EXOSAT observatory at ESTEC in the Netherlands and is accessible remotely via a captive account. The database management system was specifically developed to efficiently access the database and to allow the user to perform statistical studies on large samples of astronomical objects as well as to retrieve scientific and bibliographic information on single sources. The system was designed to be mission independent and includes timing, image processing, and spectral analysis packages as well as software to allow the easy transfer of analysis results and products to the user's own institute. The archive at ESTEC comprises a subset of the EXOSAT observations, stored on magnetic tape. Observations of particular interest were copied in compressed format to an optical jukebox, allowing users to retrieve and analyze selected raw data entirely from their terminals. Such analysis may be necessary if the user's needs are not accommodated by the products contained in the database (in terms of time resolution, spectral range, and the finesse of the background subtraction, for instance). Long-term archiving of the full final observation data is taking place at ESRIN in Italy as part of the ESIS program, again using optical media, and ESRIN have now assumed responsibility for distributing the data to the community. Tests showed that raw observational data (typically several tens of megabytes for a single target) can be transferred via the existing networks in reasonable time.

Reynolds, A. P.↗
NASA NTRS2004

Genesis Sample Material Subdividing Plans

Subdivision of Genesis collectors is necessary to maximize the science return of the mission and retain a representative subset of the collection to archive for the future. The ability to verify important scientific results and resolve experiment anomalies through experiment replication and independent experimental techniques requires a method for subdividing individual sample collectors so that equivalent samples are made available. In addition, subdividing the Genesis collectors is required to accommodate the size limitations of many analytical systems. Samples allocated for analysis are likely to be small sections of individual collectors. Subdivision of returned Genesis solar wind samples must be performed in a manner having minimum impact on the accuracy and interpretation of analytical information obtained from these samples. It is also necessary that we ensure that consequences of any processing are well understood and documented. half of the collectors are actually thin coatings on silicon and sapphire substrates. Most of these are merely physically deposited coatings with limited adhesion and hardness, complicating the handling of these collectors. Finally, the concentrator targets, which are to be evaluated to address the highest priority goals of the mission, represent all of these difficulties and require cutting to greater precision as a result of the extremely limited quantities available.

McNamara, K. M.↗
NASA NTRS2004

Digital Archive Issues from the Perspective of an Earth Science Data Producer

Contents include the following: Introduction. A Producer Perspective on Earth Science Data. Data Producers as Members of a Scientific Community. Some Unique Characteristics of Scientific Data. Spatial and Temporal Sampling for Earth (or Space) Science Data. The Influence of the Data Production System Architecture. The Spatial and Temporal Structures Underlying Earth Science Data. Earth Science Data File (or Relation) Schemas. Data Producer Configuration Management Complexities. The Topology of Earth Science Data Inventories. Some Thoughts on the User Perspective. Science Data User Communities. Spatial and Temporal Structure Needs of Different Users. User Spatial Objects. Data Search Services. Inventory Search. Parameter (Keyword) Search. Metadata Searches. Documentation Search. Secondary Index Search. Print Technology and Hypertext. Inter-Data Collection Configuration Management Issues. An Archive View. Producer Data Ingest and Production. User Data Searching and Distribution. Subsetting and Supersetting. Semantic Requirements for Data Interchange. Tentative Conclusions. An Object Oriented View of Archive Information Evolution. Scientific Data Archival Issues. A Perspective on the Future of Digital Archives for Scientific Data. References Index for this paper.

Barkstrom, Bruce R.↗
NASA NTRS2001

Ultraviolet Signatures of Stars and Gas in Starburst Galaxies

We extracted about 500 short wavelength prime (SWP) (high-dispersion) International Ultraviolet Explorer (IUE) spectra of B0 to B8 stars of all luminosity classes for the purpose of creating a stellar library. Several hundred scientifically useful high-dispersion B-star spectra are in the IUE Archive. The data were taken either with the purpose of performing stellar-wind studies or investigations of interstellar absorption lines. Due to the nature of these projects, the S/N of the data is the maximum achievable with the IUE satellite. Rountree and Sonneborn and Walborn, Parker, and Nichols extracted a subset of B spectral classification standards from the IUE Archive to construct atlases of IUE high-dispersion spectra. Their work served as our guideline for the data extraction.

Leitherer, Claus↗
NASA NTRSDate not supplied

NASA’s Atmospheric Science Data Center’s Approach to a Cloud-Based Model of Ingest, Archival, and Distribution of TEMPO Data: Methods, Challenges, and Best Practices

The National Aeronautics and Space Administration's (NASA) Atmospheric Science Data Center (ASDC) at NASA Langley Research Center in Hampton, VA provides atmospheric science data products and services to the science community, including enhanced search and subsetting capabilities for numerous datasets. The ASDC is the official Distributed Active Archive Center (DAAC) of record for the upcoming Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. TEMPO will be situated on a geostationary satellite positioned at a longitude near the center of the conterminous United States and focused on North America, making hourly swaths of its field of regard from east to west. ASDC’s data products are currently hosted locally and services (e.g., spatial and temporal subsetting) are managed on premises. The ASDC is planning to provide TEMPO data and services in the cloud through the Earthdata Search platform. This presentation will discuss the ASDC’s approach to a cloud-based model of ingest, archival, and distribution of TEMPO data. Methods, challenges, best practices, lessons learned, and future plans will be discussed.

Iman Nasif↗
NASA NTRS1986

Archive data base and handling system for the Orbiter flying qualities experiment program

The OFQ archives data base and handling system assembled as part of the Orbiter Flying Qualities (OFQ) research of the Orbiter Experiments Program (EOX) are described. The purpose of the OFQ archives is to preserve and document shuttle flight data relevant to vehicle dynamics, flight control, and flying qualities in a form that permits maximum use for qualified users. In their complete form, the OFQ archives contain descriptive text (general information about the flight, signal descriptions and units) as well as numerical time history data. Since the shuttle program is so complex, the official data base contains thousands of signals and very complex entries are required to obtain data. The OFQ archives are intended to provide flight phase oriented data subsets with relevant signals which are easily identified for flying qualities research.

Myers, T. T.↗
NASA NTRS1987

Satellite data management for effective data access

The management of data generated from satellite missions has not always led to effective access of that data by the scientific community. NASA has tried to alleviate this problem for ocean scientists, by initiating a program, the NASA Ocean Data System (NODS). The menu-based user interface that NODS employs allows a user to make request and receive answers within a short time of accessing the system. A catalog system, which holds information about oceanographic data sets may be queried to determine the suitability of a particular data set. Once a candidate data set is found, the user is directed to the person or place which actually holds the data. NODS also has an archive system that holds data from ocean-observing satellites. The archive may be queried to obtain a manageable data subset that can be delivered in a useful form.

Hogan, Patrick D.↗
NASA NTRS1993

TOPEX/POSEIDON GPS Demonstration Experiment Data

Global Positioning Satellite (GPS) measurements obtained from the on-board TOPEX/POSEIDON demonstration flight receiver and a global network of ground receivers are available from the GPS Data Handling Facility (GDHF) archive. Ground receiver observations are derived from a six station subset of the GPS global tracking network. The six TOPEX ground stations yielded 95 percent of the expected data during the six month demonstration phase from November, 1992 to April, 1993. For the same period, 98 percent of the available P-code tracking from the TOPEX flight receiver was obtained. Raw carrier phase and P-code pseudorange measurements are provided for the ground and flight receivers along with an additional file of calibrated and compressed flight receiver data...

Guinn, J. R.↗
NASA NTRS2013

Intragroup and Galaxy-linked Diffuse X-ray Emission In Hickson Compact Groups

Isolated compact groups (CGs) of galaxies present a range of dynamical states, group velocity dispersions, and galaxy morphologies with which to study galaxy evolution, particularly the properties of gas both within the galaxies and in the intragroup medium. As part of a large, multiwavelength examination of CGs, we present an archival study of diffuse X-ray emission in a subset of nine Hickson compact groups (HCGs) observed with the Chandra X-Ray Observatory. We find that seven of the groups in our sample exhibit detectable diffuse emission. However, unlike large-scale emission in galaxy clusters, the diffuse features in the majority of the detected groups are linked to the individual galaxies, in the form of both plumes and halos likely as a result of vigourous star formation or activity in the galaxy nucleus, as well as in emission from tidal features. Unlike previous studies from earlier X-ray missions, HCGs 31, 42, 59, and 92 are found to be consistent with the L(sub X-Tau) relationship from clusters within the errors, while HCGs 16 and 31 are consistent with the cluster L(sub X-sigma) relation, though this is likely coincidental given that the hot gas in these two systems is largely due to star formation. We find that L(sub X) increases with decreasing group Hi to dynamical-mass ratio with tentative evidence for a dependence in X-ray luminosity on Hi morphology whereby systems with intragroup Hi indicative of strong interactions are considerably more X-ray luminous than passively evolving groups. We also find a gap in the L(sub X) of groups as a function of the total group specific star formation rate. Our findings suggest that the hot gas in these groups is not in hydrostatic equilibrium and these systems are not low-mass analogs of rich groups or clusters, with the possible exception of HCG 62.

Desjardins, Tyler D.↗
NASA NTRS2017

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗
NASA NTRS2017

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly. Reviewed by Mark McInerney ESDIS Deputy Project Manager.

Lynnes, Christopher↗
NASA NTRS1994

Use of data description languages in the interchange of data

The Consultative Committee for Space Data Systems (CCSDS) is developing Standards for the interchange of information between systems, including those operating under different environments. The objective is to perform the interchange automatically, i.e. in a computer interpretable manner. One aspect of the concept developed by CCSDS is the use of a separate data description to specify the data being transferred. Using the description, data can then be automatically parsed by the receiving computer. With a suitably expressive Data Description Language (DDL), data formats of arbitrary complexity can be handled. The advantages of this approach are: (1) that the description need only be written and distributed once to all users, and (2) new software does not need to be written for each new format, provided generic tools are available to support writing and interpretation of descriptions and the associated data instances. Consequently, the effort of 'hard coding' each new format is avoided and problems of integrating multiple implementations of a given format by different users are avoided. The approach is applicable in any context where computer parsable description of data could enhance efficiency (e.g. within a spacecraft control system, a data delivery system or an archive). The CCSDS have identified several candidate DDL's: EAST (Extended Ada Subset), TSDN (Transfer Syntax Data Notation) and MADEL (Modified ASN.1 as a Data Description Language -- a DDL based on the Abstract Syntax Notation One - ASN.1 - specified in the ISO/IEC 8824). This paper concentrates on ESA's development of MADEL. ESA have also developed a 'proof of concept' prototype of the required support tools, implemented on a PC under MS-DOS, which has successfully demonstrated the feasibility of the approach, including the capability within an application of retrieving and displaying particular data elements, given its MADEL description (i.e. a data description written in MADEL). This paper outlines the work done to date and assesses the applicability of this modified ASN.1 as a DDL. The feasibility of the approach is illustrated with several examples.

Pignede, M.↗