Search NASA⌕ Search

SEARCH · Search NASA

Results for “data archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Data compression for data archival, browse or quick-look

Soon after space and Earth science data is collected, it is stored in one or more archival facilities for later retrieval and analysis. Since the purpose of the archival process is to keep an accurate and complete record of data, any data compression used in an archival system must be lossless, and protect against propagation of error in the storage media. A browse capability for space and Earth science data is needed to enable scientists to check the appropriateness and quality of particular data sets before obtaining the full data set(s) for detailed analysis. Browse data produced for these purposes could be used to facilitate the retrieval of data from an archival facility. Quick-look data is data obtained directly from the sensor for either previewing the data or for an application that requires very timely analysis of the space or Earth science data. Two main differences between data compression techniques appropriate to browse and quick-look cases, are that quick-look can be more specifically tailored, and it must be limited in complexity by the relatively limited computational power available on space platforms.

Dozier, Jeff↗

Community recommendations on cryoEM data archiving and validation

In January 2020, a workshop was held at EMBL-EBI (Hinxton, UK) to discuss data requirements for the deposition and validation of cryoEM structures, with a focus on single-particle analysis. The meeting was attended by 47 experts in data processing, model building and refinement, validation, and archiving of such structures. This report describes the workshop's motivation and history, the topics discussed, and the resulting consensus recommendations. Some challenges for future methods-development efforts in this area are also highlighted, as is the implementation to date of some of the recommendations.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Global digital data sets of soil type, soil texture, surface slope and other properties: Documentation of archived data tape

The file structure and coding of four soils data sets derived from the Zobler (1986) world soil file is described. The data were digitized on a one-degree square grid. They are suitable for large-area studies such as climate research with general circulation models, as well as in forestry, agriculture, soils, and hydrology. The first file is a data set of codes for soil unit, land-ice, or water, for all the one-degree square cells on Earth. The second file is a data set of codes for texture, land-ice, or water, for the same soil units. The third file is a data set of codes for slope, land-ice, or water for the same units. The fourth file is the SOILWRLD data set, containing information on soil properties of land cells of both Matthews' and Food and Agriculture Organization (FAO) sources. The fourth file reconciles land-classification differences between the two and has missing data filled in.

Staub, B.↗

Subband coding for image data archiving

The use of subband coding on image data is discussed. An overview of subband coding is given. Advantages of subbanding for browsing and progressive resolution are presented. Implementations for lossless and lossy coding are discussed. Algorithm considerations and simple implementations of subband are given.

Glover, D.↗

Subband coding for image data archiving

The use of subband coding on image data is discussed. An overview of subband coding is given. Advantages of subbanding for browsing and progressive resolution are presented. Implementations for lossless and lossy coding are discussed. Algorithm considerations and simple implementations of subband systems are given.

Glover, Daniel↗

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗

NASA participation in the 1980 PEPE/NEROS project: Data archive

Eight experimental air quality measurement systems were investigated during July and August 1980 as part of the EPA PEPE/NEROS fiel measurement program. Data from those efforts have been entered into an archive that may be accessed by other researchers. The data sets consists of airborne measurements of regional mixed layer heights and aerosol and ozone distributions as well as point measurements of meteorological parameters and ozone obtained during diurnal transitions in the planetary boundary layer. This report gives a discussion of each measurement system, a preliminary assessment of data quality, a description of the archive format for each data set, and a summary of several proposed scientific studies which will utilize these data.

Brewer, D. A.↗

gPhoton: THE GALEX PHOTON DATA ARCHIVE

gPhoton is a new database product and software package that enables analysis of GALEX ultraviolet data at the photon level. The project’s stand-alone, pure-Python calibration pipeline reproduces the functionality of the original mission pipeline to reduce raw spacecraft data to lists of time-tagged, sky-projected photons, which are then hosted in a publicly available database by the Mikulski Archive at Space Telescope. This database contains approximately 130 terabytes of data describing approximately 1.1 trillion sky-projected events with a timestamp resolution of five milliseconds. A handful of Python and command-line modules serve as a front end to interact with the database and to generate calibrated light curves and images from the photon-level data at user-defined temporal and spatial scales. The gPhoton software and source code are in active development and publicly available under a permissive license. We describe the motivation, design, and implementation of the calibration pipeline, database, and tools, with emphasis on divergence from prior work, as well as challenges created by the large data volume. We summarize the astrometric and photometric performance of gPhoton relative to the original mission pipeline. For a brief example of short time-domain science capabilities enabled by gPhoton, we show new flares from the known M-dwarf flare star CR Draconis. The gPhoton software has permanent object identifiers with the ASCL (ascl:1603.004) and DOI (doi:10.17909/T9CC7G). This paper describes the software as of version v1.27.2.

79 ASTRONOMY AND ASTROPHYSICS↗

Jet energy spectrum and substructure in $e^+e^–$ collisions at 91.2 GeV with ALEPH Archived Data

The first measurements of energy spectra and substructure of anti-k T jets in hadronic Z 0 decays in e + e – collisions are presented. The archived e + e – annihilation data at a center-of-mass energy of 91.2 GeV were collected with the ALEPH detector at LEP in 1994. In addition to inclusive jet and leading dijet energy spectra, various jet substructure observables are analyzed as a function of jet energy which includes groomed and ungroomed jet mass to jet energy ratios, groomed momentum sharing, and groomed jet radius. The results are compared with perturbative QCD calculations and predictions from the Sherpa, Herwig v7.1.5, Pythia6, Pythia8 and Pyquen event generators. The jet energy spectra agree with perturbative QCD calculations which include the treatment of logarithms of the jet radius and threshold logarithms. None of the event generators give a fully satisfactory description of the data.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

THE MID-INFRARED PERIOD-LUMINOSITY RELATIONS FOR THE SMALL MAGELLANIC CLOUD CEPHEIDS DERIVED FROM SPITZER ARCHIVAL DATA

In this paper, we derive the Spitzer IRAC band period-luminosity (P-L) relations for the Small Magellanic Cloud (SMC) Cepheids, by matching the Spitzer archival SAGE-SMC data with the OGLE-III SMC Cepheids. We find that the 3.6 {mu}m and 4.5 {mu}m band P-L relations can be better described using two P-L relations with a break period at log(P) = 0.4: this is consistent with similar results at optical wavelengths for SMC P-L relations. The 5.8 {mu}m and 8.0 {mu}m band P-L relations do not extend to sufficiently short periods to enable a similar detection of a slope change at log(P) = 0.4. The slopes of the SMC P-L relations, for log(P) > 0.4, are consistent with their Large Magellanic Cloud counterparts that were derived from a similar data set. They are also in agreement with those obtained from a small sample of Galactic Cepheids with parallax measurements.

79 ASTRONOMY AND ASTROPHYSICS↗

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗

Conceptual Study of Intelligent Data Archives of the Future

A conceptual architecture study is under way to address the problem of getting the most scientific value from the large volumes of Earth and space science data that NASA expects to accumulate in the future. This involves efficient storage and access, but goes beyond that to facilitate intelligent data understanding and utilization through modeling realistic virtual entities with predictive capabilities. The objective of the study is to formulate ideas and concepts and to provide recommendations that lead to prototyping and implementation in the period from 2010 to 2020. The approach consists of the definition of future scenarios and needs for data usage in applications (in consultation with scientific and applications users), projection of advances in technologies, and an abstraction of an intelligent archive architecture. Strategic evolution is considered in various areas such as storage, data, information and knowledge management, data ingest and mining, user interfaces, and advances in intelligent data understanding algorithms.

Ramapriyan, H. K.↗

NGDS Data Archiver

This is a Node.js command line utility for scraping XML metadata from CSW and WFS, downloading linkage data from CSW and WFS, pinging hosts and returning status codes, pinging data linkages and returning status codes, writing ping status to CSV files, and uploading data to Amazon S3.

Source record↗

NGDS Data Archiver

This is a Node.js command line utility for scraping XML metadata from CSW and WFS, downloading linkage data from CSW and WFS, pinging hosts and returning status codes, pinging data linkages and returning status codes, writing ping status to CSV files, and uploading data to Amazon S3.

Source record↗

Model Data Archive for Manuscript Titled "Evaluation of a Coupled Surface–Subsurface Hydrologic Model Using Dense Water‑Level Sensors in a Mixed Urban–Rural Watershed"

This archive provides scripts, input files, and datasets used for the implementation and evaluation of a fully coupled surface–subsurface hydrologic model in the Neches River Basin, southeast Texas. The study uses the Advanced Terrestrial Simulator (ATS) to simulate coupled surface–subsurface hydrologic processes over a mixed urban–rural watershed and evaluates model performance using a dense network of 136 in situ water-level sensors, nine U.S. Geological Survey (USGS) stream gauges, and SSEBop-derived evapotranspiration estimates during the period October 2014–June 2024. The workflow is implemented primarily in Python 3 using the Watershed Workflow package. The Jupyter notebooks can be executed using open-source software such as Anaconda JupyterLab or Visual Studio Code. Other data files include TXT, CSV, XML, SHP, TIF, NetCDF, HDF5, and ExodusII files, which can be processed using the provided Python scripts. ATS input files are provided in XML format and can be edited using any commonly used text editor. This archive contains: *Scripts and input files used to generate the ATS model setup, including watershed discretization, mesh generation, parameter mapping, and model configuration. *Jupyter notebooks used for preprocessing observational data, evaluating streamflow, water levels, and evapotranspiration, computing performance metrics, and generating the figures presented in the manuscript. *ATS simulation outputs and processed observational datasets, including OneRain and DD6 water-level sensors, USGS streamflow observations, GIS data, and supporting spatial datasets used throughout the study.

Dense water-level sensor network↗

Status of Mars Global Surveyor Science Data Archives

The Mars Global Surveyor has been in orbit around Mars since September 1997, completing its primary mission on January 31, 2001. As of that date the spacecraft had completed more than 8000 mapping orbits. Data from its science instruments, radio science experiment, and SPICE files have been released regularly to the NASA Planetary Data System (PDS) as described in the MGS Archive Plan and Addendum and are available online. Additional information is contained in the original extended abstract.

Slavney, S.↗