Search NASA⌕ Search

SEARCH · Search NASA

Results for “data preservation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Information preserving coding for multispectral data

A general formulation of the data compression system is presented. A method of instantaneous expansion of quantization levels by reserving two codewords in the codebook to perform a folding over in quantization is implemented for error free coding of data with incomplete knowledge of the probability density function. Results for simple DPCM with folding and an adaptive transform coding technique followed by a DPCM technique are compared using ERTS-1 data.

Duan, J. R.↗

Seasat data restoration

Explore the source record for details and available documents.

data restoration data preservation long-term data ↗

The Transition of High-Resolution NASA MODIS Sea Surface Temperatures into the WRF Environmental Modeling System

The NASA Short-term Prediction Research and Transition (SPoRT) Center has developed a Moderate Resolution Imaging Spectroradiometer (MODIS) sea surface temperature (SST) composite at 2-km resolution that has been implemented in version 3 of the National Weather Service (NWS) Weather Research and Forecasting (WRF) Environmental Modeling System (EMS). The WRF EMS is a complete, full physics numerical weather prediction package that incorporates dynamical cores from both the Advanced Research WRF (ARW) and the Non-hydrostatic Mesoscale Model (NMM). The installation, configuration, and execution of either the ARW or NMM models is greatly simplified by the WRF EMS to encourage its use by NWS Weather Forecast Offices (WFOs) and the university community. The WRF EMS is easy to run on most Linux workstations and clusters without the need for compilers. Version 3 of the WRF EMS contains the most recent public release of the WRF-NMM and ARW modeling system (version 3 of the ARW is described in Skamarock et al. 2008), the WRF Pre-processing System (WPS) utilities, and the WRF Post-Processing program. The system is developed and maintained by the NWS National Science Operations Officer Science and Training Resource Coordinator. To initialize the WRF EMS with high-resolution MODIS SSTs, SPoRT developed the composite product consisting of MODIS SSTs over oceans and large lakes with the NCEP Real-Time Global (RTG) filling data over land points. Filling the land points is required due to minor inconsistencies between the WRF land-sea mask and that used to generate the MODIS SST composites. This methodology ensures a continuous field that adequately initializes all appropriate arrays in WRF. MODIS composites covering the Gulf of Mexico, western Atlantic Ocean and the Caribbean are generated daily at 0400, 0700, 1600, and 1900 UTC corresponding to overpass times of the NASA Aqua and Terra polar orbiting satellites. The MODIS SST product is output in gridded binary-1 (GRIB-1) data format for a seamless incorporation into WRF via the WPS utilities. The full-resolution, 1-km MODIS product is sub-sampled to 2-km grid spacing due to limitations in handling very large dimensions in the GRIB-1 data format. The GRIB-1 files are posted online at ftp://ftp.nsstc.org/sstcomp/WRF/, which is directly accessed by the WRF EMS scripts. The MODIS SST composites are also downloaded to the EMS data server, which is accessible by the WRF EMS users and NWS WFOs. The SPoRT MODIS SST composite provides the model with superior detail of the ocean gradients around Florida and surrounding waters, whereas the operational RTG SST typically depicts a relatively smooth field and is not able to capture sharp horizontal gradients in SST. Differences of 2-3 C are common over small horizontal distances, leading to enhanced SST gradients on either side of the Gulf Stream and along the edges of the cooler shelf waters. These sharper gradients can in turn produce atmospheric responses in simulated temperature and wind fields as depicted in LaCasse et al. Differences in atmospheric verification statistics over a several month study were generally small in the vicinity of south Florida; however, the validation of SSTs at specific buoy locations revealed important improvements in the biases and RMS errors, especially in the vicinity of the cooler shelf waters off the east-central Florida coast. A current weakness in the MODIS SST product is the occurrence of occasional discontinuities caused by high latency in SST coverage due to persistent cloud cover. An enhanced method developed by Jedlovec et al. (2009, GHRSST User Symposium) reduces the occurrence of these problems by adding Advanced Microwave Scanning Radiometer -- EOS (AMSR-E) SST data to the compositing process. Enhanced SST composites are produced over the ocean regions surrounding the Continental U.S. at four times each day corresponding to Terra and Aqua equator crossing times. For a given day and overpass time, both MODInd AMSR-E data from the previous seven days form a collection used in the compositing. At each MODIS pixel, cloud-free SST values from the collection are used to form a weighted average based on their latency (number of days from the current day). In this way, recent SST data are given more weight than older data. One of the primary issues involved in incorporating the AMSR-E microwave data in the composites is the tradeoff between the decreased spatial resolution of the AMSR-E data (25 km) and the increased coverage due to its near all-weather capability. Currently, the AMSR-E is given a weight of 20% compared to MODIS data, thereby preserving the spatial structure observed in the MODIS data. Day-time (night-time) AMSR-E SST data from Aqua are used with both Terra and Aqua MODIS day-time (night-time) SST data sets.

Case, Jonathan L.↗

Evaluating lossy data compression on climate simulation data within a large ensemble

High-resolution Earth system model simulations generate enormous data volumes, and retaining the data from these simulations often strains institutional storage resources. Further, these exceedingly large storage requirements negatively impact science objectives, for example, by forcing reductions in data output frequency, simulation length, or ensemble size. To lessen data volumes from the Community Earth System Model (CESM), we advocate the use of lossy data compression techniques. While lossy data compression does not exactly preserve the original data (as lossless compression does), lossy techniques have an advantage in terms of smaller storage requirements. To preserve the integrity of the scientific simulation data, the effects of lossy data compression on the original data should, at a minimum, not be statistically distinguishable from the natural variability of the climate system, and previous preliminary work with data from CESM has shown this goal to be attainable. However, to ultimately convince climate scientists that it is acceptable to use lossy data compression, we provide climate scientists with access to publicly available climate data that have undergone lossy data compression. In particular, we report on the results of a lossy data compression experiment with output from the CESM Large Ensemble (CESM-LE) Community Project, in which we challenge climate scientists to examine features of the data relevant to their interests, and attempt to identify which of the ensemble members have been compressed and reconstructed. We find that while detecting distinguishing features is certainly possible, the compression effects noticeable in these features are often unimportant or disappear in post-processing analyses. In addition, we perform several analyses that directly compare the original data to the reconstructed data to investigate the preservation, or lack thereof, of specific features critical to climate science. Overall, we conclude that applying lossy data compression to climate simulation data is both advantageous in terms of data reduction and generally acceptable in terms of effects on scientific results.

54 ENVIRONMENTAL SCIENCES↗

Preservation of Provenance and Context to Ensure Future Understandability of Airborne Earth Observations and Derived Data Products

Open-source science goes beyond making data from scientific projects (e.g., on-orbit/satellite missions, airborne and field investigations, and other data producing activities) openly available after they are generated, but involves and open sharing of information throughout the project lifecycle. Preservation of the data and associated information required for understanding and reusing the data well after the scientific projects is a contributor to open-source science as well. Considering the high investment in the on-orbit/satellite missions, we had developed a document titled “NASA Earth Science Data Preservation Content Specification (PCS)” in 2011. This document has been used as a requirement for recent on-orbit/satellite missions by NASA. Recently it became clear that the specifications should be applied to other scientific projects as well. Therefore, the document was revised to cover other types of projects, and a Preservation Content Implementation Guidance (PCIG) document was also developed. The revised PCS, and the PCIG, were published in 2022. The purpose of this presentation is to highlight the contents of these documents as they apply to suborbital/airborne investigations. The PCS calls for content preservation in eight general categories - Measuring Instrument/Platform Description, Instrument and Science Data Products and Metadata, Science Raw Data, Product and Algorithm Documentation, Instrument Calibration, Science Algorithm Software, Science Data Product Algorithm Inputs, Science Data Product Validation, and Science Data Access and Analysis Tools. While all these categories apply to various types of projects, a few clarifying sentences have been added to the descriptions of contents in each of the categories to show which categories are especially important to airborne and field investigations and where some contents are not applicable (or difficult to obtain). The PCIG document provides some general guidance applicable to all types of projects and specific guidance in a separate section for airborne and field investigations. This section calls out typical artifacts produced during such investigations that can meet the spirit of the various PCS categories.

remote sensing↗

KEPLER ECLIPSING BINARY STARS. III. CLASSIFICATION OF KEPLER ECLIPSING BINARY LIGHT CURVES WITH LOCALLY LINEAR EMBEDDING

We present an automated classification of 2165 Kepler eclipsing binary (EB) light curves that accompanied the second Kepler data release. The light curves are classified using locally linear embedding, a general nonlinear dimensionality reduction tool, into morphology types (detached, semi-detached, overcontact, ellipsoidal). The method, related to a more widely used principal component analysis, produces a lower-dimensional representation of the input data while preserving local geometry and, consequently, the similarity between neighboring data points. We use this property to reduce the dimensionality in a series of steps to a one-dimensional manifold and classify light curves with a single parameter that is a measure of 'detachedness' of the system. This fully automated classification correlates well with the manual determination of morphology from the data release, and also efficiently highlights any misclassified objects. Once a lower-dimensional projection space is defined, the classification of additional light curves runs in a negligible time and the method can therefore be used as a fully automated classifier in pipeline structures. The classifier forms a tier of the Kepler EB pipeline that pre-processes light curves for the artificial intelligence based parameter estimator.

79 ASTRONOMY AND ASTROPHYSICS↗

Multidimensional scaling informed by F -statistic: Visualizing grouped microbiome data with inference

Multidimensional scaling (MDS) is a widely used dimensionality reduction technique in microbial ecology data analysis that captures the multivariate structure of the data while preserving pairwise distances between samples. While improvements in MDS have enhanced the ability to reveal group-specific data patterns, these MDS-based methods require prior assumptions for inference, limiting their application in general microbiome analysis. Here, in this study, we introduce a new MDS-based ordination method, “F-informed MDS,” which configures the data distribution based on the F-statistic, the ratio of dispersion between groups sharing common and different characteristics. Using semisynthetic datasets, we demonstrate that the proposed method is robust to hyperparameter selection while maintaining statistical significance throughout the ordination process. Various quality metrics for evaluating dimensionality reduction confirm that F-informed MDS is comparable to state-of-the-art methods in preserving both local and global data structures. Its application to a diatom-associated bacterial community suggests the role of this new method in interpreting the community’s response to the host. Our approach offers a well-founded refinement of MDS that aligns with statistical test results, which can be beneficial for broader multidimensional data analyses in microbiology and ecology. This new visualization tool can be incorporated into standard microbiome data analyses.

Biological and medical sciences↗

An Ontology Driven Information Architecture for Interoperable Disparate Data Sources

The mission of the Planetary Data System is to facilitate achievement of NASA's planetary science goals by efficiently collecting, archiving, and making accessible digital data produced by or relevant to NASA's planetary missions, research programs, and data analysis programs. The vision is: (1) To gather and preserve the data obtained from exploration of the Solar System by the U.S. and other nations (2) To facilitate new and exciting discoveries by providing access to and ensuring usability of those data to the worldwide community (3) To inspire the public through availability and distribution of the body of knowledge reflected in the PDS data collection PDS is a federation of heterogeneous nodes including science and support nodes

planetary data systems↗

Enabling Computation on Sensitive Data in International Safeguards with Privacy-Preserving Encryption Techniques

Privacy-preserving machine learning is a field of study that explores how to protect and preserve the privacy of sensitive data while allowing the data to be used by machine learning algorithms. This field has had substantial industry investment due to heightened concerns about privacy in the technology industry, with a focus in two broad application areas: financial services and healthcare. Numerous privacy-preserving methods have also been proposed for international safeguards, but they have been difficult to enact because the data they require is con- sidered sensitive or proprietary by the nuclear facility operator. This work examines how current privacy-preserving approaches might be used to enable the International Atomic Energy Agency (IAEA) to use that data to contribute to a safeguards conclusion about a state while giving nuclear operators confidence that their sensitive data is adequately protected. This paper begins by exploring several broad categories of privacy-preserving techniques including homomorphic encryption, secure multiparty computation, secure enclaves, and zero-knowledge proofs. Then we discuss some of the security considerations related to using these methods, potential use cases, and a conceptual system design for applying privacy-preserving methods in international safeguards.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Provenance-Enabling DOE Visualization Applications

VisTrails, Inc.’s goal main charter has been to develop provenance management solutions for software tools. As the use of computation increases, keeping track of computational processes and the data they consume and derive is essential for many applications. Provenance (also referred to as audit trail, lineage, and pedigree) captures information about the steps used to generate a given data product. Such information provides important documentation that is key to preserving the data, to determining the data’s quality and authorship, and to reproducing as well as validating the results. Given the growing complexity of the computational processes and the volume of data they manipulate, manual approaches to maintaining provenance are both tedious and error-prone. At VisTrails, Inc., we have implemented infrastructure that can be used to provenance-enable existing tools. We have developed a series of plugins for specific applications, including ParaView and VisIt, which enable these applications to capture provenance. While developing these plugins, we identified general patterns across different applications, and these were used to inform the design of a general framework: the VisTrails Provenance Software Development Kit (SDK). The SDK encapsulates a set of core functionality including methods for capture, navigation, analysis and re-use of provenance. It allows application or plugin developers to more effectively provide provenance support: they can use the core functionality and write code only to handle application specific components. The SDK includes features that can be used in any application, for example, a refined interface for browsing the intermediate and final results captured. While the application-specific plugins were written in Python, the SDK was written from scratch in C++. This has led not only to substantial gains in performance but also the ability to integrate the SDK with applications written in other languages, including Java. To ensure that the SDK supported all of the functionality in the application-specific plugins, we have rewritten some of the existing plugins using the SDK. The SDK can be downloaded from www.vistrals.com/sdk.html.

97 MATHEMATICS AND COMPUTING↗

A Survey on Error-Bounded Lossy Compression for Scientific Datasets

Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. They are designed with distinct compression models and principles, such that each of them features particular pros and cons. In this article, we provide a comprehensive survey of emerging error-bounded lossy compression techniques. The key contribution is fourfold. (1) We summarize a novel taxonomy of lossy compression into six classic models. (2) We provide a comprehensive survey of 10 commonly used compression components/modules. (3) We summarized pros and cons of 47 state-of-the-art lossy compressors and present how state-of-the-art compressors are designed based on different compression techniques. (4) We discuss how customized compressors are designed for specific scientific applications and use-cases. We believe this survey is useful to multiple communities including scientific applications, high-performance computing, lossy compression, and big data.

Error-Bounded Lossy Compression↗

NETL's Energy Data eXchange

A brief tour around NETL's Energy Data Exchange site, where researchers can upload data or look at data from another researcher.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NDMAS: Data storage, visualization, analysis, delivery, and more

Slides representing Nuclear Data Management and Analysis System (NDMAS) Data for properly preserving our publicly funded data, including DOE Public Access Plan and the users of NDMAS. From Capture, Storage, Delivery & Analysis, Archival, and Qualification of Fuel fabrication, PIE, ART Experiment monitoring and operations, and processes for recording work and for looking ahead.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

The Nuclear Energy Knowledge and Validation Center Summary of Activities Conducted in FY16

The Nuclear Energy Knowledge and Validation Center (NEKVaC) is a new initiative by the Department of Energy (DOE) and Idaho National Laboratory (INL) to coordinate and focus the resources and expertise that exist with the DOE toward solving issues in modern nuclear code validation and knowledge management. In time, code owners, users, and developers will view the NEKVaC as a partner and essential resource for acquiring the best practices and latest techniques for validating codes, providing guidance in planning and executing experiments, facilitating access to and maximizing the usefulness of existing data, and preserving knowledge for continual use by nuclear professionals and organizations for their own validation needs. The scope of the NEKVaC covers many interrelated activities that will need to be cultivated carefully in the near term and managed properly once the NEKVaC is fully functional. Three areas comprise the principal mission: (1) identify and prioritize projects that extend the field of validation science and its application to modern codes, (2) develop and disseminate best practices and guidelines for high-fidelity multiphysics/multiscale analysis code development and associated experiment design, and (3) define protocols for data acquisition and knowledge preservation and provide a portal for access to databases currently scattered among numerous organizations. These mission areas, while each having a unique focus, are interdependent and complementary. Likewise, all activities supported by the NEKVaC, both near term and long term, must possess elements supporting all three areas. This cross cutting nature is essential to ensuring that activities and supporting personnel do not become “stove piped” (i.e., focused a specific function that the activity itself becomes the objective rather than achieving the larger vision). This report begins with a description of the mission areas; specifically, the role played by each major committee and the types of activities for which they are responsible. It then lists and describes the proposed near term tasks upon which future efforts can build.

##CAI suggestions##↗