Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

ZFP: A compressed array representation for numerical computations

HPC trends favor algorithms and implementations that reduce data motion relative to FLOPS. We investigate the use of lossy compressed data arrays in place of traditional IEEE floating point arrays to store the primary data of calculations. Simulation is fundamentally an exercise in controlled approximation, and error introduced by finite-precision arithmetic (or lossy compression) is just one of several sources of error that need to be managed to ensure sufficient accuracy in a computed result. We describe ZFP, a compressed numerical format designed for in-memory storage of multidimensional arrays, and summarize theoretical results that demonstrate that the error of repeated lossy compression can be bounded and controlled. Furthermore, we establish a relationship between grid resolution and compression-induced errors and show that, contrary to conventional floating point, ZFP reduces finite-difference errors with finer grids. We present example calculations that demonstrate data reduction by 4x or more with negligible impact on solution accuracy. Our results further demonstrate several orders-of-magnitude increase in accuracy using ZFP over IEEE floating point and Posits for the same storage budget.

Lindstrom, Peter↗
DOE OSTI2024

Integrated photonic encoder for low power and high-speed image processing

Abstract Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

Deep compressed seismic learning for fast location and moment tensor inferences with natural and induced seismicity

Fast detection and characterization of seismic sources is crucial for decision-making and warning systems that monitor natural and induced seismicity. However, besides the laying out of ever denser monitoring networks of seismic instruments, the incorporation of new sensor technologies such as Distributed Acoustic Sensing (DAS) further challenges our processing capabilities to deliver short turnaround answers from seismic monitoring. In response, this work describes a methodology for the learning of the seismological parameters: location and moment tensor from compressed seismic records. In this method, data dimensionality is reduced by applying a general encoding protocol derived from the principles of compressive sensing. The data in compressed form is then fed directly to a convolutional neural network that outputs fast predictions of the seismic source parameters. Thus, the proposed methodology can not only expedite data transmission from the field to the processing center, but also remove the decompression overhead that would be required for the application of traditional processing methods. An autoencoder is also explored as an equivalent alternative to perform the same job. We observe that the CS-based compression requires only a fraction of the computing power, time, data and expertise required to design and train an autoencoder to perform the same task. Implementation of the CS-method with a continuous flow of data together with generalization of the principles to other applications such as classification are also discussed.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Lifting MGARD: Construction of (pre)wavelets on the interval using polynomial predictors of arbitrary order

MGARD (MultiGrid Adaptive Reduction of Data) is an algorithm for compressing and refactoring scientific data, based on the theory of multigrid methods. The core algorithm is built around stable multilevel decompositions of conforming piecewise linear $C^0$ finite element spaces, enabling accurate error control in various norms and derived quantities of interest. In this work, we extend this construction to arbitrary order Lagrange finite elements $\mathbb{Q}_p$, $p \geq 0$, and propose a reformulation of the algorithm as a lifting scheme with polynomial predictors of arbitrary order. Additionally, a new formulation using a compactly supported wavelet basis is discussed, and an explicit construction of the proposed wavelet transform for uniform dyadic grids is described.

Reshniak, Viktor [Oak Ridge National Laboratory (O↗
DOE OSTI2020

Byte select cache compression

Techniques are disclosed for designing cache compression algorithms that control how data in caches are compressed. The techniques generate a custom “byte select algorithm” by applying repeated transforms applied to an initial compression algorithm until a set of suitability criteria is met. The suitability criteria include that the “cost” is below a threshold and that a metadata constraint is met. The “cost” is the number of blocks that can be compressed by an algorithm as compared with the “ideal” algorithm. The metadata constraint is the number of bits required for metadata.

Das, Shomit N.↗
DOE OSTI2022

Compressed Natural Gas Component Leak Frequency Estimation

The frequency of unintended releases in a compressed natural gas system is an important aspect of the system quantitative risk assessment. The frequencies for possible release scenarios, along with engineering models, are utilized to quantify the risks for compressed natural gas facilities. This report documents component leakage frequencies representative of compressed natural gas components that were estimated as a function of the normalized leak size. A Bayesian statistical method was used which results in leak frequency distributions for each component which represent variation and uncertainty in the leak frequency. The analysis shows that there is high uncertainty in the estimated leak frequencies due to sparsity in compressed natural gas data. These leak frequencies may still be useful in compressed natural gas system risk assessments, as long as this high uncertainty is acknowledged and considered appropriately.

03 NATURAL GAS↗
DOE OSTI2024

Community detection in hypergraphs via mutual information maximization

Abstract The hypergraph community detection problem seeks to identify groups of related vertices in hypergraph data. We propose an information-theoretic hypergraph community detection algorithm which compresses the observed data in terms of community labels and community-edge intersections. This algorithm can also be viewed as maximum-likelihood inference in a degree-corrected microcanonical stochastic blockmodel. We perform the compression/inference step via simulated annealing. Unlike several recent algorithms based on canonical models, our microcanonical algorithm does not require inference of statistical parameters such as vertex degrees or pairwise group connection rates. Through synthetic experiments, we find that our algorithm succeeds down to recently-conjectured thresholds for sparse random hypergraphs. We also find competitive performance in cluster recovery tasks on several hypergraph data sets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Leaf spectral reflectance and transmittance, San Lorenzo, Panama, 2020

This data package contains full-range (350 – 2500 nm) leaf reflectance and transmittance spectra of leaves collected at the San Lorenzo forest canopy crane site in Panama (PA-SLZ) from January to March 2020. Around 1000 leaves were measured from 50 species and various leaf phenological stages. Leaves were sampled from both the top of the canopy and multiple heights within the canopy from 10 vertical profiles and for six core species, from the top of the canopy. This data package includes processed leaf spectra (*.csv), and raw data from the spectroradiometer (compressed as .zip). Metadata files include data descriptions (_dd.csv) for tabular data and sample information and a description of the field campaign protocol (PA-SLZ_2020_Protocol.pdf). In addition to the leaf spectral measurements included here, these samples were also used for measurement of leaf gas exchange, carbon and nitrogen content and leaf mass per unit leaf area (LMA) contained in related data packages.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

What to Support When You’re Compressing

Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications’ data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications’ needs and compressors’ capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

Error-Bounded Lossy Compression↗
DOE OSTI2021

Equation-of-state, sound speed, and reshock of shock-compressed fluid carbon dioxide

Mechanical equation-of-state data of initially liquid and solid CO2 shock-compressed to terapascal conditions are reported. Diamond-sapphire anvil cells were used to vary the initial density and state of CO2 samples that were then further compressed with laser-driven shock waves, resulting in a data set from which precise derivative quantities, including Grüneisen parameter and sound speed, are determined. Reshock states are measured to 800 GPa and map the same pressure-density conditions as the single shock using different thermodynamic paths. The compressibility data reported here do not support current density-functional-theory calculations, but are better represented by tabular equation-of-state models.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

StOKeDMD: Streaming Occupation kernel dynamic mode decomposition

Dynamic mode decomposition (DMD) has become a common technique for constructing surrogate models for dynamical systems from observed system states. The Occupation Kernel DMD (OKDMD) method proposed in (Rosenfeld et al., 2022) and (Rosenfeld et al., 2024) is a Liouville operator based method that builds surrogate models from system state trajectories. Here, this paper proposes an extension of OKDMD to the case when the system states are observed in a streaming fashion, i.e., only a small fraction of the state trajectory is available at a given time. The developed method, Streaming Occupation Kernel DMD (StOKeDMD), accommodates the streaming data input by leveraging properties of specific choices of kernel functions and occupation kernels. We apply the StoKeDMD method as a compression method for streaming data, analyze the memory complexity, and demonstrate the performance of StoKeDMD in the compression of streaming data generated from a Lorenz system and a fluid flow simulation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Optimising the processing and storage of visibilities using lossy compression

The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, largely through increasing the number of stations in the array and the frequency span sampled. The two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and the constraints imposed by I/O, as I/O bandwidths drop significantly on cold storage. An example of this is the data deluge expected from the SKA Telescopes of more than 60 PB per day, all to be stored on the buffer filesystem. While compressing the data is an obvious solution, the impacts on the final data products are hard to predict. In this paper, we chose an error-controlled compressor – MGARD – and applied it to simulated SKA-Mid and real pathfinder visibility data, in noise-free and noise-dominated regimes. As the data have an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. MGARD ensures the compression incurred errors adhere to the user-prescribed tolerance. To measure the degradation of images reconstructed using the lossy compressed data, we proposed a list of diagnostic measures, exploring the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images for continuum and spectral line examples. We found relative error bounds of as much as 10%, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS, whereas a 1% error bound (compression ratio of 8) introduces an increase in noise of about an order of magnitude less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a 0.1% error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO, in both the impacts on the images and in the relative flexibility. MGARD provides better compression for similar error bounds and has a host of potentially powerful additional features.

Techniques: interferometric↗
DOE OSTI2023

HarDWR - Water Management Area (WMA) Shapefiles

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. Borders of all Water Management Areas (WMAs) across the 11 Western-most states of the continental United States are available filtered through a single source. The legal name for this set of boundaries varies state-by-state. The data is provided as two compressed shapefiles. One, stateWMAs, contains data for all 11 states. For 10 of those states, Arizona being the exception, the polygons represent the legal management boundaries used by those states to manage their surface and groundwater resources respectively. Arizona is unique among this collection of states in that surface and groundwater resources are managed using two separate sets of boundaries. During our followup analysis, due to technical reasons we decided to focus on one set of boundaries, those for surface water. Due to this, the Arizona surface WMAs are included within stateWMAs. The Arizona groundwater WMAs are provided as the second file azGroundWMAs as a companion to the first file for completeness and general reference, although the boundaries were no longer referenced in the analysis (Grogan et al., in review) past a certain point. The columns for both shapefiles are: basinNum: the state provided unique numerical ID basinName: the state provided English name of the area, where applicable state: the state name uniID: a unique identifier we created by concatenating the state name, and underscore, and the state numerical ID

Economics↗
DOE OSTI2020

HarDWR - Water Management Area (WMA) Shapefiles

A dataset within the Harmonized Database of Western U.S. Water Rights (HarDWR). For a detailed description of the database, please see the meta-record v2.0. Changelog v2.0 - No changes v1.0 - Initial public release Description Borders of all Water Management Areas (WMAs) across the 11 western-most states of the coterminous United States are available filtered through a single source. The legal name for this set of boundaries varies state-by-state. The data is provided as two compressed shapefiles. One, stateWMAs, contains data for all 11 states. For 10 of those states, Arizona being the exception, the polygons represent the legal management boundaries used by those states to manage their surface and groundwater resources respectively. WMAs refer to the set of boundaries a particular state uses to manage its water resources. Each set of boundaries was collected from the states individually, and then merged into one spatial layer. The merging process included renaming some columns to enable merging with all other source layers, as well as removing columns deemed not required for followup analysis. The retained columns for each boundary are: basinNum - the state provided unique numerical ID; basinName - the state provided English name of the area, where applicable; state - the state name; and uniID - a unique identifier we created by concatenating the state name, and underscore, and the state numerical ID. Arizona is unique within this collection of states in that surface and groundwater resources are managed using two separate sets of boundaries. During our followup analysis (Grogan et al., in review) we decided to focus on one set of boundaries, those for surface water. This is due to the recommendation of our hydrologists that the surface water boundary set is a more realist representation of how water moves across the landscape, as a few of the groundwater boundaries are based on political and/or economic considerations. Therefore, the Arizona surface WMAs are included within stateWMAs. The Arizona groundwater WMAs are provided as a separate file, azGroundWMAs, as a companion to the first file for completeness and general reference. WMA spatial boundary data sources by state: Arizona: Arizona Surface Water Watersheds; Collected February, 2020; https://gisdata2016-11-18t150447874z-azwater.opendata.arcgis.com/datasets/surface-watershed/explore?location=34.158174%2C-111.970823%2C7.50 Arizona: Arizona Ground Water Basins; Collected February, 2020; https://gisdata2016-11-18t150447874z-azwater.opendata.arcgis.com/datasets/groundwater-basin-2/explore?location=34.158174%2C-111.970823%2C7.50 California: California CalWater 2.2.1; Collected February, 2020; https://www.mlml.calstate.edu/mpsl-mlml/data-center/data-entry-tools/data-tools/gis-shapefile-layers/ Colorado: Colorado Water District Boundaries; Collected February, 2020; https://www.colorado.gov/pacific/cdss/gis-data-category Idaho: Idaho Department of Water Resources (IDWR) Administrative Basins; Collected November, 2015; https://data-idwr.opendata.arcgis.com/datasets/fb0df7d688a04074bad92ca8ef74cc26_4/explore?location=45.018686%2C-113.862284%2C6.93 Montana: Collected June, 2019; Directly contacted Montana Department of Natural Resources and Conservation (DNRC) Office of Information Technology (OIT) Nevada: Nevada State Engineer Admin Basin Boundaries; Collected April, 2020 https://ndwr.maps.arcgis.com/apps/mapviewer/index.html?layers=1364d0c3a0284fa1bcd90f952b2b9f1c New Mexico: New Mexico Office of the State Engineer (OSE) Declared Groundwater Basins; Collected April, 2020 https://geospatialdata-ose.opendata.arcgis.com/datasets/ose-declared-groundwater-basins/explore?location=34.179783%2C-105.996542%2C7.51 Oregon: Oregon Water Resources Department (OWRD) Administrative Basins; Collected February, 2020; https://www.oregon.gov/OWRD/access_Data/Pages/Data.aspx Utah: Utah Adjudication Books; Collected April, 2020; https://opendata.gis.utah.gov/datasets/utahDNR::utah-adjudication-books/explore?location=39.497165%2C-111.587782%2C-1.00 Washington: Washington Water Resource Inventory Areas (WRIA); Collected June, 2017; https://ecology.wa.gov/Research-Data/Data-resources/Geographic-Information-Systems-GIS/Data Wyoming: Wyoming State Engineer's Office Board of Control Water Districts; Collected June, 2019; Directly contacted Wyoming State Engineer's Office

Economics↗
DOE OSTI2021

Data Science and Computation for Rapid and Dynamic Compression Experiment Workflows at Experimental Facilities (Workshop Overview) [Slides]

Motivation for this workshop stems from these items: Compression experiments are important to LANL's core mission. The scientific community is sometimes frustrated at the pace of discovery. Data analytics for other experimental regimes are advancing. Now is the time as facilities are upgraded and coming online. Advances in data analytics and computational resources could push compression experiment discoveries to a new level.

97 MATHEMATICS AND COMPUTING↗