Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

A General Framework for Progressive Data Compression and Retrieval

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. Here, this paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Data compression and covariance matrix inspection: Cosmic shear

Covariance matrices are among the most difficult pieces of end-to-end cosmological analyses. In principle, for two-point functions, each component involves a four-point function, and the resulting covariance often has hundreds of thousands of elements. We investigate various compression mechanisms capable of vastly reducing the size of the covariance matrix in the context of cosmic shear statistics. This helps identify which of its parts are most crucial to parameter estimation. We start with simple compression methods, by isolating and “removing” 200 modes associated with the lowest eigenvalues, then those with the lowest signal-to-noise ratio, before moving on to more sophisticated schemes like compression at the tomographic level and, finally, with the massively optimized parameter estimation and data compression (MOPED). We find that, while most of these approaches prove useful for a few parameters of interest, like Ω m , the simplest yield a loss of constraining power on the intrinsic alignment (IA) parameters as well as S 8 . For the case considered—cosmic shear from the first year of data from the Dark Energy Survey—only MOPED was able to replicate the original constraints in the 16-parameter space. Finally, we apply a tolerance test to the elements of the compressed covariance matrix obtained with MOPED and confirm that the IA parameter A IA is the most susceptible to inaccuracies in the covariance matrix.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS1991

Video data compression using artificial neural network differential vector quantization

An artificial neural network vector quantizer is developed for use in data compression applications such as Digital Video. Differential Vector Quantization is used to preserve edge features, and a new adaptive algorithm, known as Frequency-Sensitive Competitive Learning, is used to develop the vector quantizer codebook. To develop real time performance, a custom Very Large Scale Integration Application Specific Integrated Circuit (VLSI ASIC) is being developed to realize the associative memory functions needed in the vector quantization algorithm. By using vector quantization, the need for Huffman coding can be eliminated, resulting in superior performance against channel bit errors than methods that use variable length codes.

Krishnamurthy, Ashok K.↗
NASA NTRS2002

Implementation of CCSDS Lossless Data Compression in HDF

The Earth Science Data and Information System (ESDIS) handles over one terabyte (10(exp 12) bytes) of data daily and is using the Hierarchical Data Format (EDF) for data archiving and distribution. This report provides the progress and status of our effort to alleviate bandwidth and storage burdens by first performing compression studies on various science data products and later integrating the selected compression scheme into HDF.

Pen-Shu Yeh↗
NASA NTRS2008

Fast Plasma Instrument for MMS: Data Compression Simulation Results

Magnetospheric Multiscale (MMS) mission will study small-scale reconnection structures and their rapid motions from closely spaced platforms using instruments capable of high angular, energy, and time resolution measurements. To meet these requirements, the Fast Plasma Instrument (FPI) consists of eight (8) identical half top-hat electron sensors and eights (8) identical ion sensors and an Instrument Data Processing Unit (IDPU). The sensors (electron or ion) are grouped into pairs whose 6 deg x 180 deg fields-of-view (FOV) are set 90 deg apart. Each sensor is equipped with electrostatic aperture steering to allow the sensor to scan a 45 deg x 180 deg fan about its nominal viewing (0 deg deflection) direction. Each pair of sensors, known as the Dual Electron Spectrometer (DES) and the Dual Ion Spectrometer (DIS), occupies a quadrant on the MMS spacecraft and the combination of the eight electron/ion sensors, employing aperture steering, image the full-sky every 30-ms (electrons) and 150-ms (ions), respectively. To probe the results in the DES complement of a given spacecraft generating 6.5-Mbs(exp -1) of electron data while the DIS generates 1.1-Mbs(exp -1) of ion data yielding an FPI total data rate of 6.6-MBs(exp -1). The FPI electron/ion data is collected by the IDPU then transmitted to the Central Data Instrument Processor (CIDP) on the spacecraft for science interest ranking. Only data sequences that contain the greatest amount of temporal/spatial structure will be intelligently down-linked by the spacecraft. Currently, the FPI data rate allocation to the CIDP is 1.5-Mbs(exp -1). Consequently, the FPI-IDPU must employ data/image compression to meet this CIDP telemetry allocation. Here, we present simulations of the CCSDS 122.0-B-1 algorithm-based compression of the FPI-DES electron data. Compression analysis is based upon a seed of re-processed Cluster/PEACE electron measurements. Topics to be discussed include: review of compression algorithm; data quality; data formatting/organization; and, implications for data/matrix pruning. To conclude a presentation of the base-lined FPI data compression approach is provided.

Barrie, A.↗
NASA NTRS1991

The moderate resolution imaging spectrometer: An EOS facility instrument candidate for application of data compression methods

The Moderate Resolution Imaging Spectrometer (MODIS) observing facility will operate on the Earth Observing System (EOS) in the late 1990's. It is estimated that this observing facility will produce over 200 gigabytes of data per day requiring a storage capability of just over 300 gigabytes per day. Archiving, browsing, and distributing the data associated with MODIS represents a rich opportunity for testing and applying both lossless and lossy data compression methods.

Salomonson, Vincent V.↗
NASA NTRS1994

Comparison of transform coding methods with an optimal predictor for the data compression of digital elevation models

Statistical encoding techniques enable the reduction of the number of bits required to encode a set of symbols, and are derived from their probabilities. Huffman encoding is an example of statistical encoding that has been used for error-free data compression. The degree of compression given by Huffman encoding in this application can be improved by the use of prediction methods. These replace the set of elevations by a set of corrections that have a more advantageous probability distribution. In particular, the method of Lagrange Multipliers for minimization of the mean square error has been applied to local geometrical predictors. Using this technique, an 8-point predictor achieved about a 7 percent improvement over an existing simple triangular predictor.

Lewis, Michael↗
NASA NTRS1991

Data compression for full motion video transmission

Clearly transmission of visual information will be a major, if not dominant, factor in determining the requirements for, and assessing the performance of, the SEI communications systems. Projected image/video requirements which are currently anticipated for SEI mission scenarios are presented. Based on this information and projected link performance figures, the image/video data compression requirements which would allow link closure are identified. Finally several approaches which could satisfy some of the compression requirements are presented and possible future approaches which show promise for more substantial compression performance improvement are discussed.

Whyte, Wayne A., Jr.↗
NASA NTRS1991

Data compression for full motion video transmission

Clearly transmission of visual information will be a major, if not dominant, factor in determining the requirements for, and assessing the performance of the Space Exploration Initiative (SEI) communications systems. Projected image/video requirements which are currently anticipated for SEI mission scenarios are presented. Based on this information and projected link performance figures, the image/video data compression requirements which would allow link closure are identified. Finally several approaches which could satisfy some of the compression requirements are presented and possible future approaches which show promise for more substantial compression performance improvement are discussed.

Whyte, Wayne A., Jr.↗
NASA NTRS1994

A comparison of spectral decorrelation techniques and performance evaluation metrics for a wavelet-based, multispectral data compression algorithm

Future space-based, remote sensing systems will have data transmission requirements that exceed available downlinks necessitating the use of lossy compression techniques for multispectral data. In this paper, we describe several algorithms for lossy compression of multispectral data which combine spectral decorrelation techniques with an adaptive, wavelet-based, image compression algorithm to exploit both spectral and spatial correlation. We compare the performance of several different spectral decorrelation techniques including wavelet transformation in the spectral dimension. The performance of each technique is evaluated at compression ratios ranging from 4:1 to 16:1. Performance measures used are visual examination, conventional distortion measures, and multispectral classification results. We also introduce a family of distortion metrics that are designed to quantify and predict the effect of compression artifacts on multi spectral classification of the reconstructed data.

Matic, Roy M.↗
NASA NTRS1994

Evaluating the effect of online data compression on the disk cache of a mass storage system

A trace driven simulation of the disk cache of a mass storage system was used to evaluate the effect of an online compression algorithm on various performance measures. Traces from the system at NASA's Center for Computational Sciences were used to run the simulation and disk cache hit ratios, number of files and bytes migrating to tertiary storage were measured. The measurements were performed for both an LRU and a size based migration algorithm. In addition to seeing the effect of online data compression on the disk cache performance measure, the simulation provided insight into the characteristics of the interactive references, suggesting that hint based prefetching algorithms are the only alternative for any future improvements to the disk cache hit ratio.

Pentakalos, Odysseas I.↗
NASA NTRS1991

A progressive data compression scheme based upon adaptive transform coding: Mixture block coding of natural images

A method for efficiently coding natural images using a vector-quantized variable-blocksized transform source coder is presented. The method, mixture block coding (MBC), incorporates variable-rate coding by using a mixture of discrete cosine transform (DCT) source coders. Which coders are selected to code any given image region is made through a threshold driven distortion criterion. In this paper, MBC is used in two different applications. The base method is concerned with single-pass low-rate image data compression. The second is a natural extension of the base method which allows for low-rate progressive transmission (PT). Since the base method adapts easily to progressive coding, it offers the aesthetic advantage of progressive coding without incorporating extensive channel overhead. Image compression rates of approximately 0.5 bit/pel are demonstrated for both monochrome and color images.

Rost, Martin C.↗
NASA NTRS1991

Microgravity science requirements and the need for data compression

The Microgravity Science and Applications Div. (MSAD) of the NASA Office of Space Science and Applications (OSSA) is responsible for encouraging and directing the research of a wide range of physical phenomena in reduced gravity. Under MSAD's direction, NASA-Lewis is presently developing the concept of a multiuser facility which will perform combustion science experiments in space. This facility, the Combustion Experiment Module (CEM), will be located in either the Shuttle Spacelab or the Space Station Freedom lab and will be operational by mid-1997. In addition to standard instrumentation to measure temperature, pressure, and acceleration, CEM shall use a variety of imaging and optical diagnostic techniques. Images shall be the primary source of experimental data. These images create an enormous amount of data which must be archived on orbit for later analysis. Also, ground based investigators will require enough data from the orbiting facility to determine if the experimental parameters need to be changed before proceeding with the next run. The storage and transmission of this data present a major challenge to the CEM design. Data compression will play a major role in the design of the CEM diagnostics system.

Hartz, William G.↗