Demonstration of a quantile system for compression of data from deep space probes
Quantile system of data compression for space telemetry
COMPRESSION↗
SEARCH · Search NASA
Results for “Data Compression”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Demonstration of a quantile system for compression of data from deep space probes.
Quantile system of data compression for space telemetry, discussing operation, efficiency and advantages
DATA PROCESSING↗
Performance of a Discrete Wavelet Transform for Compressing Plasma Count Data and its Application to the Fast Plasma Investigation on NASA's Magnetospheric Multiscale Mission
Plasma measurements in space are becoming increasingly faster, higher resolution, and distributed over multiple instruments. As raw data generation rates can exceed available data transfer bandwidth, data compression is becoming a critical design component. Data compression has been a staple of imaging instruments for years, but only recently have plasma measurement designers become interested in high performance data compression. Missions will often use a simple lossless compression technique yielding compression ratios of approximately 2:1, however future missions may require compression ratios upwards of 10:1. This study aims to explore how a Discrete Wavelet Transform combined with a Bit Plane Encoder (DWT/BPE), implemented via a CCSDS standard, can be used effectively to compress count information common to plasma measurements to high compression ratios while maintaining little or no compression error. The compression ASIC used for the Fast Plasma Investigation (FPI) on board the Magnetospheric Multiscale mission (MMS) is used for this study. Plasma count data from multiple sources is examined: resampled data from previous missions, randomly generated data from distribution functions, and simulations of expected regimes. These are run through the compression routines with various parameters to yield the greatest possible compression ratio while maintaining little or no error, the latter indicates that fully lossless compression is obtained. Finally, recommendations are made for future missions as to what can be achieved when compressing plasma count data and how best to do so.
Particles↗
Eureka: Enabling Fine-Grained Access and Range Queries on Compressed Scientific Data via Data-Index Co-Compression
Handling large-scale scientific data in high-performance computing (HPC) environments poses significant challenges, including excessive I/O, high storage costs, and slow query performance. Traditional approaches often require full data decompression and scans, making them impractical for real-time or interactive analysis. To address these limitations, we introduce Eureka, a unified data-index co-compression framework that enables fine-grained access and efficient range queries on compressed scientific datasets. Eureka integrates spatial domain decomposition with block-wise error-bounded lossy compression to support selective decompression. It constructs a hierarchical AVL-tree index during compression to capture block-level value ranges, enabling fast pruning during query execution. To reduce metadata overhead, the index itself is also compressed while ensuring recall-preserving results. Experiments on six diverse HPC simulation datasets show that Eureka achieves up to 25x data compression and over 300x index compression, surpassing state-of-the-art compressors such as SZ3 and ZFP in rate-distortion performance. Additionally, Eureka delivers over 30x speedup for low-selectivity range queries, making it a scalable and efficient solution for modern scientific data analysis.
Yan, Ning↗
Compressing TV-image data
Compressing technique calculates activity estimator for each segment of image line. Estimator is used in conjunction with allowable bits per line, N, to determine number of bits necessary to code each segment and which segments can tolerate truncation. Preprocessed line data are then passed to adaptive variable-length coder, which selects optimum transmission code. Method increases capacity of broadcast and cable television transmissions and helps reduce size of storage medium for video and digital audio recordings.
Hilbert, E. E.↗
Error Analysis of ZFP Compression for Floating-Point Data
Compression of floating-point data will play an important role in high-performance computing as data bandwidth and storage become dominant costs. Lossy compression of floating-point data is powerful, but theoretical results are needed to bound its errors when used to store look-up tables, simulation results, or even the solution state during the computation. In this paper, we analyze the round-off error introduced by ZFP, a lossy compression algorithm. Here, the stopping criteria for ZFP depends on the compression mode specified by the user: fixed rate, fixed accuracy, or fixed precision [P. Lindstrom, ZFP 0.5.3 Documentation, 2018]. While most of our discussion is focused on the fixed precision mode of ZFP, we establish a bound on the error introduced by all three compression modes. In order to tightly capture the error, first we introduce a vector space that allows us to work with binary representations of components. Under this vector space, we define operators that implement each step of the ZFP compression and decompression to establish a bound on the error caused by ZFP. To conclude, numerical tests are provided to demonstrate the accuracy of the established bounds.
97 MATHEMATICS AND COMPUTING↗
Hardware compression using common portions of data
Methods and devices are provided for data compression. Data compression can include receiving a plurality of data chunks, sampling at least some of the plurality of data chunks extracting a common portion from a number of the plurality of data chunks based on the sampling, and storing a remainder of the plurality of data chunks in memory.
97 MATHEMATICS AND COMPUTING↗
Stability Analysis of Inline ZFP Compression for Floating-Point Data in Iterative Methods
Currently, the dominating constraint in many high performance computing applications is data capacity and bandwidth, in both internode communications and even moreso in intranode data motion. A new approach to address this limitation is to make use of data compression in the form of a compressed data array. Storing data in a compressed data array and converting to standard IEEE-754 types as needed during a computation can reduce the pressure on bandwidth and storage. However, repeated conversions (lossy compression and decompression) introduce additional approximation errors, which need to be shown to not significantly affect the simulation results. Here, we extend recent work that analyzed the error of a single use of compression and decompression of the ZFP compressed data array representation to the case of time-stepping and iterative schemes, where an advancement operator is repeatedly applied in addition to the conversions. We show that the accumulated error for iterative methods involving fixed-point and time evolving iterations is bounded under standard constraints. An upper bound is established on the number of additional iterations required for the convergence of stationary fixed-point iterations. An additional analysis of traditional forward and backward error of stationary iterative methods using ZFP compressed arrays is also presented. The results of several 1D, 2D, and 3D test problems are provided to demonstrate the correctness of the theoretical bounds.
97 MATHEMATICS AND COMPUTING↗
Syndrome source coding and its universal generalization
A method of using error-correcting codes to obtain data compression, called syndrome-source-coding, is described in which the source sequence is treated as an error pattern whose syndrome forms the compressed data. It is shown that syndrome-source-coding can achieve arbitrarily small distortion with the number of compressed digits per source digit arbitrarily close to the entropy of a binary memoryless source. A universal generalization of syndrome-source-coding is formulated which provides robustly-effective, distortionless, coding of source ensembles.
Ancheta, T. C., Jr.↗
Syndrome-source-coding and its universal generalization
A method of using error-correcting codes to obtain data compression, called syndrome-source-coding, is described in which the source sequence is treated as an error pattern whose syndrome forms the compressed data. It is shown that syndrome-source-coding can achieve arbitrarily small distortion with the number of compressed digits per source digit arbitrarily close to the entropy of a binary memoryless source. A 'universal' generalization of syndrome-source-coding is formulated which provides robustly effective distortionless coding of source ensembles. Two examples are given, comparing the performance of noiseless universal syndrome-source-coding to (1) run-length coding and (2) Lynch-Davisson-Schalkwijk-Cover universal coding for an ensemble of binary memoryless sources.
Ancheta, T. C., Jr.↗
Estimating Lossy Compressibility of Scientific Data Using Deep Neural Networks
Simulation based scientific applications generate increasingly large amounts of data on high-performance computing (HPC) systems. To allow data to be stored and analyzed efficiently, data compression is often utilized to reduce the volume and velocity of data. However, a question often raised by domain scientists is the level of compression that can be expected so that they can make more informed decisions, balancing between accuracy and performance. In this letter, we propose a deep neural network based approach for estimating the compressibility of scientific data. To train the neural network, we build both general features as well as compressor-specific features so that the characteristics of both data and lossy compressors are captured in training. Our approach is demonstrated to outperform a prior analytical model as well as a sampling based approach in the case of a biased estimation, i.e., for SZ. However, for the unbiased estimation (i.e., ZFP), the sampling based approach yields the best accuracy, despite the high overhead involved in sampling the target dataset.
97 MATHEMATICS AND COMPUTING↗
An advanced imaging communication system for planetary exploration
An advanced imaging communication system (AICS) for planetary exploration is presented. The system offers 'end-to-end' information rate improvements of 3 to 5 times over existing systems in addition to extensive user flexibility to adapt his rate/fidelity priorities to fit a particular mission. AICS contains two major system elements. The first is a concatenated Reed-Solomon/Viterbi coded channel. This provides a powerful, yet practical, solution to the usual 'error vulnerability' problem associated with compressed data. The second major element is an extremely adaptive image data compression algorithm called RM2. The details of this algorithm as presently simulated are discussed in considerable detail. Used in conjunction with the virtually error free performance of the Reed-Solomon/Viterbi channel the stated AICS advantages are obtained.
Rice, R. F.↗
Data processing assessment for the Lunar Geoscience Observer imaging spectrometer
On the Lunar Geoscience Observer project, a Visible and Infrared Mapping Spectrometer instrument has been proposed. This instrument will have science data input rates in the hundreds of kilobits per second (kbps) and an average telemetry output data rate of 4 kbps. Techniques that can be used to reduce the throughput of the instrument are editing, summing and averaging, data compression, data preprocessing, pattern recognition and snapshot data taking. Due to instrument limitations in the buffer memory size and processing speeds, a careful selection of the available techniques must be made.
Irigoyen, R. E.↗
Adaptive Instrument Module: Space Instrument Controller "Brain" through Programmable Logic Devices
The Adaptive Instrument Module (AIM) will be the first true demonstration of reconfigurable computing with field-programmable gate arrays (FPGAs) in space, enabling the 'brain' of the system to evolve or adapt to changing requirements. In partnership with NASA Goddard Space Flight Center and the Australian Cooperative Research Centre for Satellite Systems (CRC-SS), APL has built the flight version to be flown on the Australian university-class satellite FEDSAT. The AIM provides satellites the flexibility to adapt to changing mission requirements by reconfiguring standardized processing hardware rather than incurring the large costs associated with new builds. This ability to reconfigure the processing in response to changing mission needs leads to true evolveable computing, wherein the instrument 'brain' can learn from new science data in order to perform state-of-the-art data processing. The development of the AIM is significant in its enormous potential to reduce total life-cycle costs for future space exploration missions. The advent of RAM-based FPGAs whose configuration can be changed at any time has enabled the development of the AIM for processing tasks that could not be performed in software. The use of the AIM enables reconfiguration of the FPGA circuitry while the spacecraft is in flight, with many accompanying advantages. The AIM demonstrates the practicalities of using reconfigurable computing hardware devices by conducting a series of designed experiments. These include the demonstration of implementing data compression, data filtering, and communication message processing and inter-experiment data computation. The second generation is the Adaptive Processing Template (ADAPT) which is further described in this paper. The next step forward is to make the hardware itself adaptable and the ADAPT pursues this challenge by developing a reconfigurable module that will be capable of functioning efficiently in various applications. ADAPT will take advantage of radiation tolerant RAM-based field programmable gate array (FPGA) technology to develop a reconfigurable processor that combines the flexibility of a general purpose processor running software with the performance of application specific processing hardware for a variety of high performance computing applications.
Darrin, Ann Garrison↗
Electronic device and method for compressing video data
A computer-implemented method for compressing video data comprises receiving a sequence of video data values, each video data value being a digital value from a successive one of a plurality of pixels that form a video sensor, the sequence of video data values resulting from successive frames of video captured by the video sensor; extracting the video data values for each pixel in turn to create a plurality of pixel data streams, each pixel data stream including the video data value for each frame of captured video for the pixel; and applying data compression to each pixel data stream to create compressed data for each pixel data stream.
Tohlen, Michael Aaron↗
Electronic device and method for compressing video data
A computer-implemented method for compressing video data comprises receiving a sequence of video data values, each video data value being a digital value from a successive one of a plurality of pixels that form a video sensor, the sequence of video data values resulting from successive frames of video captured by the video sensor; extracting the video data values for each pixel in turn to create a plurality of pixel data streams, each pixel data stream including the video data value for each frame of captured video for the pixel; and applying data compression to each pixel data stream to create compressed data for each pixel data stream.
Tohlen, Michael Aaron↗
Competitive Parallel Processing For Compression Of Data
Momentarily-best compression algorithm selected. Proposed competitive-parallel-processing system compresses data for transmission in channel of limited band-width. Likely application for compression lies in high-resolution, stereoscopic color-television broadcasting. Data from information-rich source like color-television camera compressed by several processors, each operating with different algorithm. Referee processor selects momentarily-best compressed output.
Diner, Daniel B.↗
High performance compression of science data
In the future, NASA expects to gather over a tera-byte per day of data requiring space for levels of archival storage. Data compression will be a key component in systems that store this data (e.g., optical disk and tape) as well as in communications systems (both between space and Earth and between scientific locations on Earth). We propose to develop algorithms that can be a basis for software and hardware systems that compress a wide variety of scientific data with different criteria for fidelity/bandwidth tradeoffs. The algorithmic approaches we consider are specially targeted for parallel computation where data rates of over 1 billion bits per second are achievable with current technology.
Storer, James A.↗