Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Homomorphic data compression for real time photon correlation analysis

The construction of highly coherent X-ray sources, combined with next-generation detectors that are larger and faster, has enabled new research opportunities across the scientific landscape. Among the techniques that benefit most from these advancements is X-ray photon correlation spectroscopy (XPCS), where faster acquisition unlocks the ability to study faster dynamics within samples. However, faster acquisition on larger detectors also introduces unprecedented challenges for online data processing and offline data storage. Such challenges are particularly prominent for XPCS, where real time analyses require simultaneous calculation of all the previously acquired data in the time series. We present a homomorphic compression scheme to effectively reduce the computational time and memory space required for XPCS analysis. Leveraging similarities in the mathematical expression between a matrix-based compression algorithm and the correlation calculation, our approach allows direct operation on the compressed data without their decompression. The offline compression scheme extends storage capacity by a factor of 40 while preserving key features in the lossy compressed data. Meanwhile, the online compression scheme reduces the computational time to below 1 ms, enabling real time calculation of the correlation functions at kHz framerate. Our demonstration of a homomorphic compression of scientific data provides an effective solution to the big data challenge at coherent light sources. Beyond the example shown in this work, the framework can be extended to facilitate real-time operations directly on a compressed data stream for other techniques.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Physics-Driven Convolutional Autoencoder Approach for CFD Data Compressions: Preprint

With the growing size and complexity of turbulent flow models, data compression approaches are of the utmost importance to analyze, visualize, or restart the simulations. Recently, in-situ autoencoder-based compression approaches have been proposed and shown to be effective at producing reduced representations of turbulent flow data. However, these approaches focus solely on training the model using point-wise sample reconstruction losses that do not take advantage of the physical properties of turbulent flows. In this paper, we show that training autoencoders with additional physics-informed regularizations, e.g., enforcing incompressibility and preserving enstrophy, improves the compression model in three ways: (i) the compressed data better conform to known physics for homogeneous isotropic turbulence without negatively impacting point-wise reconstruction quality, (ii) inspection of the gradients of the trained model uncovers changes to the learned compression mapping that can facilitate the use of explainability techniques, and (iii) as a performance byproduct, training losses are shown to converge up to 12x faster than the baseline model.

auto-encoders↗
NASA NTRS1990

VLSI chip-set for data compression using the Rice algorithm

A full custom VLSI implementation of a data compression encoder and decoder which implements the lossless Rice data compression algorithm is discussed in this paper. The encoder and decoder reside on single chips. The data rates are to be 5 and 10 Mega-samples-per-second for the decoder and encoder respectively.

Venbrux, J.↗
NASA NTRS1993

Cassini radio and plasma wave investigation - Data compression and scientific applications

The Radio and Plasma Wave Science (RPWS) experiment being built for the Cassini spacecraft will study a wide range of plasma and radio wave phenomena in the magnetosphere of Saturn and will also make valuable measurements during the cruise phase and at other encounters. A feature of data from wave receivers is the capability of producing vastly more data than the spacecraft telemetry link is capable of transmitting back to the Earth. Thus, techniques of on-board data compression and data reduction are important. The RPWS instrument has one processor dedicated to data compression tasks.

Woolliscroft, L. J. C.↗
NASA NTRS1993

Cassini radio and plasma wave investigation: Data compression and scientific applications

The Radio and Plasma Wave Science (RPWS) experiment being built for the Cassini spacecraft will study a wide range of plasma and radio wave phenomena in the magnetosphere of Saturn and will also make valuable measurements during the cruise phase and at other encounters. A feature of data from wave receivers is the capability of producing vastly more data than the spacecraft telemetry link is capable of transmitting back to the Earth. Thus, techniques of on-board data compression and data reduction are important. The RPWS instrument has one processor dedicated to data compression tasks.

Woolliscroft, L. J. C.↗
NASA NTRS2009

Fast and Adaptive Lossless On-Board Hyperspectral Data Compression System for Space Applications

Efficient on-board lossless hyperspectral data compression reduces the data volume necessary to meet NASA and DoD limited downlink capabilities. The techniques also improves signature extraction, object recognition and feature classification capabilities by providing exact reconstructed data on constrained downlink resources. At JPL a novel, adaptive and predictive technique for lossless compression of hyperspectral data was recently developed. This technique uses an adaptive filtering method and achieves a combination of low complexity and compression effectiveness that far exceeds state-of-the-art techniques currently in use. The JPL-developed 'Fast Lossless' algorithm requires no training data or other specific information about the nature of the spectral bands for a fixed instrument dynamic range. It is of low computational complexity and thus well-suited for implementation in hardware, which makes it practical for flight implementations of pushbroom instruments. A prototype of the compressor (and decompressor) of the algorithm is available in software, but this implementation may not meet speed and real-time requirements of some space applications. Hardware acceleration provides performance improvements of 10x-100x vs. the software implementation (about 1M samples/sec on a Pentium IV machine). This paper describes a hardware implementation of the JPL-developed 'Fast Lossless' compression algorithm on a Field Programmable Gate Array (FPGA). The FPGA implementation targets the current state of the art FPGAs (Xilinx Virtex IV and V families) and compresses one sample every clock cycle to provide a fast and practical real-time solution for Space applications.

image compression↗
NASA NTRS2013

Effects of Tunable Data Compression on Geophysical Products Retrieved from Surface Radar Observations with Applications to Spaceborne Meteorological Radars

This paper presents results and analyses of applying an international space data compression standard to weather radar measurements that can easily span 8 orders of magnitude and typically require a large storage capacity as well as significant bandwidth for transmission. By varying the degree of the data compression, we analyzed the non-linear response of models that relate measured radar reflectivity and/or Doppler spectra to the moments and properties of the particle size distribution characterizing clouds and precipitation. Preliminary results for the meteorologically important phenomena of clouds and light rain indicate that for a 0.5 dB calibration uncertainty, typical for the ground-based pulsed-Doppler 94 GHz (or 3.2 mm, W-band) weather radar used as a proxy for spaceborne radar in this study, a lossless compression ratio of only 1.2 is achievable. However, further analyses of the non-linear response of various models of rainfall rate, liquid water content and median volume diameter show that a lossy data compression ratio exceeding 15 is realizable. The exploratory analyses presented are relevant to future satellite missions, where the transmission bandwidth is premium and storage requirements of vast volumes of data, potentially problematic.

cloud properties↗
NASA NTRS1989

Orbital maneuvering vehicle teleoperation and video data compression

The Orbital Maneuvering Vehicle (OMV) and concepts of teleoperation and video data compression as applied to OMV design and operation are described. The OMV provides spacecraft delivery, retrieval, reboost, deboost and viewing services, with ground-control or Space Station operation, through autonomous navigation and pilot controlled maneuvers. Communications systems are comprised of S-band RF command, telemetry, and compressed video data links through the TDRSS and GSTDN networks. The control console video monitors display a monochrome image at an update rate of five frames per second. Depending upon the mode of operation selected by the pilot, the video resolution is either 255 x 244 pixels, or 510 x 244 pixels. Since practically all video image redundancy is removed by the compression process, the video reconstruction is particularly sensitive to data transmission bit errors. Concatenated Reed-Solomon and convolution coding are used with helical data interleaving for error detection and correction, and an error-containment process minimizes the propagation of error effects throughout the video image. Video sub-frame replacement is used, in the case of a non-correctable error or error burst, to minimize the visual impact to the pilot.

Jones, Steve↗
DOE OSTI2021

Unbalanced Parallel I/O: An Often-Neglected Side Effect of Lossy Scientific Data Compression

Lossy compression techniques have demonstrated promising results in significantly reducing the scientific data size while guaranteeing the compression error bounds. However, one important yet often neglected side effect of lossy scientific data compression is its impact on the performance of parallel I/O. Our key observation is that the compressed data size is often highly skewed across processes in lossy scientific compression. To understand this behavior, we conduct extensive experiments where we apply three lossy compressors MGARD, ZFP, and SZ, which are specifically designed and optimized for scientific data, to three real-world scientific applications Gray-Scott simulation, WarpX, and XGC. Our analysis result demonstrates that the size of the compressed data is always skewed even if the original data is evenly decomposed among processes. Such skewness widely exists in different scientific applications using different compressors as long as the information density of the data varies across processes. We then systematically study how this side effect of lossy scientific data compression impacts the performance of parallel I/O. We observe that the skewness in the sizes of the compressed data often leads to I/O imbalance, which can significantly reduce the efficiency of I/O bandwidth utilization if not properly handled. In addition, writing data concurrently to a single shared file through MPI-IO library is more sensitive to the unbalanced I/O loads. Therefore, we believe our research community should pay more attention to the unbalanced parallel I/O caused by lossy scientific data compression.

Wang, Xinying↗
DOE OSTI2021

A Reconfigurable Neural Network ASIC for Detector Front-End Data Compression at the HL-LHC

Despite advances in the programmable logic capabilities of modern trigger systems, a significant bottleneck remains in the amount of data to be transported from the detector to off-detector logic where trigger decisions are made. We demonstrate that a neural network (NN) autoencoder model can be implemented in a radiation-tolerant application-specific integrated circuit (ASIC) to perform lossy data compression alleviating the data transmission problem while preserving critical information of the detector energy profile. For our application, we consider the high-granularity calorimeter from the Compact Muon Solenoid (CMS) experiment at the CERN Large Hadron Collider. The advantage of the machine learning approach is in the flexibility and configurability of the algorithm. By changing the NN weights, a unique data compression algorithm can be deployed for each sensor in different detector regions and changing detector or collider conditions. To meet area, performance, and power constraints, we perform quantization-aware training to create an optimized NN hardware implementation. The design is achieved through the use of high-level synthesis tools and the hls4ml framework and was processed through synthesis and physical layout flows based on a low-power (LP)-CMOS 65-nm technology node. The flow anticipates 200 Mrad of ionizing radiation to select gates and reports a total area of 3.6 mm 2 and consumes 95 mW of power. The simulated energy consumption per inference is 2.4 nJ. Furthermore, this is the first radiation-tolerant on-detector ASIC implementation of an NN that has been designed for particle physics applications.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS1970

Entropy and data compression

Optimum compression ratio defined for Markov sources for compression algorithm efficiency and data compression schemes

Farr, E. H.↗
NASA NTRS2004

Feature-Identification and Data-Compression Software

A report discusses the continuing development of Windows Interface for Nominal Displacement Selection (WINDS), a computer program for automated analysis of images of the Sun and planets acquired by scientific instruments aboard spacecraft. WINDS is intended to afford capabilities for identification of features, measurement of displacements and velocities, analysis of terrain and of atmospheres, and synthesis of animation sequences of images of terrains and atmospheres from small sets of samples by use of velocity-based interpolation. A major element of WINDS will be a nonlinear correlator capable of tracking small features in complex image sequences. For dynamic image sequences, the correlator will enable compression of data by factors >100. In processing image data, WINDS will take account of such factors as texture in image data, rotation of features during measurement intervals, effects of viewing and solar illumination angles, and vertical structures of atmospheres. WINDS will also take account of positions, aiming directions, and fields of view of cameras to determine three-dimensional feature structures by use of triangulation and stereoscopic analysis techniques.

Jong, Eric De↗
DOE OSTI2025

A 28 nm multiply-accumulate ASIC architecture for on-chip data compression in MHz frame rate X-ray and electron pixel detectors

Modern X-ray detector systems urgently require compact, efficient, and fast data compression schemes to handle the transmission of big data from pixel arrays, enabling frame rates in the MHz regime. Here, in this work, a data compression ASIC that implements a streaming fixed-length lossy compression scheme is introduced and analyzed, proving the feasibility and benefits of on-chip compression. The compression scheme utilizes a vector matrix product logic, which performs a number of floating-point multiplications, additions, and accumulations. The logic is verified, synthesized, and shown to fit in the area resource available for the X-ray detector under study, which comprises 192 × 168 pixels each of 12-bit width, and having a total area of 20 mm× 20 mm, about 2 mm× 20 mm of which are available for the digital logic. Several system architectures, precisions, and compression ratios ranging from 100 to 250 were analyzed to pave the way for on-chip fixed-length compression (e.g., principal component analysis, singular value decomposition) and data reduction (e.g., azimuthal integration) for X-ray and electron detectors.

Data compression↗
NASA NTRS2003

A High Performance Image Data Compression Technique for Space Applications

A highly performing image data compression technique is currently being developed for space science applications under the requirement of high-speed and pushbroom scanning. The technique is also applicable to frame based imaging data. The algorithm combines a two-dimensional transform with a bitplane encoding; this results in an embedded bit string with exact desirable compression rate specified by the user. The compression scheme performs well on a suite of test images acquired from spacecraft instruments. It can also be applied to three-dimensional data cube resulting from hyper-spectral imaging instrument. Flight qualifiable hardware implementations are in development. The implementation is being designed to compress data in excess of 20 Msampledsec and support quantization from 2 to 16 bits. This paper presents the algorithm, its applications and status of development.

Yeh, Pen-Shu↗