Search NASA⌕ Search

SEARCH · Search NASA

Results for “lossy data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2002

The CCDS Data Compression Recommendations: Development and Status

The Consultative Committee for Space Data Systems (CCSDS) has been engaging in recommending data compression standards for space applications. The first effort focused on a lossless scheme that was adopted in 1997. Since then, space missions benefiting from this recommendation range from deep space probes to near Earth observatories. The cost savings result not only from reduced onboard storage and reduced bandwidth, but also in ground archive of mission data. In many instances, this recommendation also enables more science data to be collected for added scientific value. Since 1998, the compression sub-panel of CCSDS has been investigating lossy image compression schemes and is currently working towards a common solution for a single recommendation. The recommendation will fulfill the requirements for remote sensing conducted on space platforms.

Yeh, Pen-Shu↗
DOE OSTI2025

Eureka: Enabling Fine-Grained Access and Range Queries on Compressed Scientific Data via Data-Index Co-Compression

Handling large-scale scientific data in high-performance computing (HPC) environments poses significant challenges, including excessive I/O, high storage costs, and slow query performance. Traditional approaches often require full data decompression and scans, making them impractical for real-time or interactive analysis. To address these limitations, we introduce Eureka, a unified data-index co-compression framework that enables fine-grained access and efficient range queries on compressed scientific datasets. Eureka integrates spatial domain decomposition with block-wise error-bounded lossy compression to support selective decompression. It constructs a hierarchical AVL-tree index during compression to capture block-level value ranges, enabling fast pruning during query execution. To reduce metadata overhead, the index itself is also compressed while ensuring recall-preserving results. Experiments on six diverse HPC simulation datasets show that Eureka achieves up to 25x data compression and over 300x index compression, surpassing state-of-the-art compressors such as SZ3 and ZFP in rate-distortion performance. Additionally, Eureka delivers over 30x speedup for low-selectivity range queries, making it a scalable and efficient solution for modern scientific data analysis.

Yan, Ning↗
NASA NTRS1991

Emerging standards for still image compression: A software implementation and simulation study

The software implementation is described of an emerging standard for the lossy compression of continuous tone still images. This software program can be used to compress planetary images and other 2-D instrument data. It provides a high compression image coding capability that preserves image fidelity at compression rates competitive or superior to most known techniques. This software implementation confirms the usefulness of such data compression and allows its performance to be compared with other schemes used in deep space missions and for data based storage.

Pollara, F.↗
NASA NTRS1992

Locally adaptive vector quantization: Data compression with feature preservation

A study of a locally adaptive vector quantization (LAVQ) algorithm for data compression is presented. This algorithm provides high-speed one-pass compression and is fully adaptable to any data source and does not require a priori knowledge of the source statistics. Therefore, LAVQ is a universal data compression algorithm. The basic algorithm and several modifications to improve performance are discussed. These modifications are nonlinear quantization, coarse quantization of the codebook, and lossless compression of the output. Performance of LAVQ on various images using irreversible (lossy) coding is comparable to that of the Linde-Buzo-Gray algorithm, but LAVQ has a much higher speed; thus this algorithm has potential for real-time video compression. Unlike most other image compression algorithms, LAVQ preserves fine detail in images. LAVQ's performance as a lossless data compression algorithm is comparable to that of Lempel-Ziv-based algorithms, but LAVQ uses far less memory during the coding process.

Cheung, K. M.↗
NASA NTRS1998

Data Compression Techniques for Advanced Space Transportation Systems

Advanced space transportation systems, including vehicle state of health systems, will produce large amounts of data which must be stored on board the vehicle and or transmitted to the ground and stored. The cost of storage or transmission of the data could be reduced if the number of bits required to represent the data is reduced by the use of data compression techniques. Most of the work done in this study was rather generic and could apply to many data compression systems, but the first application area to be considered was launch vehicle state of health telemetry systems. Both lossless and lossy compression techniques were considered in this study.

Bradley, William G.↗
DOE OSTI2025

Real Time implementation of Artificial Intelligence compression algorithm for High-Speed Streaming Readout signals

The new generation of high-energy physics experiments plans to acquire data in streaming mode. With this approach, it is possible to access the information of the whole detector (organized in time slices) for optimal and lossless triggering of data acquisitions. With this approach, data rates, especially in large detectors, are often very high, and the network is likely to be the bottleneck for the entire Streaming Read Out system. The aim of this work is to study the implementation of a lossy compression algorithm based on Artificial Intelligence: an Autoencoder. With Machine Learning it is possible to achieve a high compression ratio and fast inference time with only a small degradation of the signals, almost negligible for the specific application. This work explores different configurations of the Autoencoder and the implementation on different hardware. Different Autoencoder configurations are explored to find the best trade-off between compression ratio and reconstruction loss, both for signals and energy spectrum. Different hardware implementations are also explored to find the best platform to achieve real-time performance for the specific application.

Rossi, Fabio (ORCID:0009000385713885)↗
NASA NTRS1993

High-performance compression of astronomical images

Astronomical images have some rather unusual characteristics that make many existing image compression techniques either ineffective or inapplicable. A typical image consists of a nearly flat background sprinkled with point sources and occasional extended sources. The images are often noisy, so that lossless compression does not work very well; furthermore, the images are usually subjected to stringent quantitative analysis, so any lossy compression method must be proven not to discard useful information, but must instead discard only the noise. Finally, the images can be extremely large. For example, the Space Telescope Science Institute has digitized photographic plates covering the entire sky, generating 1500 images each having 14000 x 14000 16-bit pixels. Several astronomical groups are now constructing cameras with mosaics of large CCD's (each 2048 x 2048 or larger); these instruments will be used in projects that generate data at a rate exceeding 100 MBytes every 5 minutes for many years. An effective technique for image compression may be based on the H-transform (Fritze et al. 1977). The method that we have developed can be used for either lossless or lossy compression. The digitized sky survey images can be compressed by at least a factor of 10 with no noticeable losses in the astrometric and photometric properties of the compressed images. The method has been designed to be computationally efficient: compression or decompression of a 512 x 512 image requires only 4 seconds on a Sun SPARCstation 1. The algorithm uses only integer arithmetic, so it is completely reversible in its lossless mode, and it could easily be implemented in hardware for space applications.

White, Richard L.↗
NASA NTRS1993

Digital mammography, cancer screening: Factors important for image compression

The use of digital mammography for breast cancer screening poses several novel problems such as development of digital sensors, computer assisted diagnosis (CAD) methods for image noise suppression, enhancement, and pattern recognition, compression algorithms for image storage, transmission, and remote diagnosis. X-ray digital mammography using novel direct digital detection schemes or film digitizers results in large data sets and, therefore, image compression methods will play a significant role in the image processing and analysis by CAD techniques. In view of the extensive compression required, the relative merit of 'virtually lossless' versus lossy methods should be determined. A brief overview is presented here of the developments of digital sensors, CAD, and compression methods currently proposed and tested for mammography. The objective of the NCI/NASA Working Group on Digital Mammography is to stimulate the interest of the image processing and compression scientific community for this medical application and identify possible dual use technologies within the NASA centers.

Clarke, Laurence P.↗
NASA NTRS1993

Adjustable lossless image compression based on a natural splitting of an image into drawing, shading, and fine-grained components

The compression, or efficient coding, of single band or multispectral still images is becoming an increasingly important topic. While lossy compression approaches can produce reconstructions that are visually close to the original, many scientific and engineering applications require exact (lossless) reconstructions. However, the most popular and efficient lossless compression techniques do not fully exploit the two-dimensional structural links existing in the image data. We describe here a general approach to lossless data compression that effectively exploits two-dimensional structural links of any length. After describing in detail two main variants on this scheme, we discuss experimental results.

Novik, Dmitry A.↗
NASA NTRS1994

System considerations for efficient communication and storage of MSTI image data

The Ballistic Missile Defense Organization has been developing the capability to evaluate one or more high-rate sensor/hardware combinations by incorporating them as payloads on a series of Miniature Seeker Technology Insertion (MSTI) flights. This publication represents the final report of a 1993 study to analyze the potential impact f data compression and of related communication system technologies on post-MSTI 3 flights. Lossless compression is considered alone and in conjunction with various spatial editing modes. Additionally, JPEG and Fractal algorithms are examined in order to bound the potential gains from the use of lossy compression. but lossless compression is clearly shown to better fit the goals of the MSTI investigations. Lossless compression factors of between 2:1 and 6:1 would provide significant benefits to both on-board mass memory and the downlink. for on-board mass memory, the savings could range from $5 million to $9 million. Such benefits should be possible by direct application of recently developed NASA VLSI microcircuits. It is shown that further downlink enhancements of 2:1 to 3:1 should be feasible thorough use of practical modifications to the existing modulation system and incorporation of Reed-Solomon channel coding. The latter enhancement could also be achieved by applying recently developed VLSI microcircuits.

Rice, Robert F.↗
NASA NTRS2000

Effects of Digitization and JPEG Compression on Land Cover Classification Using Astronaut-Acquired Orbital Photographs

Studies that utilize astronaut-acquired orbital photographs for visual or digital classification require high-quality data to ensure accuracy. The majority of images available must be digitized from film and electronically transferred to scientific users. This study examined the effect of scanning spatial resolution (1200, 2400 pixels per inch [21.2 and 10.6 microns/pixel]), scanning density range option (Auto, Full) and compression ratio (non-lossy [TIFF], and lossy JPEG 10:1, 46:1, 83:1) on digital classification results of an orbital photograph from the NASA - Johnson Space Center archive. Qualitative results suggested that 1200 ppi was acceptable for visual interpretive uses for major land cover types. Moreover, Auto scanning density range was superior to Full density range. Quantitative assessment of the processing steps indicated that, while 2400 ppi scanning spatial resolution resulted in more classified polygons as well as a substantially greater proportion of polygons < 0.2 ha, overall agreement between 1200 ppi and 2400 ppi was quite high. JPEG compression up to approximately 46:1 also did not appear to have a major impact on quantitative classification characteristics. We conclude that both 1200 and 2400 ppi scanning resolutions are acceptable options for this level of land cover classification, as well as a compression ratio at or below approximately 46:1. Auto range density should always be used during scanning because it acquires more of the information from the film. The particular combination of scanning spatial resolution and compression level will require a case-by-case decision and will depend upon memory capabilities, analytical objectives and the spatial properties of the objects in the image.

Robinson, Julie A.↗
DOE OSTI2026

Physics-aware adaptive checkpointing with shadow systems for nonlinear PDE simulations

Large-scale simulations of nonlinear partial differential equations (PDEs) that exhibit strongly transient behavior and pattern-forming dynamics produce enormous amounts of data, which, even with modern storage systems, cannot be stored for later curation. Current I/O strategies either write dense time series of snapshots, which is often prohibitive in I/O and storage, or store a few checkpoints that enable restart but incur expensive recomputation cost and provide no control over post-restart error growth, especially when lossy compression is used. Moreover, most, if not all, existing strategies take no account of the actual physical state of the system. Here, we present a simple physics-aware I/O framework in which a low-cost shadow system adaptively triggers lossy checkpoints when the shadow system deviates from the fine-scale simulation. The shadow system can be a coarsened replica of the fine-scale simulation that evolves concurrently. This means that checkpoints are taken based on the physical state of the system: fewer checkpoints are triggered when the system is quiescent while more are taken when the system undergoes a rapid change. This type of behavior is observed in many systems such as Brusselator and FitzHugh–Nagumo. We illustrate that our framework maintains stable restarts, keeps fine-scale restart errors bounded by shadow errors, and reconstructs the time history with significantly lower error and storage than interpolating fixed-interval snapshots, with low-cost shadow replay and modest online synchronization overhead.

Gong, Qian [ORNL] (ORCID:0000000235704142)↗
DOE OSTI2026

QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control

Scientific applications generate an unprecedented volume of data, overwhelming the network and file systems’ bandwidth and posing challenges for efficient and scalable data retrieval and analysis. Progressive data compression offers a promising solution by enabling on-demand retrieval at reduced size. However, existing progressive methods either fail to bound the errors in essential quantities of interest (QoIs) derived from raw data or suffer from suboptimal retrieval efficiency. In this work, we propose QProR, an efficient QoI-based progressive framework that optimizes progressive retrieval for target QoIs. Our key contributions include: (1) a systematic framework that integrates error-controlled lossy compressors with bitplane encoding while decoupling the two processes for high flexibility and adaptability; (2) a novel weighted bitplane encoding method which incorperates QoI knowledge into data refactoring to enhance retrieval efficiency; (3) an optimized retrieval strategy that accounts for the varying impacts of different variables on multivariate QoIs; (4) comprehensive evaluations using six real-world datasets from multiple scientific applications and thorough comparisons against state of the arts. Experimental results demonstrate that QProR achieves up to 80.38% reduction in the retrieval size under the same requested QoI error tolerance, when compared with the best-performing existing methods. When transferring 384 GB of scientific data to remote sites, QProR delivers up to 1.68 × speedup in the end-to-end data transfer performance.

Li, Wenbo [University of Kentucky]↗
DOE OSTI2025

Understanding and Estimating Error Propagation in Neural Networks for Scientific Data Analysis

Neural networks are increasingly integrated into scientific discovery, where input data reduction and model quantization play a key role in accelerating inference. However, understanding and mitigating the impact of these techniques on output error is critical for ensuring reliable results, particularly in tasks demanding high numerical precision. This paper introduces a comprehensive framework for optimizing neural network inference in scientific computing by combining data reduction and weight quantization while maintaining error-controlled outcomes. We develop theoretical analyses to bound error propagation under these reductions and propose a framework that balances computational performance with error constraints. Evaluation on real-world learning-based combustion simulations and satellite image classification demonstrates that our derived error bounds accurately predict observed errors while enabling significant computational speedup under our framework. This work highlights the potential for further leveraging advancements in modern lossy compression algorithms and hardware accelerators that support lower-precision formats.

He, Weiming [New Jersey Institute of Technology]↗
NASA NTRS1992

Compression of color-mapped images

In a standard image coding scenario, pixel-to-pixel correlation nearly always exists in the data, especially if the image is a natural scene. This correlation is what allows predictive coding schemes (e.g., DPCM) to perform efficient compression. In a color-mapped image, the values stored in the pixel array are no longer directly related to the pixel intensity. Two color indices which are numerically adjacent (close) may point to two very different colors. The correlation still exists, but only via the colormap. This fact can be exploited by sorting the color map to reintroduce the structure. The sorting of colormaps is studied and it is shown how the resulting structure can be used in both lossless and lossy compression of images.

Hadenfeldt, A. C.↗
NASA NTRSDate not supplied

Assimilation of Reconstructed Radiances from IASI Principal Component Scores into the GEOS-ADAS

Hyperspectral Infrared sounders such as IASI, AIRS, and CrIS have long been an integral part of radiance assimilation in numerical weather prediction (NWP), providing vertical profiles of water vapor and temperature information. Principal Component Scores (PCS) are a lossy form of compression that retains most information, such as temperature and moisture, by using a large training set of atmospheric profiles. However, PCS may not well represent profiles which are rare events, such as volcanic eruptions, and drops some sources of random noise. There has been an increased interest in the use of PCS as EUMETSAT plans to distribute future geostationary sounder radiances from MTG-IRS via PCS only. NWP centers use two approaches to deal with PCS: direct assimilation of the PCS by modifying the radiative transfer model to produce PCS and the associated Jacobians, or a simpler approach of decompressing the PCS and reconstructing the radiances back into channel space to allow assimilating radiances without modifications to the data assimilation system. EUMETSAT has developed a PCS product for IASI that has been operational since 2011. We utilize this product opting for the simpler approach, decompressing IASI PCS into channel space, and assimilating those radiances using the GEOS-ADAS. We then compare this with a control using the standard IASI radiance product. Resulting differences in global forecast statistics, differences in Forecast Sensitivity to Observation Impact, along with implications for implementation and quality control are discussed.

Bryan M. Karpowicz↗
NASA NTRS1992

Efficient transfer of images over networks

Effective remote observing requires sending large images over long distances. The usual approach to the transfer problem is to require high bandwidth transmission links, which are expensive to install and operate. An alternative approach is to use existing low-bandwidth connections, such as phone lines or the Internet, in a highly efficient manner by compressing the images. The combined use of existing low-cost infrastructure and standard networking software means that remote observing can be made practical even for small observatories with limited network resources. The authors have implemented such a scheme based on the H-transform compression method developed for astronomical images, which are often resistant to compression because they are noisy. The H-transform can be used for either lossy or lossless compression, and compression factors of at least 10 can be achieved with no noticeable losses in the astrometric or photometric properties of the compressed images. The H-transform allows us to organize the information in an image so that the 'useful' information can be sent first, followed by the noise, which makes up the bulk of the transmission. The receiver can invert a partially received set of H-coefficients, creating an image that improves with time. The H-transform is particularly well-suited to this style of incremental reconstruction, because the spatially localized nature of the basis functions of the H-transorm prevents the appearance of artifacts such as ringing around point sources and edges. The authors' implementation uses the WIYN Telescope Control System's TCP-based communications protocol. An 800x800 16-bit astronomical image was sent over a 2400 baud connection, which would normally take about 71 minutes; after only 60 seconds, the partially received H-transform produced an image that did not differ appreciably from the original. This poster presents a quantification of the efficiencies, as well as examples of images reconstructed from partial data.

Percival, J. W.↗