Search NASA⌕ Search

SEARCH · Search NASA

Results for “compression algorithm”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Global and Local Distortion Inference During Embedded Zerotree Wavelet Decompression

This paper presents algorithms for inferring global and spatially local estimates of the squared-error distortion measures for the Embedded Zerotree Wavelet (EZW) image compression algorithm. All distortion estimates are obtained at the decoder without significantly compromising EZW's rate-distortion performance. Two methods are given for propagating distortion estimates from the wavelet domain to the spatial domain, thus giving individual estimates of distortion for each pixel of the decompressed image. These local distortion estimates seem to provide only slight improvement in the statistical characterization of EZW compression error relative to the global measure, unless actual squared errors are propagated. However, they provide qualitative information about the asymptotic nature of the error that may be helpful in wavelet filter selection for low bit rate applications.

Huber, A. Kris↗

TTHRESH: Tensor Compression for Multidimensional Visual Data

Memory and network bandwidth are decisive bottlenecks when handling high-resolution multidimensional data sets in visualization applications, and they increasingly demand suitable data compression strategies. In this paper, we introduce a novel lossy compression algorithm for multidimensional data over regular grids. It leverages the higher-order singular value decomposition (HOSVD), a generalization of the SVD to three dimensions and higher, together with bit-plane, run-length and arithmetic coding to compress the HOSVD transform coefficients. Our scheme degrades the data particularly smoothly and achieves lower mean squared error than other state-of-the-art algorithms at low-to-medium bit rates, as it is required in data archiving and management for visualization purposes. Further advantages of the proposed algorithm include very fine bit rate selection granularity and the ability to manipulate data at very small cost in the compression domain, for example to reconstruct filtered and/or subsampled versions of all (or selected parts) of the data set.

97 MATHEMATICS AND COMPUTING↗

A seismic data compression system using subband coding

This article presents a study of seismic data compression techniques and a compression algorithm based on subband coding. The algorithm includes three stages: a decorrelation stage, a quantization stage that introduces a controlled amount of distortion to allow for high compression ratios, and a lossless entropy coding stage based on a simple but efficient arithmetic coding method. Subband coding methods are particularly suited to the decorrelation of nonstationary processes such as seismic events. Adaptivity to the nonstationary behavior of the waveform is achieved by dividing the data into separate blocks that are encoded separately with an adaptive arithmetic encoder. This is done with high efficiency due to the low overhead introduced by the arithmetic encoder in specifying its parameters. The technique could be used as a progressive transmission system, where successive refinements of the data can be requested by the user. This allows seismologists to first examine a coarse version of waveforms with minimal usage of the channel and then decide where refinements are required. Rate-distortion performance results are presented and comparisons are made with two block transform methods.

Kiely, A. B.↗

Multilevel Techniques for Compression and Reduction of Scientific Data-Quantitative Control of Accuracy in Derived Quantities

Although many compression algorithms are focused on preserving pointwise values of the data, application scientists are generally more concerned with derived quantities. Equally well, the user may even be willing to accept a high level of lossiness in the compression provided that the compressed data respect certain invariants, such as mass conservation. In the current work, we develop a mathematical framework and techniques that enable data to be adaptively compressed while maintaining a specified tolerance on a class of user-prescribed quantities. Here, the algorithm is used to augment the functionality of the data reduction package MGARD developed in previous work and the functionality is illustrated by a range of application including data from computational simulation of autocatalytic reaction simulation, turbulent combustion simulation, experimental data obtained from magnetic confinement fusion experiment, and simulation of turbulent flow along a rectangular channel. In each case, we consider one or more relevant quantities of interest and reduce the data so as to preserve these quantities.

97 MATHEMATICS AND COMPUTING↗

Automatic Generation of Algorithms for High-Speed Reliable Lossy Data Compression (Final Report)

Fast reliable data compression is urgently needed for many leading-edge scientific instruments and for exascale high-performance computing applications because they produce vast amounts of data at extremely high rates. The goal of this project has been to develop a framework named LC that is able to automatically generate high-speed lossless and reliable lossy compression and decompression algorithms that can be customized for different kinds of data. The resulting LC framework is freely available on GitHub. To achieve high-speed operation, LC outputs optimized and parallelized CPU and GPU implementations of the generated algorithms. To ensure the quality of lossily compressed data, LC guarantees the user-provided error bound. To be able to customize the compression algorithm to various use cases, LC can synthesize millions of different algorithms and automatically search for the one that works best for the given data. We have already employed LC to create state-of-the-art lossless and lossy compressors for scientific data as well as leading lossless compressors for images. We hope that LC and the customized, fast, reliable, and CPU/GPU-compatible compression algorithms that it can generate will greatly benefit the many scientific applications that need not only high trustworthiness but also high performance.

97 MATHEMATICS AND COMPUTING↗

Benchmarking and Testing of Qualcomm Snapdragon System-on-Chip for JPL Space Applications and Missions

As some space missions become more challenging due to new environments, greater distances, or more limited size, weight, and power (SWaP) constraints, spacecraft avionics must adapt to allow the spacecraft to be more autonomous and agile---eliminating the Spacecraft-Earth-Spacecraft feedback loop whenever possible. Prime examples of such missions include Aerobots (such as Ingenuity with extremely low SWaP constraints and demanding signal/image processing during flight) and landers in possibly hostile environments (such as a Europa lander mission, with limited communication capacity, high latency, and constrained power budget). To address these challenges, JPL worked with Qualcomm to demonstrate the use of their Snapdragon 801 system-on-chip (SoC) onboard the Ingenuity Helicopter on Mars. The Qualcomm Snapdragon SoC contains various subsystems, including an ARM cluster, a Graphics processing unit, a Digital Signal Processing subsystem, a Neural Processing Engine, Image Signal Processing subsystem, among others. Since the success of Ingenuity, JPL is continuing to work with Qualcomm to address other applications of the Snapdragon SoC technology. This includes the deployment of two 855 Snapdragon development boards onboard the International Space Station (ISS) for successful in-situ benchmarking of applications in space (beyond those tested on Ingenuity). In this paper, we will examine the performance of various applications that have been identified to benefit from greater onboard computational capability. These applications include (among others): machine vision algorithms that are expected to be critical in autonomous entry-descent-and-landing scenarios and real-time Aerobot flight navigation; Hyperspectral compression algorithms; Synthetic Aperture Radar Processing along with various instrument processing algorithms. We discuss how the infusion of Qualcomm's Snapdragon SoC is capable of enabling missions that may not have been able to achieve their goals with traditional flight computing. In addition, we also show that for some algorithms, the software implementation on the Snapdragon SoC outperforms traditional FPGA implementations.

Cretu, Vlad↗

Synthetic aperture radar signal data compression using block adaptive quantization

This paper describes the design and testing of an on-board SAR signal data compression algorithm for ESA's ENVISAT satellite. The Block Adaptive Quantization (BAQ) algorithm was selected, and optimized for the various operational modes of the ASAR instrument. A flexible BAQ scheme was developed which allows a selection of compression ratio/image quality trade-offs. Test results show the high quality of the SAR images processed from the reconstructed signal data, and the feasibility of on-board implementation using a single ASIC.

Kuduvalli, Gopinath↗

Joint pattern recognition/data compression concept for ERTS multispectral imaging

This paper describes a new technique which jointly applies clustering and source encoding concepts to obtain data compression. The cluster compression technique basically uses clustering to extract features from the measurement data set which are used to describe characteristics of the entire data set. In addition, the features may be used to approximate each individual measurement vector by forming a sequence of scalar numbers which define each measurement vector in terms of the cluster features. This sequence, called the feature map, is then efficiently represented by using source encoding concepts. A description of a practical cluster compression algorithm is given and experimental results are presented to show trade-offs and characteristics of various implementations. Examples are provided which demonstrate the application of cluster compression to multispectral image data of the Earth Resources Technology Satellite.

Hilbert, E. E.↗

1D-VAR Retrieval Using Superchannels

Since modern ultra-spectral remote sensors have thousands of channels, it is difficult to include all of them in a 1D-var retrieval system. We will describe a physical inversion algorithm, which includes all available channels for the atmospheric temperature, moisture, cloud, and surface parameter retrievals. Both the forward model and the inversion algorithm compress the channel radiances into super channels. These super channels are obtained by projecting the radiance spectra onto a set of pre-calculated eigenvectors. The forward model provides both super channel properties and jacobian in EOF space directly. For ultra-spectral sensors such as Infrared Atmospheric Sounding Interferometer (IASI) and the NPOESS Airborne Sounder Testbed Interferometer (NAST), a compression ratio of more than 80 can be achieved, leading to a significant reduction in computations involved in an inversion process. Results will be shown applying the algorithm to real IASI and NAST data.

Liu, Xu↗

Block adaptive rate controlled image data compression

A block adaptive rate controlled (BARC) image data compression algorithm is described. It is noted that in the algorithm's principal rate controlled mode, image lines can be coded at selected rates by combining practical universal noiseless coding techniques with block adaptive adjustments in linear quantization. Compression of any source data at chosen rates of 3.0 bits/sample and above can be expected to yield visual image quality with imperceptible degradation. Exact reconstruction will be obtained if the one-dimensional difference entropy is below the selected compression rate. It is noted that the compressor can also be operated as a floating rate noiseless coder by simply not altering the input data quantization. Here, the universal noiseless coder ensures that the code rate is always close to the entropy. Application of BARC image data compression to the Galileo orbiter mission of Jupiter is considered.

Rice, R. F.↗

Improving I/O Throughput with PRIMACY: Preconditioning ID-Mapper for Compressing Incompressibility

The ability to efficiently handle massive amounts of data is necessary for the continuing development towards exascale scientific data-mining applications and database systems. Unfortunately, recent years have shown a growing gap between the size and complexity of data produced from scientific applications and the limited I/O bandwidth available on modern high-performance computing systems. Utilizing data compression in order to lower the degree of I/O activity offers a promising means to addressing this problem. However, the standard compression algorithms previously explored for such use offer limited gains on both the end-to-end throughput and storage fronts. In this paper, we introduce an in-situ compression scheme aimed at improving end-to-end I/O throughput as well as reduction of dataset size. Our technique, PRIMACY (Preconditioning Id-MApper for Compressing incompressibility), acts as a preconditioner for standard compression libraries by modifying representation of original floating-point scientific data to increase byte-level repeatability, allowing standard loss less compressors to take advantage of their entropy-based byte-level encoding schemes. We additionally present a theoretical model for compression efficiency in high-performance computing environments and evaluate the efficiency of our approach via comparative analysis. Based on our evaluations on 20 real-world scientific datasets, PRIMACY achieved up to 38% and 22% improvements upon standard end-to-end write and read throughputs respectively in addition to a 25% increase in compression ratios paired with 3-to-4-fold improvement in both compression and decompression throughput over general purpose compressors.

Computer Science↗

VLSI chip-set for data compression using the Rice algorithm

A full custom VLSI implementation of a data compression encoder and decoder which implements the lossless Rice data compression algorithm is discussed in this paper. The encoder and decoder reside on single chips. The data rates are to be 5 and 10 Mega-samples-per-second for the decoder and encoder respectively.

Venbrux, J.↗

Digital mammography, cancer screening: Factors important for image compression

The use of digital mammography for breast cancer screening poses several novel problems such as development of digital sensors, computer assisted diagnosis (CAD) methods for image noise suppression, enhancement, and pattern recognition, compression algorithms for image storage, transmission, and remote diagnosis. X-ray digital mammography using novel direct digital detection schemes or film digitizers results in large data sets and, therefore, image compression methods will play a significant role in the image processing and analysis by CAD techniques. In view of the extensive compression required, the relative merit of 'virtually lossless' versus lossy methods should be determined. A brief overview is presented here of the developments of digital sensors, CAD, and compression methods currently proposed and tested for mammography. The objective of the NCI/NASA Working Group on Digital Mammography is to stimulate the interest of the image processing and compression scientific community for this medical application and identify possible dual use technologies within the NASA centers.

Clarke, Laurence P.↗

ROOT I/O compression improvements for HEP analysis

We overview recent changes in the ROOT I/O system, increasing performance and enhancing it and improving its interaction with other data analysis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment's software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically increased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are significant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space.

Shadura, Oksana↗

Towards Feature-Preserving 2D and 3D Vector Field Compression

The objective of this work is to develop error-bounded lossy compression methods to preserve topological features in 2D and 3D vector fields. Specifically, we explore the preservation of critical points in piecewise linear vector fields. We define the preservation of critical points as, without any false positive, false negative, or false type change in the decompressed data, (1) keeping each critical point in its original cell and (2) retaining the type of each critical point (e.g., saddle and attracting node). The key to our method is to adapt a vertex-wise error bound for each grid point and to compress input data together with the error bound field using a modified lossy compressor. Our compression algorithm can be also embarrassingly parallelized for large data handling and in situ processing. We benchmark our method by comparing it with existing lossy compressors in terms of false positive/negative/type rates, compression ratio, and various vector field visualizations with several scientific applications.

critical points↗

ExaTN

A challenging scientific problem that will benefit from exascale computing is predictive simulation of two- and three-dimensional quantum many-body Hamiltonians in condensed matter physics, together with the related strong entanglement problems in quantum chemistry and quantum computing. Efficient numerical simulations will require new insights into wavefunction compression algorithms that scale on extremely heterogeneous HPC architectures. To address this need, we propose to develop a scalable math library of parallel multi-linear algebraic numerical primitives for hierarchical tensor compression. Our library will subsequently be used for building an efficient, performance-portable HPC framework for simulating strongly correlated quantum systems. In particular, our will will be the first to deliver a massively parallel implementation of the multiscale entanglement renormalization ansatz (MERA), a hierarchical tensor decomposition quantum scheme capable of expressing local expectation values of strongly entangled higher-dimensional quantum systems efficiently. To address the extreme complexity and high computational cost of such schemes, we will automate their parallel implementation by delivering an integrated software framework, ExaTN (Exascale Tensor Networks), that will process hierarchical tensor representations on exascale HPC systems via an asymchronous task-based programming model. As a specific application, our framework will provide scientists with a capability of simulating strongly entangled quantum systems, thus positioning ORNL as a strategic leader in modeling of strongly correlated materials.

McCaskey, AlexanderJ↗

A Framework of Hyperspectral Image Compression using Neural Networks

Hyperspectral image analysis has gained considerable attention due to its wide range of applications. Hyperspectral images provide a vast amount of information about underlying objects in an image by using a large range of the electromagnetic spectrum for each pixel. However, since the same image is taken multiple times using distinct electromagnetic bands, the size of such images tend to be significant, which leads to greater processing requirements. The aim of this paper is to present a proposed framework for image compression and to study the possible effects of spatial compression on quality of unmixing results. Image compression allows us to reduce the dimensionality of an image while still preserving most of the original information, which could lead to faster image processing. This paper presents preliminary results of different training techniques used in Artificial Neural Network (ANN) based compression algorithm.

97 MATHEMATICS AND COMPUTING↗

Toward Quantity-of-Interest Preserving Lossy Compression for Scientific Data

Today's scientific simulations and instruments are producing a large amount of data, leading to difficulties in storing, transmitting, and analyzing these data. While error-controlled lossy compressors are effective in significantly reducing data volumes and efficiently developing databases for multiple scientific applications, they mainly support error controls on raw data, which leaves a significant gap between the data and user's downstream analysis. This may cause unqualified uncertainties in the outcomes of the analysis, a.k.a quantities of interest (QoIs), which are the major concerns of users in adopting lossy compression in practice. In this paper, we propose rigorous mathematical theories to preserve four families of QoIs that are widely used in scientific analysis during lossy compression along with practical implementations. Specifically, we first develop the error control theory for univariate QoIs which are essential for computing physical properties such as kinetic energy, followed by multivariate QoIs that are more commonly used in real-world applications. The proposed method is integrated into a state-of-the-art compression framework in a modular fashion, which could easily adapt to new QoIs and new compression algorithms. Experiments on real-world datasets demonstrate that the proposed method provides faithful error control on important QoIs including kinetic energy, regional average, and isosurface without trials and errors, while offering compression ratios that are up to 4x of the compression ratios provided by state-of-the-art compressors.

Jiao, Pu↗