Adaptive buffer design for data compression systems.
Data compression system buffer fill control by adaptive compression algorithm aperture
Schwarz, G. R.↗
SEARCH · Search NASA
Results for “Data compression”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
On the theoretical analysis of data compression techniques.
Data compression techniques, analyzing prediction and interpolation algorithms for time discrete continuous valued stationary Gaussian sequence
Davisson, L. D.↗
Investigation of data compression techniques
Data compression algorithms for processing electocardiogram and vectorcardiogram signals
Source record↗
Study of adaptive methods for data compression of scanner data
The performance of adaptive image compression techniques and the applicability of a variety of techniques to the various steps in the data dissemination process are examined in depth. It is concluded that the bandwidth of imagery generated by scanners can be reduced without introducing significant degradation such that the data can be transmitted over an S-band channel. This corresponds to a compression ratio equivalent to 1.84 bits per pixel. It is also shown that this can be achieved using at least two fairly simple techniques with weight-power requirements well within the constraints of the LANDSAT-D satellite. These are the adaptive 2D DPCM and adaptive hybrid techniques.
Source record↗
Framework of compressive sensing and data compression for 4D-STEM
Four-dimensional Scanning Transmission Electron Microscopy (4D-STEM) is a powerful technique for high-resolution and high-precision materials characterization at multiple length scales, including the characterization of beam-sensitive materials. However, the field of view of 4D-STEM is relatively small, which in absence of live processing is limited by the data size required for storage. Furthermore, the rectilinear scan approach currently employed in 4D-STEM places a resolution- and signal-dependent dose limit for the study of beam sensitive materials. Improving 4D-STEM data and dose efficiency, by keeping the data size manageable while limiting the amount of electron dose, is thus critical for broader applications. Here we introduce a general method for reconstructing 4D-STEM data with subsampling in both real and reciprocal spaces at high fidelity. The approach is first tested on the subsampled datasets created from a full 4D-STEM dataset, and then demonstrated experimentally using random scan in real-space. The same reconstruction algorithm can also be used for compression of 4D-STEM datasets, leading to a large reduction (100 times or more) in data size, while retaining the fine features of 4D-STEM imaging, for crystalline samples.
4D-STEM↗
Data compression system using base values and methods thereof
In some embodiments, a memory controller in a processor includes a base value cache, a compressor, and a metadata cache. The compressor is coupled to the base value cache and the metadata cache. The compressor compresses a data block using at least a base value and delta values. The compressor determines whether the size of the data block exceeds a data block threshold value. Based on the determination of whether the size of the compressed data block generated by the compressor exceeds the data block threshold value, the memory controller transfers only a set of the compressed delta values to memory for storage. A decompressor located in the lower level cache of the processor decompresses the compressed data block using the base value stored in the base value cache, metadata stored in the metadata cache and the delta values stored in memory.
Seyedzadehdelcheh, SeyedMohammad↗
Data compression system using base values and methods thereof
In some embodiments, a memory controller in a processor includes a base value cache, a compressor, and a metadata cache. The compressor is coupled to the base value cache and the metadata cache. The compressor compresses a data block using at least a base value and delta values. The compressor determines whether the size of the data block exceeds a data block threshold value. Based on the determination of whether the size of the compressed data block generated by the compressor exceeds the data block threshold value, the memory controller transfers only a set of the compressed delta values to memory for storage. A decompressor located in the lower level cache of the processor decompresses the compressed data block using the base value stored in the base value cache, metadata stored in the metadata cache and the delta values stored in memory.
Seyedzadehdelcheh, Seyed Mohammad↗
Multimode Data-Compression System
Data-compression system developed to satisfy need for high-speed, high-performance compression of data from sources as diverse as medical images, high-definition television images, audio signals, readouts from scientific instruments, and binary data files. Maximum data-transmission capability of communication channel or storage capacity of storage device multiplied by approximately compression ratio. Various combinations of lossless and lossy compression chosen to suit various data streams.
Fang, Wai-Chi↗
Data compression system
A data compression system is described in which TV PCM data for each line scan is received in the form of a succession of multibit pixel words. All or selected bits of each word are compressed by providing difference values between successive pixel words and coding the difference values of a selected number of pixel words forming a block into a fundamental sequence (FS). The FS, based on its length and the number of words per block, is either transmitted as the compressed data or is used to generate a code FS or its complement is used to generate a code FS bar. When the code FS is generated, its length is compared with the original block PCM and only if the former is the shorter of the two is the code transmitted. Selected bits per pixel word may be compressed, while the remaining bits may be transmitted directly, or some of them may be omitted altogether.
Rice, R. F.↗
Data Compression in the NEXT-100 Data Acquisition System
NEXT collaboration detectors are based on energy measured by an array of photomultipliers (PMT) and topological event filtering based on an array of silicon photomultipliers (SiPMs). The readout of the PMT sensors for low-frequency noise effects and detector safety issues requires a grounded cathode connection that makes the readout AC-couple with variations in the signal baseline. Strict detector requirements of energy resolution better than 1% FWHM require a precise baseline reconstruction that is performed offline for data analysis and detector performance characterization. Baseline variations make it inefficient to apply traditional lossy data compression techniques, such as zero-suppression, that help to minimize data throughput and, therefore, the dead time of the system. However, for the readout of the SiPM sensors with less demanding requirements in terms of accuracy, a traditional zero-suppression is currently applied with a configuration that allows for a compression ratio of around 71%. The third stage in the NEXT detectors program, the NEXT-100 detector, is a 100 kg detector that instruments approximately five times more PMT sensors and twice the number of SiPM sensors than its predecessor, the NEXT-White detector, putting more pressure in the DAQ throughput, expected to be over 900 MB/s with the current configuration, which will worsen the dead time of the acquisition data system. This paper describes the data compression techniques applied to the sensor data in the NEXT-100 detector, which reduces data throughput and minimizes dead time while maintaining the event rate to the level of its predecessor, around 50 Hz.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Data compression
An approach to data compression is discussed in which the effect achieved by compression is evaluated by the closeness of the approach to the minimum possible volume. An attempt is made to systematize the known results on data compression. The review contains: description of methods of data compression based on statistical coding and information theory; application of methods of interpolation and extrapolation; a specific compression method (related to description of the histogram of a sample); some criteria of effectiveness and methods of service information representation; and discussion of models suggested for theoretical analysis.
Babkin, V. F.↗
The theoretical analysis of data compression systems.
Data statistical models and compression techniques, discussing figures of merit, algorithms and effects of imperfect channel
Davisson, L. D.↗
Space data management at the NSSDC (National Space Sciences Data Center): Applications for data compression
The National Space Science Data Center (NSSDC), established in 1966, is the largest archive for processed data from NASA's space and Earth science missions. The NSSDC manages over 120,000 data tapes with over 4,000 data sets. The size of the digital archive is approximately 6,000 gigabytes with all of this data in its original uncompressed form. By 1995 the NSSDC digital archive is expected to more than quadruple in size reaching over 28,000 gigabytes. The NSSDC digital archive is expected to more than quadruple in size reaching over 28,000 gigabytes. The NSSDC is beginning several thrusts allowing it to better serve the scientific community and keep up with managing the ever increasing volumes of data. These thrusts involve managing larger and larger amounts of information and data online, employing mass storage techniques, and the use of low rate communications networks to move requested data to remote sites in the United States, Europe and Canada. The success of these thrusts, combined with the tremendous volume of data expected to be archived at the NSSDC, clearly indicates that innovative storage and data management solutions must be sought and implemented. Although not presently used, data compression techniques may be a very important tool for managing a large fraction or all of the NSSDC archive in the future. Some future applications would consist of compressing online data in order to have more data readily available, compress requested data that must be moved over low rate ground networks, and compress all the digital data in the NSSDC archive for a cost effective backup that would be used only in the event of a disaster.
Green, James L.↗
Selecting a general-purpose data compression algorithm
The National Space Science Data Center's Common Data Formate (CDF) is capable of storing many types of data such as scalar data items, vectors, and multidimensional arrays of bytes, integers, or floating point values. However, regardless of the dimensionality and data type, the data break down into a sequence of bytes that can be fed into a data compression function to reduce the amount of data without losing data integrity and thus remaining fully reconstructible. Because of the diversity of data types and high performance speed requirements, a general-purpose, fast, simple data compression algorithm is required to incorporate data compression into CDF. The questions to ask are how to evaluate and compare compression algorithms, and what compression algorithm meets all requirements. The object of this paper is to address these questions and determine the most appropriate compression algorithm to use within the CDF data management package that would be applicable to other software packages with similar data compression needs.
Mathews, Gary Jason↗
Real time telemetry and data compression
The generation of telemetry and data compression by the flight program was verified. The adequacy of flight program telemetry control and timing is proven by the analysis of simulation laboratory runs of past programs through the use of telemetry. The telemetry data are correctly issued using specific tags called process input/output (P10) tags. Verification is accomplished by ensuring that a specific P10 tag and mode register setting identifies the correct parameter and that the data are properly scaled for subsequent ground station reduction. It was checked that the LVDC telemetry correctly adheres to the general requirements specified by the EDD. Data compression specifications are verified using compressed data from nominal flight simulations and from a series of perturbations designed to test data table overflows, data dump rates, and compression of data for occurrence events.
Source record↗
An Algorithmic and Software Pipeline for Very Large Scale Scientific Data Compression with Error Guarantees
Efficient data compression is becoming increasingly critical for storing scientific data because many scientific applications produce vast amounts of data. This paper presents an end-to-end algorithmic and software pipeline for data compression that guarantees both error bounds on primary data (PD) and derived data, known as Quantities of Interest (QoI).We demonstrate the effectiveness of the pipeline by compressing fusion data generated by a large-scale fusion code, XGC, which produces tens of petabytes of data in a single day. We demonstrate that the compression is conducted by setting aside computational resources known as staging nodes, and does not impact the simulation performance. For efficient parallel I/O, the pipeline uses ADIOS2, which many codes such as XGC already use for their parallel I/O. We show that our approach can compress the data by two orders of magnitude while guaranteeing high accuracy on both the PD and the QoIs. Further, the amount of resources required by compression is a few percent of the resources required by simulation while ensuring that the compression time for each stage is less than the corresponding simulation time.This pipeline consists of three main steps. The first step decomposes the data using domain decomposition into small subdomains. Each subdomain is then compressed independently to achieve a high level of parallelism. The second step uses existing techniques that guarantee error bounds on the primary data for each subdomain. The third step uses a post-processing optimization technique based on Lagrange multipliers to reduce the QoI errors for data corresponding to each subdomain. The Lagrange multipliers generated can be further quantized or truncated to increase the compression level. All of the above characteristics of our approach make it highly practical to apply on-the-fly compression while guaranteeing errors on QoIs that are critical to the scientists.
Banerjee, Tania↗
Ocelot: An Interactive, Efficient Distributed Compression-As-a-Service Platform With Optimized Data Compression Techniques
Large volumes of data generated by scientific simulations, genome sequencing, and other applications need to be moved among clusters for data collection/analysis. Data compression techniques have effectively reduced data storage and transfer costs. However, users' requirements on interactively controlling both data quality and compression ratios are non-trivial to fulfill. Here, we propose a novel Compression-as-a-Service (CaaS) platform called Ocelot with four important contributions: (1) It offers real-time visualization, interactive compression, and transfer of scientific datasets. (2) It incorporates new strategies for compressing diverse types of datasets more effectively than traditional methods. (3) It provides an effective method for estimating the compression ratio and execution time of compression tasks. (4) Experiments on multiple real-world datasets on geographically distributed computers show that Ocelot can significantly improve data transfer efficiency with a performance gain of more than 10x in computing clusters with relatively slow networks.
compression as a service (CaaS)↗
Combined Industry, Space and Earth Science Data Compression Workshop
The sixth annual Space and Earth Science Data Compression Workshop and the third annual Data Compression Industry Workshop were held as a single combined workshop. The workshop was held April 4, 1996 in Snowbird, Utah in conjunction with the 1996 IEEE Data Compression Conference, which was held at the same location March 31 - April 3, 1996. The Space and Earth Science Data Compression sessions seek to explore opportunities for data compression to enhance the collection, analysis, and retrieval of space and earth science data. Of particular interest is data compression research that is integrated into, or has the potential to be integrated into, a particular space or earth science data information system. Preference is given to data compression research that takes into account the scien- tist's data requirements, and the constraints imposed by the data collection, transmission, distribution and archival systems.
Kiely, Aaron B.↗