Search NASA⌕ Search

SEARCH · Search NASA

Results for “lossless compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)

Abstract. Geoscientific models and measurements generate false precision (scientifically meaningless data bits) that wastes storage space. False precision can mislead (by implying noise is signal) and be scientifically pointless, especially for measurements. By contrast, lossy compression can be both economical (save space) and heuristic (clarify data limitations) without compromising the scientific integrity of data. Data quantization can thus be appropriate regardless of whether space limitations are a concern. We introduce, implement, and characterize a new lossy compression scheme suitable for IEEE floating-point data. Our new Bit Grooming algorithm alternately shaves (to zero) and sets (to one) the least significant bits of consecutive values to preserve a desired precision. This is a symmetric, two-sided variant of an algorithm sometimes called Bit Shaving that quantizes values solely by zeroing bits. Our variation eliminates the artificial low bias produced by always zeroing bits, and makes Bit Grooming more suitable for arrays and multi-dimensional fields whose mean statistics are important. Bit Grooming relies on standard lossless compression to achieve the actual reduction in storage space, so we tested Bit Grooming by applying the DEFLATE compression algorithm to bit-groomed and full-precision climate data stored in netCDF3, netCDF4, HDF4, and HDF5 formats. Bit Grooming reduces the storage space required by initially uncompressed and compressed climate data by 25–80 and 5–65 %, respectively, for single-precision values (the most common case for climate data) quantized to retain 1–5 decimal digits of precision. The potential reduction is greater for double-precision datasets. When used aggressively (i.e., preserving only 1–2 digits), Bit Grooming produces storage reductions comparable to other quantization techniques such as Linear Packing. Unlike Linear Packing, whose guaranteed precision rapidly degrades within the relatively narrow dynamic range of values that it can compress, Bit Grooming guarantees the specified precision throughout the full floating-point range. Data quantization by Bit Grooming is irreversible (i.e., lossy) yet transparent, meaning that no extra processing is required by data users/readers. Hence Bit Grooming can easily reduce data storage volume without sacrificing scientific precision or imposing extra burdens on users.

58 GEOSCIENCES↗

Real-Time Adaptive Lossless Hyperspectral Image Compression using CCSDS on Parallel GPGPU and Multicore Processor Systems

The proposed CCSDS (Consultative Committee for Space Data Systems) Lossless Hyperspectral Image Compression Algorithm was designed to facilitate a fast hardware implementation. This paper analyses that algorithm with regard to available parallelism and describes fast parallel implementations in software for GPGPU and Multicore CPU architectures. We show that careful software implementation, using hardware acceleration in the form of GPGPUs or even just multicore processors, can exceed the performance of existing hardware and software implementations by up to 11x and break the real-time barrier for the first time for a typical test application.

realtime↗

Optimized clustering estimators for BAO measurements accounting for significant redshift uncertainty

Here, we determine an optimized clustering statistic to be used for galaxy samples with significant redshift uncertainty, such as those that rely on photometric redshifts. To do so, we study the baryon acoustic oscillation (BAO) information content as a function of the orientation of galaxy clustering modes with respect to their angle to the line-of-sight (LOS). The clustering along the LOS, as observed in a redshift-space with significant redshift uncertainty, has contributions from clustering modes with a range of orientations with respect to the true LOS. For redshift uncertainty $\sigma_z \geq 0.02(1+z)$ we find that while the BAO information is confined to transverse clustering modes in the true space, it is spread nearly evenly in the observed space. Thus, measuring clustering in terms of the projected separation (regardless of the LOS) is an efficient and nearly lossless compression of the signal for $\sigma_z \geq 0.02(1+z)$. For reduced redshift uncertainty, a more careful consideration is required. We then use more than 1700 realizations (combining two separate sets) of galaxy simulations mimicking the Dark Energy Survey Year 1 sample to validate our analytic results and optimized analysis procedure. We find that using the correlation function binned in projected separation, we can achieve uncertainties that are within 10 per cent of those predicted by Fisher matrix forecasts. We predict that DES Y1 should achieve a 5 per cent distance measurement using our optimized methods. We expect the results presented here to be important for any future BAO measurements made using photometric redshift data.

79 ASTRONOMY AND ASTROPHYSICS↗

Extreme digitization for ground-based cosmic microwave background experiments

The large size of the time-ordered data of cosmic microwave background experiments presents challenges for mission planning and data analysis. These issues are particularly significant for Antarctica- and space-based experiments, which depend on satellite links to transmit data. We explore the viability of reducing the time-ordered data to a few bit numbers to address these challenges. Unlike lossless compression, few bit digitization introduces additional noise into the data. We present a set of one, two, and three bit digitization schemes and measure the increase in noise in the cosmic microwave background temperature and polarization power spectra. The digitization noise is independent of angular scale and is well described as a constant percentage of the original detector noise. Three bit digitization increases the map noise level by <2 percent, while reducing the data volume by a factor of 10 relative to 32-bit floats. Extreme digitization is a promising strategy for upcoming experiments.

79 ASTRONOMY AND ASTROPHYSICS↗

HPDR: High-Performance Portable Scientific Data Reduction Framework

The rapid growth in scientific data generation is outpacing advancements in computing systems necessary for efficient storage, transfer, and analysis, particularly in the context of exascale computing. With the deployment of first-generation exascale computing systems and next-generation experimental facilities, this gap is widening and necessitates effective data reduction techniques to manage enormous data volumes. Over the past decade, various data reduction methods, including lossless compression, error-controlled lossy compression, and data refactoring, have been developed to accelerate I/O in scientific workflows. Despite significant reductions in data volume, these methods introduce considerable computational overhead, which can become the new bottleneck in data processing. To mitigate this, GPU-accelerated data reduction algorithms have been introduced. However, challenges remain in their integration into exascale workflows, including limited portability across different GPU architectures, substantial memory transfer overhead, and reduced scalability on dense multi-GPU systems. To address these challenges, we propose HPDR, a high-performance and portable data reduction framework. HPDR is designed to enable the execution of state-of-the-art reduction algorithms across diverse processor architectures while reducing memory transfer overhead to 2.3 % of the original, resulting in up to 3.5× faster throughput compared to existing solutions. It also achieves up to 96% of the theoretical speedup in multi-GPU settings. In addition, evaluations on accelerating I/O operations at scale up to 1,024 nodes of the Frontier supercomputer demonstrate that HPDR can achieve up to 103 TB/s reduction throughput, providing up to 4× acceleration in parallel I/O performance compared to existing data reduction routines. This work highlights the potential of HPDR to significantly enhance data reduction efficiency in exascale computing environments.

Chen, Jieyang [University of Oregon]↗

FELIX-Based Readout of the Single-Phase ProtoDUNE Detector

Large liquid argon (LAr) time projection chambers (TPCs) have been adopted for the Deep Underground Neutrino Experiment (DUNE) experiment's far detector, which will be composed of four 17-kton detectors situated 1.5 km underground at the Sanford Underground Research Facility. This represents a large increase in scale compared to existing experiments. Both single- and dual-phase technologies will be validated at CERN, in cryostats capable of accommodating full-size detector modules, and exposed to low-energy charged particle beams. This program, called ProtoDUNE, also allows for extensive tests of data acquisition strategies. The Front-End LInk eXchange (FELIX) readout system was initially developed within the ATLAS collaboration and is based on custom field-programmable gate array (FPGA)-based Peripheral Component Interconnect Express input/output cards, connected through point-to-point links to the detector front end and hosted in commodity servers. FELIX will be used in the single-phase ProtoDUNE setup to read the data coming from 2560 anode wires organized in a single anode plane assembly (APS) structure. With a continuous readout at a sampling rate of 2 MHz, the system must deal with an input rate of 96 Gb/s. An external trigger will preselect time windows of 5 ms with interesting activity expected inside the detector. Event building will occur for triggered events, at a target rate of 25 Hz; the readout system will form fragments from the data samples matching the time window, carry out lossless compression, and forward the data to event building nodes over 10-Gb/s Ethernet. This paper discusses the design and implementation of this readout system as well as the first operational experience.

47 OTHER INSTRUMENTATION↗

Propagation Pattern for Moment Representation of the Lattice Boltzmann Method

A propagation pattern for the moment representation of the regularized lattice Boltzmann method (LBM) in three dimensions is presented. Using effectively lossless compression, the simulation state is stored as a set of moments of the lattice Boltzmann distribution function, instead of the distribution function itself. An efficient cache-aware propagation pattern for this moment representation has the effect of substantially reducing both the storage and memory bandwidth required for LBM simulations. This article extends recent work with the moment representation by expanding the performance analysis on central processing unit (CPU) architectures, considering how boundary conditions are implemented, and demonstrating the effectiveness of the moment representation on a graphics processing unit (GPU) architecture.

42 ENGINEERING↗

Enhancing ZFP: A Statistical Approach to Understanding and Reducing Error Bias in a Lossy Floating-Point Compression Algorithm

The amount of data generated and gathered in scientific simulations and data collection applications is continuously growing, putting mounting pressure on storage and bandwidth concerns. A means of reducing such issues is data compression; but, lossless data compression is typically ineffective when applied to floating-point data. Thus, users tend to apply a lossy data compressor, which allows for small deviations from the original data. It is essential to understand how the error from lossy compression impacts the accuracy of the data analytics. Thus, we must analyze not only the compression properties but the error as well. In this paper, we provide a statistical analysis of the error caused by ZFP compression, a state-of-the-art, lossy compression algorithm explicitly designed for floating-point data. We show that the error is indeed biased and propose simple modifications to the algorithm to neutralize the bias and further reduce the resulting error.

97 MATHEMATICS AND COMPUTING↗

Design and Implementation of an In?Cache Archival Entropy Coder for the Sonoma Persistent Surveillance System

The Sonoma sensor, developed at Lawrence Livermore National Laboratory, is an electro-optical (EO) sensor composed of a tiled arrangement of cameras, where each camera sends 20 megabytes (MB) of raw data per frame. Depending on the sensor configuration, each sensor image, in aggregate, can total up to 320 MB. I discuss here the design and implementation of a high-speed entropy coder utilizing the Lead-1 encoding method, intended to quickly and losslessly compress sensor data before it is written to storage. More sophisticated (and slower) compression techniques may be performed off-line. The coder, favoring speed over coding efficiency, is simple in principle and tiny in practice: the amount of allocated memory required by the coder can be as small as 26 bytes. On a 2.7 GHz PowerMac G5 this coder is capable of encoding sensor data at a rate of 163 megabytes per second. The overall compression rate is better than that of BZIP2, GZIP, and a range coder, and the total execution speed is superior.

99 GENERAL AND MISCELLANEOUS↗

High spatial resolution Fe xii observations of solar active regions

We use UV spectral observations of active regions with the Interface Region Imaging Spectrograph (IRIS) to investigate the properties of the coronal Fe xii 1349.4 Å emission at unprecedented high spatial resolution (∼0.33″). We find that by using appropriate observational strategies (i.e., long exposures, lossless compression), Fe xii emission can be studied with IRIS at high spatial and spectral resolution, at least for high-density plasma (e.g., post-flare loops and active region moss). We find that upper transition region (TR; moss) Fe xii emission shows very small average Doppler redshifts (v{sub D} ∼ 3 km s{sup −1}) as well as modest non-thermal velocities (with an average of ∼24 km s{sup −1} and the peak of the distribution at ∼15 km s{sup −1}). The observed distribution of Doppler shifts appears to be compatible with advanced three-dimensional radiative MHD simulations in which impulsive heating is concentrated at the TR footpoints of a hot corona. While the non-thermal broadening of Fe xii 1349.4 Å peaks at similar values as lower resolution simultaneous Hinode Extreme Ultraviolet Imaging Spectrometer (EIS) measurements of Fe xii 195 Å, IRIS observations show a previously undetected tail of increased non-thermal broadening that might be suggestive of the presence of subarcsecond heating events. We find that IRIS and EIS non-thermal line broadening measurements are affected by instrumental effects that can only be removed through careful analysis. Our results also reveal an unexplained discrepancy between observed 195.1/1349.4 Å Fe xii intensity ratios and those predicted by the CHIANTI atomic database.

79 ASTRONOMY AND ASTROPHYSICS↗

A very high speed lossless compression/decompression chip set

A chip is described that will perform lossless compression and decompression using the Rice Algorithm. The chip set is designed to compress and decompress source data in real time for many applications. The encoder is designed to code at 20 M samples/second at MIL specifications. That corresponds to 280 Mbits/second at maximum quantization or approximately 500 Mbits/second under nominal conditions. The decoder is designed to decode at 10 M samples/second at industrial specifications. A wide range of quantization levels is allowed (4...14 bits) and both nearest neighbor prediction and external prediction are supported. When the pre and post processors are bypassed, the chip set performs high speed entropy coding and decoding. This frees the chip set from being tied to one modeling technique or specific application. Both the encoder and decoder are being fabricated in a 1.0 micron CMOS process that has been tested to survive 1 megarad of total radiation dosage. The CMOS chips are small, only 5 mm on a side, and both are estimated to consume less than 1/4 of a Watt of power while operating at maximum frequency.

Venbrux, Jack↗

Real-time transmission of digital video using variable-length coding

Huffman coding is a variable-length lossless compression technique where data with a high probability of occurrence is represented with short codewords, while 'not-so-likely' data is assigned longer codewords. Compression is achieved when the high-probability levels occur so frequently that their benefit outweighs any penalty paid when a less likely input occurs. One instance where Huffman coding is extremely effective occurs when data is highly predictable and differential coding can be applied (as with a digital video signal). For that reason, it is desirable to apply this compression technique to digital video transmission; however, special care must be taken in order to implement a communication protocol utilizing Huffman coding. This paper addresses several of the issues relating to the real-time transmission of Huffman-coded digital video over a constant-rate serial channel. Topics discussed include data rate conversion (from variable to a fixed rate), efficient data buffering, channel coding, recovery from communication errors, decoder synchronization, and decoder architectures. A description of the hardware developed to execute Huffman coding and serial transmission is also included. Although this paper focuses on matters relating to Huffman-coded digital video, the techniques discussed can easily be generalized for a variety of applications which require transmission of variable-length data.

Bizon, Thomas P.↗

End-to-end communication test on variable length packet structures utilizing AOS testbed

This paper describes a communication test, which successfully demonstrated the transfer of losslessly compressed images in an end-to-end system. These compressed images were first formatted into variable length Consultative Committee for Space Data Systems (CCSDS) packets in the Advanced Orbiting System Testbed (AOST). The CCSDS data Structures were transferred from the AOST to the Radio Frequency Simulations Operations Center (RFSOC), via a fiber optic link, where data was then transmitted through the Tracking and Data Relay Satellite System (TDRSS). The received data acquired at the White Sands Complex (WSC) was transferred back to the AOST where the data was captured and decompressed back to the original images. This paper describes the compression algorithm, the AOST configuration, key flight components, data formats, and the communication link characteristics and test results.

Miller, Warner H.↗

A packet data compressor

We are in the preliminary stages of creating an operational system for losslessly compressing packet data streams. The end goal is to reduce costs. Real world constraints include transmission in the presence of error, tradeoffs between the costs of compression and the costs of transmission and storage, and imperfect knowledge of the data streams to be transmitted. The overall method is to bring together packets of similar type, split the data into bit fields, and test a large number of compression algorithms. Preliminary results are very encouraging, typically offering compression factors substantially higher than those obtained with simpler generic byte stream compressors, such as Unix Compress and HA 0.98.

Grunes, Mitchell R.↗

Incorporating the APS Catalog of the POSS I and Image Archive in ADS

The primary purpose of this contract was to develop the software to both create and access an on-line database of images from digital scans of the Palomar Sky Survey. This required modifying our DBMS (called Star Base) to create an image database from the actual raw pixel data from the scans. The digitized images are processed into a set of coordinate-reference index and pixel files that are stored in run-length files, thus achieving an efficient lossless compression. For efficiency and ease of referencing, each digitized POSS I plate is then divided into 900 subplates. Our custom DBMS maps each query into the corresponding POSS plate(s) and subplate(s). All images from the appropriate subplates are retrieved from disk with byte-offsets taken from the index files. These are assembled on-the-fly into a GIF image file for browser display, and a FITS format image file for retrieval. The FITS images have a pixel size of 0.33 arcseconds. The FITS header contains astrometric and photometric information. This method keeps the disk requirements manageable while allowing for future improvements. When complete, the APS Image Database will contain over 130 Gb of data. A set of web pages query forms are available on-line, as well as an on-line tutorial and documentation. The database is distributed to the Internet by a high-speed SGI server and a high-bandwidth disk system. URL is http://aps.umn.edu/IDB/. The image database software is written in perl and C and has been compiled on SGI computers with MIX5.3. A copy of the written documentation is included and the software is on the accompanying exabyte tape.

Humphreys, Roberta M.↗

The Imager for Mars Pathfinder Insurance Pan

The Imager for Mars Pathfinder (IMP) obtained a full panorama of the Sagan Memorial Station landing site on Sol 2, before the IMP mast was deployed. The images in this panorama were taken in 4 filters (including stereo) and losslessly compressed to provide a high-quality multispectral survey of the landing site even if the IMP mast did not successfully deploy; this data set was therefore called the Insurance Pan. It was completed late in the afternoon of Sol 2, just before the IMP mast was (successfully) deployed. The data were stored in memory and returned to Earth after it became clear that downlink rates were higher than expected. The Insurance Pan horizontal (azimuth) coverage is nearly complete, with gaps caused by pointing errors and data packet losses. Stereo data were acquired in the blue (445 nm) filter, as well as right-eye green (531 nm), orange (600 nm), and near-infrared (752 nm) data.

K E Herkenhoff↗

NASA Tech Briefs, March 2010

Topics covered include: Software Tool Integrating Data Flow Diagrams and Petri Nets; Adaptive Nulling for Interferometric Detection of Planets; Reducing the Volume of NASA Earth-Science Data; Reception of Multiple Telemetry Signals via One Dish Antenna; Space-Qualified Traveling-Wave Tube; Smart Power Supply for Battery-Powered Systems; Parallel Processing of Broad-Band PPM Signals; Inexpensive Implementation of Many Strain Gauges; Constant-Differential-Pressure Two-Fluid Accumulator; Inflatable Tubular Structures Rigidized with Foams; Power Generator with Thermo-Differential Modules; Mechanical Extraction of Power From Ocean Currents and Tides; Nitrous Oxide/Paraffin Hybrid Rocket Engines; Optimized Li-Ion Electrolytes Containing Fluorinated Ester Co-Solvents; Probabilistic Multi-Factor Interaction Model for Complex Material Behavior; Foldable Instrumented Bits for Ultrasonic/Sonic Penetrators; Compact Rare Earth Emitter Hollow Cathode; High-Precision Shape Control of In-Space Deployable Large Membrane/Thin-Shell Reflectors; Rapid Active Sampling Package; Miniature Lightweight Ion Pump; Cryogenic Transport of High-Pressure-System Recharge Gas; Water-Vapor Raman Lidar System Reaches Higher Altitude; Compact Ku-Band T/R Module for High-Resolution Radar Imaging of Cold Land Processes; Wide-Field-of-View, High-Resolution, Stereoscopic Imager; Electrical Capacitance Volume Tomography with High-Contrast Dielectrics; Wavefront Control and Image Restoration with Less Computing; Polarization Imaging Apparatus; Stereoscopic Machine-Vision System Using Projected Circles; Metal Vapor Arcing Risk Assessment Tool; Performance Bounds on Two Concatenated, Interleaved Codes; Parameterizing Coefficients of a POD-Based Dynamical System; Confidence-Based Feature Acquisition; Algorithm for Lossless Compression of Calibrated Hyperspectral Imagery; Universal Decoder for PPM of any Order; Algorithm for Stabilizing a POD-Based Dynamical System; Mission Reliability Estimation for Repairable Robot Teams; Processing AIRS Scientific Data Through Level 3; Web-Based Requesting and Scheduling Use of Facilities; AutoGen Version 5.0; Time-Tag Generation Script; PPM Receiver Implemented in Software; Tropospheric Emission Spectrometer Product File Readers; Reporting Differences Between Spacecraft Sequence Files; Coordinating "Execute" Data for ISS and Space Shuttle; Database for Safety-Oriented Tracking of Chemicals; Apparatus for Cold, Pressurized Biogeochemical Experiments; Growing B Lymphocytes in a Three-Dimensional Culture System; Tissue-like 3D Assemblies of Human Broncho-Epithelial Cells; Isolation of Resistance-Bearing Microorganisms; Oscillating Cell Culture Bioreactor; and Liquid Cooling/Warming Garment.

Source record↗

NASA Tech Briefs, August 2006

Topics covered include: Measurement and Controls Data Acquisition System IMU/GPS System Provides Position and Attitude Data Using Artificial Intelligence to Inform Pilots of Weather Fast Lossless Compression of Multispectral-Image Data Developing Signal-Pattern-Recognition Programs Implementing Access to Data Distributed on Many Processors Compact, Efficient Drive Circuit for a Piezoelectric Pump; Dual Common Planes for Time Multiplexing of Dual-Color QWIPs; MMIC Power Amplifier Puts Out 40 mW From 75 to 110 GHz; 2D/3D Visual Tracker for Rover Mast; Adding Hierarchical Objects to Relational Database General-Purpose XML-Based Information Managements; Vaporizable Scaffolds for Fabricating Thermoelectric Modules; Producing Quantum Dots by Spray Pyrolysis; Mobile Robot for Exploring Cold Liquid/Solid Environments; System Would Acquire Core and Powder Samples of Rocks; Improved Fabrication of Lithium Films Having Micron Features; Manufacture of Regularly Shaped Sol-Gel Pellets; Regulating Glucose and pH, and Monitoring Oxygen in a Bioreactor; Satellite Multiangle Spectropolarimetric Imaging of Aerosols; Interferometric System for Measuring Thickness of Sea Ice; Microscale Regenerative Heat Exchanger Protocols for Handling Messages Between Simulation Computers Statistical Detection of Atypical Aircraft Flights NASA's Aviation Safety and Modeling Project Multimode-Guided-Wave Ultrasonic Scanning of Materials Algorithms for Maneuvering Spacecraft Around Small Bodies Improved Solar-Radiation-Pressure Models for GPS Satellites Measuring Attitude of a Large, Flexible, Orbiting Structure

Source record↗