Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

Compression of tokamak boundary plasma simulation data using a maximum volume algorithm for matrix skeleton decomposition

This report demonstrates satisfactory data compression of SOLPS-ITER simulation output ranging from 2D fields, 1D profiles, and 0D scalar variables with a novel matrix decomposition approach. The singular value decomposition (SVD) scales poorly for large matrix sizes and is unsuited to the application on high dimensional data common to fusion plasma physics simulation. In this work, we employ the columns-submatrix-rows (CUR) matrix factorization technique in order to compute a low-rank approximation up to two orders of magnitude faster than the SVD, but within a nominal L2-norm relative error of ε = 10 –2 . In addition, the CUR approach maintains the original format of the data, in its extracted columns and rows, allowing for interpretable data storage at the original resolution of the simulation. We utilize an iterative algorithm to compute the CUR decomposition of simulation output by maximizing the volume, or linearly independent information content, of a low-rank submatrix contained within the data. Experiments over $\textit{n} × \textit{n}$ randomized test matrices with embedded rank-deficient features show that this maximum volume implementation of CUR matrix approximation has reduced asymptotic computational complexity on the order of n compared to the SVD, which scales approximately as $n^3$. These results show that the CUR technique can be used to effectively select time step snapshots (columns) of over 140 SOLPS-ITER output variables and the associated discretized coordinate timeseries (rows) allowing for reconstruction of the complete simulation dynamics.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Fast digital lossy compression for X-ray ptychographic data

Increases in X-ray brightness from synchrotron light sources lead to a requirement for higher frame rates from hybrid pixel array detectors (HPADs), while also favoring charge integration over photon counting. However, transfer of the full uncompressed data will begin to constrain detector design, as well as limit the achievable continuous frame rate. Here a data compression scheme that is easy to implement in a HPAD's application-specific integrated circuit (ASIC) is described, and how different degrees of compression affect image quality in ptychography, a commonly employed coherent imaging method, is examined. Using adaptive encoding quantization, it is shown in simulations that one can digitize signals up to 16383 photons per pixel (corresponding to 14 bits of information) using only 8 or 9 bits for data transfer, with negligible effect on the reconstructed image.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Lossless Compression of Data into Fixed-Length Packets

A computer program effects lossless compression of data samples from a one-dimensional source into fixed-length data packets. The software makes use of adaptive prediction: it exploits the data structure in such a way as to increase the efficiency of compression beyond that otherwise achievable. Adaptive linear filtering is used to predict each sample value based on past sample values. The difference between predicted and actual sample values is encoded using a Golomb code.

Kiely, Aaron B.↗

Vector quantization for volume rendering

Volume rendering techniques typically process volumetric data in raw, uncompressed form. As algorithmic and architectural advances improve rendering speeds, however, larger data sets will be evaluated requiring consideration of data storage and transmission issues. In this paper, we analyze the data compression requirements for volume rendering applications and present a solution based on vector quantization. The proposed system compresses volumetric data and then renders images directly from the new data format. Tests on a fluid flow data set demonstrate that good image quality may be achieved at a compression ratio of 17:1 with only a 5 percent cost in additional rendering time.

Ning, Paul↗

Scientific requirements for space science data systems

In the 1990's space plasma physics studies will increasingly involve correlative analysis of observations from multiple instruments and multiple spacecraft. The solar terrestrial physics missions in the 1990's will be designed around simultaneous observations from spacecraft monitoring the solar wind, the polar magnetosphere and the near and distant magnetotail. Within these regions, clusters of spacecraft flying in formation will increasingly involve comparative magnetospheric studies. No single lab will have the expertise to process and analyze all of the different types of data so the data repositories will be distributed. Catalog and browse systems will be required to help select events for study. Data compression techniques may be useful in designing the data bases used for selecting events for study. Data compression onboard the spacecraft will be necessary, since instrument data rates will be much larger than available telemetry rates. However, considerable care will be necessary to avoid losing valuable data when applying data compression algorithms.

Walker, Raymond J.↗

Improved Compression of Wavelet-Transformed Images

A recently developed data-compression method is an adaptive technique for coding quantized wavelet-transformed data, nominally as part of a complete image-data compressor. Unlike some other approaches, this method admits a simple implementation and does not rely on the use of large code tables. A common data compression approach, particularly for images, is to perform a wavelet transform on the input data, and then losslessly compress a quantized version of the wavelet-transformed data. Under this compression approach, it is common for the quantized data to include long sequences, or runs, of zeros. The new coding method uses prefixfree codes for the nonnegative integers as part of an adaptive algorithm for compressing the quantized wavelet-transformed data by run-length coding. In the form of run-length coding used here, the data sequence to be encoded is parsed into strings consisting of some number (possibly 0) of zeros, followed by a nonzero value. The nonzero value and the length of the run of zeros are encoded. For a data stream that contains a sufficiently high frequency of zeros, this method is known to be more effective than using a single variable length code to encode each symbol. The specific prefix-free codes used are from two classes of variable-length codes: a class known as Golomb codes, and a class known as exponential-Golomb codes. The codes within each class are indexed by a single integer parameter. The present method uses exponential-Golomb codes for the lengths of the runs of zeros, and Golomb codes for the nonzero values. The code parameters within each code class are determined adaptively on the fly as compression proceeds, on the basis of statistics from previously encoded values. In particular, a simple adaptive method has been devised to select the parameter identifying the particular exponential-Golomb code to use. The method tracks the average number of bits used to encode recent runlengths, and takes the difference between this average length and the code parameter. When this difference falls outside a fixed range, the code parameter is updated (increased or decreased). The Golomb code parameter is selected based on the average magnitude of recently encoded nonzero samples. The coding method requires no floating- point operations, and more readily adapts to local statistics than other methods. The method can also accommodate arbitrarily large input values and arbitrarily long runs of zeros. In practice, this means that changes in the dynamic range or size of the input data set would not require a change to the compressor. The algorithm has been tested in computational experiments on test images. A comparison with a previously developed algorithm that uses large code tables (generated via Huffman coding on training data) suggests that the data-compression effectiveness of the present algorithm is comparable to the best performance achievable by the previously developed algorithm.

Kiely, Aaron↗

Studies of optical and biological properties of terrestrial land cover using multispectral linear array technology

A series of experiments to study the optical and biological properties of terrestrial land cover are planned for late 1987 using a six-channel imaging spectroradiometer based on newly developed multispectral linear array (MLA) detector technology. Data from selected portions of the Sahel and rain forests of Africa and South America will be used to delineate biomass classes and estimate spherical albedos. A spatial resolution of 15 meters in the four visible-near IR channels and 30 meters in two shortwave IR channels, including a 'new' channel centered at 1.24 micrometers when combined with a spectral width of 20 nm for all channels, will be used to investigate possible improvements in land cover classification. Technology demonstrations include a test of data compression on data quality, the first spaceborne utilization of short wave infrared Schottky barrier Pd2Si detector arrays, and the use of close-butted, multi-array modules with attached spectral filters.

Barnes, W. L.↗

Compressing Aviation Data in XML Format

Design, operations and maintenance activities in aviation involve analysis of variety of aviation data. This data is typically in disparate formats making it difficult to use with different software packages. Use of a self-describing and extensible standard called XML provides a solution to this interoperability problem. XML provides a standardized language for describing the contents of an information stream, performing the same kind of definitional role for Web content as a database schema performs for relational databases. XML data can be easily customized for display using Extensible Style Sheets (XSL). While self-describing nature of XML makes it easy to reuse, it also increases the size of data significantly. Therefore, transfemng a dataset in XML form can decrease throughput and increase data transfer time significantly. It also increases storage requirements significantly. A natural solution to the problem is to compress the data using suitable algorithm and transfer it in the compressed form. We found that XML-specific compressors such as Xmill and XMLPPM generally outperform traditional compressors. However, optimal use of Xmill requires of discovery of optimal options to use while running Xmill. This, in turn, depends on the nature of data used. Manual disc0ver.y of optimal setting can require an engineer to experiment for weeks. We have devised an XML compression advisory tool that can analyze sample data files and recommend what compression tool would work the best for this data and what are the optimal settings to be used with a XML compression tool.

Patel, Hemil↗

Transform coding for space applications

Data compression coding requirements for aerospace applications differ somewhat from the compression requirements for entertainment systems. On the one hand, entertainment applications are bit rate driven with the goal of getting the best quality possible with a given bandwidth. Science applications are quality driven with the goal of getting the lowest bit rate for a given level of reconstruction quality. In the past, the required quality level has been nothing less than perfect allowing only the use of lossless compression methods (if that). With the advent of better, faster, cheaper missions, an opportunity has arisen for lossy data compression methods to find a use in science applications as requirements for perfect quality reconstruction runs into cost constraints. This paper presents a review of the data compression problem from the space application perspective. Transform coding techniques are described and some simple, integer transforms are presented. The application of these transforms to space-based data compression problems is discussed. Integer transforms have an advantage over conventional transforms in computational complexity. Space applications are different from broadcast or entertainment in that it is desirable to have a simple encoder (in space) and tolerate a more complicated decoder (on the ground) rather than vice versa. Energy compaction with new transforms are compared with the Walsh-Hadamard (WHT), Discrete Cosine (DCT), and Integer Cosine (ICT) transforms.

Glover, Daniel↗

Screening IMS Seismic Detections using Dynamic Correlation Processor: Progress report for 2020

LLNL completed the FY20 workplan entailing application of the Dynamic Correlation Processor (DCP) to International Monitoring System (IMS) primary seismic data in 2019 and 2020. The effective analysis period was curtailed to January 1 through November 12, 2019 due to a data compression change that rendered later data unusable. Fixing the data compression error is a priority for our data source, and we plan to extend the period of data analysis when the fix is implemented. We found that the percentage overlap between detections generated by the International Data Centre (IDC) DFX software and detections generated by DCP varied widely by station. Overlap is 30-40% at some stations, suggesting that a large percentage of DFX detections could be screened (removed) from the detection-event association algorithm. However, detection overlap was much lower, a few percent, at many stations and the percentage overlap was low at stations known to have many mines and other local sources of seismicity. Unexpectedly low detection overlap at many stations prompted us to examine results for the ARCES station in detail. There is considerable mine activity near ARCES, but our initial FY20 analysis found only ~2% DCP/DFX detection overlap. Differences in the pre-processing (e.g. beam recipes) used by DCP and DFX appear to be the cause of low detection overlap at ARCES. DCP uses wideband filters to improve detector robustness and the IDC uses narrow-band pre-filtering to improve sensitivity. After reprocessing ARCES using the IDC beam recipe, DCP/DFX overlap increased to ~74%. This result shows that we must reprocess the IMS network using the IDC station-specific beam recipes if we are to effectively screen DFX detections and improve automatic event building performance at the IDC.

58 GEOSCIENCES↗

Subband/transform functions for image processing

Functions for image data processing written for use with the MATLAB(TM) software package are presented. These functions provide the capability to transform image data with block transformations (such as the Walsh Hadamard) and to produce spatial frequency subbands of the transformed data. Block transforms are equivalent to simple subband systems. The transform coefficients are reordered using a simple permutation to give subbands. The low frequency subband is a low resolution version of the original image, while the higher frequency subbands contain edge information. The transform functions can be cascaded to provide further decomposition into more subbands. If the cascade is applied to all four of the first stage subbands (in the case of a four band decomposition), then a uniform structure of sixteen bands is obtained. If the cascade is applied only to the low frequency subband, an octave structure of seven bands results. Functions for the inverse transforms are also given. These functions can be used for image data compression systems. The transforms do not in themselves produce data compression, but prepare the data for quantization and compression. Sample quantization functions for subbands are also given. A typical compression approach is to subband the image data, quantize it, then use statistical coding (e.g., run-length coding followed by Huffman coding) for compression. Contour plots of image data and subbanded data are shown.

Glover, Daniel↗

Image Compression Algorithm Altered to Improve Stereo Ranging

A report discusses a modification of the ICER image-data-compression algorithm to increase the accuracy of ranging computations performed on compressed stereoscopic image pairs captured by cameras aboard the Mars Exploration Rovers. (ICER and variants thereof were discussed in several prior NASA Tech Briefs articles.) Like many image compressors, ICER was designed to minimize a mean-square-error measure of distortion in reconstructed images as a function of the compressed data volume. The present modification of ICER was preceded by formulation of an alternative error measure, an image-quality metric that focuses on stereoscopic-ranging quality and takes account of image-processing steps in the stereoscopic-ranging process. This metric was used in empirical evaluation of bit planes of wavelet-transform subbands that are generated in ICER. The present modification, which is a change in a bit-plane prioritization rule in ICER, was adopted on the basis of this evaluation. This modification changes the order in which image data are encoded, such that when ICER is used for lossy compression, better stereoscopic-ranging results are obtained as a function of the compressed data volume.

Kiely, Aaron↗

Total variation vs L1 regularization: a comparison of compressive sensing optimization methods for chemical detection

One of the fundamental assumptions of compressive sensing (CS) is that a signal can be reconstructed from a small number of samples by solving an optimization problem with the appropriate regularization term. Two standard regularization terms are the L1 norm and the total variation (TV) norm. We present a comparison of CS reconstruction results based on these two approaches in the context of chemical detection, and we demonstrate that optimization based on the L1 norm outperforms optimization based on the TV norm. Our comparison is driven by CS sampling, reconstruction, and chemical detection in two real-world datasets: the Physical Sciences Inc. Fabry-Perot interferometer sensor multispectral dataset and the Johns Hopkins Applied Physics Lab FTIR-based longwave infrared sensor hyperspectral dataset. Both datasets contain the release of a chemical simulant such as glacial acetic acid, triethyl phosphate, and sulfur hexafluoride. For chemical detection we use the adaptive coherence estimator (ACE) and bulk coherence, and we propose algorithmic ACE thresholds to define the presence or absence of a chemical of interest in both un-compressed data cubes and reconstructed data cubes. The un-compressed data cubes provide an approximate ground truth. We demonstrate that optimization based on either the L1 norm or TV norm results in successful chemical detection at a compression rate of 90%, but we show that L1 optimization is preferable. We present quantitative comparisons of chemical detection on reconstructions from the two methods, with an emphasis on the number of pixels with an ACE value above the threshold.

Hyperspectral imagery, L1-norm, total variation, A↗

Effects of compression and random noise on multispectral data.

Two problem areas are often encountered in the use of multispectral scanner data: (1) large quantities of man and machine time are required to analyze and store the volumes of data gathered and (2) the quality of the data is usually reduced by the introduction of unwanted random noise. This paper will present some effects of data compression and random noise on multispectral data, particularly as they apply to pattern recognition and picture quality.

Ready, P. J.↗

Real-time and post-hoc compression for data from Distributed Acoustic Sensing

Distributed Acoustic Sensing (DAS) is an emerging sensing technology that records the strain-rate along fiber optic cables at high spatial and temporal resolution. This technique is becoming a popular tool in seismology, hydrology, and other subsurface monitoring applications. However, due to the large coverage (10’s of km) and high density of measurements (1m spacing at 100’s of Hz), a DAS installation could produce terabytes of data records per day. Because many DAS instruments are deployed in remote locations, this large data size poses significant challenges to its transfer and storage. In this paper, we explore lossless compression methods to reduce the storage requirement in both real-time and post-hoc scenarios. Here we propose a two-stage compression method to improve the compression ratio and compression speed. This two-stage compression method could reduce the storage requirement by 40%, which is 20% more than other lossless methods, such as ZSTD. We demonstrate that the compression method could complete its operation well before the DAS instrument needs to output the next file, making it suitable for real-time DAS acquisition. We also implement a parallel compression method for a post-hoc scenario and demonstrate that our method could effectively utilize a parallel computer. With 256 CPU cores, our parallel compression method achieves the speed of 26GB/second.

58 GEOSCIENCES↗

A VLSI design for universal noiseless coding

The practical, noiseless and efficient data-compression technique presented involves a conceptual VLSI design which is capable of meeting real-time processing rates and meets low-power, low-weight, and small-volume requirements. This form of data compression is applicable to image data compression aboard future low-budget spaceflight missions, for such instruments as visual-IR mapping spectrometers and high-resolution imaging spectrometers.

Lee, Jun-Ji↗

Bounding Methods for Heterogeneous Lidar-derived Navigational Geofences

Safe Unmanned Aerial Vehicle (UAV) operations near the ground require navigation methods that avoid fixed obstacles such as buildings, power lines and trees. Aerial lidar surveys of ground structures are available with the precision and accuracy to geolocate obstacles, but the high volume of raw survey data can exceed the compute power of onboard processors and the rendering ability of ground-based flight planning maps. Representing ground structures with bounding polyhedra instead of point clouds greatly reduces the data size and can enable effective obstacle avoidance, as long as the bounding geometry envelopes the structures with high spatial fidelity. This report describes in detail four methods to compute bounding geometries of ground obstacles from lidar point clouds. The four methods are: 1) 2.5D Maximum Elevation Box, 2) 2.5D Ground Map Extrusion, 3) 3D Bounding Cylinder, and 4) 3D Bounding Box. The methods are applied to five point cloud datasets from lidar surveys of UAV flight research sites in Georgia and Virginia with an average point spacing that ranges from 0.1m to 0.6m. The methods are assessed using survey areas with geometrically heterogeneous ground structures: buildings, vegetation, power lines, and sub-meter structures such as road signs and guy wires. The 2.5D Maximum Elevation Box method is useful for simple structures. The 2.5D Ground Map Extrusion method efficiently encloses vegetation, but requires handdrawn ground footprints. The 3D Bounding Cylinder method excels at enclosing linear structures such as power lines and fences. The 3D Bounding Box method excels at enclosing planar structures such as buildings. The methods are compared on the basis of data compression and boundary fidelity on selected areas. The 2.5D methods yield the highest data compression but the polyhedra produced by them enclose significant amounts of empty space. Boundary fidelity is superior for the 3D methods, though this fidelity comes at the cost of a roughly thirtyfold lower data compression ratio than the 2.5D Maximum Elevation Box method. A mix of these output geometries is proposed for autonomous UAV navigation with limited on-board computing. Both the accuracy and spatial detail of emerging satellite-based survey technology lower than that of aerial lidar scanning survey technology. Sub-meter structures and thin linear structures are not reliably mapped at present by satellite-based surveys.

Moore, Andrew J.↗

Inverse Method for Estimation of Composite Kink-Band Toughness from Open-Hole Compression Strength Data

Fiber-reinforced polymer matrix composite materials can fail by kink-band propagation mechanism when subjected to in-plane compressive loading. This mode of failure is especially prevalent in compressive loading of laminates with holes, cut-outs, or impact damage. Most of the successful models for predicting compressive strength of such laminates require “fracture” toughness associated with kink-band propagation under in-plane compression. However, this property is difficult to measure experimentally, limiting the use of such models in design practice. In this paper an inverse method is proposed to estimate the kink-band toughness of the laminate from its open-hole compression strength data, which is an easier property to measure experimentally. Furthermore, a scaling relationship is proposed to estimate kink-band toughness for other laminate configurations of the same material.

Luke Borkowski↗