SEARCH · Search NASA

Results for “data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28

All sources NASA NTRS DOE OSTI

DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗

NASA NTRS2010

Reconfigurable Hardware for Compressing Hyperspectral Image Data

High-speed, low-power, reconfigurable electronic hardware has been developed to implement ICER-3D, an algorithm for compressing hyperspectral-image data. The algorithm and parts thereof have been the topics of several NASA Tech Briefs articles, including Context Modeler for Wavelet Compression of Hyperspectral Images (NPO-43239) and ICER-3D Hyperspectral Image Compression Software (NPO-43238), which appear elsewhere in this issue of NASA Tech Briefs. As described in more detail in those articles, the algorithm includes three main subalgorithms: one for computing wavelet transforms, one for context modeling, and one for entropy encoding. For the purpose of designing the hardware, these subalgorithms are treated as modules to be implemented efficiently in field-programmable gate arrays (FPGAs). The design takes advantage of industry- standard, commercially available FPGAs. The implementation targets the Xilinx Virtex II pro architecture, which has embedded PowerPC processor cores with flexible on-chip bus architecture. It incorporates an efficient parallel and pipelined architecture to compress the three-dimensional image data. The design provides for internal buffering to minimize intensive input/output operations while making efficient use of offchip memory. The design is scalable in that the subalgorithms are implemented as independent hardware modules that can be combined in parallel to increase throughput. The on-chip processor manages the overall operation of the compression system, including execution of the top-level control functions as well as scheduling, initiating, and monitoring processes. The design prototype has been demonstrated to be capable of compressing hyperspectral data at a rate of 4.5 megasamples per second at a conservative clock frequency of 50 MHz, with a potential for substantially greater throughput at a higher clock frequency. The power consumption of the prototype is less than 6.5 W. The reconfigurability (by means of reprogramming) of the FPGAs makes it possible to effectively alter the design to some extent to satisfy different requirements without adding hardware. The implementation could be easily propagated to future FPGA generations and/or to custom application-specific integrated circuits.

Aranki, Nazeeh↗

NASA NTRS1995

Alternate physical formats for storing data in HDF

Since its inception HDF has evolved to meet new demands by the scientific community to support new kinds of data and data structures, larger data sets, and larger numbers of data sets. The first generation of HDF supported simple objects and simple storage schemes. These objects were used to build more complex objects such as raster images and scientific data sets. The second generation of HDF provided alternate methods of storing data elements, making it possible to do such things as store extendible objects with in HDF, to store data externally from HDF files, and support data compression effectively. As we look to the next generation of HDF, we are considering fundamental changes to HDF, including a redefinition of the basic HDF object from a simple object to a more general, higher-level scientific data object that has certain characteristics, such as dimensionality, a more general atomic number type, and attributes. These changes suggest corresponding changes to the HDF file format itself.

Folk, Mike↗

DOE OSTI2025

A comparison between ShapeFit compression and Full-Modelling method with PyBird for DESI 2024 and beyond

DESI aims to provide one of the tightest constraints on cosmological parameters by analysing the clustering of more than thirty million galaxies. However, obtaining such constraints requires special care in validating the methodology and efforts to reduce the computational time required through data compression and emulation techniques. In this work, we perform a rigorous validation of the PyBird power spectrum modelling code with both a traditional emulated Full-Modelling approach and the model-independent ShapeFit compression approach. By using cubic box simulations that accurately reproduce the clustering and precision of the DESI survey, we find that the cosmological constraints from ShapeFit and Full-Modelling are consistent with each other at the ∼ 0.5σ level for the ΛCDM model. Both ShapeFit and Full-Modelling are also consistent with the true ΛCDM simulation cosmology down to a scale of k max = 0.20 hMpc -1 even after including the hexadecapole. For extended models such as the wCDM and the oCDM models, we find that including the hexadecapole can significantly improve the constraints and reduce the modelling errors with the same k max . While their discrepancies between the constraints from ShapeFit and Full-Modelling are more significant than ΛCDM, they remain consistent within 0.7σ. Lastly, we also show that the constraints on cosmological parameters with the correlation function evaluated from PyBird down to s min = 30h -1 Mpc are unbiased and consistent with the constraints from the power spectrum.

79 ASTRONOMY AND ASTROPHYSICS↗

NASA NTRS1990

Data transmission networks

A task order was written by the High Resolution, High Frame Rate Video Technology (HHVT) project engineers to investigate data compression techniques that could be applied to the HHVT system, and both existing and planned downlink/uplink capabilities of the Space Shuttle and Space Station Freedom. The following tasks were included: (1) Investigate signal channel availability and determine both the maximum possible data rate and the average data rate; (2) Identify time blocks for HHVT video transmission assuming time sharing and interruptions in the communication links; (3) Determine the bit error rates to be expected; and (4) Define the transmit and receive interfaces. A summary chart of the data transmission capabilities for Tracking and Data Relay Satellite System (TDRSS), the Space Shuttle, Space Station Freedom, Spacelab, and USLab are also presented.

Alexovich, Robert↗

NASA NTRS2009

TRMM .25 deg x .25 deg Gridded Precipitation Text Product

Since the launch of the Tropical Rainfall Measuring Mission (TRMM), the Precipitation Measurement Missions science team has endeavored to provide TRMM precipitation retrievals in a variety of formats that are more easily usable by the broad science community than the standard Hierarchical Data Format (HDF) in which TRMM data is produced and archived. At the request of users, the Precipitation Processing System (PPS) has developed a .25 x .25 gridded product in an easily used ASCII text format. The entire TRMM mission data has been made available in this format. The paper provides the details of this new precipitation product that is designated with the TRMM designator 3G68.25. The format is packaged into daily files. It provides hourly precipitation information from the TRMM microwave imager (TMI), precipitation radar (PR), and TMI/PR combined rain retrievals. A major advantage of this approach is the inclusion only of rain data, compression when a particular grid has no rain from the PR or combined, and its direct ASCII text format. For those interested only in rain retrievals and whether rain is convection or stratiform, these products provide a huge reduction in the data volume inherent in the standard TRMM products. This paper provides examples of the 3G68 data products and their uses. It also provides information about C tools that can be used to aggregate daily files into larger time samples. In addition, it describes the possibilities inherent in the spatial sampling which allows resampling into coarser spatial sampling. The paper concludes with information about downloading the gridded text data products.

Stocker, Erich↗

NASA NTRS1986

A user's guide for the signal processing software for image and speech compression developed in the Communications and Signal Processing Laboratory (CSPL), version 1

A complete documentation of the software developed in the Communication and Signal Processing Laboratory (CSPL) during the period of July 1985 to March 1986 is provided. Utility programs and subroutines that were developed for a user-friendly image and speech processing environment are described. Additional programs for data compression of image and speech type signals are included. Also, programs for the zero-memory and block transform quantization in the presence of channel noise are described. Finally, several routines for simulating the perfromance of image compression algorithms are included.

Kumar, P.↗

NASA NTRS1986

Archival of Poker Flat MST radar data

The Poker Flat MST (mesosphere stratosphere troposphere) radar has operated from early 1979 to 1985. The data recorded during that time resides on some 1100 magnetic tapes. A second set containing only the derived parameters of velocity, width and signal to noise of the primary echo at each height, plus the noise spectra, occupies another 250 tapes. While processing to generate the compressed data set does correct some known errors, no attempt has been made to identify or remove spurious echoes. Because the Poker Flat data set is such a unique and valuable resource, a proposal is made to archive the data in forms more useful for analysis. The archived data set would contain only the parameters for significant echoes with contamination from airplanes, unwanted ionospheric returns, frequently aliased Doppler signals and other sources removed. The archived set should be as good or better than the example shown and may occupy only 25 to 50 tapes.

Riddle, A. C.↗

NASA NTRS1996

ICASE/LaRC Symposium on Visualizing Time-Varying Data

Time-varying datasets present difficult problems for both analysis and visualization. For example, the data may be terabytes in size, distributed across mass storage systems at several sites, with time scales ranging from femtoseconds to eons. In response to these challenges, ICASE and NASA Langley Research Center, in cooperation with ACM SIGGRAPH, organized the first symposium on visualizing time-varying data. The purpose was to bring the producers of time-varying data together with visualization specialists to assess open issues in the field, present new solutions, and encourage collaborative problem-solving. These proceedings contain the peer-reviewed papers which were presented at the symposium. They cover a broad range of topics, from methods for modeling and compressing data to systems for visualizing CFD simulations and World Wide Web traffic. Because the subject matter is inherently dynamic, a paper proceedings cannot adequately convey all aspects of the work. The accompanying video proceedings provide additional context for several of the papers.

Banks, D. C.↗

DOE OSTI2026

Ensemble Kalman filter for data assimilation coupled with low-resolution computations techniques applied in fluid dynamics

This paper presents an innovative Reduced-order model (ROM) for merging experimental and simulation data using data assimilation (DA) to estimate the "True" state of a fluid dynamics system, leading to more accurate predictions. Our methodology introduces a novel approach by implementing the ensemble Kalman filter (EnKF) within a reduced-dimensional framework, grounded in a robust theoretical foundation and applied to fluid dynamics. To address the substantial computational demands of DA, the proposed ROM employs low-resolution (LR) techniques to drastically reduce computational costs. This innovative approach involves downsampling datasets for DA computations, followed by an advanced reconstruction technique based on low-cost singular value decomposition (lcSVD). The lcSVD method, a key innovation in this paper, has never been applied to DA before and offers a highly efficient way to enhance resolution with minimal computational resources. Our results demonstrate significant reductions in both computation time and RAM usage through these LR techniques without compromising the accuracy of the estimations. For instance, in a turbulent test case, for a data compression rate of 15.9, the LR approach can achieve a speed-up of 13.7 and a RAM compression of 90.9% while maintaining a low relative root mean square error (RRMSE) of 2.6%, compared to 0.8% in the high-resolution (HR) reference. Furthermore, we highlight the effectiveness of the EnKF in estimating and predicting the state of fluid flow systems based on limited observations and given low-fidelity numerical data. This paper highlights the potential of the proposed DA method in fluid dynamics applications, particularly for improving computational efficiency in CFD and related fields. Its ability to balance accuracy with low computational and memory costs makes it especially suitable for large-scale and real-time applications, such as environmental monitoring or engineering design. This method will be incorporated into ModelFLOWs-app.

Data Assimilation↗

NASA NTRS1990

Learning and diagnosing faults using neural networks

Neural networks have been employed for learning fault behavior from rocket engine simulator parameters and for diagnosing faults on the basis of the learned behavior. Two problems in applying neural networks to learning and diagnosing faults are (1) the complexity of the sensor data to fault mapping to be modeled by the neural network, which implies difficult and lengthy training procedures; and (2) the lack of sufficient training data to adequately represent the very large number of different types of faults which might occur. Methods are derived and tested in an architecture which addresses these two problems. First, the sensor data to fault mapping is decomposed into three simpler mappings which perform sensor data compression, hypothesis generation, and sensor fusion. Efficient training is performed for each mapping separately. Secondly, the neural network which performs sensor fusion is structured to detect new unknown faults for which training examples were not presented during training. These methods were tested on a task of fault diagnosis by employing rocket engine simulator data. Results indicate that the decomposed neural network architecture can be trained efficiently, can identify faults for which it has been trained, and can detect the occurrence of faults for which it has not been trained.

Whitehead, Bruce A.↗

NASA NTRS1981

Entropy encoding applied to a video DM bit stream

This paper investigates bandwidth compression of delta modulated video signals using a variety of novel techniques. The video signal is encoded at 16 MHz with an adaptive delta modulator (ADM) using the Song video mode algorithm. One method of achieving bandwidth compression is through the detection and reduction of steady state patterns in the DM bit stream. Further reduction is obtained by deleting the horizontal and vertical drive information. Finally, a modified ADM algorithm is introduced to increase the frequency of steady state patterns. When the modified ADM is combined with entropy coding a 30-50% data compression can be realized.

Barba, J.↗

NASA NTRS1989

Deep space communications, weather effects, and error control

Deep space telemetry is and will remain signal-to-noise limited and vulnerable to interference. A need exists to increase received signal power and decrease noise. This includes going to Ka-band in the mid-1990's to increase directivity. The effects of a wet atmosphere can increase the noise temperature by a factor of 5 or more, even at X-band, but the order of magnitude increase in average data rate obtainable at Ka-band relative to X-band makes the increased uncertainty a good trade. Lowbit error probabilities required by data compression are available both theoretically and practically with coding, at an infinitesimal power penalty rather than the 10 to 15 dB more power required to reduce error probabilities without coding. Advances are coming rapidly in coding, as with the new constraint-length 15 rate 1/4 convolutional code concatenated with the already existing Reed-Solomon code to be demonstrated on Galileo. In addition, high density spacecraft data storage will allow selective retransmissions, even from the edge of the Solar System, to overcome weather effects. In general, deep space communication was able to operate, and will continue to operate, closer to theoretical limits than any other form of communication. These include limits in antenna area and directivity, system noise temperature, coding efficiency, and everything else. The deep space communication links of the mid-90's and beyond will be compatible with new instruments and compression algorithms and represent a sensible investment in an overall end-to-end information system design.

Posner, Edward C.↗

NASA NTRS1995

Method for coding low entrophy data

A method of lossless data compression for efficient coding of an electronic signal of information sources of very low information rate is disclosed. In this method, S represents a non-negative source symbol set, (s(sub 0), s(sub 1), s(sub 2), ..., s(sub N-1)) of N symbols with s(sub i) = i. The difference between binary digital data is mapped into symbol set S. Consecutive symbols in symbol set S are then paired into a new symbol set Gamma which defines a non-negative symbol set containing the symbols (gamma(sub m)) obtained as the extension of the original symbol set S. These pairs are then mapped into a comma code which is defined as a coding scheme in which every codeword is terminated with the same comma pattern, such as a 1. This allows a direct coding and decoding of the n-bit positive integer digital data differences without the use of codebooks.

Yeh, Pen-Shu↗

NASA NTRS1985

An algorithm for computing the number of distinct spectral vectors in thematic mapper data

A computationally efficient method was developed to compute the number of distinct spectral vectors and their frequency of occurrence in Landsat-4 Thematic Mapper (TM) data. The algorithm first partitions the image into spectrally disjoint subsets and then computes the frequency distribution of distinct spectral vectors within each subset from a multidimensional histogram. The overall frequency distribution is tabulated by accumulating the results from each subset. The number of distinct spectral vectors could be used as a measure of potential storage compaction of alternate data representations for data compression, or as a measure of information content in the comparison of spectral band combinations and/or spatial resolutions for an image. Results from processing three 512 x 512 pixel Landsat-4 TM images and one Landsat-4 Multispectral Scanner (MSS) image are presented as examples. An algorithm for computing the frequency distribution of distinct spectral vectors in MSS data is given in the Appendix.

Wharton, S. W.↗

NASA NTRS1988

Identification of visual evoked response parameters sensitive to pilot mental state

Systems analysis techniques were developed and demonstrated for modeling the electroencephalographic (EEG) steady state visual evoked response (ssVER), for use in EEG data compression and as an indicator of mental workload. The study focused on steady state frequency domain stimulation and response analysis, implemented with a sum-of-sines (SOS) stimulus generator and an off-line describing function response analyzer. Three major tasks were conducted: (1) VER related systems identification material was reviewed; (2) Software for experiment control and data analysis was developed and implemented; and (3) ssVER identification and modeling was demonstrated, via a mental loading experiment. It was found that a systems approach to ssVER functional modeling can serve as the basis for eventual development of a mental workload indicator. The review showed how transient visual evoked response (tVER) and ssVER research are related at the functional level, the software development showed how systems techniques can be used for ssVER characterization, and the pilot experiment showed how a simple model can be used to capture the basic dynamic response of the ssVER, under varying loads.

Zacharias, G. L.↗

NASA NTRS1973

Image processing techniques and applications to the Earth Resources Technology Satellite program

The Earth Resources Technology Satellite system is studied, with emphasis on sensors, data processing requirements, and image data compression using the Fast Fourier and Hadamard transforms. The ERTS-A system and the fundamentals of remote sensing are discussed. Three user applications (forestry, crops, and rangelands) are selected and their spectral signatures are described. It is shown that additional sensors are needed for rangeland management. An on-board information processing system is recommended to reduce the amount of data transmitted.

Polge, R. J.↗

NASA NTRS1991

A neural net based architecture for the segmentation of mixed gray-level and binary pictures

A neural-net-based architecture is proposed to perform segmentation in real time for mixed gray-level and binary pictures. In this approach, the composite picture is divided into 16 x 16 pixel blocks, which are identified as character blocks or image blocks on the basis of a dichotomy measure computed by an adaptive 16 x 16 neural net. For compression purposes, each image block is further divided into 4 x 4 subblocks; a one-bit nonparametric quantizer is used to encode 16 x 16 character and 4 x 4 image blocks; and the binary map and quantizer levels are obtained through a neural net segmentor over each block. The efficiency of the neural segmentation in terms of computational speed, data compression, and quality of the compressed picture is demonstrated. The effect of weight quantization is also discussed. VLSI implementations of such adaptive neural nets in CMOS technology are described and simulated in real time for a maximum block size of 256 pixels.

Tabatabai, Ali↗