Search NASA⌕ Search

SEARCH · Search NASA

Results for “DATA COMPRESSOR”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Enhancing ZFP: A Statistical Approach to Understanding and Reducing Error Bias in a Lossy Floating-Point Compression Algorithm

The amount of data generated and gathered in scientific simulations and data collection applications is continuously growing, putting mounting pressure on storage and bandwidth concerns. A means of reducing such issues is data compression; but, lossless data compression is typically ineffective when applied to floating-point data. Thus, users tend to apply a lossy data compressor, which allows for small deviations from the original data. It is essential to understand how the error from lossy compression impacts the accuracy of the data analytics. Thus, we must analyze not only the compression properties but the error as well. In this paper, we provide a statistical analysis of the error caused by ZFP compression, a state-of-the-art, lossy compression algorithm explicitly designed for floating-point data. We show that the error is indeed biased and propose simple modifications to the algorithm to neutralize the bias and further reduce the resulting error.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Automatic Generation of Algorithms for High-Speed Reliable Lossy Data Compression (Final Report)

Fast reliable data compression is urgently needed for many leading-edge scientific instruments and for exascale high-performance computing applications because they produce vast amounts of data at extremely high rates. The goal of this project has been to develop a framework named LC that is able to automatically generate high-speed lossless and reliable lossy compression and decompression algorithms that can be customized for different kinds of data. The resulting LC framework is freely available on GitHub. To achieve high-speed operation, LC outputs optimized and parallelized CPU and GPU implementations of the generated algorithms. To ensure the quality of lossily compressed data, LC guarantees the user-provided error bound. To be able to customize the compression algorithm to various use cases, LC can synthesize millions of different algorithms and automatically search for the one that works best for the given data. We have already employed LC to create state-of-the-art lossless and lossy compressors for scientific data as well as leading lossless compressors for images. We hope that LC and the customized, fast, reliable, and CPU/GPU-compatible compression algorithms that it can generate will greatly benefit the many scientific applications that need not only high trustworthiness but also high performance.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

High-temperature seals for supercritical carbon-dioxide (sCO 2 ) turbines (Final Report)

This is the final report for project DE-FE0031924 titled “High-temperature seals for supercritical carbon-dioxide (sCO 2 ) turbines.” The report provides a summary of the entire project efforts from October 2020 through December 2024 including the high-temperature commercial dry gas seal (DGS) tests and thermal modeling of Task 2, as well as the high-temperature, large-diameter seal design and high-temperature tests of large-diameter seals in Task 3. A key outcome of Task 2 was the testing completion of specially instrumented commercial DGS in the GE-SwRI Apollo sCO 2 compressor (27,000 rpm). Test data from the DGS showed elevated temperatures upwards of 350 o F, which are close to the higher operating temperature limit of the DGS. The temperature measurements provide insight into the expected thermal loads on DGS operating in high-speed sCO 2 compressor and provided test data for validation of an in-house thermal model of the compressor/seal. Under Task 2.0, this report also presents the development of a steady-state conjugate heat-transfer model of the DGS operating in the sCO 2 compressor – a first of its kind model for modeling heat transfer of sCO 2 in an actual operating compressor. The findings of the thermal model show a reasonable match between temperature predictions of the model and the measured temperature data, also pointing out the validity of the approach and assumptions made in modeling the flows, heat transfer coefficients and windage modeling in the rig. Under Task 3.0, this report presents the preliminary design of a large-diameter hybrid face seal (14 inch and 26-inch diameter) for field testing in a land-based GE turbine. The preliminary seal design effort presented in this report under project DE-FE0031924 builds on the development and successful laboratory testing for such large diameter hybrid face seal under the prior DE-FE0024007 project. Key aspects of seal fluid analyses with CFD, mechanical design considerations and assembly considerations in a land-based turbine are presented. Finally, under Task 3.0, this report also presents the continued high-temperature testing of the 14-inch diameter hybrid face seal developed previously under the DE-FE0024007 program. Specifically, test data demonstrating successful non-contact seal operation and seal effective leakage of 0.001-inch with seal inlet temperatures above 700 o F are presented in this report. Successful hybrid seal operation in a laboratory environment for a large diameter (14-inch) seal at temperatures above 700 o F is a major technological milestone for this technology.

01 COAL, LIGNITE, AND PEAT↗
DOE OSTI2024

Tuning the Interpolation Basis in a Multigrid Decomposition for Local Error Control

In the compression of scientific data, error-controlled compressors enable to considerably decrease the size of the dataset while maintaining adequate levels of accuracy. In this paper, we note that multi-level refactoring scheme such as MGARD i) rely on an approximation of the data based on the interpolation of coefficients, ii) estimate the resulting error with global metrics on the dataset. To improve on these two aspects, we propose a method that aims to divide the original dataset into blocks based on their smoothness and refactors each block separately with the most relevant interpolation order. We show the relevance of such a method on tailored datasets and the benefits and challenges when applying it to large scientific data.

Vidal, Nicolas [ORNL]↗
DOE OSTI2025

DFAT: A web-based toolkit for estimating demand flexibility in building-to-grid integration

Demand Flexibility Assessment Tool (DFAT) is an open source web-based tool that estimates the demand flexibility potential of common control strategies in commercial buildings. The toolkit features a demand flexibility estimation tool that contains two calculators, basic and advanced, based on the level of input of customer data. The basic version calculates demand shed metrics for the control strategy “global temperature adjustment” and “cycle on/off compressors” using customer building information, local weather data, and electrical meter data. The advanced version, which uses detailed HVAC equipment data, calculates demand flexibility metrics for control strategies such as static pressure reset, global temperature adjustment, and cycle on/off compressors. In addition to the demand flexibility estimation tool, this toolkit offers a benchmarking tool that helps facility operators, aggregators, and utility resource managers assess demand flexibility opportunities, quantify/verify performance, and compare their performance against that of their peers.

Leong, Michael↗
DOE OSTI2025

Scalable Hybrid Learning Techniques for Scientific Data Compression

Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data (PD), scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Here, this article presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

ITER↗
DOE OSTI2024

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization

Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. These optimizations can improve the compression ratio of SOTA approaches by up to 3.3× under the same data quality loss. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements.

Wang, Daoce↗
DOE OSTI2025

A Survey on Error-Bounded Lossy Compression for Scientific Datasets

Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. They are designed with distinct compression models and principles, such that each of them features particular pros and cons. In this article, we provide a comprehensive survey of emerging error-bounded lossy compression techniques. The key contribution is fourfold. (1) We summarize a novel taxonomy of lossy compression into six classic models. (2) We provide a comprehensive survey of 10 commonly used compression components/modules. (3) We summarized pros and cons of 47 state-of-the-art lossy compressors and present how state-of-the-art compressors are designed based on different compression techniques. (4) We discuss how customized compressors are designed for specific scientific applications and use-cases. We believe this survey is useful to multiple communities including scientific applications, high-performance computing, lossy compression, and big data.

Error-Bounded Lossy Compression↗
DOE OSTI2025

Optimising the processing and storage of visibilities using lossy compression

The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, largely through increasing the number of stations in the array and the frequency span sampled. The two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and the constraints imposed by I/O, as I/O bandwidths drop significantly on cold storage. An example of this is the data deluge expected from the SKA Telescopes of more than 60 PB per day, all to be stored on the buffer filesystem. While compressing the data is an obvious solution, the impacts on the final data products are hard to predict. In this paper, we chose an error-controlled compressor – MGARD – and applied it to simulated SKA-Mid and real pathfinder visibility data, in noise-free and noise-dominated regimes. As the data have an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. MGARD ensures the compression incurred errors adhere to the user-prescribed tolerance. To measure the degradation of images reconstructed using the lossy compressed data, we proposed a list of diagnostic measures, exploring the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images for continuum and spectral line examples. We found relative error bounds of as much as 10%, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS, whereas a 1% error bound (compression ratio of 8) introduces an increase in noise of about an order of magnitude less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a 0.1% error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO, in both the impacts on the images and in the relative flexibility. MGARD provides better compression for similar error bounds and has a host of potentially powerful additional features.

Techniques: interferometric↗
DOE OSTI2025

Advancing 1D thermo-hydraulic tools for large cryogenic facilities

The Cryogenic Division at Fermilab develops large-scale cryogenic systems for particle accelerators and superconducting test facilities. To support design and diagnostics, a Python-based code was created to calculate pressure drops in components such as valves and pipes. This paper presents recent enhancements to the code, including new heat transfer functions that improve the accuracy of thermal and hydraulic predictions. The first application models pressure and temperature changes in PIP-II relief pipes, aiding pipe sizing and protecting relief valves. The second example analyzes heat load evolution in a pipe carrying sub-atmospheric helium, helping interpret temperature sensor data and understand gas return behavior to cold compressors. These improvements significantly expand the tool’s capabilities, offering a practical resource for designing and troubleshooting cryogenic systems under dynamic thermal and flow conditions.

Beckwith, Rosalyn [Fermilab]↗
DOE OSTI2025

Advancing 1D thermo-hydraulic tools for large cryogenic facilities

The Cryogenic Division at Fermilab develops large-scale cryogenic systems for particle accelerators and superconducting magnet test facilities. To support design and diagnostics, a Python-based code was created to calculate pressure drops in components such as valves and pipes. This paper presents recent enhancements to the code, including new heat transfer functions that improve the accuracy of thermal and hydraulic predictions. The first application models pressure and temperature changes in Proton Improvement Plant PIP-II relief pipes, aiding pipe sizing and protecting relief valves. The second example analyses heat load evolution in a pipe carrying sub-atmospheric helium, helping interpret temperature sensor data and understand gas return behavior to cold compressors. These improvements significantly expand the tool's capabilities, offering a practical resource for designing and troubleshooting cryogenic systems under dynamic thermal and flow conditions.

Beckwith, R. [Fermilab]↗
DOE OSTI2025

Eureka: Enabling Fine-Grained Access and Range Queries on Compressed Scientific Data via Data-Index Co-Compression

Handling large-scale scientific data in high-performance computing (HPC) environments poses significant challenges, including excessive I/O, high storage costs, and slow query performance. Traditional approaches often require full data decompression and scans, making them impractical for real-time or interactive analysis. To address these limitations, we introduce Eureka, a unified data-index co-compression framework that enables fine-grained access and efficient range queries on compressed scientific datasets. Eureka integrates spatial domain decomposition with block-wise error-bounded lossy compression to support selective decompression. It constructs a hierarchical AVL-tree index during compression to capture block-level value ranges, enabling fast pruning during query execution. To reduce metadata overhead, the index itself is also compressed while ensuring recall-preserving results. Experiments on six diverse HPC simulation datasets show that Eureka achieves up to 25x data compression and over 300x index compression, surpassing state-of-the-art compressors such as SZ3 and ZFP in rate-distortion performance. Additionally, Eureka delivers over 30x speedup for low-selectivity range queries, making it a scalable and efficient solution for modern scientific data analysis.

Yan, Ning↗
DOE OSTI2025

What to Support When You’re Compressing

Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications’ data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications’ needs and compressors’ capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

Error-Bounded Lossy Compression↗
DOE OSTI2026

QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control

Scientific applications generate an unprecedented volume of data, overwhelming the network and file systems’ bandwidth and posing challenges for efficient and scalable data retrieval and analysis. Progressive data compression offers a promising solution by enabling on-demand retrieval at reduced size. However, existing progressive methods either fail to bound the errors in essential quantities of interest (QoIs) derived from raw data or suffer from suboptimal retrieval efficiency. In this work, we propose QProR, an efficient QoI-based progressive framework that optimizes progressive retrieval for target QoIs. Our key contributions include: (1) a systematic framework that integrates error-controlled lossy compressors with bitplane encoding while decoupling the two processes for high flexibility and adaptability; (2) a novel weighted bitplane encoding method which incorperates QoI knowledge into data refactoring to enhance retrieval efficiency; (3) an optimized retrieval strategy that accounts for the varying impacts of different variables on multivariate QoIs; (4) comprehensive evaluations using six real-world datasets from multiple scientific applications and thorough comparisons against state of the arts. Experimental results demonstrate that QProR achieves up to 80.38% reduction in the retrieval size under the same requested QoI error tolerance, when compared with the best-performing existing methods. When transferring 384 GB of scientific data to remote sites, QProR delivers up to 1.68 × speedup in the end-to-end data transfer performance.

Li, Wenbo [University of Kentucky]↗
DOE OSTI2025

Supporting Special Values in ZFP

This white paper outlines potential approaches to supporting special values in the ZFP numerical compressor without breaking backwards compatibility. Other than infinities and NaNs, special values are often used to indicate the absence of data, where no value is defined, for example by designating finite but extreme “fill values” as special. Such fill values are commonly used in earth system science, among other applications, but if left as is during compression lead to artifacts and loss of precision in nearby true values. Multiple candidate solutions that would allow ZFP to recognize special values are here proposed. Until such support is available, we also sketch available workarounds.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

High throughput screening of ultra-thin electrocaloric materials enabled by additive manufacturing

Current HVAC systems can constitute as much as half of a building’s total energy consumption, depending on its use. More specifically cooling, particularly in hotter climates or for energy intensive applications, like data centers, can comprise between 10-30% of the total energy use. These cooling systems are largely comprised of compressor-based direct expansion systems and chilled water systems that use refrigerants or water, respectively, to reject heat. . Electrocaloric (EC) cycles present an eco-friendly alternative to these conventional processes.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Efficiently predicting pressure-composition-temperature diagrams to discover low-stability metal hydrides

Quantitatively accurate computational predictions of metal hydride thermodynamics are challenging but critical for alloy performance optimization across a multitude of technological domains, including hydrogen storage, compression, purification, and getters. Recent machine learning approaches have demonstrated great success in this area, but can potentially suffer from several shortcomings since they rely on imbalanced experimental training data and can have poor out-of-distribution (ood) test performance. Here, in this study, we circumvent such pitfalls by developing a computationally efficient, first principles-based workflow for direct prediction of metal hydride phase equilibrium, i.e., the pressure-composition-temperature (PCT) diagram. We then demonstrate its utility on predicting low stability hydrides derived from compositionally complex C14 Laves phase AB2 alloys. Specifically, we computationally predict and then experimentally validate an AB 2 alloy series (z < 0.6 for Ti 2−z Zr z CrMnFeNi) with ideal hydriding thermodynamics for a two-stage metal hydride-based compressor for pressurizing boil off from liquefied hydrogen. Importantly, this study lays the groundwork for accurate and efficient discovery/optimization of ood, low-stability hydrides for which purely data-driven approaches lack sufficient accuracy.

08 HYDROGEN↗
DOE OSTI2025

FedCSpc: A Cross-Silo Federated Learning System With Error-Bounded Lossy Parameter Compression

Cross-Silo federated learning is widely used for scaling deep neural network (DNN) training over data silos from different locations worldwide while guaranteeing data privacy. Communication has been identified as the main bottleneck when training large-scale models due to large-volume model parameters and gradient transmission across public networks with limited bandwidth. Most previous works focus on gradient compression, while limited work tries to compress parameters that can not be ignored and extremely affect communication performance during the training. Here, to bridge this gap, we propose FedCSpc: an efficient cross-silo federated learning system with an XAI-driven adaptive parameter compression strategy for large-scale model training. Our work substantially differs from existing gradient compression techniques due to the distinct data features of gradient and parameter. The key contributions of this paper are fourfold. (1) Our designed FedCSpc proposes to compress the parameter during the training using the state-of-the-art error-bounded lossy compressor – SZ3. (2) We develop an adaptive compression error bound adjustment algorithm to guarantee the model accuracy effectively. (3) We exploit an efficient approach to utilize the idle CPU resources of clients to compress the parameters. (4) We perform a comprehensive evaluation with a wide range of models and benchmarks on a GPU cluster with 65 GPUs. Results show that FedCSpc can achieve the same model accuracy as FedAvg while reducing the data volume of parameters and gradients in communication by up to 7.39× and 288×, respectively. With 32 clients on a 4 Gb size model, FedCSpc significantly outperforms FedAvg in wall-clock time in the emulated WAN environment (at the bandwidth of 1 Gbps or lower without loss of generality).

SZ3↗