Search NASA⌕ Search

SEARCH · Search NASA

Results for “Hybrid compression algorithm”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Lossy Compression: An Online Multi-Stage Technology for High-Fidelity Synchro- Waveform Measurements

Effective real-time monitoring and analysis of distributed grids necessitate the use of synchro-waveform measurements, which capture almost all high-frequency disturbances and transient phenomena. However, due to limitations in high-speed measurements and network bandwidth, it is challenging to transfer all high-fidelity synchro-waveforms losslessly and successfully. To cope with these challenges, a hybrid-based online multi-stage compression algorithm is proposed to significantly improve the compression efficiency for synchro-waveform measurements. Initially, the multiple discrete Wavelet transformation is deployed to deconstruct the waveform components. The delta encoding is further developed to decrease the magnitude. In conjunction with the Lempel-Ziv-Markov chain, the hybrid compression algorithm is implemented to achieve real-time compression for the synchro-waveform measurements. Moreover, an innovative error index that synergizes the time and frequency domain error and correlation is formulated to evaluate the waveform distortion. By integrating compression ratio, suitable parameters can be optimally selected. Finally, the simulation, laboratory experiments, as well as field tests across a spectrum of sampling frequencies and time intervals are conducted to substantiate the efficacy of the proposed method. Here, the outcomes demonstrated that a compression ratio of approximately 15.5 and 17.83 can be reached for 0.5 s and 1 s data under both offline and online scenarios, which equates to a substantial 93.5% to 94.39% reduction in data storage requirements.

High-fidelity synchro-waveform measurements↗
DOE OSTI2024

A hybrid finite volume method and smoothed particle hydrodynamics approach for efficient and accurate blast simulations

Modeling strong shock waves in fluids remains a persistent challenge in computational physics. Essential to research efforts in industry and defense, numerous methods have been devised to improve the accuracy and efficiency of shock simulations. A novel, hybrid Finite Volume Method (FVM)-Smoothed Particle Hydrodynamics (SPH) approach is capable of further improving efficiency and retaining accuracy by exploiting the favorable characteristics of each respective method. This hybrid approach is presented for shock capturing in compressible fluids. The Python framework Pyro2 is employed to simulate a coarse FVM mesh, while the Python framework PySPH is utilized to model the fluid in regions with high gradients through SPH particles. The performance of the hybrid FVM-SPH scheme, compared to the individual FVM and SPH methods, is assessed in 1 kt and 10 kt blast simulations. Our results indicate that the hybrid approach offers higher computational efficiency than SPH while preserving its accuracy and characteristics. The hybrid approach had a relative speedup of 11.3x and 22.3x over the FVM and SPH approaches for the 1 kt simulation and a relative speedup of 14.7x and 20.9x over the FVM and SPH approaches for the 10 kt simulation. The hybrid SPH algorithm enables future compressible fluid simulations with more extensive capabilities than grid-based methods alone, presenting potential applications in modeling fluid-structure interactions and solid deformation and fracturing in blast simulations.

Myers, Conner↗
DOE OSTI2019

Parallel membership queries on very large scientific data sets using bitmap indexes

© 2019 John Wiley & Sons, Ltd. Many scientific applications produce very large amounts of data as advances in hardware fuel computing and experimental facilities. Managing and analyzing massive quantities of scientific data is challenging as data are often stored in specific formatted files, such as HDF5 and NetCDF, which do not offer appropriate search capabilities. In this research, we investigated a special class of search capability, called membership query, to identify whether queried elements of a set are members of an attribute. Attributes that naturally have classification values appear frequently in scientific domains such as category and object type as well as in daily life such as zip code and occupation. Because classification attribute values are discrete and require random data access, performing a membership query on a large scientific data set creates challenges. We applied bitmap indexing and parallelization to membership queries to overcome these challenges. Bitmap indexing provides high performance not only for low cardinality attributes but also for high cardinality attributes, such as floating-point variables, electric charge, or momentum in a particle physics data set, due to compression algorithms such as Word-Aligned Hybrid. We conducted experiments, in a highly parallelized environment, on data obtained from a particle accelerator model and a synthetic data set.

Yildiz, Beytullah↗
DOE OSTI2025

Scalable Hybrid Learning Techniques for Scientific Data Compression

Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data (PD), scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Here, this article presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

ITER↗
DOE OSTI2022

A weighted state redistribution algorithm for embedded boundary grids

State redistribution is an algorithm that stabilizes cut cells for embedded boundary grid methods. This work extends the earlier algorithm in several important ways. First, state redistribution is extended to three spatial dimensions. Second, we discuss several algorithmic changes and improvements motivated by the more complicated cut cell geometries that can occur in higher dimensions. In particular, we introduce a weighted version with less dissipation in an easily generalizable framework. Third, we demonstrate that state redistribution can also stabilize a solution update that includes both advective and diffusive contributions. Notably, the stabilization algorithm is shown to be effective for incompressible as well as compressible reacting flows. Finally, we discuss the implementation of the algorithm for several exascale-ready simulation codes based on AMReX, demonstrating ease of use in combination with domain decomposition, hybrid parallelism and complex physics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Hybrid learning techniques for scientific data reduction with performance guarantees

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Restricted Boltzmann Machines for galaxy morphology classification with a quantum annealer

We present the application of Restricted Boltzmann Machines (RBMs) to the task of astronomical image classification using a quantum annealer built by D-Wave Systems. Morphological analysis of galaxies provides critical information for studying their formation and evolution across cosmic time scales. We compress the images using principal component analysis to fit a representation on the quantum hardware. Then, we train RBMs with discriminative and generative algorithms, including contrastive divergence and hybrid generative-discriminative approaches. We compare these methods to Quantum Annealing (QA), Markov Chain Monte Carlo (MCMC) Gibbs Sampling, Simulated Annealing (SA) as well as machine learning algorithms like gradient boosted decision trees. We find that RBMs implemented on D-wave hardware perform well, and that they show some classification performance advantages on small datasets, but they don't offer a broadly strategic advantage for this task. During this exploration, we analyzed the steps required for Boltzmann sampling with the D-Wave 2000Q, including a study of temperature estimation, and examined the impact of qubit noise by comparing and contrasting the original D-Wave 2000Q to the lower-noise version recently made available. While these analyses ultimately had minimal impact on the performance of the RBMs, we include them for reference.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

1.2 Mfps standalone X-ray detector for Time-Resolved Experiments

We present a standalone and autonomous X-ray detector capable of operation with the speed of up to1.2Mfps. The detector utilizes UFXC32k hybrid pixel detectors for sensing X-rays, Spartan-6 LX45 FPGA placed in commercially available sbRIO 9628 controller for data acquisition and processing including a compression with zero-suppression algorithm. A Linux-RT system working on the 400 MHz Dual-Core CPU is used for FPGA control and data streaming to the higher-level system over 1 Gbps Ethernet connection. 1.2 M frames per second is achieved in so-called burst mode of operation while in zerodead-time mode 70 kfps is possible. Due to efficient data compression in FPGA there’s no need of using high-speed transceivers and Frame-Grabber cards on the data server side and the detector can stream the data infinitely over standard 1 Gbps network connection. Operation modes were tested at Advanced Photon Source Synchrotron at Argonne National Laboratory.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

A reinforcement learning approach to long-horizon operations, health, and maintenance supervisory control of advanced energy systems

In this work, we develop a Reinforcement Learning (RL) approach to the supervisory control problem for advanced energy systems, such as novel nuclear reactors and other demand-driven, mission-critical, and component-health-sensitive energy plants. The inclusive problem landscape considered captures the stochastic confluence of plant performance, component health evolution, power demand from the grid, diverse maintenance actions, and operator-defined goals and constraints, all considered over meaningfully long-enough reasoning horizons. Key aspects of the proposed approach are a receding horizon control-inspired technique dictating time- or event-triggered supervisory policy (re-)constructions, as well as additional capability-enabling contributions such as timescale compression, to handle long reasoning horizons and uncertainty in parts of the problem, and practical yet demonstrably-effective handling of hybrid action spaces with continuous and discrete decision variables. The resulting algorithm consists of a simulation-based RL agent constructing stochastic supervisory control policies over nontrivial action spaces and for long horizons, applying the learned policy to the system for a much shorter interval, and perpetually repeating, to construct the next long-horizon policy. That next policy will only be applied, again, for a short interval, yet originally far-in-time events move progressively closer, their associated uncertainty decreases, and new events and aspects enter the reasoning horizon. The proposed methodology bridges fundamental receding horizon concepts with the unequivocally stronger and more scalable reasoning of contemporary RL. Numerical examples using Soft Actor–Critic Deep RL illustrate the operation and efficacy of the proposed technique for a power plant tasked with health-aware load following missions in a dynamic electricity market landscape.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Fast Algorithms for Scientific Data Compression

Many scientific simulations and experiments generate terabytes to petabytes of data daily, necessitating data compression techniques. Unlike video and image compression, scientists require methods that accurately preserve primary data (PD) and derived quantities of interest (QoIs). In our previous work, we demonstrated the effectiveness of hybrid compression techniques that combine machine learning with traditional approaches. This paper presents innovative computational techniques aimed at expediting the compression pipeline. Our experiments, conducted on two distinct platforms with a large-scale XGC-based fusion simulation, demonstrate that the overhead incurred by these new approaches is less than one percent of the computational resources needed for the simulation.

Banerjee, Tania↗
DOE OSTI2023

Demonstration of a quantum-classical coprocessing protocol for simulating nuclear reactions

Quantum computers hold great promise for exact simulations of nuclear dynamical processes (e.g., scattering and reactions), which are paramount to the study of nuclear matter at the limit of stability and in the formation of chemical elements in stars. However, quantum simulations of the unitary (real) time dynamics of fermionic many-body systems require a currently prohibitive number of reliable and long-lived qubits. Here we propose a co-processing algorithm for the simulation of real-time dynamics in which the time evolution of the spatial coordinates is carried out on a classical processor, while the evolution of the spin degrees of freedom is carried out on quantum hardware. We demonstrate this hybrid scheme with the simulation of two neutrons scattering at the Lawrence Berkeley National Laboratory's Advanced Quantum Testbed. After implementing error mitigation strategies to improve the accuracy of the algorithm in addition to a combination of circuit compression techniques and tomography as methods to elucidate the onset of decoherence, our results validate the principle of the proposed co-processing scheme. A generalization of this present scheme will open the way for (real-time) path integral simulations of nuclear scattering.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Final report- UFL - RAPIDS2: A SciDAC Institute for Computer Science, Data, and Artificial Intelligence

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Sparse matrix‐vector and matrix‐multivector products for the truncated SVD on graphics processors

Summary Many practical algorithms for numerical rank computations implement an iterative procedure that involves repeated multiplications of a vector, or a collection of vectors, with both a sparse matrix and its transpose. Unfortunately, the realization of these sparse products on current high performance libraries often deliver much lower arithmetic throughput when the matrix involved in the product is transposed. In this work, we propose a hybrid sparse matrix layout, named CSRC, that combines the flexibility of some well‐known sparse formats to offer a number of appealing properties: (1) CSRC can be obtained at low cost from the popular CSR (compressed sparse row) format; (2) CSRC has similar storage requirements as CSR; and especially, (3) the implementation of the sparse product kernels delivers high performance for both the direct product and its transposed variant on modern graphics accelerators thanks to a significant reduction of atomic operations compared to a conventional implementation based on CSR. This solution thus renders considerably higher performance when integrated into an iterative algorithm for the truncated singular value decomposition (SVD), such as the randomized SVD or, as demonstrated in the experimental results, the block Golub–Kahan–Lanczos algorithm.

Aliaga, José I.↗
DOE OSTI2024

Hybrid algorithm for the time-dependent Hartree–Fock method using the Yang–Baxter equation on quantum computers *

Abstract The time-dependent Hartree–Fock (TDHF) method is an approach to simulate the mean field dynamics of electrons within the assumption that the electrons move independently in their self-consistent average field and within the space of single Slater determinants. One of the major advantages of performing time dynamics within Hartree–Fock theory is the free fermionic nature of the problem, which makes TDHF classically simulatable in polynomial time. Here, we present a hybrid TDHF implementation for quantum computers. This quantum circuit grows with time; but with our recent work on circuit compression via the Yang–Baxter equation (YBE), the resulting circuit is constant depth. This study provides a new way to simulate TDHF with the aid of a quantum device as well as provides a new direction for the application of YBE symmetry in quantum chemistry simulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Validating quantum-classical programming models with tensor network simulations

The exploration of hybrid quantum-classical algorithms and programming models on noisy near-term quantum hardware has begun. As hybrid programs scale towards classical intractability, validation and benchmarking are critical to understanding the utility of the hybrid computational model. In this paper, we demonstrate a newly developed quantum circuit simulator based on tensor network theory that enables intermediate-scale verification and validation of hybrid quantum-classical computing frameworks and programming models. We present our tensor-network quantum virtual machine (TNQVM) simulator which stores a multi-qubit wavefunction in a compressed (factorized) form as a matrix product state, thus enabling single-node simulations of larger qubit registers, as compared to brute-force statevector simulators. Our simulator is designed to be extensible in both the tensor network form and the classical hardware used to run the simulation (multicore, GPU, distributed). The extensibility of the TNQVM simulator with respect to the simulation hardware type is achieved via a pluggable interface for different numerical backends (e.g., ITensor and ExaTENSOR numerical libraries). We demonstrate the utility of our TNQVM quantum circuit simulator through the verification of randomized quantum circuits and the variational quantum eigensolver algorithm, both expressed within the eXtreme-scale ACCelerator (XACC) programming model.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

AMM: Adaptive Multilinear Meshes

Adaptive representations are increasingly indispensable for reducing the in-memory and on-disk footprints of large-scale data. Usual solutions are designed broadly along two themes: reducing data precision, e.g., through compression, or adapting data resolution, e.g., using spatial hierarchies. Additionally, recent research suggests that combining the two approaches, i.e., adapting both resolution and precision simultaneously, can offer significant gains over using them individually. However, there currently exist no practical solutions to creating and evaluating such representations at scale. In this work, we present a new resolution-precision-adaptive representation to support hybrid data reduction schemes and offer an interface to existing tools and algorithms. Through novelties in spatial hierarchy, our representation, Adaptive Multilinear Meshes (AMM), provides considerable reduction in the mesh size. AMM creates a piecewise multilinear representation of uniformly sampled scalar data and can selectively relax or enforce constraints on conformity, continuity, and coverage, delivering a flexible adaptive representation. AMM also supports representing the function using mixed-precision values to further the achievable gains in data reduction. We describe a practical approach to creating AMM incrementally using arbitrary orderings of data and demonstrate AMM on six types of resolution and precision datastreams. By interfacing with state-of-the-art rendering tools through VTK, we demonstrate the practical and computational advantages of our representation for visualization techniques. With an open-source release of our tool to create AMM, we make such evaluation of data reduction accessible to the community, which we hope will foster new opportunities and future data reduction schemes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Improved melt model for power flow

Accelerators that drive z-pinch experiments transport current densities in excess of 1 MA/cm 2 in order to melt or ionize the target and implode it on axis. These high current densities stress the transmission lines upstream from the target, where rapid electrode heating causes plasma formation, melt, and possibly vaporization. These plasmas negatively impact accelerator efficiency by diverting some portion of the current away from the target, referred to as “current loss”. Simulations that are able to reproduce this behavior may be applied to improving the efficiency of existing accelerators and to designing systems operating at ever higher current densities. The relativistic particle-in-cell code CHICAGO ® is the primary code for modeling power flow on Sandia National Laboratories’ Z accelerator. We report here on new algorithms that incorporate vaporization and melt into the standard power-flow simulation framework. Taking a hybrid approach, the CHICAGO® kinetic/multi-fluid treatment has been expanded to include vaporization while the quasi-neutral equation-of-motion has been updated for melt at high current-densities. For vaporization, a new one-dimensional substrate model provides a more accurate calculation of electrode thermal, mass, and magnetic field diffusion as well as a means of emitting absorbed contaminants and vaporized metal ions. A quasi-fluid model has been implemented expressly to mimic the motion of imploding liners for accurate inductance histories. For melt, a multi-ion Hall-MHD option has been implemented and benchmarked against Alegra MHD. This new model is described with sufficient detail to reproduce these algorithms in any hybrid kinetic code. Physics results from the new code are also presented. A CHICAGO ® Hall-MHD simulation of a radial transmission line demonstrates that Hall physics, not included in Alegra, has no significant impact on the diffusion of electrode material. When surface contaminant desorption is mocked in as a hydrogen surface plasma, both the surface and bulk-material plasmas largely compress under the influence of the j × B force. Similar results are seen in Alegra, which also shows magnetic and material diffusion scaling with peak current. Test vaporization simulations using MagLIF and a power-flow experimental geometry show Fe + ions diffuse only a few hundred µm from the electrodes, so present models of Z power flow remain valid.

43 PARTICLE ACCELERATORS↗