Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Fiducial-cosmology-dependent systematics for the DESI 2024 BAO analysis

When measuring the Baryon Acoustic Oscillations (BAO) scale from galaxy surveys, one typically assumes a fiducial cosmology when converting redshift measurements into comoving distances and also when defining input parameters for the reconstruction algorithm. A parameterised template for the model to be fitted is also created based on a (possibly different) fiducial cosmology. This model reliance can be considered a form of data compression, and the data is then analysed allowing that the true answer is different from the fiducial cosmology assumed. In this study, we evaluate the impact of the fiducial cosmology assumed in the BAO analysis of the Dark Energy Spectroscopic Instrument (DESI) survey Data Release 1 (DR1) on the final measurements in DESI 2024 III. We utilise a suite of mock galaxy catalogues with survey realism that mirrors the DESI DR1 tracers: the bright galaxy sample (BGS), the luminous red galaxies (LRG), the emission line galaxies (ELG) and the quasars (QSO), spanning a redshift range from 0.1 to 2.1. We compare the four secondary AbacusSummit cosmologies against DESI's fiducial cosmology (Planck 2018). The secondary cosmologies explored include a lower cold dark matter density, a thawing dark energy universe, a higher number of effective species, and a lower amplitude of matter clustering. The mocks are processed through the BAO pipeline by consistently iterating the grid, template, and reconstruction reference cosmologies. We determine a conservative systematic contribution to the error of 0.1% for both the isotropic and anisotropic dilation parameters αiso and αAP. We then directly test the impact of the fiducial cosmology on DESI DR1 data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Deep learning for electron and scanning probe microscopy: From materials design to atomic fabrication

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with applications ranging from theory and materials prediction to high-throughput data analysis. In parallel, the recent successes in applying ML/AI methods for autonomous systems from robotics through self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiment in imaging. Here, we discuss recent progress in application of machine learning methods in scanning transmission electron microscopy and scanning probe microscopy, from applications such as data compression and exploratory data analysis to physics learning to atomic fabrication.

36 MATERIALS SCIENCE↗
DOE OSTI2024

DLSIA: Deep Learning for Scientific Image Analysis

DLSIA (Deep Learning for Scientific Image Analysis) is a Python-based machine learning library that empowers scientists and researchers across diverse scientific domains with a range of customizable convolutional neural network (CNN) architectures for a wide variety of tasks in image analysis to be used in downstream data processing. DLSIA features easy-to-use architectures, such as autoencoders, tunable U-Nets and parameter-lean mixed-scale dense networks (MSDNets). Additionally, this article introduces sparse mixed-scale networks (SMSNets), generated using random graphs, sparse connections and dilated convolutions connecting different length scales. For verification, several DLSIA-instantiated networks and training scripts are employed in multiple applications, including inpainting for X-ray scattering data using U-Nets and MSDNets, segmenting 3D fibers in X-ray tomographic reconstructions of concrete using an ensemble of SMSNets, and leveraging autoencoder latent spaces for data compression and clustering. As experimental data continue to grow in scale and complexity, DLSIA provides accessible CNN construction and abstracts CNN complexities, allowing scientists to tailor their machine learning approaches, accelerate discoveries, foster interdisciplinary collaboration and advance research in scientific image analysis.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Compression behavior of microcrystalline cellulose spheres: Single particle compression and confined bulk compression across regimes

Particle characteristics can drastically influence the process-structure-property-performance aspects of granular materials in compression. In this study, we aim to computationally simulate the mechanical processes of stress redistribution in compacts including the kinematics of particle rearrangement during densification and particle deformation leading to fragmentation. Confined compression experiments are conducted with three sets of commercial microcrystalline cellulose particles nearly spherical in shape with different mean particle size. Experimentally measured compression curves from tall powder columns are fitted with the Kenkre et al. (J. of American Chemical Society, Vol. 79, No. 12) model. This model provides a basis to derive several common two-parameter literature models and as a framework to incorporate statistical representations of critical particle behaviors. We focus on the low-stress compression data and the model comparisons typically not discussed in the literature. Furthermore, additional single particle compressions report fracture strength with particle size for comparison to the apparent particle strength extracted from bulk compression data.

42 ENGINEERING↗
DOE OSTI2020

Machine Learning for Memory Reduction in the Implicit Monte Carlo Simulations of Thermal Radiative Transfer [Slides]

Project Goal: Use parametric Machine Learning methods in order to reduce memory requirements at checkpointing & restarting in the IMC simulations of Thermal Radiative Transfer using: Expectation Maximization and Weighted Gaussian Mixture Model-based approach for `particle-data compression', introduced in Plasma Physics to model Maxwellian particle distributions by Luis Chacon and Guangye Chen; Expectation Maximization with Weighted Hyper-Erlang Model in order to compress isotropic IMC particle data in the frequency domain; and Expectation Maximization and von Mises-Fisher Mixture Model for compression of anisotropic IMC particle data in the angular domain (work-in-progress).

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Variational autoencoders for at-source data reduction and anomaly detection in high energy particle detectors

Detectors in next-generation high-energy physics experiments face several daunting requirements, such as high data rates, damaging radiation exposure, and stringent constraints on power, space, and latency. To address these challenges, machine learning in readout electronics can be leveraged for smart detector designs, enabling intelligent inference and data reduction at-source. Variational autoencoders (VAEs) offer a variety of benefits for front-end readout; an on-sensor encoder can perform efficient lossy data compression while simultaneously providing a latent space representation that can be used for anomaly detection. Results are presented from low-latency and resource-efficient VAEs for front-end data processing in a futuristic silicon pixel detector. Encoder-based data compression is found to preserve good performance of off-detector analysis while significantly reducing the off-detector data rate as compared to a similarly sized data filtering approach. Furthermore, the latent space information is found to be a useful discriminator in the context of real-time sensor defect monitoring. Together, these results highlight the multifaceted utility of autoencoder-based front-end readout schemes and motivate their consideration in future detector designs.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Data-Driven Compression of Electron-Phonon Interactions

First-principles calculations of electron interactions in materials have seen rapid progress in recent years, with electron-phonon ( e − ph ) interactions being a prime example. However, these techniques use large matrices encoding the interactions on dense momentum grids, which reduces computational efficiency and obscures interpretability. For e − ph interactions, existing interpolation techniques leverage locality in real space, but the high dimensionality of the data remains a bottleneck to balance cost and accuracy. Here we show an efficient way to compress e − ph interactions based on singular value decomposition (SVD), a widely used matrix and image compression technique. Leveraging (un)constrained SVD methods, we accurately predict material properties related to e − ph interactions—including charge mobility, spin relaxation times, band renormalization, and superconducting critical temperature—while using only a small fraction (1%–2%) of the interaction data. These findings unveil the hidden low-dimensional nature of e − ph interactions. Furthermore, they accelerate state-of-the-art first-principles e − ph calculations by about 2 orders of magnitude without sacrificing accuracy. Our Pareto-optimal parametrization of e − ph interactions can be readily generalized to electron-electron and electron-defect interactions, as well as to other couplings, advancing quantitative studies of condensed matter. Published by the American Physical Society 2024

Physics↗
DOE OSTI2022

Femtosecond diffraction studies of the sodium chloride phase diagram under laser shock compression

The phase diagram of sodium chloride (NaCl) under laser shock compression has been studied at Linac Coherent Light Source (LCLS) at the x-ray free-electron laser facility. Both solid–solid (B1 → B2) and solid–liquid (B2 → liquid) transitions have been observed along the Hugoniot over nanosecond time scales. By combining structural measurements through in situ x-ray diffraction, pressure determination through velocimetry, and a thermal equation-of-state, the shock-compressed data are used to constrain the phase diagram of NaCl. Transformation into the B2 phase is found to occur at 28(2) GPa, and B2–liquid coexistence is observed between 54(4) and 66(6) GPa, with near full melt at 66(6) GPa. Late-time pressure release from an initial shocked B2-state results in a B2 → B1 back transformation. Our results show agreement with previous static compression data, suggesting that the time scale for melting is very rapid and that equilibrium states in NaCl are being accessed over nanosecond time scales. Finally, a multiphase equation-of-state description of NaCl incorporated into a one-dimensional hydrocode is used to interpret pressure and temperature evolution over these rapid time scales.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Dark Energy Survey Year 3 results: optimized $w$CDM simulation-based inference with weak lensing map-level hybrid statistics

We present cosmological constraints from the Dark Energy Survey Year 3 (DES Y3) weak lensing data using hierarchical hybrid statistics within a Bayesian simulation-based inference framework that is based on the Gower Street simulations. To maximize the precision of the inference, we have developed a new, information-theory based, data compression of the weak lensing maps to just seven highly informative summary statistics. The hybrid scheme exploits the high information content of the power spectrum, compressing both the power spectrum and neural-based summaries that are designed to extract further information. Our simulation-based approach enables principled forward modelling of all major sources of systematic uncertainty and survey properties into realistic mock observations, including the survey mask, photometric redshift uncertainties, intrinsic galaxy alignments, multiplicative shear calibration bias, source galaxy clustering, non-Gaussian shape noise, and non-linear structure formation. The summary statistics are then used in a Bayesian simulation-based inference pipeline. The inference is validated through coverage tests and checks for robustness against baryonic feedback. Assuming a $w$CDM cosmology, our analysis yields $S_8 = 0.808 \pm 0.017$, $Ω_{\rm m} = 0.325 \pm 0.024$, and $w < -0.766$ (marginalized posterior 68 per cent credible intervals). This rigorous combination of information theory, physics- and neural network-based extreme data compression, and principled Bayesian analysis improves the figure of merit for $(Ω_{\rm m}, S_8, w)$ by 60 per cent over the previous state-of-the-art, and by almost a factor of 3 over two-point analyses of the same data. They are the most precise joint constraints on $(Ω_{\rm m}, S_8, w)$ from weak gravitational lensing data alone of any survey to date. We intend to apply this analysis to the more recent DES Y6 data.

Williamson, J. [University Coll. London]↗
DOE OSTI2020

Compression behavior of dense H 2 -He mixtures up to 160 GPa

We have studied the compression behavior of H 2 -He mixtures in comparison with pure H 2 and He using powder synchrotron x-ray diffraction and present the pressure-volume (PV) compression data of H 2 -He mixtures to 160 GPa. The results indicate that both H 2 and He in H 2 -He mixtures remain in hcp to the maximum pressure studied, yet develop a substantial level of lattice distortion in the (100) plane, most profound in He-rich solids and below 66 GPa. The measured PV data also indicate softening of He (or H 2 )-rich lattice upon increasing the level of the guest H 2 (or He) concentration. We suggest that the observed softening and lattice distortion are due to a substitutional incorporation of H2 (guest) molecules into the basal plane of hcp-He (host) lattice and, thereby, reflect the miscibility between H 2 and He in H 2 -He mixtures. Interestingly, solid He exhibits a lesser degree of preferred orientation in H2-He mixtures than in pure He, likely due to the presence of solid H 2 disturbing the crystalline ordering of He-rich solids. Finally, the present PV compression data of H 2 -rich and He-rich solids to 160 GPa deviate from those of pure H 2 and pure He above ~70 and 45 GPa respectively, providing new constraints for development of the EOS for H 2 -He mixtures for planetary models.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2025

GRUMDN: A Multi-Task Model for Predicting Human Patterns-of-Life from Stay Transition Data

Understanding human patterns-of-life (PoL) is essential towards ensuring safe and secure indoor facility environment as well as outdoor urban environment. Prediction of human movement in between places of interest is vital in understanding human PoL. Movement between spaces maybe represented and detected in one of the two forms: 1) trajectories: locations measured at regular time intervals by mobile sensors, bluetooth or GPS sensors; or 2) stay transitions: semantic PoI (points of interest) and stay duration data measurable by eventbased sensors that collect data when a check-in or check-out event is detected. Stay transition data provides a more compressed data format compared to trajectories data, especially in situations with longer stay durations, while preserving the information necessary for PoL analysis. Now as introduced briefly in the paper, our deployed end application (Digital Twin of a facility with non-player characters, besides the interactive user in virtual reality) needed a well-performing and validated AI/ML model for simulating high quality stay transitions behavior. In this study we thus primarily present our findings with developing and validating that model, which is a multi-task neural network for stay transition prediction. The neural network consists of two heads, for corresponding two tasks of stay category prediction and stay duration prediction. We evaluated gated recurrent units and multi-layer perceptrons of varying network sizes for stay category prediction; while mixture density networks, noisy generator-only networks, and generative adversarial networks of varying network sizes for stay duration prediction. We have then evaluated four multi-task models, constructed by combining these specialized models, on their ability to predict stay transition data. We tested our models on datasets from two different cases: 1) a simulation-generated dataset of indoor movement within the HFIR (high flux isotope reactor) nuclear reactor facility at Oak Ridge National Laboratory (ORNL); and 2) the GeoLife human mobility dataset of outdoor urban movement available in literature. Our results indicate that GRUMDN, which combines gated recurrent units (GRU) for stay category prediction task, and mixture density networks (MDN) for stay duration prediction task, did overall outperform other multitask models and the current state-of-the-art.

Gunaratne, Chathika [ORNL] (ORCID:0000000225088745↗
DOE OSTI2023

HQ-Sim: High-performance State Vector Simulation of Quantum Circuits on Heterogeneous HPC Systems

Quantum circuit simulations are applied in more and more circum- stances as the quantum computing community becomes broader. It helps researchers to evaluate quantum algorithms and relieve the burden of limited quantum computing resources. However, most of the state-of-the-art quantum simulators utilize either CPU or GPU to store and calculate the state vector, which results in resource starvation. Moreover, the maximum number of qubits supported by the simulator is bounded by the memory, since the memory utilization increases exponentially with the number of qubits. In this study, we leverage Heterogeneous computing to utilize both CPU and GPU to store and update state vectors. We also integrate lossy data compression to reduce memory requirements. Specifically, we develop a heterogeneous framework that has a dynamic scheduler to fully utilize the computing resources. We apply lossy compression to chunked state vector to make the maximum number of qubits higher than the regular simulators, the compression also benefits the data movement between CPU and GPU.

Zhang, Boyuan↗
DOE OSTI2024

Haar-Like Wavelets on Hierarchical Trees

Here, discrete wavelet methods, originally formulated in the setting of regularly sampled signals, can be adapted to data defined on a point cloud if some multiresolution structure is imposed on the cloud. A wide variety of hierarchical clustering algorithms can be used for this purpose, and the multiresolution structure obtained can be encoded by a hierarchical tree of subsets of the cloud. Prior work introduced the use of Haar-like bases defined with respect to such trees for approximation and learning tasks on unstructured data. This paper builds on that work in two directions. First, we present an algorithm for constructing Haar-like bases on general discrete hierarchical trees. Second, with an eye towards data compression, we present thresholding techniques for data defined on a point cloud with error controlled in the $L$ $\infty$ norm and in a Hölder-type norm. In a concluding trio of numerical examples, we apply our methods to compress a point cloud dataset, study the tightness of the $L$ $\infty$ error bound, and use thresholding to identify MNIST classifiers with good generalizability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Exploring Autoencoder-based Error-bounded Compression for Scientific Data

Error-bounded lossy compression is becoming an indispensable technique for the success of today's scientific projects with vast volumes of data produced during the simulations or instrument data acquisitions. Not only can it significantly reduce data size, but it also can control the compression errors based on user-specified error bounds. Autoencoder (AE) models have been widely used in image compression, but few AE-based compression approaches support error-bounding features, which are highly required by scientific applications. To address this issue, we explore using convolutional autoencoders to improve error-bounded lossy compression for scientific data, with the following three key contributions. (1) We provide an in-depth investigation of the characteristics of various autoencoder models and develop an error-bounded autoencoder-based framework in terms of the SZ model. (2) We optimize the compression quality for main stages in our designed AE-based error-bounded compression framework, fine-tuning the block sizes and latent sizes and also optimizing the compression efficiency of latent vectors. (3) We evaluate our proposed solution using five real-world scientific datasets and comparing them with six other related works. Experiments show that our solution exhibits a very competitive compression quality from among all the compressors in our tests. In absolute terms, it can obtain a much better compression quality (100%similar to 800% improvement in compression ratio with the same data distortion) compared with SZ2.1 and ZFP in cases with a high compression ratio.

Liu, Jinyang↗
DOE OSTI2026

Parity-odd four-point correlation function from the DESI data release 1 luminous red galaxy sample

The parity-odd four-point function provides a unique probe of fundamental symmetries and potential new physics in the large-scale structure of the Universe. We present measurements of the parity-odd four-point function using the Dark Energy Spectroscopic Instrument (DESI) DR1 luminous red galaxy (LRG) sample and assess its detection significance. Our analysis considers both auto- and cross-correlations, using two complementary approaches to the covariance: (i) the full analytic covariance matrix applied to the uncompressed data vector, and (ii) a compressed data vector combined with a hybrid covariance matrix constructed from simulations and analytic estimates. When using the full analytic covariance matrix without corrections, we observe apparent auto-correlation signals with significance up to 4⁢𝜎. However, this excess is also consistent with a mismatch between the statistical fluctuations estimated from the simulations and those present in the real data. Our findings therefore suggest that the parity-odd signal in the current DESI DR1 LRG sample is consistent with zero. We note, however, that the low completeness of this sample may have a non-negligible impact on the detection sensitivity. Future data releases with improved completeness will be crucial for further investigation.

Hou, Jiamin [Ludwig-Maximilians-Universität; Unive↗
DOE OSTI2020

waveSZ: A Hardware-Algorithm Co-Design of Efficient Lossy Compression for Scientific Data

Error-bounded lossy compression is critical to the success of extreme-scale scientific research because of ever-increasing volumes of data produced by today's high-performance computing (HPC) applications. Not only can error-controlled lossy compressors significantly reduce the I/O and storage burden but they can retain high data fidelity for post analysis. Existing state-of-the-art lossy compressors, however, generally suffer from relatively low compression and decompression throughput (up to hundreds of megabytes per second on a single CPU core), which considerably restrict the adoption of lossy compression by many HPC applications especially those with a fairly high data production rate. In this paper, we propose a highly efficient lossy compression approach based on field programmable gate arrays (FPGAs) under the state-of-the-art lossy compression model SZ. Our contributions are fourfold. (1) We adopt a wavefront memory layout to alleviate the data dependency during the prediction for higher-dimensional predictors, such as the Lorenzo predictor. (2) We propose a co-design framework named WAVESZ based on the wavefront memory layout and the characteristics of SZ algorithm and carefully implement it by using high-level synthesis. (3) We propose a hardware-algorithm co-optimization method to improve the performance. (4) We evaluate our proposed WAVESZ on three real-world HPC simulation datasets from the Scientific Data Reduction Benchmarks and compare it with other state-of-the-art methods on both CPUs and FPGAs. Experiments show that our WAVESZ can improve SZ's compression throughput by 6.9x similar to 8.7x over the production version running on a state-of-the-art CPU and improve the compression ratio and throughput by 2.1x and 5.8x on average, respectively, respectively, compared with the state-of-the-art FPGA design.

Compression Ratio↗
DOE OSTI2024

Shifting Between Compute and Memory Bounds: A Compression-Enabled Roofline Model

In the evolving landscape of high-performance computing, especially to fight the end of Moore’s Law and Dennard’s Scaling, the ability to shift between compute-bound and memory-bound states is critical for enhancing adaptability and flexibility to diverse system and domain-specific architectures. Such capability is vital for optimizing performance across distinguished hardware configurations, such as accelerators, memory hierarchies, and cache systems. Despite that ad hoc optimization techniques, such as compressed/approximate computation, have been enabled for compute-/data-intensive computing for improved performance in distinct hardware settings, there lacks an understanding of 1) the rational behind performance improvement; 2) capability of different optimizations; 3) what optimization to respond to specific computational and memory demands. This work proposes a compression-enabled roofline model to facilitate this adaptability with data compression techniques to balance and transform between computational and memory demands. This model enables applications to adjust in response to the specific strengths and limitations of the underlying hardware and system to optimize resource utilization. The effectiveness of this approach is demonstrated with matrix multiplication kernels on different input sizes, with turning on/off various compression techniques, including 1) low-precision floating point; 2) sparse matrix formulation; and 3) compressed arrays with ZFP. By reducing memory transfer volumes and cache misses and increasing data locality and computational intensity through compression, the specific roofline model can transform between compute and memory bounds to align more efficiently with system capabilities. This advancement not only improves overall performance but also maximizes adaptability in diverse computing environments.

Naraparaju, Ramasoumya [University of Washington]↗
DOE OSTI2026

Goated: goal-oriented tensor decompositions in python

SAND2026-20464O Goated performs goal-oriented tensor decompositions in Python, enabling efficient compression of multi-dimensional simulation data. It extends common tensor decomposition methods by incorporating domain-specific knowledge, such as conservation laws in physics, through a penalty term in the optimization process. This approach improves data compression and modeling accuracy across various applications, including physics simulations, by using specialized algorithms and structure-aware subroutines to accelerate solver performance. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗