SEARCH · Search NASA

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

DOE OSTI2014

Prior data assisted compressed sensing: A novel MR imaging strategy for real time tracking of lung tumors

Purpose: Hybrid radiotherapy-MRI devices promise real time tracking of moving tumors to focus the radiation portals to the tumor during irradiation. This approach will benefit from the increased temporal resolution of MRI's data acquisition and reconstruction. In this work, the authors propose a novel spatial-temporal compressed sensing (CS) imaging strategy for the real time MRI–-prior data assisted compressed sensing (PDACS), which aims to improve the image quality of the conventional CS without significantly increasing reconstruction times. Methods: Conventional 2D CS requires a random sampling of partial k-space data, as well as an iterative reconstruction that simultaneously enforces the image's sparsity in a transform domain as well as maintains the fidelity to the acquired k-space. PDACS method requires the additional acquisition of the prior data, and for reconstruction, it additionally enforces fidelity to the prior k-space domain similar to viewsharing. In this work, the authors evaluated the proposed PDACS method by comparing its results to those obtained from the 2D CS and viewsharing methods when performed individually. All three methods are used to reconstruct images from lung cancer patients whose tumors move and who are likely to benefit from lung tumor tracking. The patients are scanned, using a 3T MRI, under free breathing using the fully sampled k-space with 2D dynamic bSSFP sequence in a sagittal plane containing lung tumor. These images form a reference set for the evaluation of the partial k-space methods. To create partial k-space, the fully sampled k-space is retrospectively undersampled to obtain a range of acquisition acceleration factors, and reconstructed with 2D-CS, PDACS, and viewshare methods. For evaluation, metrics assessing global image artifacts as well as tumor contour shape fidelity are determined from the reconstructed images. These analyses are performed both for the original 3T images and those at a simulated 0.5T equivalent noise level. Results: In the 3.0T images, the PDACS strategy is shown to give superior results compared to viewshare and conventional 2D CS using all metrics. The 2D-CS tends to perform better than viewshare at the low acceleration factors, while the opposite is true at the high acceleration factors. At simulated 0.5T images, PDACS method performs only marginally better than the viewsharing method, both of which are superior compared to 2D CS. The PDACS image reconstruction time (0.3 s/image) is similar to that of the conventional 2D CS. Conclusions: The PDACS method can potentially improve the real time tracking of moving tumors by significantly increasing MRI's data acquisition speeds. In 3T images, the PDACS method does provide a benefit over the other two methods in terms of both the overall image quality and the ability to accurately and automatically contour the tumor shape. MRI's data acquisition may be accelerated using the simpler viewsharing strategy at the lower, 0.5T magnetic field, as the marginal benefit of the PDACS method may not justify its additional reconstruction times.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2018

Adaptive wavelet compression of large additive manufacturing experimental and simulation datasets

New manufacturing technologies such as additive manufacturing require research and development to minimize the uncertainties in the produced parts. The research involves experimental measurements and large simulations, which result in huge quantities of data to store and analyze. We address this challenge by alleviating the data storage requirements using lossy data compression. We select wavelet bases as the mathematical tool for compression. Unlike images, additive manufacturing data is often represented on irregular geometries and unstructured meshes. Thus, we use Alpert tree-wavelets as bases for our data compression method. We first analyze different basis functions for the wavelets and find the one that results in maximal compression and miminal error in the reconstructed data. We then devise a new adaptive thresholding method that is data-agnostic and allows a priori estimation of the reconstruction error. Finally, we propose metrics to quantify the global and local errors in the reconstructed data. One of the error metrics addresses the preservation of physical constraints in reconstructed data fields, such as divergence-free stress field in structural simulations. While our compression and decompression method is general, we apply it to both experimental and computational data obtained from measurements and thermal/structural modeling of the sintering of a hollow cylinder from metal powders using a Laser Engineered Net Shape process. The results show that monomials achieve optimal compression performance when used as wavelet bases. The new thresholding method results in compression ratios that are two to seven times larger than the ones obtained with commonly used thresholds. Overall, adaptive Alpert tree-wavelets can achieve compression ratios between one and three orders of magnitude depending on the features in the data that are required to preserve. Furthermore, these results show that Alpert tree-wavelet compression is a viable and promising technique to reduce the size of large data structures found in both experiments and simulations.

42 ENGINEERING↗

DOE OSTI2022

The Exploitation of Data Reduction for Visualization

The disparity between the computational speed and storage bandwidth, as demonstrated in Figure 1, is a well known problem that grows with each successive generation. The visualization community is principally responding to this issue by using in situ to reduce which data must be written to storage. However, other communities are taking different, possibly complementary approaches. In particular, data compression is a common general approach to reduce storage demands. Data compression technologies are typically not designed with post processing in mind. The principal metrics measured are compression ratio, the improved bandwidth to storage, and the error introduced. It is assumed that data is inflated to its full size before any post processing can happen. Although when talking about bandwidth disparities, HPC’s dirty little secret is that no part of the memory nor interconnect hardware is increasing at the rate of computation. For example, the Summit supercomputer has a peak computation rate almost 10 times its predecessor, Titan, but only about 4 times the memory, less than twice the aggregate memory bandwidth, and almost no improvement in the interconnect bisection bandwidth. Naively inflating data for post processing does not help with limitations in the memory and interconnect systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Malicious activity detection in a memory

A method and apparatus for monitoring a volatile memory in a computer system. Samples of compressed data from locations in the volatile memory in the computer system are read. Data in the volatile memory is reconstructed using the samples of compressed data. The data is an image of the volatile memory. The image enables determining whether an undesired process is present in the volatile memory.

Wheeler, Jason W.↗

DOE OSTI2025

Fiducial-cosmology-dependent systematics for the DESI 2024 BAO analysis

When measuring the Baryon Acoustic Oscillations (BAO) scale from galaxy surveys, one typically assumes a fiducial cosmology when converting redshift measurements into comoving distances and also when defining input parameters for the reconstruction algorithm. A parameterised template for the model to be fitted is also created based on a (possibly different) fiducial cosmology. This model reliance can be considered a form of data compression, and the data is then analysed allowing that the true answer is different from the fiducial cosmology assumed. In this study, we evaluate the impact of the fiducial cosmology assumed in the BAO analysis of the Dark Energy Spectroscopic Instrument (DESI) survey Data Release 1 (DR1) on the final measurements in DESI 2024 III. We utilise a suite of mock galaxy catalogues with survey realism that mirrors the DESI DR1 tracers: the bright galaxy sample (BGS), the luminous red galaxies (LRG), the emission line galaxies (ELG) and the quasars (QSO), spanning a redshift range from 0.1 to 2.1. We compare the four secondary AbacusSummit cosmologies against DESI's fiducial cosmology (Planck 2018). The secondary cosmologies explored include a lower cold dark matter density, a thawing dark energy universe, a higher number of effective species, and a lower amplitude of matter clustering. The mocks are processed through the BAO pipeline by consistently iterating the grid, template, and reconstruction reference cosmologies. We determine a conservative systematic contribution to the error of 0.1% for both the isotropic and anisotropic dilation parameters αiso and αAP. We then directly test the impact of the fiducial cosmology on DESI DR1 data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2016

Damaris: Addressing performance variability in data management for post-petascale simulations

With exascale computing on the horizon, reducing performance variability in data management tasks (storage, visualization, analysis, etc.) is becoming a key challenge in sustaining high performance. Here, this variability significantly impacts the overall application performance at scale and its predictability over time. In this article, we present Damaris, a system that leverages dedicated cores in multicore nodes to offload data management tasks, including I/O, data compression, scheduling of data movements, in situ analysis, and visualization. We evaluate Damaris with the CM1 atmospheric simulation and the Nek5000 computational fluid dynamic simulation on four platforms, including NICS’s Kraken and NCSA’s Blue Waters. Our results show that (1) Damaris fully hides the I/O variability as well as all I/O-related costs, thus making simulation performance predictable; (2) it increases the sustained write throughput by a factor of up to 15 compared with standard I/O approaches; (3) it allows almost perfect scalability of the simulation up to over 9,000 cores, as opposed to state-of-the-art approaches that fail to scale; and (4) it enables a seamless connection to the VisIt visualization software to perform in situ analysis and visualization in a way that impacts neither the performance of the simulation nor its variability. In addition, we extended our implementation of Damaris to also support the use of dedicated nodes and conducted a thorough comparison of the two approaches—dedicated cores and dedicated nodes—for I/O tasks with the aforementioned applications.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Deep learning for electron and scanning probe microscopy: From materials design to atomic fabrication

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with applications ranging from theory and materials prediction to high-throughput data analysis. In parallel, the recent successes in applying ML/AI methods for autonomous systems from robotics through self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiment in imaging. Here, we discuss recent progress in application of machine learning methods in scanning transmission electron microscopy and scanning probe microscopy, from applications such as data compression and exploratory data analysis to physics learning to atomic fabrication.

36 MATERIALS SCIENCE↗

DOE OSTI2024

DLSIA: Deep Learning for Scientific Image Analysis

DLSIA (Deep Learning for Scientific Image Analysis) is a Python-based machine learning library that empowers scientists and researchers across diverse scientific domains with a range of customizable convolutional neural network (CNN) architectures for a wide variety of tasks in image analysis to be used in downstream data processing. DLSIA features easy-to-use architectures, such as autoencoders, tunable U-Nets and parameter-lean mixed-scale dense networks (MSDNets). Additionally, this article introduces sparse mixed-scale networks (SMSNets), generated using random graphs, sparse connections and dilated convolutions connecting different length scales. For verification, several DLSIA-instantiated networks and training scripts are employed in multiple applications, including inpainting for X-ray scattering data using U-Nets and MSDNets, segmenting 3D fibers in X-ray tomographic reconstructions of concrete using an ensemble of SMSNets, and leveraging autoencoder latent spaces for data compression and clustering. As experimental data continue to grow in scale and complexity, DLSIA provides accessible CNN construction and abstracts CNN complexities, allowing scientists to tailor their machine learning approaches, accelerate discoveries, foster interdisciplinary collaboration and advance research in scientific image analysis.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Fourier band-power E/B-mode estimators for cosmic shear

We introduce new Fourier band-power estimators for cosmic shear data analysis and E/B-mode separation. We consider both the case where one performs E/B-mode separation and the case where one does not. The resulting estimators have several nice properties which make them ideal for cosmic shear data analysis. First, they can be written as linear combinations of the binned cosmic shear correlation functions. Secondly, they account for the survey window function in real-space. Thirdly, they are unbiased by shape noise since they do not use correlation function data at zero separation. Fourthly, the band-power window functions in Fourier space are compact and largely non-oscillatory. Fifthly, they can be used to construct band-power estimators with very efficient data compression properties. In particular, we find that all of the information on the parameters Ωm, σ8 and ns in the shear correlation functions in the range of ~10–400 arcmin for single tomographic bin can be compressed into only three band-power estimates. Finally, we can achieve these rates of data compression while excluding small-scale information where the modelling of the shear correlation functions and power spectra is very difficult. Given these desirable properties, these estimators will be very useful for cosmic shear data analysis.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

Compression behavior of microcrystalline cellulose spheres: Single particle compression and confined bulk compression across regimes

Particle characteristics can drastically influence the process-structure-property-performance aspects of granular materials in compression. In this study, we aim to computationally simulate the mechanical processes of stress redistribution in compacts including the kinematics of particle rearrangement during densification and particle deformation leading to fragmentation. Confined compression experiments are conducted with three sets of commercial microcrystalline cellulose particles nearly spherical in shape with different mean particle size. Experimentally measured compression curves from tall powder columns are fitted with the Kenkre et al. (J. of American Chemical Society, Vol. 79, No. 12) model. This model provides a basis to derive several common two-parameter literature models and as a framework to incorporate statistical representations of critical particle behaviors. We focus on the low-stress compression data and the model comparisons typically not discussed in the literature. Furthermore, additional single particle compressions report fracture strength with particle size for comparison to the apparent particle strength extracted from bulk compression data.

42 ENGINEERING↗

DOE OSTI2019

SWinzip v. 2.0

SWinzip is a Matlab and C++ library for scientific lossy data compression and reconstruction using compressed sensing and tree-wavelets transforms. These methods are known for their large compression and usefulness in data analytics such as features extraction. Compressed sensing and wavelets methods rely heavily on sparse and dense linear algebra operations implemented through the Boost codes in our library. SWinzip accommodates data represented on both regular grids (e.g. image data) and point-clouds (e.g. unstructured meshes). In SWinzip v2.0, there is more focus on wavelet compression and the following features are added: - The matrix-free (direct) wavelet transform is implemented in both C++ and Matlab. A threshold-based compression can now be used. Examples for distributed compression are also added. - Python implementation of Alpert wavelet transforms - Field data comparison using wavelets in Matlab with global and local metrics

Solloum, Maher↗

DOE OSTI2020

Machine Learning for Memory Reduction in the Implicit Monte Carlo Simulations of Thermal Radiative Transfer [Slides]

Project Goal: Use parametric Machine Learning methods in order to reduce memory requirements at checkpointing & restarting in the IMC simulations of Thermal Radiative Transfer using: Expectation Maximization and Weighted Gaussian Mixture Model-based approach for `particle-data compression', introduced in Plasma Physics to model Maxwellian particle distributions by Luis Chacon and Guangye Chen; Expectation Maximization with Weighted Hyper-Erlang Model in order to compress isotropic IMC particle data in the frequency domain; and Expectation Maximization and von Mises-Fisher Mixture Model for compression of anisotropic IMC particle data in the angular domain (work-in-progress).

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2016

SWinzip v. 1.0

Salloum, Maher↗

DOE OSTI2025

Variational autoencoders for at-source data reduction and anomaly detection in high energy particle detectors

Detectors in next-generation high-energy physics experiments face several daunting requirements, such as high data rates, damaging radiation exposure, and stringent constraints on power, space, and latency. To address these challenges, machine learning in readout electronics can be leveraged for smart detector designs, enabling intelligent inference and data reduction at-source. Variational autoencoders (VAEs) offer a variety of benefits for front-end readout; an on-sensor encoder can perform efficient lossy data compression while simultaneously providing a latent space representation that can be used for anomaly detection. Results are presented from low-latency and resource-efficient VAEs for front-end data processing in a futuristic silicon pixel detector. Encoder-based data compression is found to preserve good performance of off-detector analysis while significantly reducing the off-detector data rate as compared to a similarly sized data filtering approach. Furthermore, the latent space information is found to be a useful discriminator in the context of real-time sensor defect monitoring. Together, these results highlight the multifaceted utility of autoencoder-based front-end readout schemes and motivate their consideration in future detector designs.

47 OTHER INSTRUMENTATION↗

DOE OSTI2024

Data-Driven Compression of Electron-Phonon Interactions

First-principles calculations of electron interactions in materials have seen rapid progress in recent years, with electron-phonon ( e − ph ) interactions being a prime example. However, these techniques use large matrices encoding the interactions on dense momentum grids, which reduces computational efficiency and obscures interpretability. For e − ph interactions, existing interpolation techniques leverage locality in real space, but the high dimensionality of the data remains a bottleneck to balance cost and accuracy. Here we show an efficient way to compress e − ph interactions based on singular value decomposition (SVD), a widely used matrix and image compression technique. Leveraging (un)constrained SVD methods, we accurately predict material properties related to e − ph interactions—including charge mobility, spin relaxation times, band renormalization, and superconducting critical temperature—while using only a small fraction (1%–2%) of the interaction data. These findings unveil the hidden low-dimensional nature of e − ph interactions. Furthermore, they accelerate state-of-the-art first-principles e − ph calculations by about 2 orders of magnitude without sacrificing accuracy. Our Pareto-optimal parametrization of e − ph interactions can be readily generalized to electron-electron and electron-defect interactions, as well as to other couplings, advancing quantitative studies of condensed matter. Published by the American Physical Society 2024

Physics↗

DOE OSTI2014

Experimental verification of the high pressure crystal structures in NH{sub 3}BH{sub 3}

A detailed high-pressure study on NH{sub 3}BH{sub 3} has been carried out using in situ synchrotron X-ray diffraction (XRD) and Raman scattering with a diamond anvil cell up to 20 and 33 GPa, respectively. The Rietveld refinement based on the XRD pattern and analysis of Raman data indicate two first-order phase transitions from the ambient pressure I4 mm structure (α-NH{sub 3}BH{sub 3}) to a high pressure Cmc2{sub 1} phase (β-NH{sub 3}BH{sub 3}) at 2.14 GPa, and further into a monoclinic P2{sub 1} (Z = 2) phase (γ-NH{sub 3}BH{sub 3}) at 9.67 GPa. Fitting the measured volumetric compression data to the third order Birch-Murnaghan equation of state reveals a bulk modulus of B{sub 0} = 9.9 ± 0.5 and 17.0 ± 3.0 GPa (with fixed B{sub 0}{sup ′} = 4) for the β-NH{sub 3}BH{sub 3} below and above 5 GPa, respectively. Still, with the splitting of the NBH rock mode in Raman experiment, it is concluded that a second-order isostructural phase transition occurs at 5 GPa. By analyzing the dihydrogen bonding framework, the origin of the isostructural phase transition is attributed to the number of dihydrogen bondings per molecule in the Cmc2{sub 1} phase increasing from 12 to 14 at 5 GPa.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2022

Femtosecond diffraction studies of the sodium chloride phase diagram under laser shock compression

The phase diagram of sodium chloride (NaCl) under laser shock compression has been studied at Linac Coherent Light Source (LCLS) at the x-ray free-electron laser facility. Both solid–solid (B1 → B2) and solid–liquid (B2 → liquid) transitions have been observed along the Hugoniot over nanosecond time scales. By combining structural measurements through in situ x-ray diffraction, pressure determination through velocimetry, and a thermal equation-of-state, the shock-compressed data are used to constrain the phase diagram of NaCl. Transformation into the B2 phase is found to occur at 28(2) GPa, and B2–liquid coexistence is observed between 54(4) and 66(6) GPa, with near full melt at 66(6) GPa. Late-time pressure release from an initial shocked B2-state results in a B2 → B1 back transformation. Our results show agreement with previous static compression data, suggesting that the time scale for melting is very rapid and that equilibrium states in NaCl are being accessed over nanosecond time scales. Finally, a multiphase equation-of-state description of NaCl incorporated into a one-dimensional hydrocode is used to interpret pressure and temperature evolution over these rapid time scales.

36 MATERIALS SCIENCE↗

DOE OSTI2026

Dark Energy Survey Year 3 results: optimized $w$CDM simulation-based inference with weak lensing map-level hybrid statistics

We present cosmological constraints from the Dark Energy Survey Year 3 (DES Y3) weak lensing data using hierarchical hybrid statistics within a Bayesian simulation-based inference framework that is based on the Gower Street simulations. To maximize the precision of the inference, we have developed a new, information-theory based, data compression of the weak lensing maps to just seven highly informative summary statistics. The hybrid scheme exploits the high information content of the power spectrum, compressing both the power spectrum and neural-based summaries that are designed to extract further information. Our simulation-based approach enables principled forward modelling of all major sources of systematic uncertainty and survey properties into realistic mock observations, including the survey mask, photometric redshift uncertainties, intrinsic galaxy alignments, multiplicative shear calibration bias, source galaxy clustering, non-Gaussian shape noise, and non-linear structure formation. The summary statistics are then used in a Bayesian simulation-based inference pipeline. The inference is validated through coverage tests and checks for robustness against baryonic feedback. Assuming a $w$CDM cosmology, our analysis yields $S_8 = 0.808 \pm 0.017$, $Ω_{\rm m} = 0.325 \pm 0.024$, and $w < -0.766$ (marginalized posterior 68 per cent credible intervals). This rigorous combination of information theory, physics- and neural network-based extreme data compression, and principled Bayesian analysis improves the figure of merit for $(Ω_{\rm m}, S_8, w)$ by 60 per cent over the previous state-of-the-art, and by almost a factor of 3 over two-point analyses of the same data. They are the most precise joint constraints on $(Ω_{\rm m}, S_8, w)$ from weak gravitational lensing data alone of any survey to date. We intend to apply this analysis to the more recent DES Y6 data.

Williamson, J. [University Coll. London]↗