Results for “Data Compression” — Page 9

DOE OSTI2017

Scientific data interpolation with low dimensional manifold model

Here, we propose to apply a low dimensional manifold model to scientific data interpolation from regular and irregular samplings with a significant amount of missing information. The low dimensionality of the patch manifold for general scientific data sets has been used as a regularizer in a variational formulation. The problem is solved via alternating minimization with respect to the manifold and the data set, and the Laplace–Beltrami operator in the Euler–Lagrange equation is discretized using the weighted graph Laplacian. Various scientific data sets from different fields of study are used to illustrate the performance of the proposed algorithm on data compression and interpolation from both regular and irregular samplings.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Lidar - Massachusetts - Leosphere Windcube 866 (120) - Raw Data

These are the data collected during the validation period. The buoy is scheduled to be deployed near Martha's Vinyard in mid-January 2020. The Leosphere Windcube 866 lidar systems output 5 different compressed data files. Compression is explained below.

17 WIND ENERGY↗

DOE OSTI2023

Lidar - Massachusetts - Leosphere Windcube 866 (130) - Raw Data

These are the data collected during the validation period. The buoy is scheduled to be deployed near Martha's Vinyard in mid-January 2020. The Leosphere Windcube 866 lidar systems output 5 different compressed data files. Compression is explained below.

17 WIND ENERGY↗

DOE OSTI2026

Precision Cascade: A novel algorithm for multi-precision extreme compression

Explore the source record for details and available documents.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

TDD LoRa and Delta Encoding in Low-Power Networks of Environmental Sensor Arrays for Temperature and Deformation Monitoring

Abstract Densely distributed sensor networks can revolutionize environmental observations by providing real-time data with an unprecedented spatiotemporal resolution. However, field deployments often pose unique challenges in terms of power provisions and wireless connectivity. We present a framework for wirelessly connected distributed sensor arrays for near-surface temperature and/or deformation monitoring. Our research focuses on a novel time division duplex implementation of the LoRa protocol, enabling battery powered base stations and avoiding collisions within the network. In order to minimize transmissions and improve battery life throughout the network, we propose a dedicated delta encoding algorithm that utilizes the spatial and temporal similarity in the acquired data sets. We implemented the developed technologies in a AA battery powered hardware platform that can be used as a wireless data logger or base station, and we conducted an assessment of the power consumption. Without data compression, the projected battery life for a data logger is 4.74 years, and a wireless base stations can last several weeks or months depending on the amount of network traffic. The delta encoding algorithm can further improve this battery life with a factor of up to 3.50. Our results demonstrate the viability of the proposed methods for low-power environmental wireless sensor networks.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2018

Deployable sensor system using mesh networking and satellite communication

A sensor system may be configured for continuous operation in a low resource environment and/or in extreme environmental conditions. The sensor system may have sufficient processing capabilities to provide scientific computing for pre-processing, quality control, statistical analysis, event classification, data compression and corrections (e.g., spikes in the data), autonomous decisions and actions, triggering other nodes, and information assurance functions that provide data confidentiality, data integrity, authentication, and non-repudiation. The hardware may have both mesh networking and satellite and cellular communication capability, and may be available for relatively low cost. Such a network provides the flexibility to have potentially any number of nodes be completely independent from one another. Thus, the network may scale across a diverse terrain.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Deployable sensor system using mesh networking and satellite communication

A sensor system may be configured for continuous operation in a low resource environment and/or in extreme environmental conditions. The sensor system may have sufficient processing capabilities to provide scientific computing for pre-processing, quality control, statistical analysis, event classification, data compression and corrections (e.g., spikes in the data), autonomous decisions and actions, triggering other nodes, and information assurance functions that provide data confidentiality, data integrity, authentication, and non-repudiation. The hardware may have both mesh networking and satellite and cellular communication capability, and may be available for relatively low cost. Such a network provides the flexibility to have potentially any number of nodes be completely independent from one another. Thus, the network may scale across a diverse terrain.

Frigo, Janette↗

DOE OSTI2020

SnowPac : a multiscale cubic B-spline wavelet compressor for astronomical images

ABSTRACT As more advanced and complex survey telescopes are developed, the size and scale of data being captured grows at increasing rates. Across various domains, data compression through wavelets has enabled the reduction of data size and increase in computation efficiency. In this paper, we provide qualitative and quantitative tests of a new wavelet-based image compression method compared against the current standard for astronomical images. The analysis is improved by making use of state-of-the-art object detection systems to accurately measure the impact of the compression. We find that a combination of lossy wavelet-based methods, efficient quantization, and lossless dictionary compressors can preserve up to 98 per cent of astronomical objects at a 10:1 compression ratio. This significant reduction in file size also preserves astronomical object properties better than existing methods. These methods help further reduce future workloads for image-heavy processing pipelines.

Pulido, Jesus↗

DOE OSTI2020

Impacts of swinging door lossy compression of synchrophasor data

As the number of operational Phasor Measurement Units (PMUs) is increasing, compression is becoming crucial for efficiently transferring and storing enormous amount of synchrophasor data continuously measured from a power system. Some utilities in the U.S. have already adopted a nonlinear compression technique, called Swinging Door Algorithm (SDA), assuming compressed data meets technical and regulatory requirements without resulting in any degradation of the measurements. This paper studies impacts of this compression algorithm on field measured and simulated PMU data, and presents results from spectral analyses of raw and decompressed measurements. The results show blind application of the SDA compression alters frequency content in the measurements and introduces false harmonics and aliasing. These can lead to false conclusions being drawn from the measurements about the power system condition.

42 ENGINEERING↗

DOE OSTI2021

A multiresolution adaptive wavelet method for nonlinear partial differential equations

We report the multiscale complexity of modern problems in computational science and engineering can prohibit the use of traditional numerical methods in multi-dimensional simulations. Therefore, novel algorithms are required in these situations to solve partial differential equations (PDEs) with features evolving on a wide range of spatial and temporal scales. To meet these challenges, we present a multiresolution wavelet algorithm to solve PDEs with significant data compression and explicit error control. We discretize in space by projecting fields and spatial derivative operators onto wavelet basis functions. We provide error estimates for the wavelet representation of fields and their derivatives. Then, our estimates are used to construct a sparse multiresolution discretization which guarantees the prescribed accuracy. Additionally, we embed a predictor-corrector procedure within the temporal integration to dynamically adapt the computational grid and maintain the accuracy of the solution of the PDE as it evolves. We present examples to highlight the accuracy and adaptivity of our approach.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Bayesian Linear Regression for Hugoniot Data

This repository provides the code and datasets used in the paper Bayesian Analysis of Linear Shock Compression Data. This paper analyzes publicly available shock compression datasets on copper, argon, and nickel from Marsh (1980) using Bayesian linear regression, and compares the results with those obtained using bootstrapping methods. References: - Marsh, S. P. (1980). LASL shock Hugoniot data (Vol. 5). Univ of California Press.

Bernstein, JasonA [Lawrence Livermore National Lab↗

DOE OSTI2025

SNAPRed: Reduction of multidimensional neutron time-of-flight diffraction data

SNAP is a neutron time-of-flight diffractometer at the Spallation Neutron Source operated by Oak Ridge National Laboratory. It generates large arrays of neutron detection events that encode the crystalline atomic structure of materials under study. SNAPRed is an application that makes these datasets accessible to end users by orchestrating the process of data reduction while automatically managing the variable neutron instrumentation configuration. It supports arbitrary grouping and masking of individual detector pixels and includes custom-developed data compression approaches to accommodate the large volumes of data generated by the SNAP instrument.

Diffraction↗

DOE OSTI2019

Multilevel Techniques for Compression and Reduction of Scientific Data---The Multivariate Case

We develop a technique for multigrid adaptive reduction of data (MGARD). Special attention is given to the case of tensor product grids, where our approach permits the use of nonuniformly spaced grids in each direction, which can prove problematic for many types of data reduction methods. An important feature of our approach is the provision of guaranteed, computable bounds on the loss incurred by the reduction of the data. Many users are leery of lossy algorithms and will only consider using them provided that numerical bounds on the pointwise difference between the original and the reduced datasets are given. Accordingly, we develop techniques for bounding the loss measured in the $L^{\infty}(\Omega)$ norm, and we show that these bounds are realistic in the sense that they do not significantly overestimate the actual loss. Here, the resulting loss indicators are used to guide the adaptive reduction of the data so that the reduced dataset meets a user-prescribed tolerance or memory constraint. Illustrative numerical examples, including the reduction of data arising from the simulation of a nonlinear reaction-diffusion problem, a turbulent channel flow, and a climate simulation, are provided.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

HBMax: Optimizing Memory Efficiency for Parallel Influence Maximization on Multicore Architectures

The goal of influence maximization is to select k most-influential vertices or seeds in a network, where influence is defined by a given diffusion process. The problem has a number of important applications such as viral marketing, information spread, and epidemic control. Although computing optimal seed set is NP-Hard, due to the submodular nature of the problem efficient approximation algorithms exist. However, even state-of-the-art parallel implementations are limited by a sampling step that incurs large memory footprints. This in turn limits the problem size reach and approximation quality. In this work, we study the memory footprint of the sampling process collecting reverse reachability information in the IMM algorithm over large real-world social networks. We present an adaptive and memory-efficient optimization approach for a state-of-the-art multi-threaded parallel influence maximization algorithm. Our approach,HuffMax, uses a portion of the reverse reachable (RR) sets collected by the algorithm to learn the characteristics of the graph. Then, it compresses the intermediate reverse reachability information with Huffman coding, and queries directly on the compressed data to preserve the memory savings obtained through compression. We also propose an efficient sampling strategy based on the distribution of RR sets, which can further reduce the computation time for typical social networks with long-tail distributions. Considering a NUMA architecture, we scale up our solution on 128-core CPUs and reduce the memory footprint by up to 45.7% with negligible time overhead (or even faster) and without perceivable loss of accuracy.

Chen, Xinyu↗

DOE OSTI2026

buhito

buhito is a Python library for graph analysis and machine learning. Graphs can represent networks with objects as nodes and their relationships as edges. buhito focuses on graphlet methods that study graphs through enumerating their component subgraphs to enable interpretable and fast models of complex systems. The package provides tools for different algorithmic designs for computing, analyzing, and applying graphlets to research problems such as machine learning, data compression, and anomaly detection in graph-structured data. A central feature is performing decomposition data analysis on graphs for machine learning models. Implemented in Python and built upon open-source scientific libraries such as NetworkX, NumPy, and SciPy, buhito provides high-performance methods for researchers exploring the mathematical and computational foundations of graphlet analysis applicable to systems of different sizes.

Pimonova, Yulia↗

DOE OSTI2025

Hybrid learning techniques for scientific data reduction with performance guarantees

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Final report- UFL - RAPIDS2: A SciDAC Institute for Computer Science, Data, and Artificial Intelligence

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Compressing unstructured mesh data from simulations using machine learning

The amount of data output from a computer simulation has grown to terabytes and petabytes as increasingly complex simulations are being run on massively parallel systems. As we approach exaflop computing in the next decade, it is expected that the I/O subsystem will not be able to write out these large volumes of data. In this paper, we explore the use of machine learning to compress the data before it is written out. Despite the computational constraints that limit us to using very simple learning algorithms, our results show that machine learning is a viable option for compressing unstructured data. Furthermore, we demonstrate that by simply using a better sampling algorithm to generate the training set, we can obtain more accurate results compared to random sampling, but at no extra cost. Further, by carefully selecting and incorporating points with high prediction error, we can improve reconstruction accuracy without sacrificing the compression rate.

97 MATHEMATICS AND COMPUTING↗