Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data compression (Computer science)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Region-adaptive, Error-controlled Scientific Data Compression using Multilevel Decomposition

The increase of computer processing speed is significantly outpacing improvements in network and storage bandwidth, leading to the big data challenge in modern science, where scientific applications can quickly generate much more data than that can be transferred and stored. As a result, big scientific data must be reduced by a few orders of magnitude while the accuracy of the reduced data needs to be guaranteed for further scientific explorations. Moreover, scientists are often interested in some specific spatial/temporal regions in their data, where higher accuracy is required. The locations of the regions requiring high accuracy can sometimes be prescribed based on application knowledge, while other times they must be estimated based on general spatial/temporal variation. In this paper, we develop a novel multilevel approach which allows users to impose region-wise compression error bounds. Our method utilizes the byproduct of a multilevel compressor to detect regions where details are rich and we provide the theoretical underpinning for region-wise error control. With spatially varying precision preservation, our approach can achieve significantly higher compression ratios than single-error bounded compression approaches and control errors in the regions of interest.We conduct the evaluations on two climate use cases – one targeting small-scale, node features and the other focusing on long, areal features. For both use cases, the locations of the features were unknown ahead of the compression. By selecting approximately 16% of the data based on multi-scale spatial variations and compressing those regions with smaller error tolerances than the rest, our approach improves the accuracy of post-analysis by approximately 2 × compared to single-error-bounded compression at the same compression ratio. Using the same error bound for the region of interest, our approach can achieve an increase of more than 50% in overall compression ratio.

Gong, Qian↗

IoGET: Internet of Geophysical and Environmental Things

The objective of this project is to provide novel and fast reduced-order models for onboard computation at sensor nodes for real-time analysis. The approach will require that LANL perform high-fidelity numerical simulations, construct simple reduced-order models (ROMs) using machine learning and signal processing algorithms, and use real-time data analysis for ROMs and compressive sensing at sensor nodes.

54 ENVIRONMENTAL SCIENCES↗

Extreme Compression for Large Scale Data Store

For the last 5 years Accelogic pioneered and perfected a radically new theory of numerical computing codenamed “Compressive Computing”, which has an extremely profound impact on real-world computer science [1]. At the core of this new theory is the discovery of one of its fundamental theorems which states that, under very general conditions, the vast majority (typically between 70% and 80%) of the bits used in modern large-scale numerical computations are absolutely irrelevant for the accuracy of the end result. This theory of Compressive Computing provides mechanisms able to identify (with high intelligence and surgical accuracy) the number of bits (i.e., the precision) that can be used to represent numbers without affecting the substance of the end results, as they are computed and vary in real time. The bottom line outcome would be to provide a state-of-the-art compression algorithm that surpasses those currently available in the ROOT framework, with the purpose of enabling substantial economic and operational gains (including speedup) for High Energy and Nuclear Physics data storage/analysis. In our initial studies, a factor of nearly x4 (3.9) compression was achieved with RHIC/STAR data where ROOT compression managed only x1.4.In this contribution, we will present our concepts of “functionally lossless compression”, have a glance at examples and achievements in other communities, present the results and outcome of our current, ongoing R&D, as well as present a high-level view of our plan to move forward with a ROOT implementation that would deliver a basic solution readily integrated into HENP applications. As a collaboration of experimental scientists, private industry, and the ROOT Team, our aim is to capitalize on the substantial success delivered by the initial effort and produce a robust technology properly packaged as an open-source tool that could be used by virtually every experiment around the world as means for improving data management and accessibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

In Situ Indexing and Query Processing of AMR Data

We have developed the first, holistic, Adaptive Mesh Refinement (AMR) capable, in situ query framework to ensure the tractability of the current- and next-generation AMR simulation analysis on the path to extreme scale. We have addressed key issues pertaining to in situ analysis, memory hierarchy, generalized parallel indexing, index-compression, and real-time access pattern detection. We have published the following research papers in high profile peer reviewed computer science conferences: (a) Parallel In Situ Detection of Connected Components in Adaptive Mesh Refinement Data; (b) AMR-aware In Situ Indexing and Scalable Querying; (c) Pattern-driven dynamic data layout optimization; (d) In situ Storage Layout Optimization for AMR Spatio-temporal Read Accesses; (e) AMRZone: A Runtime AMR Data Sharing Framework for Scientific Applications; (f) Exploring Memory Hierarchy and Network Topology for Runtime AMR Data Sharing Across Scientific Applications.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

A lithium projector augmented wave potential suitable for use in VASP at high compression and temperature

Density Functional Theory (DFT) based Equation of State (EOS) construction is a prominent part of Sandia’s capabilities to support engineering sciences. This capability is based on augmenting experimental data with information gained from computational investigations, especially in those parts of the phase space where experimental data is hard, dangerous, or expensive to obtain. A key part of the success of the Sandia approach is the fundamental science work supporting the computational capability. Not only does this work enhance the capability to perform highly accurate calculations but it also provides crucial insight into the limitations of the computational tools, providing high confidence in the results even where results can not be, or have not yet been, validated by experimental data. This report concerns the key ingredient of projector augmented-wave potentials for use in pseudo-potential computational codes. While the insights are general for all materials and codes, I specifically address calculations of highly compressed, high temperature, lithium using the Vienna Ab-initio Simulation Package.

Mattsson, Ann Elisabet↗

Parallel Tensor Compression for Large-Scale Scientific Data.

As parallel computing trends towards the exascale, scientific data produced by high-fidelity simulations are growing increasingly massive. For instance, a simulation on a three-dimensional spatial grid with 512 points per dimension that tracks 64 variables per grid point for 128 time steps yields 8 TB of data. By viewing the data as a dense five way tensor, we can compute a Tucker decomposition to find inherent low-dimensional multilinear structure, achieving compression ratios of up to 10000 on real-world data sets with negligible loss in accuracy. So that we can operate on such massive data, we present the first-ever distributed memory parallel implementation for the Tucker decomposition, whose key computations correspond to parallel linear algebra operations, albeit with nonstandard data layouts. Our approach specifies a data distribution for tensors that avoids any tensor data redistribution, either locally or in parallel. We provide accompanying analysis of the computation and communication costs of the algorithms. To demonstrate the compression and accuracy of the method, we apply our approach to real-world data sets from combustion science simulations. We also provide detailed performance results, including parallel performance in both weak and strong scaling experiments.

59 BASIC BIOLOGICAL SCIENCES↗

Maximizing efficiency of dataset compression for machine learning potentials with information theory

Machine learning interatomic potentials (MLIPs) balance high accuracy and lower costs compared to density functional theory calculations, but their performance often depends on the size and diversity of training datasets. Large datasets improve model accuracy and generalization but are computationally expensive to produce and train on, while smaller datasets risk discarding rare but important atomic environments and compromising MLIP accuracy/reliability. Here, we develop an information-theoretical framework to quantify the efficiency of dataset compression methods and propose an algorithm that maximizes this efficiency. By framing atomistic dataset compression as an instance of the minimum set cover (MSC) problem over atom-centered environments, our method identifies the smallest subset of structures that contains as much information as possible from the original dataset while pruning redundant information. The approach is extensively demonstrated on the GAP-20 and TM23 datasets and validated on 64 varied datasets from the ColabFit repository. Across all cases, MSC consistently retains outliers, preserves dataset diversity, and reproduces the long-tail distributions of forces even at high compression rates, outperforming other subsampling methods. Furthermore, MLIPs trained on MSC-compressed datasets exhibit reduced error for out-of-distribution data even in low-data regimes. We explain these results using an outlier analysis and show that such quantitative conclusions could not be achieved with conventional dimensionality reduction methods. The algorithm is implemented in the open-source QUESTS package and can be used for several tasks in atomistic modeling, from data subsampling, outlier detection, and training improved MLIPs at a lower cost.

36 MATERIALS SCIENCE↗

CESDIS

CESDIS, the Center of Excellence in Space Data and Information Sciences was developed jointly by NASA, Universities Space Research Association (USRA), and the University of Maryland in 1988 to focus on the design of advanced computing techniques and data systems to support NASA Earth and space science research programs. CESDIS is operated by USRA under contract to NASA. The Director, Associate Director, Staff Scientists, and administrative staff are located on-site at NASA's Goddard Space Flight Center in Greenbelt, Maryland. The primary CESDIS mission is to increase the connection between computer science and engineering research programs at colleges and universities and NASA groups working with computer applications in Earth and space science. Research areas of primary interest at CESDIS include: 1) High performance computing, especially software design and performance evaluation for massively parallel machines; 2) Parallel input/output and data storage systems for high performance parallel computers; 3) Data base and intelligent data management systems for parallel computers; 4) Image processing; 5) Digital libraries; and 6) Data compression. CESDIS funds multiyear projects at U. S. universities and colleges. Proposals are accepted in response to calls for proposals and are selected on the basis of peer reviews. Funds are provided to support faculty and graduate students working at their home institutions. Project personnel visit Goddard during academic recess periods to attend workshops, present seminars, and collaborate with NASA scientists on research projects. Additionally, CESDIS takes on specific research tasks of shorter duration for computer science research requested by NASA Goddard scientists.

Source record↗

Automated pipeline processing X-ray diffraction data from dynamic compression experiments on the Extreme Conditions Beamline of PETRA III

Presented and discussed here is the implementation of a software solution that provides prompt X-ray diffraction data analysis during fast dynamic compression experiments conducted within the dynamic diamond anvil cell technique. It includes efficient data collection, streaming of data and metadata to a high-performance cluster (HPC), fast azimuthal data integration on the cluster, and tools for controlling the data processing steps and visualizing the data using the DIOPTAS software package. This data processing pipeline is invaluable for a great number of studies. The potential of the pipeline is illustrated with two examples of data collected on ammonia–water mixtures and multiphase mineral assemblies under high pressure. The pipeline is designed to be generic in nature and could be readily adapted to provide rapid feedback for many other X-ray diffraction techniques, e.g. large-volume press studies, in situ stress/strain studies, phase transformation studies, chemical reactions studied with high-resolution diffraction etc.

97 MATHEMATICS AND COMPUTING↗

Catching the whispers from Uranus

Sophisticated telecommunications techniques are described that were used to acquire images of Uranus, its 14 moons and ten narrow rings darker than coal. The images, equal in quality to those transmitted from Saturn several years earlier despite the signal being weaker by 6 dB due to the increased distance, were received from Voyager 2 during its January 24, 1986 flyby of Uranus. Solutions to the problem of the weakening signal were found in modifications to Voyager's image processing system and NASA's ground tracking network. In April 1985, Voyager's prime flight data computer was reconfigured to accept only nonimaging science data, and its backup, only imaging data; the latter was reprogrammed to determine only arithmetic differences between adjacent pixel intensities rather than absolute intensities. By image compression, equivalent imaging information could be sent at lower bit rates. Instead of Golay coding, Reed-Solomon onboard encoding was used. These techniques gained the equivalent of 4-dB in imaging yield. Additional improvements were gained by using earth station antennas in pairs (the Parkes radio telescope and the Canberra ground station antenna). Moves under way to prepare for the Voyager encounter with Neptune in 1989 are described (using additional antennas and arrays, scaling up the Deep Space Network antennas from 64 m to 70 m, etc.) to assure almost Saturn-equivalent pictures despite a further 3.5-dB drop in signal strength.

Bartok, C. D.↗

Hugoniot Models for Na and LiF from LEOS

In this document, we provide the Hugoniot for sodium from two models: LEOS table L110 and Lynx table 110. We also provide the Hugoniot for lithium fluoride from LEOS (L2240) and Lynx (2240). The Hugoniot pressures are supplied for temperatures between 338.0 and 1.16×10 9 Kelvin and densities between 0.968 and 11.5 g/cc. These LEOS models were developed by the quotidian EOS methodology, which is a widely used and robust method for producing tabular EOS data. Tables list the model data for LEOS 110, Lynx 110, LEOS 2240, and Lynx 2240. The Lynx models follow the same methodology as the LEOS models; however, the Purgatorio average-atom DFT code was used to compute the electron thermal part of the EOS. The models for Lynx are only listed at high compression due to known issues with the Lynx library at lower pressures.

36 MATERIALS SCIENCE↗

Determination of the Critical Parameters for Remote Microscope Control

As part of a program to determine the capabilities of Telescience as applied to Microgravity Materials Science the need for a remotely controlled microscope was recognized. For this purpose we equipped a microscope with an X-Y-Z positioning device and motors on the zoom and focus controls. Computer control of these devices allowed remote operation. A standard TV camera was mounted to the computer controlled video board which could compress the image in resolution and grey scale. The operator control console was programmed to display three still video pictures as well as provide command access. A standard data transfer network was used to transmit the video data files and the command interaction was via a high speed phone modem. This system, with the microscope in the Microgravity Materials Science Laboratory (MMSL) at LeRC and the control at RPI, was used to determine the accuracy of setting, time required to achieve setting and the operator ease factor. It was found that the focus setting could be established well within the resolution limit of the TV system and that each motion took about 50 seconds and approximately 12 minutes was required to reach ?best? focus. These times could be reduced significantly with operator experience. The operators were provided with ancillary equipment which provided assistance in making the necessary decisions and they reported satisfaction with the control.

Hahn, R. C.↗

Data compression for data archival, browse or quick-look

Soon after space and Earth science data is collected, it is stored in one or more archival facilities for later retrieval and analysis. Since the purpose of the archival process is to keep an accurate and complete record of data, any data compression used in an archival system must be lossless, and protect against propagation of error in the storage media. A browse capability for space and Earth science data is needed to enable scientists to check the appropriateness and quality of particular data sets before obtaining the full data set(s) for detailed analysis. Browse data produced for these purposes could be used to facilitate the retrieval of data from an archival facility. Quick-look data is data obtained directly from the sensor for either previewing the data or for an application that requires very timely analysis of the space or Earth science data. Two main differences between data compression techniques appropriate to browse and quick-look cases, are that quick-look can be more specifically tailored, and it must be limited in complexity by the relatively limited computational power available on space platforms.

Dozier, Jeff↗