Search NASA⌕ Search

SEARCH · Search NASA

Results for “Methods: data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 919 records · Page 51
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Small Angle X-Ray Scattering and Polymers

Polymers are some of the most versatile and useful materials on this planet. Properties of polymers are a result of their processing history, chemical makeup, and physical structure. Therefore, characterization is necessary to develop new polymer formulations and processes. Many established techniques for polymer characterization are destructive, and alternative methods that don’t destroy samples are in high demand. Thermal analysis techniques such as differential scanning calorimetry (DSC) and thermogravimetric analysis (TGA) yield important characteristic information for polymer samples while destroying the sample. Non-destructive strategies such as X ray diffraction (XRD) or small angle X-ray scattering (SAXS) can also find polymer characteristics while preserving a sample’s attributes. Sample preservation is desirable when investigating novel formulation and processing methods. SAXS analysis will be done on two different polymers: polyethylene, and a triblock copolymer. Using these two polymers allows for confidence between tests as polyethylene is very common, and well understood. Polyethylene analysis will be done first, and when methods have been refined so that the SAXS data matches accepted thermal analysis values for HDPE, there will be confidence to use the same analysis on block copolymers. Block copolymers remain relatively unexplored, so SAXS data will yield valuable information on their characteristics. Understanding the mechanical properties of these polymers is paramount to their application in upcycling, a process by which waste plastic can be converted into a higher value commodity. For this project, the upcycling process will be done via 3-dimensional (3d) printing.

36 MATERIALS SCIENCE↗
DOE OSTI2013

Predicting Radiation Pneumonitis After Chemoradiation Therapy for Lung Cancer: An International Individual Patient Data Meta-analysis

Background: Radiation pneumonitis is a dose-limiting toxicity for patients undergoing concurrent chemoradiation therapy (CCRT) for non-small cell lung cancer (NSCLC). We performed an individual patient data meta-analysis to determine factors predictive of clinically significant pneumonitis. Methods and Materials: After a systematic review of the literature, data were obtained on 836 patients who underwent CCRT in Europe, North America, and Asia. Patients were randomly divided into training and validation sets (two-thirds vs one-third of patients). Factors predictive of symptomatic pneumonitis (grade {>=}2 by 1 of several scoring systems) or fatal pneumonitis were evaluated using logistic regression. Recursive partitioning analysis (RPA) was used to define risk groups. Results: The median radiation therapy dose was 60 Gy, and the median follow-up time was 2.3 years. Most patients received concurrent cisplatin/etoposide (38%) or carboplatin/paclitaxel (26%). The overall rate of symptomatic pneumonitis was 29.8% (n=249), with fatal pneumonitis in 1.9% (n=16). In the training set, factors predictive of symptomatic pneumonitis were lung volume receiving {>=}20 Gy (V{sub 20}) (odds ratio [OR] 1.03 per 1% increase, P=.008), and carboplatin/paclitaxel chemotherapy (OR 3.33, P<.001), with a trend for age (OR 1.24 per decade, P=.09); the model remained predictive in the validation set with good discrimination in both datasets (c-statistic >0.65). On RPA, the highest risk of pneumonitis (>50%) was in patients >65 years of age receiving carboplatin/paclitaxel. Predictors of fatal pneumonitis were daily dose >2 Gy, V{sub 20}, and lower-lobe tumor location. Conclusions: Several treatment-related risk factors predict the development of symptomatic pneumonitis, and elderly patients who undergo CCRT with carboplatin-paclitaxel chemotherapy are at highest risk. Fatal pneumonitis, although uncommon, is related to dosimetric factors and tumor location.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2023

Understanding Biases in Sample Preparation Techniques for Coupled Scanning Electron Microscopy and MAMA PuO 2 Morphological Analysis

In this project, the scanning electron microscopy (SEM) sampling method used during the statistical design study (SDS) was investigated to determine if any sampling biases were present in the analyzed data. Using standard particle size distribution powders from the National Institute of Standards and Technology (NIST 1984 standard reference material) with the origin wet dispersion method, it was determined that a bias to smaller particles was present. This was supported by theoretical calculations using Stokes’ law to determine the settling rate of spherical particles of roughly the same size and mass as those found in the SDS. Based on the theoretical calculations, it was determined that the settling rate for each of the 76 powder sets in the SDS could be unique based on specific particle shape and mass distributions, making a universal correction factor/formula not applicable. Therefore, priority shifted to developing an improved wet dispersion method that significantly reduced the particle settling rate for all particle size and shapes. This was achieved by replacing the original solvent (isopropyl alcohol) with a heavy liquid (lithium heteropolytungstates), which dramatically slowed the settling rate and allowed for the capture of a suitable homogeneous aliquot. SEM imaging and Morphological Analysis for Material Attribution (MAMA) software analysis were conducted on the NIST standard, and the SEM/MAMA data were compared to data captured by a dynamic image analysis particle size analyzer. The resulting data confirmed that the new wet dispersion method does indeed deliver an improved representative aliquot to the SEM stub. For instance, in the NIST certificate, the average particle size is ~17.1 µm ± 2.2 µm with a normal distribution. The initial wet dispersion method resulted in a drastically reduced average particle size of 6.1 µm in addition to a non-representative heavy bi-modal distribution whereas the improved LST wet dispersion method resulting in an average particle size that was much closer to the NIST certificate (12.7 µm) with a similar normal distribution. Although the improved method was still short of the NIST certificate average, atomic force microscopy analysis determined that the resulting ~20-25% reduction in size was due to particles sinking into the carbon sticky tape used for SEM imaging. It is believed that that this bias can be calibrated in a much more predicable manner than the original settling rate bias. In addition, the matching normal distribution curves between the NIST certificate and the heavy liquid method indicate a much-improved representative aliquot has been sampled and imaged. A surrogate CeO 2 powder was used to reflect PuO 2 more accurately and to aid in implementing radiological controls and shielding. The resulting data sets from the SEM/MAMA method and the particle size analyzer give almost identical average particle sizes and particle distribution statistics. Future work will re-analyze several select runs from the SDS to determine if morphological signatures can be found with the improved sampling method.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2017

Dark matter constraints from an observation of dSphs and the LMC with the Baikal NT200

We have analyzed the neutrino events recoded in the deep-water neutrino experiment NT200 in Lake Baikal in five years of observations toward dark dwarf spheroidal galaxies (dSphs) in the southern hemisphere and the Large Magellanic Cloud (LMC). This analysis completes the series of works based on NT200 data in the search for a dark matter annihilation signal in astrophysical objects. We have found no significant excess in the number of observed events relative to the expected background from atmospheric neutrinos in all tested directions, in 22 dSphs and the LMC. For a sample of five selected dwarf galaxies we have performed a joint analysis of the data by the maximum likelihood method. We have obtained a correspondence of the observational data to the null hypothesis about the presence of only background events and established 90% confidence-level upper limits for the annihilation cross sections of dark matter particles with a mass from 30 GeV to 10 TeV in several annihilation channels both in the joint analysis of the selected sample of galaxies and in the analysis toward the LMC. The strongest constraints at a level of 7 × 10{sup –21} cm{sup 3} s{sup –1} have been obtained for the direction toward the LMC in the channel of annihilation into a pair of neutrinos.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

Multi-Gas Sensors for Enhanced Reliability of SOFC Operation

GE Research, in partnership with SUNY Polytechnic Institute (SUNY Poly), designed, built, and tested gas sensors for in situ monitoring of H 2 and CO anode tail gases produced with on-site steam reforming in solid oxide fuel cell (SOFC) systems. The knowledge of the H 2 /CO ratio of these anode tail gases should allow accurate determination and control of the efficiency of the reforming process in the SOFC system and should deliver a lower operating cost for SOFC customers. The project objectives were to achieve multi-gas monitoring capability with a single multivariable sensor, and to sustain this performance in the presence of gaseous interferences and a potential poison for the sensor. The duration of the project was 24 months with the project structure that included three technical tasks such as (1) development of design rules of photonic nanostructures for H 2 and CO gas detection, (2) laboratory validation of photonic nanostructures for selective H 2 and CO gas detection, and (3) validation of photonic nanostructures for initial stability and poison-resistance against H 2 S. To build multivariable sensors for selective H 2 and CO gas detection in the presence of interferences, we expanded our earlier knowledge of multi-gas sensors into new fabrication and functionalization methodologies as well as into new methodologies for the spectral data analysis of multi-gas responses. We have advanced our design rules of the three-dimensional (3D) photonic nanostructures that allowed detection of H 2 and CO at high temperatures as individual gases and as their mixtures and rejection of interferences such as CO 2 , H 2 O, CH 4 , and other hydrocarbons for SOFC applications. Our advanced design rules should be attractive for building the new generation of cost-effective industrial sensors. Stability and poison resistance of our 3D photonic nanostructures was tested in the laboratory conditions. While initially we utilized conventional machine learning data analysis tools, we have found that they were unable to correct for the sensor drift. Thus, we have implemented new methods of machine learning for the analysis of our spectral data. These learnings pave the way to move the future studies into advanced testing of effects of interferences, aging and field tests. In future, our work will continue to advance our sensing designs to operate in conditions with known and unknown interferences by implementing nanostructures with enhanced spectral diversity of responses to gaseous species of interest and interferences. Our systematic reduction of technical risks in this completed project and in future studies will ensure transition of this sensing technology to commercialization.

03 NATURAL GAS↗
DOE OSTI2024

Commercial, industrial, and institutional discount rate estimation for efficiency standards analysis: Sector-level data 1998–2023

Underlying each of the U.S. Department of Energy’s (DOE’s) federal appliance and equipment energy conservation standards are a set of complex analyses of the projected costs and benefits of regulation. Any new or amended standard must be designed to achieve significant additional energy conservation, provided that it is technologically feasible and economically justified (42 U.S.C. 6295(o)(2)(A)). DOE determines economic justification based on whether the benefits exceed the burdens, considering a variety of factors, including the economic impact of the standard on consumers of the product and the savings in lifetime operating cost compared to any increase in price or maintenance expenses (42 U.S.C. 6295(o)(2)(B)). As part of this determination, DOE conducts a life-cycle cost (LCC) analysis, which models the combined impact of appliance first cost and operating cost changes on a representative commercial building sample to identify the fraction of customers achieving LCC savings or incurring net cost at the considered efficiency levels. Thus, the commercial discount rate value(s) used to calculate the present value of energy cost savings within the LCC model implicitly plays a role in estimating the economic impact of potential standard levels. This report provides an in-depth discussion of the commercial discount rate estimation process relying on the Capital Asset Pricing Model (CAPM) to estimate a business’ cost of equity, and by adding a risk adjustment factor to the risk-free rate associated with long-term U.S. Treasury bonds to estimate their cost of debt. It is an update to previous reports on estimating commercial discount rates from firm-level and sector-level financial data (e.g., Fujita, 2021, 2016). Major topics covered in this report include the following: • Discount rate estimation methods and rationale • Data sources used and data limitations • Discount rate distributions for use in standards analysis • Discount rate estimation methods and distributions specific to the small business subgroup analysis.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Extending quantum-mechanical benchmark accuracy to biological ligand-pocket interactions

Predicting the binding affinity of ligands to protein pockets is key in the drug design pipeline. The flexibility of ligand-pocket motifs arises from a range of attractive and repulsive electronic interactions during binding. Accurately accounting for all interactions requires robust quantum-mechanical (QM) benchmarks, which are scarce for ligand-pocket systems. Additionally, disagreement between “gold standard” Coupled Cluster (CC) and Quantum Monte Carlo (QMC) methods casts doubt on many benchmarks for larger non-covalent systems. We introduce the “QUantum Interacting Dimer” (QUID) benchmark framework containing 170 non-covalent (non-)equilibrium systems modeling chemically and structurally diverse ligand-pocket motifs. Symmetry-adapted perturbation theory shows that QUID broadly covers non-covalent binding motifs and energetic contributions. Robust binding energies are obtained using complementary CC and QMC methods, achieving agreement of 0.5 kcal/mol. The benchmark data analysis reveals that several dispersion-inclusive density functional approximations provide accurate energy predictions, though their atomic van der Waals forces differ in magnitude and orientation. Contrarily, semiempirical methods and empirical force fields require improvements in capturing non-covalent interactions (NCIs) for out-of-equilibrium geometries. The wide span of NCIs, highly accurate interaction energies, and analysis of molecular properties take QUID beyond the “gold standard” for QM benchmarks of ligand-protein systems.

Puleva, Mirela [University of Luxembourg, Luxembou↗
DOE OSTI2023

Combining cosmic shear data with correlated photo-z uncertainties: constraints from DESY1 and HSC-DR1

An accurate calibration of the source redshift distribution p(z) is a key aspect in the analysis of cosmic shear data. This, one way or another, requires the use of spectroscopic or high-quality photometric samples. However, the difficulty to obtain colour-complete spectroscopic samples matching the depth of weak lensing catalogs means that the analyses of different cosmic shear datasets often use the same samples for redshift calibration. This introduces a source of statistical and systematic uncertainty that is highly correlated across different weak lensing datasets, and which must be accurately characterised and propagated in order to obtain robust cosmological constraints from their combination. In this paper we introduce a method to quantify and propagate the uncertainties on the source redshift distribution in two different surveys sharing the same calibrating sample. The method is based on an approximate analytical marginalisation of the p(z) statistical uncertainties and the correlated marginalisation of residual systematics. We apply this method to the combined analysis of cosmic shear data from the DESY1 data release and the HSC-DR1 data, using the COSMOS 30-band catalog as a common redshift calibration sample. We find that, although there is significant correlation in the uncertainties on the redshift distributions of both samples, this does not change the final constraints on cosmological parameters significantly. The same is true also for the impact of residual systematic uncertainties from the errors in the COSMOS 30-band photometric redshifts. Additionally, we show that these effects will still be negligible in Stage-IV datasets. Finally, the combination of DESY1 and HSC-DR1 allows us to constrain the "clumpiness" parameter to S 8 = ${0.768}_{-0.017}^{+0.021}$. This corresponds to a ~√(2) improvement in uncertainties with respect to either DES or HSC alone.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Multivariate Analysis of the Influence of Process Parameters on Suspension Plasma-Sprayed Coatings

Suspension plasma spray (SPS) is far more complicated than conventional plasma spray and requires a deep knowledge about the influence of process parameters and their correlations. In this study, YSZ coatings were manufactured by SPS with six different process parameters such as plasma power, suspension mass load, original powder size, substrate surface topology, spray distance, and spray step. Afterwards, the porosity of as-prepared coatings was investigated by image method and X-ray transmission technique. A multivariate analysis on the collected experimental data was carried out by employing mathematical statistics methods. The results showed that: 1) Coating porosity has a negative correlation with plasma power and suspension mass load and a positive correlation with the original powder size, spray distance, spray step, and substrate roughness; 2) Spraying distance is the main factor affecting to coating porosity, followed by suspension mass load and substrate surface roughness, respectively. A linear model for porosity prediction was developed and was verified by experiments. The mechanism by which process parameters influence coating porosity is also discussed.

Zhao, Yongli↗
DOE OSTI2024

Tracking the topology of neural manifolds across populations

Neural manifolds summarize the intrinsic structure of the information encoded by a population of neurons. Advances in experimental techniques have made simultaneous recordings from multiple brain regions increasingly commonplace, raising the possibility of studying how these manifolds relate across populations. However, when the manifolds are nonlinear and possibly code for multiple unknown variables, it is challenging to extract robust and falsifiable information about their relationships. We introduce a framework, called the method of analogous cycles, for matching topological features of neural manifolds using only observed dissimilarity matrices within and between neural populations. We demonstrate via analysis of simulations and in vivo experimental data that this method can be used to correctly identify multiple shared circular coordinate systems across both stimuli and inferred neural manifolds. Conversely, the method rejects matching features that are not intrinsic to one of the systems. Further, as this method is deterministic and does not rely on dimensionality reduction or optimization methods, it is amenable to direct mathematical investigation and interpretation in terms of the underlying neural activity. We thus propose the method of analogous cycles as a suitable foundation for a theory of cross-population analysis via neural manifolds.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Monte Carlo toolkit for designing and validating step-range-filter spectrometer designs

Here, we present a Monte Carlo toolkit for validating step range filter (SRF) spectrometer designs. Geant4 is used to transport charged particles through the SRF filters to generate synthetic SRF data that include realistic CR-39 effects. Synthetic SRF spectra generated by this method inherently account for instrument response and allow for the quantification of SRF performance before shots. The usefulness of this toolkit is demonstrated through its application to a number of problems. A new broadband SRF for the ∼10 MeV wide 3He3He proton spectrum is validated, and an analysis method for analyzing 3He3He-p SRF data that accounts for instrument response is put forth. In addition, an SRF design for the compact recoil-proton spectrometer (CRS) on the Z-machine is validated. Finally, a new calibration technique for the DD-p SRF is proposed and validated.

Johnson, T. M. (ORCID:0000000193032949)↗
DOE OSTI2024

Leveraging graph clustering techniques for cyber‐physical system analysis to enhance disturbance characterisation

Abstract Cyber‐physical systems have behaviour that crosses domain boundaries during events such as planned operational changes and malicious disturbances. Traditionally, the cyber and physical systems are monitored separately and use very different toolsets and analysis paradigms. The security and privacy of these cyber‐physical systems requires improved understanding of the combined cyber‐physical system behaviour and methods for holistic analysis. Therefore, the authors propose leveraging clustering techniques on cyber‐physical data from smart grid systems to analyse differences and similarities in behaviour during cyber‐, physical‐, and cyber‐physical disturbances. Since clustering methods are commonly used in data science to examine statistical similarities in order to sort large datasets, these algorithms can assist in identifying useful relationships in cyber‐physical systems. Through this analysis, deeper insights can be shared with decision‐makers on what cyber and physical components are strongly or weakly linked, what cyber‐physical pathways are most traversed, and the criticality of certain cyber‐physical nodes or edges. This paper presents several types of clustering methods for cyber‐physical graphs of smart grid systems and their application in assessing different types of disturbances for informing cyber‐physical situational awareness. The collection of these clustering techniques provide a foundational basis for cyber‐physical graph interdependency analysis.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Decay curve analysis for data error quantification in time-domain induced polarization imaging

In recent years, the time-domain induced polarization (TDIP) imaging technique has emerged as a suitable method for the characterization and the monitoring of hydrogeologic and biogeochemical processes. However, one of the major challenges refers to the resolution of the electrical images. Hence, various studies have stressed the importance of data processing, error characterization, and the deployment of adequate inversion schemes. A widely accepted method to assess data error in electrical imaging relies on the analysis of the discrepancy between normal and reciprocal measurements. Nevertheless, the collection of reciprocals doubles the acquisition time and is only viable for a limited subset of commonly used electrode configurations (e.g., dipole-dipole [DD]). To overcome these limitations, we have developed a new methodology to quantify the data error in TDIP imaging, which is entirely based on the analysis of the recorded IP decay curve and does not require recollection of data (e.g., reciprocals). The first two steps of the methodology assess the general characteristics of the decay curves and the spatial consistency of the measurements for the detection and removal of outliers. In the third and fourth steps, we quantify the deviation of the measured decay curves from a smooth model for the estimation of random error of the total chargeability and transfer resistance measurement. The error models and imaging results obtained from this methodology - in the following referred to as "decay curve analysis" - are compared with those obtained following a conventional normal-reciprocal analysis revealing consistent results. As a result, we determine the applicability of our methodology with real field data collected at the floodplain scale (approximately 12 ha) using multiple gradient and DD configurations.

58 GEOSCIENCES↗
DOE OSTI2022

Physics-informed graph neural networks for predicting cetane number with systematic data quality analysis

Designing alternative fuels for advanced compression ignition engines necessitates a predictive model for cetane number (CN). In this study, the physics-informed graph neural networks are introduced for a reliable CN prediction by considering molecular features pertinent to the physical properties of molecules that affect CN. The reliability of measured data is another key factor to consider for improving the predictive model. Various experimental instruments for measuring CN exist, including standard and non-standard methods. In this regard, a systematic data quality analysis was carried out for the total 630 CNs collected from literature and new measurements in this study using Advanced Fuel Ignition Delay Analyzer (AFIDA). The results from this data curation process were reflected in the model by imposing lower sample weights on the data coming from less reliable measurement techniques. This approach effectively maximized the prediction accuracy while incorporating data from all available sources. Using the sample weights decreased the mean absolute error (MAE) up to 0.8 CN units. The accuracy was also improved by introducing the CN-related physical properties (the number of hydrogen bond donors and acceptors); the test set MAE is 5.74 and 7.01 for the model with and without such properties, respectively. Investigating molecular structural effects on CN was also carried out to gain chemical insights into factors used to design new fuel candidates. The dimensionality reduction analysis of feature vectors showed a clear clustering in terms of functional groups and CN and the structural effect derived from the model was consistent with the physicochemical insights. Finally, this physics-informed model and data curation would be helpful for accurate CN prediction and inform rational fuel design.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Data Mining and Machine Learning for Power System Monitoring, Understanding, and Impact Evaluation

This chapter presents results from the Big Data analysis framework to improve power system situational awareness and system reliability. For this purpose, a dataset with real-world phasor measurement unit data and historical transmission system outage data has been created and used to carry out the analysis. Several statistical analysis and machine learning methods have been developed and implemented for event and anomaly detection and modeling. Detection and analysis results for actual examples of power system events are presented. Finally, data-driven characterization and risk assessment methods for weather-related extremes in power systems are developed and demonstrated on the Bonneville Power Administration system. These applications demonstrate the capability of Machine Learning (ML) methods to monitor system abnormalities, to predict system events, and to characterize the impact of extreme events on power grid

data mining, power grid, machine learning, anomaly↗
DOE OSTI2020

A novel CMB component separation method: hierarchical generalized morphological component analysis

ABSTRACT We present a novel technique for cosmic microwave background (CMB) foreground subtraction based on the framework of blind source separation. Inspired by previous work incorporating local variation to generalized morphological component analysis (GMCA), we introduce hierarchical GMCA (HGMCA), a Bayesian hierarchical graphical model for source separation. We test our method on Nside = 256 simulated sky maps that include dust, synchrotron, free–free, and anomalous microwave emission, and show that HGMCA reduces foreground contamination by $25{{\ \rm per\ cent}}$ over GMCA in both the regions included and excluded by the Planck UT78 mask, decreases the error in the measurement of the CMB temperature power spectrum to the 0.02–0.03 per cent level at ℓ > 200 (and $\lt 0.26{{\ \rm per\ cent}}$ for all ℓ), and reduces correlation to all the foregrounds. We find equivalent or improved performance when compared to state-of-the-art internal linear combination type algorithms on these simulations, suggesting that HGMCA may be a competitive alternative to foreground separation techniques previously applied to observed CMB data. Additionally, we show that our performance does not suffer when we perturb model parameters or alter the CMB realization, which suggests that our algorithm generalizes well beyond our simplified simulations. Our results open a new avenue for constructing CMB maps through Bayesian hierarchical analysis.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

Synthesis route attribution of sulfur mustard by multivariate data analysis of chemical signatures

A multivariate model was developed here to attribute samples to a synthetic method used in the production of sulfur mustard (HD). Eleven synthetic methods were used to produce 66 samples for model construction. Three chemists working in both participating laboratories took part in the production, with the aim to introduce variability while reducing the influence of laboratory or chemist specific impurities in multivariate analysis. A gas chromatographic/mass spectrometric data set of peak areas for 103 compounds was subjected to orthogonal partial least squares - discriminant analysis to extract chemical attribution signature profiles and to construct multivariate models for classification of samples. For one- and two-step routes, model quality allowed the classification of an external test set (16/16 samples) according to synthesis conditions in the reaction yielding sulfur mustard. Classification of samples according to first-step methodology was considerably more difficult, given the high purity and uniform quality of the intermediate thiodiglycol produced in the study. Model performance in classification of aged samples was also investigated.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

A General Framework for Progressive Data Compression and Retrieval

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. Here, this paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

97 MATHEMATICS AND COMPUTING↗