Using Unsupervised Anomaly Detection for Data Reduction in Temporal and Spatial Data.
Abstract not provided.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Abstract not provided.
Smart pixels with data reduction at source.
Raw data generation for several existing and planned large physics experiments now exceeds TB/s rates, generating untenable data sets in very little time. Those data often demonstrate high dimensionality while containing limited information. Meanwhile, Machine Learning algorithms are now becoming an essential part of data processing and data analysis. Those algorithms can be used offline for post processing and post data analysis, or they can be used online for real time processing providing ultra low latency experiment monitoring. Both use cases would benefit from data throughput reduction while preserving relevant information: one by reducing the offline storage requirements by several orders of magnitude and the other by allowing ultra fast online inferencing with low complexity Machine Learning models. Moreover, reducing the data source throughput also reduces material cost, power and data management requirements. In this work we demonstrate optimized nonuniform scalar quantization for data source reduction. This data reduction allows lower dimensional representations while preserving the relevant information of the data, thus enabling high accuracy Tiny Machine Learning classifier models for online fast inferences. We demonstrate this approach with an initial proof of concept targeting the CookieBox, an array of electron spectrometers used for angular streaking, that was developed for LCLS-II as an online beam diagnostic tool. We used the Lloyd-Max algorithm with the CookieBox dataset to design an optimized nonuniform scalar quantizer. Optimized quantization lets us reduce input data volume by 69% with no significant impact on inference accuracy. When we tolerate a 2% loss on inference accuracy, we achieved 81% of input data reduction. Finally, the change from a 7-bit to a 3-bit input data quantization reduces our neural network size by 38%.
Large-scale supercomputing is revolutionizing the way science is conducted. A growing challenge, however, is understanding the massive quantities of data produced by large-scale simulations. The data, typically time-varying, multivariate, and volumetric, can occupy from hundreds of gigabytes to several terabytes of storage space. Transferring and processing volume data of such sizes is prohibitively expensive and resource intensive. Although it may not be possible to entirely alleviate these problems, data compression should be considered as part of a viable solution, especially when the primary means of data analysis is volume rendering. In this paper we present our study of multivariate compression, which exploits correlations among related variables, for volume rendering. Two configurations for multidimensional compression based on vector quantization are examined. We emphasize quality reconstruction and interactive rendering, which leads us to a solution using graphics hardware to perform on-the-fly decompression during rendering. In this paper we present a solution which addresses the need for data reduction in large supercomputing environments where data resulting from simulations occupies tremendous amounts of storage. Our solution employs a lossy encoding scheme to acrueve data reduction with several options in terms of rate-distortion behavior. We focus on encoding of multiple variables together, with optional compression in space and time. The compressed volumes can be rendered directly with commodity graphics cards at interactive frame rates and rendering quality similar to that of static volume renderers. Compression results using a multivariate time-varying data set indicate that encoding multiple variables results in acceptable performance in the case of spatial and temporal encoding as compared to independent compression of variables. The relative performance of spatial vs. temporal compression is data dependent, although temporal compression has the advantage of offering smooth animations, while spatial compression can handle volumes of larger dimensions.
OSIRIS is a near-infrared (1.0–2.4 μm) integral field spectrograph operating behind the adaptive optics system at Keck Observatory and one of the first lenslet-based integral field spectrographs. Since its commissioning in 2005, it has been a productive instrument, producing nearly half the laser guide star adaptive optics papers on Keck. The complexity of its raw data format necessitated a custom data reduction pipeline (DRP) delivered with the instrument in order to iteratively assign flux in overlapping spectra to the proper spatial and spectral locations in a data cube. Other than bug fixes and updates required for hardware upgrades, the bulk of the DRP has not been updated since initial instrument commissioning. We report on the first major comprehensive characterization of the DRP using on-sky and calibration data. We also detail improvements to the DRP, including characterization of the flux assignment algorithm, exploration of spatial rippling in the reduced data cubes, and improvements to several calibration files, including the rectification matrix, bad-pixel mask, and wavelength solution. We present lessons learned from over a decade of OSIRIS data reduction that are relevant to the next generation of integral field spectrograph hardware and data reduction software design.
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.
The presentation covers data formats expected from Gemini experiments; data quick look vs. in-depth analysis; iPDV object-oriented data storage; iPDV's traceability of analysis results; optimizing object memory usage in iPDV; and long-term archival of data objects by iPDV.
One of the goals of the Deep Underground Neutrino Experiment (DUNE) is to use the massive underground liquid argon time projection chamber (LArTPC) detectors at its far site for multimessenger astronomy (MMA), in the detection of neutrinos from core-collapse supernovae (SNe). Its current baseline trigger strategy detects activity in the detector that is consistent with supernova (SN) neutrinos and saves the raw data for further offline analysis but provides no prompt pointing information crucial for optical follow-ups by other observatories. This approach is based on the assumption that prompt pointing determination using raw data is computationally prohibitive. In this article, we demonstrate a proof-of-concept based on applying extreme data reduction on the buffered SN data in the DUNE data acquisition (DAQ) system’s front-end computers using a machine learning (ML) workflow. This reduces the data by ~5 orders of magnitude, allowing a full track reconstruction to be carried out quickly on a single server. The total time to perform the ML-based data reduction and the full track reconstruction is less than the time to transfer the SN data back to Fermilab or a high-performance computing (HPC) center. This shows that prompt processing of raw SN data is possible and, in fact, trivial once the data have been reduced to reject radiological backgrounds, paving the way to a high-quality SN pointing trigger that is based on fully reconstructed data instead of trigger primitives (TPs).
This environmental calculation file (ECF) describes the data reduction methodology for the solid waste release datasets generated by the Composite Analysis Solid Waste Release (CASWR) model (CP-62766, Model Package Report: Composite Analysis Solid Waste Release Model [CASWR Model]) and documents the resulting reduced datasets for the updated Composite Analysis (CA) baseline assessment. This ECF is in support of the updated Hanford Site CA. The purpose of the updated Hanford Site CA is to provide an estimate of the cumulative radiological impacts from active and planned low-level radioactive waste disposals and other potentially interacting radioactive waste disposal sources that will remain following Hanford Site closure. The vadose zone modeling activities associated with the CA includes solid waste forms as contaminant sources. To support the modeling of solid waste transport from the vadose zone to the groundwater at the Hanford Site, the CASWR model was designed to generate deterministic radionuclide release rates for the Hanford Site Central Plateau solid waste disposal sites. The complete CASWR model-generated dataset for the CA Baseline Assessment consists of 2,378,831 sets of time and release rates (1,073,869 sets for the 200 East Area and 1,304,962 sets for the 200 West Area), representing a course of the simulation from year 1953 to year 12,069, 16 radiological contaminants of potential concern (COPCs), and 237 solid waste sources as documented in ECFHANFORD- 19-0112, Solid Waste Release Calculations for the Composite Analysis Baseline Assessment. For several of the STOMP vadose zone models, the number of release rate data pairs required for release at all waste sites would exceed the limitations imposed by the STOMP modelling software package used to simulate contaminant transport through the vadose zone. As a result, a data reduction of the CASWR dataset is required prior to its use as a source input for the CA vadose zone modeling activities.
This Handbook of Data Reduction Procedures, Workbooks, and Exchange Templates is designed to support the Oncor geodatabase for the Columbia Estuary Ecosystem Restoration Program (CEERP). The following data categories are covered: water-surface elevation and temperature, sediment accretion rate, photo points, herbaceous wetland vegetation cover, tree plots and site summaries, fish catch and density, fish size, fish diet, fish prey, and Chinook salmon genetic stock identification. The handbook is intended for use by scientists collecting monitoring and research data for the CEERP. The ultimate goal of Oncor is to provide quality, easily accessible, geospatial data for synthesis and evaluation of the collective performance of CEERP ecosystem restoration actions at a program scale.
The data quality requirements for charge density studies on actinide compounds are extreme. Important steps in data collection and reduction required to obtain such data are summarized and evaluated. The steps involved in building an augmented Hansen–Coppens multipole model for an actinide pseudo-atom are provided. The number and choice of radial functions, in particular the definition of the core, valence and pseudo-valence terms are discussed. The conclusions in this paper are based on a re-examination and improvement of a previously reported study on [PPh 4 ][UF 6 ]. Topological analysis of the total electron density shows remarkable agreement between experiment and theory; however, there are significant differences in the Laplacian distribution close to the uranium atoms which may be due to the effective core potential employed for the theoretical calculations.
A collection of modules or tools to produce reflectivity profiles from data acquired with unpolarized and polarized neutron beams using the BL4A (MagRef) instrument at the Spallation Neutron Source. The Jupyter notebook approach offers transparency to the user on what is done with the data, and once the event mode data are stored as NumPy binary files empowers the user to perform the steps from their own laptop without the need for Mantid. The notebook offers two options for background subtraction, applies corrections for the wavelength dependence of the polarizer and flipper efficiencies, accounts for Zeeman energy splitting of the spin-flip channels, Qz-bins the data along contours of constant Qz, propagates errors and writes results in text format suitable for analysis with tools such as GenX.
Data reduction for science holds promise for addressing the challenges of moving, storing, and processing massive data sets produced by the scientific community. Pursuing the PRDs outlined here will enable advances in data streaming, fast feedback and/or autonomous control of experiments, and faster time to scientific insight. These advances will result in a significant improvement in the ability to transport, store, process and interpret experimental, observational, and computational data.
The Large Binocular Telescope Interferometer (LBTI) is a versatile instrument designed for high angular resolution and high-contrast infrared imaging (1.5–13 μ m). In this paper, we focus on the mid-infrared (8–13 μ m) nulling mode and present its theory of operation, data reduction, and on-sky performance as of the end of the commissioning phase in 2015 March. With an interferometric baseline of 14.4 m, the LBTI nuller is specifically tuned to resolve the habitable zone of nearby main-sequence stars, where warm exozodiacal dust emission peaks. Measuring the exozodi luminosity function of nearby main-sequence stars is a key milestone to prepare for future exo-Earth direct imaging instruments. Thanks to recent progress in wavefront control and phase stabilization, as well as in data reduction techniques, the LBTI demonstrated in 2015 February a calibrated null accuracy of 0.05% over a 3 hr long observing sequence on the bright nearby A3V star β Leo. This is equivalent to an exozodiacal disk density of 15–30 zodi for a Sun-like star located at 10 pc, depending on the adopted disk model. This result sets a new record for high-contrast mid-infrared interferometric imaging and opens a new window on the study of planetary systems.
Systems, methods, and apparatus are disclosed herein for data pattern detection and data reduction. Devices include an input port configured to receive data values that include a plurality of data units. The devices may also include a data analyzer configured to determine a test statistic based on at least some of the plurality of data units, the test statistic indicating a degree of difference between a first data unit and at least a second data unit, the second data unit being received at the input port before the first data unit. The data analyzer includes the first data unit in a first data block responsive to a determination that the test statistic indicates a low degree of difference, the determination being based on a comparison with a designated difference threshold, the first data block being a same data block as a second data block that includes the second data unit.
The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.
BioXTAS RAW is a free open-source program for reduction, analysis and modelling of biological small-angle scattering data. Here, the new developments in RAW version 2 are described. These include improved data reduction using pyFAI ; updated automated Guinier fitting and D max finding algorithms; automated series ( e.g. size-exclusion chromatography coupled small-angle X-ray scattering or SEC-SAXS) buffer- and sample-region finding algorithms; linear and integral baseline correction for series; deconvolution of series data using regularized alternating least squares ( REGALS ); creation of electron-density reconstructions using electron density via solution scattering ( DENSS ); a comparison window showing residuals, ratios and statistical comparisons between profiles; and generation of PDF reports with summary plots and tables for all analysis. Furthermore, there is now a RAW API, which can be used without the graphical user interface (GUI), providing full access to all of the functionality found in the GUI. In addition to these new capabilities, RAW has undergone significant technical updates, such as adding Python 3 compatibility, and has entirely new documentation available both online and in the program.
This is the final report for the project "Geometric Analysis for Data Reduction and Structure Discovery" in which insights and tools from geometric analysis were developed and exploited for their potential to large scale data challenges.