Gappy AE: A nonlinear approach for Gappy data reconstruction using auto-encoder
Not Available
97 MATHEMATICS AND COMPUTING↗
SEARCH · Search NASA
Results for “data reconstruction”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Commissioning of US HPC GPU resources for CMS Data Reconstruction
Explore the source record for details and available documents.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Hybrid Interferometric and Deflectometric Stitching and Data Reconstruction for X-ray Mirrors
Explore the source record for details and available documents.
36 MATERIALS SCIENCE↗
High-performance data format for scientific data storage and analysis
Here, in this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data analysis frameworks to use. In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete). In columnar data analysis, HiPO surpasses established data formats in performance and can be effectively applied to data analysis in other scientific fields.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
A dictionary learning algorithm for compression and reconstruction of streaming data in preset order
There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.
97 MATHEMATICS AND COMPUTING↗
UNESE Data Analyses: Reconstruction of BARNWELL Surfaces and Data Fusion for UNE Observables
Explore the source record for details and available documents.
58 GEOSCIENCES↗
Real-time reconstruction of ground motion during small magnitude earthquakes: A pilot study
This study presents a pilot investigation into a novel method for reconstructing real-time ground motion during small magnitude earthquakes (M < 4.5), removing the need for computationally expensive source characterization and simulation processes to assess ground shaking. Small magnitude earthquakes, which occur frequently and can be modeled as point sources, provide ideal conditions for evaluating real-time reconstruction methods. Utilizing sparse observation data, the method applies the Gappy Auto-Encoder (Gappy AE) algorithm for efficient field data reconstruction. This is the first study to apply the Gappy AE algorithm to earthquake ground motion reconstruction. Numerical experiments conducted with SW4 simulations demonstrate the method’s accuracy and speed across varying seismic scenarios. The reconstruction performance is further validated using real seismic data from the Berkeley area in California, USA, demonstrating the potential for practical application of real-time earthquake data reconstruction using Gappy AE. As a pilot investigation, it lays the groundwork for future applications to larger and more complex seismic events.
58 GEOSCIENCES↗
A Multi-Branch Decoder Network Approach to Adaptive Temporal Data Selection and Reconstruction for Big Scientific Simulation Data
A key challenge in scientific simulation is that the simulation outputs often require intensive I/O and storage space to store the results for effective post hoc analysis. This article focuses on a quality-aware adaptive temporal data selection and reconstruction problem where the goal is to adaptively select simulation data samples at certain key timesteps in situ and reconstruct the discarded samples with quality assurance during post hoc analysis. This problem is motivated by the limitation of current solutions that a significant amount of simulation data samples are either discarded or aggregated during the sampling process, leading to inaccurate modeling of the simulated phenomena. Two unique challenges exist: 1) the sampling decisions have to be made in situ and adapted to the dynamics of the complex scientific simulation data; 2) the reconstruction error must be strictly bounded to meet the application requirement. To address the above challenges, we develop DeepSample , an error-controlled convolutional neural network framework, that jointly integrates a set of coherent multi-branch deep decoders to effectively reconstruct the simulation data with rigorous quality assurance. The results on two real-world scientific simulation applications show that DeepSample significantly outperforms other state-of-the-art methods on both sampling efficiency and reconstructed simulation data quality.
Zhang, Yang↗
Constructing a Testing Application for GWMS
Many of Fermilab’s High Energy Physics experiments require High Throughput Computing to carry out simulations, data reconstruction, and data analysis. Glidein Workflow Management System (GWMS) is a tool that distributes High Throughput Computing. The purpose of GWMS is to provide convenient access to Grid resources and sites. GWMS is coupled to HTCondor. HTCondor is the workload management system that is used for scheduling and job control. Users submit jobs to a local HTCondor queue and GWMS will make sure the job will run on one of the many remote resources.
Neely-Brown, LeRayah↗
Enhancing Sterile Neutrino Oscillation Sensitivities using SBND-PRISM at the Short-Baseline Neutrino Programme
The Short-Baseline Neutrino (SBN) Programme at Fermilab is comprised of two detectors, SBND and ICARUS, placed at $110$\,m and $600$\,m along the Booster Neutrino Beam at Fermilab. The key physics aim of the programme is to definitively test the sterile neutrino hypothesis, a proposed fourth flavour of neutrino that may explain certain experimental anomalies seen regarding standard model neutrino oscillations. To be able to detect the existence of sterile neutrinos, the uncertainties of the programme must be well constrained. To enable this, a robust analysis must be constructed that can consistently identify the correct values of systematic parameters and generate accurate predictions of what the neutrino energy spectrum at ICARUS should look like. SBND's design allows for the introduction of a technique called PRISM. In PRISM, the detector is divided into regions of different off axis angles from the beam, forming different samples where systematics impact each one in a distinct way. This allows any fits performed to obtain a better understanding of the correct value of the systematic parameters. The first analysis included in this thesis focuses on the improvements seen to the sensitivity of SBN to sterile oscillation parameters when using a PRISM configuration instead of treating SBND as a single whole. Focusing on the $5\sigma$ exclusion contour from the $\numu$ disappearance channel using three PRISM samples, an improvement on the order of $30$\,\% is seen, extending the parameter space for which the null hypothesis can be excluded. This thesis also presents a series of mock data studies comparing the abilities of SBND and PRISM analyses, concluding that in the case of simple changes between Monte Carlo (MC) and data, PRISM produces predictions of the ICARUS event rate spectrum that are more accurate and have smaller uncertainties. When moving to more realistic mock data, using different models to create the mock data than were used for the MC, the postfit predictions at ICARUS had a smaller difference between the postfit and mock data reconstructed energy spectra across all mock data samples tested. Finally, a covariance matrix defined by the maximum discrepancy between the postfit and mock data spectra at ICARUS from each of the SBND and PRISM fits across all the samples was constructed. The resultant $1\sigma$ fractional error induced by this bias systematic on the postfit spectrum has a smaller magnitude for PRISM than SBND, with the improvements ranging from $2.21$\,\% to $6.26$\,\% depending on the mock data samples. This summarises the reduction in systematic error when using PRISM instead of treating SBND as a single detector. *************************************** AUTHOR = Slater, Bethany University of Liverpool b.slater2@liverpool.ac.uk TITLE = Enhancing Sterile Neutrino Oscillation Sensitivities using SBND-PRISM at the Short-Baseline Neutrino Programme PAGES = 218 NOTE = Ph.D. University of Liverpool March 2026 ABSTRACT = The Short-Baseline Neutrino (SBN) Programme at Fermilab is comprised of two detectors, SBND and ICARUS, placed at $110$\,m and $600$\,m along the Booster Neutrino Beam at Fermilab. The key physics aim of the programme is to definitively test the sterile neutrino hypothesis, a proposed fourth flavour of neutrino that may explain certain experimental anomalies seen regarding standard model neutrino oscillations. To be able to detect the existence of sterile neutrinos, the uncertainties of the programme must be well constrained. To enable this, a robust analysis must be constructed that can consistently identify the correct values of systematic parameters and generate accurate predictions of what the neutrino energy spectrum at ICARUS should look like. SBND's design allows for the introduction of a technique called PRISM. In PRISM, the detector is divided into regions of different off axis angles from the beam, forming different samples where systematics impact each one in a distinct way. This allows any fits performed to obtain a better understanding of the correct value of the systematic parameters. The first analysis included in this thesis focuses on the improvements seen to the sensitivity of SBN to sterile oscillation parameters when using a PRISM configuration instead of treating SBND as a single whole. Focusing on the $5\sigma$ exclusion contour from the $\numu$ disappearance channel using three PRISM samples, an improvement on the order of $30$\,\% is seen, extending the parameter space for which the null hypothesis can be excluded. This thesis also presents a series of mock data studies comparing the abilities of SBND and PRISM analyses, concluding that in the case of simple changes between Monte Carlo (MC) and data, PRISM produces predictions of the ICARUS event rate spectrum that are more accurate and have smaller uncertainties. When moving to more realistic mock data, using different models to create the mock data than were used for the MC, the postfit predictions at ICARUS had a smaller difference between the postfit and mock data reconstructed energy spectra across all mock data samples tested. Finally, a covariance matrix defined by the maximum discrepancy between the postfit and mock data spectra at ICARUS from each of the SBND and PRISM fits across all the samples was constructed. The resultant $1\sigma$ fractional error induced by this bias systematic on the postfit spectrum has a smaller magnitude for PRISM than SBND, with the improvements ranging from $2.21$\,\% to $6.26$\,\% depending on the mock data samples. This summarises the reduction in systematic error when using PRISM instead of treating SBND as a single detector.
Slater, Bethany [Liverpool U.]↗
Enhancing Sterile Neutrino Oscillation Sensitivities using SBND-PRISM at the Short-Baseline Neutrino Programme
The Short-Baseline Neutrino (SBN) Programme at Fermilab is comprised of two detectors, SBND and ICARUS, placed at $110$\,m and $600$\,m along the Booster Neutrino Beam at Fermilab. The key physics aim of the programme is to definitively test the sterile neutrino hypothesis, a proposed fourth flavour of neutrino that may explain certain experimental anomalies seen regarding standard model neutrino oscillations. To be able to detect the existence of sterile neutrinos, the uncertainties of the programme must be well constrained. To enable this, a robust analysis must be constructed that can consistently identify the correct values of systematic parameters and generate accurate predictions of what the neutrino energy spectrum at ICARUS should look like. SBND's design allows for the introduction of a technique called PRISM. In PRISM, the detector is divided into regions of different off axis angles from the beam, forming different samples where systematics impact each one in a distinct way. This allows any fits performed to obtain a better understanding of the correct value of the systematic parameters. The first analysis included in this thesis focuses on the improvements seen to the sensitivity of SBN to sterile oscillation parameters when using a PRISM configuration instead of treating SBND as a single whole. Focusing on the $5\sigma$ exclusion contour from the $\numu$ disappearance channel using three PRISM samples, an improvement on the order of $30$\,\% is seen, extending the parameter space for which the null hypothesis can be excluded. This thesis also presents a series of mock data studies comparing the abilities of SBND and PRISM analyses, concluding that in the case of simple changes between Monte Carlo (MC) and data, PRISM produces predictions of the ICARUS event rate spectrum that are more accurate and have smaller uncertainties. When moving to more realistic mock data, using different models to create the mock data than were used for the MC, the postfit predictions at ICARUS had a smaller difference between the postfit and mock data reconstructed energy spectra across all mock data samples tested. Finally, a covariance matrix defined by the maximum discrepancy between the postfit and mock data spectra at ICARUS from each of the SBND and PRISM fits across all the samples was constructed. The resultant $1\sigma$ fractional error induced by this bias systematic on the postfit spectrum has a smaller magnitude for PRISM than SBND, with the improvements ranging from $2.21$\,\% to $6.26$\,\% depending on the mock data samples. This summarises the reduction in systematic error when using PRISM instead of treating SBND as a single detector. *************************************** AUTHOR = Slater, Bethany University of Liverpool b.slater2@liverpool.ac.uk TITLE = Enhancing Sterile Neutrino Oscillation Sensitivities using SBND-PRISM at the Short-Baseline Neutrino Programme PAGES = 218 NOTE = Ph.D. University of Liverpool March 2026 ABSTRACT = The Short-Baseline Neutrino (SBN) Programme at Fermilab is comprised of two detectors, SBND and ICARUS, placed at $110$\,m and $600$\,m along the Booster Neutrino Beam at Fermilab. The key physics aim of the programme is to definitively test the sterile neutrino hypothesis, a proposed fourth flavour of neutrino that may explain certain experimental anomalies seen regarding standard model neutrino oscillations. To be able to detect the existence of sterile neutrinos, the uncertainties of the programme must be well constrained. To enable this, a robust analysis must be constructed that can consistently identify the correct values of systematic parameters and generate accurate predictions of what the neutrino energy spectrum at ICARUS should look like. SBND's design allows for the introduction of a technique called PRISM. In PRISM, the detector is divided into regions of different off axis angles from the beam, forming different samples where systematics impact each one in a distinct way. This allows any fits performed to obtain a better understanding of the correct value of the systematic parameters. The first analysis included in this thesis focuses on the improvements seen to the sensitivity of SBN to sterile oscillation parameters when using a PRISM configuration instead of treating SBND as a single whole. Focusing on the $5\sigma$ exclusion contour from the $\numu$ disappearance channel using three PRISM samples, an improvement on the order of $30$\,\% is seen, extending the parameter space for which the null hypothesis can be excluded. This thesis also presents a series of mock data studies comparing the abilities of SBND and PRISM analyses, concluding that in the case of simple changes between Monte Carlo (MC) and data, PRISM produces predictions of the ICARUS event rate spectrum that are more accurate and have smaller uncertainties. When moving to more realistic mock data, using different models to create the mock data than were used for the MC, the postfit predictions at ICARUS had a smaller difference between the postfit and mock data reconstructed energy spectra across all mock data samples tested. Finally, a covariance matrix defined by the maximum discrepancy between the postfit and mock data spectra at ICARUS from each of the SBND and PRISM fits across all the samples was constructed. The resultant $1\sigma$ fractional error induced by this bias systematic on the postfit spectrum has a smaller magnitude for PRISM than SBND, with the improvements ranging from $2.21$\,\% to $6.26$\,\% depending on the mock data samples. This summarises the reduction in systematic error when using PRISM instead of treating SBND as a single detector.
Slater, Bethany [Liverpool U.]↗
Enhancing Sterile Neutrino Oscillation Sensitivities using SBND-PRISM at the Short-Baseline Neutrino Programme
The Short-Baseline Neutrino (SBN) Programme at Fermilab is comprised of two detectors, SBND and ICARUS, placed at $110$\,m and $600$\,m along the Booster Neutrino Beam at Fermilab. The key physics aim of the programme is to definitively test the sterile neutrino hypothesis, a proposed fourth flavour of neutrino that may explain certain experimental anomalies seen regarding standard model neutrino oscillations. To be able to detect the existence of sterile neutrinos, the uncertainties of the programme must be well constrained. To enable this, a robust analysis must be constructed that can consistently identify the correct values of systematic parameters and generate accurate predictions of what the neutrino energy spectrum at ICARUS should look like. SBND's design allows for the introduction of a technique called PRISM. In PRISM, the detector is divided into regions of different off axis angles from the beam, forming different samples where systematics impact each one in a distinct way. This allows any fits performed to obtain a better understanding of the correct value of the systematic parameters. The first analysis included in this thesis focuses on the improvements seen to the sensitivity of SBN to sterile oscillation parameters when using a PRISM configuration instead of treating SBND as a single whole. Focusing on the $5\sigma$ exclusion contour from the $\numu$ disappearance channel using three PRISM samples, an improvement on the order of $30$\,\% is seen, extending the parameter space for which the null hypothesis can be excluded. This thesis also presents a series of mock data studies comparing the abilities of SBND and PRISM analyses, concluding that in the case of simple changes between Monte Carlo (MC) and data, PRISM produces predictions of the ICARUS event rate spectrum that are more accurate and have smaller uncertainties. When moving to more realistic mock data, using different models to create the mock data than were used for the MC, the postfit predictions at ICARUS had a smaller difference between the postfit and mock data reconstructed energy spectra across all mock data samples tested. Finally, a covariance matrix defined by the maximum discrepancy between the postfit and mock data spectra at ICARUS from each of the SBND and PRISM fits across all the samples was constructed. The resultant $1\sigma$ fractional error induced by this bias systematic on the postfit spectrum has a smaller magnitude for PRISM than SBND, with the improvements ranging from $2.21$\,\% to $6.26$\,\% depending on the mock data samples. This summarises the reduction in systematic error when using PRISM instead of treating SBND as a single detector. *************************************** AUTHOR = Slater, Bethany University of Liverpool b.slater2@liverpool.ac.uk TITLE = Enhancing Sterile Neutrino Oscillation Sensitivities using SBND-PRISM at the Short-Baseline Neutrino Programme PAGES = 218 NOTE = Ph.D. University of Liverpool March 2026 ABSTRACT = The Short-Baseline Neutrino (SBN) Programme at Fermilab is comprised of two detectors, SBND and ICARUS, placed at $110$\,m and $600$\,m along the Booster Neutrino Beam at Fermilab. The key physics aim of the programme is to definitively test the sterile neutrino hypothesis, a proposed fourth flavour of neutrino that may explain certain experimental anomalies seen regarding standard model neutrino oscillations. To be able to detect the existence of sterile neutrinos, the uncertainties of the programme must be well constrained. To enable this, a robust analysis must be constructed that can consistently identify the correct values of systematic parameters and generate accurate predictions of what the neutrino energy spectrum at ICARUS should look like. SBND's design allows for the introduction of a technique called PRISM. In PRISM, the detector is divided into regions of different off axis angles from the beam, forming different samples where systematics impact each one in a distinct way. This allows any fits performed to obtain a better understanding of the correct value of the systematic parameters. The first analysis included in this thesis focuses on the improvements seen to the sensitivity of SBN to sterile oscillation parameters when using a PRISM configuration instead of treating SBND as a single whole. Focusing on the $5\sigma$ exclusion contour from the $\numu$ disappearance channel using three PRISM samples, an improvement on the order of $30$\,\% is seen, extending the parameter space for which the null hypothesis can be excluded. This thesis also presents a series of mock data studies comparing the abilities of SBND and PRISM analyses, concluding that in the case of simple changes between Monte Carlo (MC) and data, PRISM produces predictions of the ICARUS event rate spectrum that are more accurate and have smaller uncertainties. When moving to more realistic mock data, using different models to create the mock data than were used for the MC, the postfit predictions at ICARUS had a smaller difference between the postfit and mock data reconstructed energy spectra across all mock data samples tested. Finally, a covariance matrix defined by the maximum discrepancy between the postfit and mock data spectra at ICARUS from each of the SBND and PRISM fits across all the samples was constructed. The resultant $1\sigma$ fractional error induced by this bias systematic on the postfit spectrum has a smaller magnitude for PRISM than SBND, with the improvements ranging from $2.21$\,\% to $6.26$\,\% depending on the mock data samples. This summarises the reduction in systematic error when using PRISM instead of treating SBND as a single detector.
Slater, Bethany [Liverpool U.]↗
DUNE Database Development
The DUNE experiment will produce vast amounts of metadata, which describe the data coming from the read-out of the primary DUNE detectors. Various databases will make up the overall DB architecture for this metadata. ProtoDUNE at CERN is the largest existing prototype for DUNE and serves as a testing ground for - among other things - possible database solutions for DUNE. The subset of all metadata that is accessed during offline data reconstruction and analysis is referred to as ‘conditions data’ and it is stored in a dedicated database. As offline data reconstruction and analysis will be deployed on HTC and HPC resources, conditions data is expected to be accessed at very high rates. It is therefore crucial to store it in a granularity that matches the expected access patterns allowing for extensive caching. This requires a good understanding of the sources and use cases of conditions data. This contribution will briefly summarize the database architecture deployed at ProtoDUNE and explain the various sources of conditions data. We will present how the conditions data is retrieved and streamed from the databases and how it is handled to match expected access patterns.
Vizcaya Hernandez, Ana Paula↗
DUNE Database Development
The DUNE experiment will produce vast amounts of metadata, which describe the data coming from the read-out of the primary DUNE detectors. Various databases will make up the overall DB architecture for this metadata. ProtoDUNE at CERN is the largest existing prototype for DUNE and serves as a testing ground for - among other things - possible database solutions for DUNE. The subset of all metadata that is accessed during offline data reconstruction and analysis is referred to as ‘conditions data’ and it is stored in a dedicated database. As offline data reconstruction and analysis will be deployed on HTC and HPC resources, conditions data is expected to be accessed at very high rates. It is therefore crucial to store it in a granularity that matches the expected access patterns allowing for extensive caching. This requires a good understanding of the sources and use cases of conditions data. This contribution will briefly summarize the database architecture deployed at ProtoDUNE and explain the various sources of conditions data. We will present how the conditions data is retrieved and streamed from the databases and how it is handled to match expected access patterns.
43 PARTICLE ACCELERATORS↗
Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints
Vast volumes of data are produced by today's scientific simulations and advanced instruments. These data cannot be stored and transferred efficiently because of limited I/O bandwidth, network speed, and storage capacity. Error-bounded lossy compression can be an effective method for addressing these issues: not only can it significantly reduce data size, but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific requirements or constraints for lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. For example, some datasets contain irrelevant data that should be isolated in particular and users often have intuition regarding value ranges, geospatial regions, and other data subsets that are crucial for subsequent analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not consider these constraints during compression, resulting in inferior compression ratios with respect to user's post hoc analysis, due to the fact that the data itself provides little or no value for post hoc analysis. In this work we address this issue by proposing an optimized framework that can preserve diverse constraints during the error-bounded lossy compression, e.g., cleaning the irrelevant data, efficiently preserving different precision for multiple value intervals, and allowing users to set diverse precision over both regular and irregular regions. We perform our evaluation on a supercomputer with up to 2,100 cores. Experiments with six real-world applications show that our proposed diverse constraints based error-bounded lossy compressor can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios compared with the traditional state-of-the-art compressor SZ. Furthermore, our experiments also demonstrate very good scalability in compression performance compared with the I/O throughput of the parallel file system.
97 MATHEMATICS AND COMPUTING↗
Data and scripts from: “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”
This data package includes data and scripts from the manuscript “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”.The study addressed common challenges faced in environmental sensing and modeling, including uncertain input data, missing sensor observations, and high-dimensional datasets with interrelated but redundant variables. Point-scaled meteorological and soil sensor observations were perturbed with noises and missing values, and denoising autoencoder (DAE) neural networks were developed to reconstruct the perturbed data and further predict evapotranspiration. This study concluded that (1) the reconstruction quality of each variable depends on its cross-correlation and alignment to the underlying data structure, (2) uncertainties from the models were overall stronger than those from the data corruption, and (3) there was a tradeoff between reducing bias and reducing variance when evaluating the uncertainty of the machine learning models.This package includes:(1) Four ipython scripts (.ipynb): “DAE_train.ipynb” trains and evaluates DAE neural networks, “DAE_predict.ipynb” makes predictions from the trained DAE models, “ET_train.ipynb” trains and evaluates ET prediction neural networks, and “ET_predict.ipynb” makes predictions from trained ET models.(2) One python file (.py): “methods.py” includes all user-defined functions and python codes used in the ipython scripts.(3) A “sub_models” folder that includes five trained DAE neural networks (in pytorch format, .pt), which could be used to ingest input data before being fed to the downstream ET models in ‘ET_train.ipynb” or ‘ET_predict.ipynb’.(4) Two data files (.csv). Daily meteorological, vegetation, and soil data is in “df_data.csv”, where “df_meta.csv” contains the location and time information of “df_data.csv”. Each row (index) in “df_meta.csv” corresponds to each row in “df_data.csv”. These data files are formatted to follow the data structure requirements and be directly used in the ipython scripts, and they have been shuffled chronologically to train machine learning models. The meteorological and soil data was collected using point sensors between 2019-2023 at(4.a) Three shrub-dominated field sites in East River, Colorado (named “ph1”, “ph2” and “sg5” in “df_meta.csv”, where “ph1” and “ph2” were located at PumpHouse Hillslopes, and “sg5” was at Snodgrass Mountain meadow) and(4.b) One outdoor, mesoscale, and herbaceous-dominated experiment in Berkeley, California (named “tb” in “df_meta.csv”, short for Smartsoils Testbed at Lawrence Berkeley National Lab).- See "df_data_dd.csv" and "df_meta_dd.csv" for variable descriptions and the Methods section for additional data processing steps. See "flmd.csv" and "README.txt" for brief file descriptions.- All ipython scripts and python files are written in and require PYTHON language software.
54 ENVIRONMENTAL SCIENCES↗
Parallelizing the Unpacking and Clustering of Detector Data for Reconstruction of Charged Particle Tracks on Multi-core CPUs and Many-core GPUs
We present results from parallelizing the unpacking and clustering steps of the raw data from the silicon strip modules for reconstruction of charged particle tracks. Throughput is further improved by concurrently processing multiple events using nested OpenMP parallelism on CPU or CUDA streams on GPU. The new implementation along with earlier work in developing a parallelized and vectorized implementation of the combinatoric Kalman filter algorithm has enabled efficient global reconstruction of the entire event on modern computer architectures. We demonstrate the performance of the new implementation on Intel Xeon and NVIDIA GPU architectures.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Gaussian process analysis of electron energy loss spectroscopy data: multivariate reconstruction and kernel control
Abstract Advances in hyperspectral imaging including electron energy loss spectroscopy bring forth the challenges of exploratory and physics-based analysis of multidimensional data sets. The multivariate linear unmixing methods generally explore similarities in the energy dimension, but ignore correlations in the spatial domain. At the same time, Gaussian process (GP) explicitly incorporate spatial correlations in the form of kernel functions but is computationally intensive. Here, we implement a GP method operating on the full spatial domain and reduced representations in the energy domain. In this multivariate GP, the information between the components is shared via a common spatial kernel structure, while allowing for variability in the relative noise magnitude or image morphology. We explore the role of kernel constraints on the quality of the reconstruction, and suggest an approach for estimating them from the experimental data. We further show that spatial information contained in higher-order components can be reconstructed and spatially localized.
36 MATERIALS SCIENCE↗