Leveraging Cluster Computing for Large-Scale Data Processing and Automated Analysis for Radiation Portal Monitor Data
Explore the source record for details and available documents.
Smith, Jacob↗
SEARCH · Search NASA
Results for “data analysis”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Computational epidemiological tools for pandemic analysis, understanding, and response
This suite of software tools is being developed to enhance and analyze computational epidemiological models that incorporate realistic disease dynamics and human behavior, with the goal of supporting epidemic and pandemic response. Specifically, the tools enable data analysis, feature extraction, data synthesis, machine learning model development, and prediction of key public health outcomes, such as cases, hospitalizations, deaths, and behavioral responses, for airborne infectious diseases like COVID-19 and influenza.
Butts, David↗
Data & Code from Phoenix CPPP Phase 2 Analysis
This data and code package supports the analysis presented in “Beyond Surface Cooling: Comprehensive Field Assessment of Reflective Pavement Thermal Performance in Phoenix, Arizona” and provides fully reproducible workflows for evaluating the thermal performance of cool pavement treatments in a hot urban environment. The dataset integrates multi-modal field measurements collected across residential and nonresidential settings, including mobile air temperature traverses, stationary air temperature monitoring, residential mean radiant temperature (MRT) measurements, subsurface temperature profiles, and controlled testbed observations. The data package contains raw and processed datasets in comma-separated value (CSV) format, accompanying metadata files describing site characteristics and measurement protocols, and R scripts (.R files) used for data cleaning, time synchronization, spatial and temporal matching, quality control filtering, statistical comparison, and figure generation. All analyses were conducted using R (version ≥ 4.2.0) with commonly available packages (e.g., tidyverse, lubridate, data.table, ggplot2). No proprietary software is required to reproduce results. Field campaigns were designed to quantify the effects of high-reflectance pavement coatings on surface temperature, near-surface air temperature, subsurface heat propagation, and radiative heat exposure. Temporal alignment procedures include standardized timestamp conversion and nearest-neighbor matching of high-frequency sensor measurements to stop-based metadata within defined tolerance windows to ensure comparability across instruments. The workflows generate summary statistics, treatment–control contrasts, depth-dependent thermal gradients, and time-series visualizations used in the associated publication. By integrating mobile, stationary, radiative, and subsurface measurements within a unified and transparent processing framework, this package enables comprehensive evaluation of cool pavement performance across multiple thermal exposure pathways and supports reuse in future urban heat mitigation and climate resilience studies.
AIR TEMPERATURE↗
SPRUCE Whole Ecosystem Warming (WEW) Environmental Data and Water Table Summaries, Marcell Experimental Forest, Minnesota, 2015-2024
This data set contains observations of photosynthetically active radiation (PAR), precipitation, soil temperature, soil volumetric water content, air temperature, relative humidity, and normalized water table depth that are summarized on a daily, weekly, monthly, and annual basis for each of the SPRUCE plots. Observations span 2015-2024. This dataset draws on several datasets (Hanson et al. 2016; Hanson et al. 2020; and Warren, unpublished data) and compiles these environmental observations into useful formats for data analysis. These environmental metrics can be used to understand the environmental conditions inside SPRUCE environmental chambers throughout the durations of the experiment and can be paired with other data for modeling and analysis. R code used to generate these files is provided as part of the data package. This dataset contains four data files in comma separate (.csv) format and a compressed folder (*.zip) containing three R (*.r) scripts. Additional metadata are provided: one data dictionary and a file-level metadata file in comma separate (.csv) format and a user guide in PDF (*.pdf) format. User note: Users must cite the original dataset/s along with this dataset when publishing any analyses using this dataset. Details on the dataset used to compile each variable are available in the header row of the files and in the user guide.
air temperature↗
Machine Learning-Driven Reliability Estimation of PV Inverters Considering Alert-Ambient Variability
Weather-induced spatio-temporal degradation limits outdoor PV inverter lifetime and reliability, necessitating advanced data analysis. This study employs a top-down, data-driven approach utilizing multiple machine learning (ML) algorithms to estimate inverter reliability in a 1.4 MW PV power plant, considering factors such as irradiance, humidity, temperature, time of day, and weather conditions. An extensive alert dataset from 17 identical inverters, including alert types, propagation, and frequency, reveals significant correlations with environmental factors and inverter output power, enabling the construction of a performance reliability model. Dual-stage supervised-ML models are evaluated for accuracy, with the ‘classification-regression’ model by an artificial neural network (ANN) tested on the averaged “Alert-Ambient” dataset, which is outperformed by ‘clustering-regression’ models using random forest (RF) and K-Nearest Neighbors (KNN) on individual inverter datasets. K-means clustering applies principal component analysis to reduce dimensions, achieving improved accuracy beyond the 80% achieved by ANN on the averaged dataset. Second-stage regression estimates inverter reliability with a mean square error of 0.0195 on the averaged dataset and as low as 0.002 on individual inverter datasets using RF. Furthermore, these findings highlight the method's suitability for estimating PV inverter output reliability under ambient conditions, essential for digital twin development and related applications.
14 SOLAR ENERGY↗
Outdoor Deployment Data for a Four-Terminal GaAs//Si Tandem Solar Mini-Module
This dataset contains the complete outdoor measurement and analysis data for a mechanically stacked, four-terminal (4T) gallium arsenide (GaAs)//silicon (Si) tandem solar mini-module deployed from October 2019 to January 2021 at the Solar Radiation Research Laboratory (SRRL) in Golden, Colorado, USA. The data support a performance modeling and degradation analysis framework for tandem photovoltaic devices, as described in the accompanying publication. The dataset includes: (1) current–voltage (J–V) characteristics of each sub-cell measured approximately every five minutes, with extracted performance parameters; (2) spectral irradiance from an EKO MS-710 WISER spectroradiometer, along with derived spectral mismatch ratios (SMR) and average photon energy (APE); (3) one-minute resolution meteorological data from the co-located SRRL weather station and GPS-derived precipitable water vapor (PWV); (4) pre-deployment laboratory characterization (external quantum efficiency, J–V curves, standard test conditions parameters); (5) outdoor-extracted temperature and PWV correction coefficients; and (6) PVcircuit equivalent-circuit simulation outputs used for model validation. Degradation rates of −4.1 ± 0.2 %/year (GaAs) and −2.5 ± 0.9 %/year (Si) were determined using a filtering and normalization methodology adapted for fixed-tilt tandem modules. All data are provided in open, portable formats (Apache Parquet, CSV, JSON) to enable full reproducibility of the published analysis.
14 SOLAR ENERGY↗
Portable Distortion Free Large Solid Angle Coverage Detector
Neutron single crystal diffractometers require large solid angle coverage for optimum performance. This can be achieved by tiling flat detectors in a cylindrical or spherical geometry, but results in large gaps in detector coverage and parallax distortion. The detector edges exhibit degraded resolution, distortion, and gamma rejection. Since the detector edge regions are a significant fraction of the detector active area, they must be removed from the experimental data set, requiring extra beam time to collect enough analysis data. A spherical detector with a continuous surface would effectively address this issue while eliminating most boundary ‘dead’ areas. Here we report on the development of a novel hemispherical shaped neutron detector using seamlessly tiled readout modules to form the desired shape. The heart of the detector is a specially developed curved neutron scintillator coupled to high resolution silicon photomultiplier (SiPM) Anger cameras via custom made fiber optic tapers (FOTs). The detector has been assembled and initial tests have been conducted at the High Flux Isotope Reactor (HFIR) beamlines at Oak Ridge National Laboratory (ORNL). Here, in this work, we describe details of the scintillator design, fabrication and characterization, evaluation of individual detector modules, the details of the detector design implementation, and evaluation of the assembled detector at ORNL beamlines.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Pando
SAND2025-02006O Pando is a distributed data analysis software tool. It is designed to handle large-scale graph analysis problems, often with a specific focus on blockchain/cryptocurrency data. Pando handles scalability by running on a distributed cluster of servers. Users can customize the output using the program’s plugin/extension design methodology. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Gabert, Kasimir↗
Probabilistic data fusion and physics-informed machine learning: A new paradigm for modeling under uncertainty, and its application to accelerating the discovery of new materials
In this report we summarize the work conducted by PI Perdikaris and his group under this Early Career project DE–SC0019116 during the period of 09/01/2018 – 08/31/2023. The central aim of the work was to introduce a new paradigm for scientific data analysis that can seamlessly synthesize rigorous mathematical modeling with data of variable fidelity (e.g., measurements at multiple scales/resolutions or predictions of variable fidelity models) and multiple modalities (e.g., images, time–series, or scattered measurements). The setting we are interested in involves complex systems that are partially observed and whose dynamical behavior could be hard to model or totally unknown. The inherent uncertainty associated with this setting necessitates a departure from the classical deterministic realm of modeling and scientific computation, and, consequently, our main building blocks can no longer be crisp deterministic numbers and governing laws, but instead we must operate with probabilistic models.
97 MATHEMATICS AND COMPUTING↗
Electrification Analysis: All Aboard America!
This one-page highlight details the key takeaways from a project that utilized NREL's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline, an electrification analysis for the Bustang motorcoach fleet operated by All Aboard America! Holdings Inc. (AAA). NREL installed logging devices and collected operational data on nine 40-foot Bustang motorcoaches operating on fixed routes from May 2022 through August 2022. The analysis determined that partial fleet electrification may be feasible with electrified motorcoach options currently on the market. While this fleet faces significant challenges to electrification given current market options due to demanding range requirements and relatively limited charging opportunities, vehicles operating on the shorter, lower-grade routes along the I-25 corridor show more immediately available electrification potential. Increases in available battery capacity and the availability of fast-charging locations along I-70 routes are likely critical for electrification of the full fleet.
AAA↗
pathSQE : an automated workflow for single-crystal inelastic neutron scattering data processing and analysis
Inelastic neutron scattering (INS) experiments utilizing modern time-of-flight spectrometers enable the comprehensive mapping of the energy (E)- and momentum (Q)-resolved dynamical structure factor of single crystals, probing both the lattice and magnetic excitations. Yet, the large size and complexity of four-dimensional INS data are challenging current analysis workflows, often resulting in an underutilization of the measured information. To help address this issue, this paper introduces new software interfaced with the Mantid framework, pathSQE, designed to streamline the processing, analysis and interpretation of 4D single-crystal INS data. By automating key tasks such as 1D/2D slicing, symmetrization, Brillouin zone folding, data visualization, prioritization and filtering, and comparisons with simulations, pathSQE facilitates and accelerates INS data analysis workflows. Here, this paper outlines the features and implementation and provides several illustrations of the use of pathSQE on data collected on single crystals using direct-geometry time-of-flight spectrometers at the Spallation Neutron Source, including Ge, FeSi, MnO and SnS single-crystal measurements on the ARCS, HYSPEC and CNCS neutron spectrometers. Beyond streamlining post-experiment data processing, pathSQE establishes an automated and modular processing pipeline that could support future real-time experiment steering.
36 MATERIALS SCIENCE↗
Discovering the Unknowns: A First Step
This article aims at discovering the unknown variables in the system through data analysis. The main idea is to use the time of data collection as a surrogate variable and try to identify the unknown variables by modeling gradual and sudden changes in the data. We use Gaussian process modeling and a sparse representation of the sudden changes to efficiently estimate the large number of parameters in the proposed statistical model. The method is tested on a realistic dataset generated using a one-dimensional implementation of a Magnetized Liner Inertial Fusion (MagLIF) simulation model, and encouraging results are obtained.
42 ENGINEERING↗
Field and Model Data Associated with the Manuscript “Drivers of Streamflow Intermittency in Humid Regions: 2. Evaluating Controls on Flow Persistence in an Urbanized Catchment”
This package contains field data, modeling files, and scripts supporting the investigation of the drivers of streamflow intermittency in an urbanized catchment. It includes the field data collected from electrical resistivity tomography (ERT) surveys, distributed temperature sensing (DTS), continuous self-potential (SP) monitoring, groundwater and stilling well. In addition, it contains the data and results of the coupled water- and electrical-flow model developed using the COMSOL Multiphysics and Advanced Terrestrial Simulator (ATS), as well as software files and Jupyter notebooks used to process the data and generate figures in the manuscript submitted for peer review. The data archive is organized in the following directories: 1) Climate Includes hourly precipitation and daily evapotranspiration time series (2024 – 2025) provided as CSV files, alongside a text file detailing dataset units. 2) Coupled_model Field_Application subfolder contains the ATS XML input scripts, data files, output data for the SP site. It also contains the Jupyter notebook (Plot_final_calib.ipynb) to visualize the results of the modeled SP, stream-groundwater exchange and moisture content. The flow model simulation is executed using the ATS XML scripts and the included Python script (generate_data_set.py) to convert ATS output to COMSOL-ready input. COMSOL Multiphysics template (.m can only be used with COMSOL with MATLAB) is executed using the ATS output data to simulate the potential field. 3) Discharge Includes the electrical conductivity (EC) time series (provided as CSV files) from salt slug injections. It also includes the Jupyter notebook (Discharge_process.ipynyb) used to estimate discharge. All discharge measurements collated into rating_curve_processed.csv 4) DTS Contains collated DTS data including raw Stokes and anti-Stokes measurement (provided as .h5 file). It also includes DTS processing.ipynb, a Jupyter notebook for calibrating the DTS data using dts_calibration Python package. cooler_calibration.csv is the DTS calibration CSV used in the calibration sequence. 5) ERT Contains raw resistivity data (provided as CSV files), spatial location of each of the electrodes (provided as CSV files), and files used for the resistivity inversion. 6) Slug_test Includes the slug test data at all the groundwater wells provided as CSV files, as well as the Jupyter notebook (Slug_test.ipynb) for calculating hydraulic conductivity. 7) SP Contains the SP data collected in field at the SP sites (provided as CSV files). 8) Well_data Contains two subfolders: 1) Raw, which provides unprocessed pressure, electrical conductivity and temperature timeseries downloaded from the loggers in all the groundwater and stilling wells, and 2) Processed, which contains sorted, QA/QC timeseries data for each well. The data archive also contains data_process.ipynb, a Jupyter notebook used for field data analysis and generating figures (plotting well, SP, climate, and discharge data, as well as calculating head gradient at sites with nested groundwater wells). Note: Code files (.ipynb, .py, .xml) can be opened in any standard code editor, .exo file can be viewed using Paraview, .h5 files can be opened using HDFView software and h5py Python package, and .resipy file can be opened with the open-source ResIPy software.
ATS↗
Field and Model Data Associated with the Manuscript “Drivers of Streamflow Intermittency in Humid Regions: 1. Evaluating Above- and Below-ground Controls of Flow Persistence in a Forested Catchment”
This package contains field data, modeling files, and scripts supporting the investigation of the drivers of streamflow intermittency in a forested catchment. It includes the field data collected from electrical resistivity tomography (ERT) surveys, ground penetrating radar (GPR), continuous self-potential (SP) monitoring, electromagnetic (EM) imaging, groundwater and stilling well. In addition, it contains the data and results of the coupled water- and electrical-flow model developed using the COMSOL Multiphysics and Advanced Terrestrial Simulator (ATS), as well as software files and Jupyter notebooks used to process the data and generate figures in the manuscript submitted for peer review. The data archive is organized in the following directories: 1) Climate Includes hourly precipitation and daily evapotranspiration time series (2024 – 2025) provided as CSV files, alongside a text file detailing dataset units. 2) Coupled_model Contains two subfolders: Synthetic and Field_Application subfolder. Synthetic subfolder contains the ATS XML input script (can be opened using any code editor) for the four synthetic hydrological cases tested (Connected and gaining, Connected and losing, Disconnected and losing, and dry stream). It also includes other experimental cases to test the influence of precipitation and concentration gradient. For each synthetic case, the flow model simulation is executed using the ATS XML scripts and the included Python script (generate_data_set.py) to convert ATS output to COMSOL-ready input. COMSOL Multiphysics template (.mph can be opened with the commercial software COMSOL and requires a license) is executed using the ATS output data to simulate the potential field. It also includes the Synthetic_model_plot.ipynb (can be opened using any code editor) to visualize the SP result and generate manuscript figures. The data subfolder contains mesh files to run both the ATS (.exo and .stl files can be viewed using Paraview; .h5 files can be opened using HDFView software and h5py Python package) and COMSOL models. Field_Application subfolder contains two subfolders: ES_MDA_inversion and Final_Model. ES_MDA_inversion contains the Python script (.py can be opened using any code editor) and SP observation data used to run the Ensemble Smoother with Multiple Data Assimilation (ES-MDA) inversion sequence to get the optimal model parameters. The Final_model subfolder contains the ATS XML input scripts, data files, output data for the two SP sites. The same workflow steps outlined for the Synthetic subfolder apply here. It also contains the Jupyter notebook (Plot_final_calib.ipynb) to visualize the results of the modeled SP, stream-groundwater exchange and moisture content. 3) Discharge Includes the electrical conductivity (EC) time series (provided as CSV files) from salt slug injections. It also includes the Jupyter notebook (Discharge_process.ipynyb) used to estimate discharge. All discharge measurements collated into rating_curve_processed.csv 4) EM Contains the CSV file of the EM data from the DUALEM-42, including spatial coordinates (x, y, z), apparent conductivity, and in-phase measurements at 2 m coil separations for horizontal coplanar (HCP) and perpendicular (PRP) geometries. 5) ERT Contains raw resistivity data (provided as CSV files), spatial location of each of the electrodes (provided as CSV files), and files used for the resistivity inversion (.resipy can be opened with the open-source ResIPy software). 6) GPR Includes GPR field datasets collected at 100 MHz and 250 MHz antenna frequencies, along with the processing/interpretation project file (GPR_process.gpz can be viewed using EKKO_Project 6, a commercial software by Sensors & Software that requires a license). 7) Slug_test Includes the slug test data at all the groundwater wells provided as CSV files, as well as the Jupyter notebook (Slug_test.ipynb) for calculating hydraulic conductivity. 8) SP Contains the SP data collected in field at the two SP sites (one in the perennial reach and the other in the intermittent reach), provided as DAT files. 9) Well_data Contains two subfolders: 1) Raw, which provides unprocessed pressure, electrical conductivity and temperature timeseries downloaded from the loggers in all the groundwater and stilling wells, and 2) Processed, which contains sorted, QA/QC timeseries data for each well. The data archive also contains data_process.ipynb, a Jupyter notebook used for field data analysis and generating figures (plotting well, SP, climate, and discharge data, as well as calculating head gradient at sites with nested groundwater wells). It also includes DTW.ipynb, a Jupyter notebook containing the code for the dynamic time warping (DTW) with sliding window to evaluate SP signal synchronicity.
ATS↗
ESnet Data and AI Workshop Report
In February 2025, the DOE user facility Energy Sciences Network (ESnet) held a three-day Data and AI Workshop in Berkeley, California. The objective of the workshop was to identify challenges within ESnet that could be addressed through data-driven methods, to help define ESnet’s data-analysis requirements, and to shape its AI strategy, guiding data-stewardship efforts and the direction of AI research and AIOps exploration for ESnet7, the next iteration of ESnet’s network. This report summarizes the multi-faceted discussions and findings and presents a set of recommendations for next steps.
97 MATHEMATICS AND COMPUTING↗
Deep learning-driven super-resolution in Raman hyperspectral imaging: Efficient high-resolution reconstruction from low-resolution data
Deep learning (DL) has become an indispensable tool in hyperspectral data analysis, automatically extracting valuable features from complex, high-dimensional datasets. Super-resolution reconstruction, an essential aspect of hyperspectral data, involves enhancing spatial resolution, particularly relevant to low-resolution hyperspectral data. Yet, the pursuit of super-resolution in hyperspectral analysis is fraught with challenges, including acquiring ground truth high-resolution data for training, generalization, and scalability. The pressing issue of extended spectral acquisition times, notably for high-resolution scans, is a significant roadblock in hyperspectral imaging. Super-resolution methods offer a promising solution by providing higher spatial resolution data to expedite data collection and yield more efficient outcomes. This paper delves into a practical application of these concepts using Raman imaging, where spectral acquisition times can be prohibitively long. In this context, DL-based super-resolution models demonstrate their efficacy by predicting and reconstructing high-resolution Raman data from low-resolution input, eliminating the need for resource-intensive high-resolution scans. While previous work often relied on substantial high-resolution datasets, this study showcases the ability to achieve similar outcomes even with limited data, presenting a more practical and cost-effective approach. In conclusion, the results offer a glimpse into the transformative potential of this technology to streamline hyperspectral imaging applications by saving valuable time and resources through the successful generation of high-resolution data from low-resolution inputs.
42 ENGINEERING↗
A Probabilistic Approach to Load Modeling for Central HVAC Systems in Large Commercial Buildings for Retrofit Decisions Under Uncertainty
Retrofitting central HVAC systems in large commercial buildings with advanced technologies like heat recovery chillers (HRCs) offers a significant opportunity to enhance energy efficiency. However, analyzing these retrofits is challenging with traditional whole-building simulation tools, which require intensive calibration and struggle to model innovative system configurations and controls. To overcome these limitations, this study proposes a load profilebased retrofit analysis framework that provides better decisions under uncertainty. The main focus of this paper is the development of a probabilistic load profile model that can be used in the framework by using exploratory data analysis (EDA) of measured building data to properly quantify its inherent variability. A non-parametric Gaussian Process (GP) model was employed to capture the time- and weather-dependent characteristics of the heating load while explicitly modeling its uncertainty. The model's effectiveness is demonstrated through strong predictive performance on unseen data and physically interpretable insights into load behavior. This data-driven, probabilistic load profile serves as a robust and flexible input for subsequent system simulations, enabling a more confident and statistically sound analysis of retrofit potential.
Ham, S W↗
The Analysis Description Language Ecosystem: Latest developments and physics applications
We present latest developments in Analysis Description Language (ADL), a declarative domain-specific language describing the physics algorithm of a HEP data analysis decoupled from software frameworks. Analyses written in ADL can be integrated into any framework for various tasks. ADL is a multipurpose construct with uses ranging from analysis design to preservation, reinterpretation, queries, visualisation, combination, etc. The most advanced infrastructure to execute ADL on events is the CutLang runtime interpreter. Recent technical developments include an automated interface with different data types, generation of the abstract syntax tree, a visualization tool that that auto-converts analysis flows to graphs, incorporation of trained machine learning models and a Jupyter-based plotting tool. We also report physics implications including a large scale LHC analysis implementation and validation effort for beyond the standard model reinterpretation purposes and studies with ATLAS and CMS open data.
Sekmen, Sezen [Kyungpook National Univ., Daegu (Ko↗