Search NASA⌕ Search

SEARCH · Search NASA

Results for “hyperparameter tuning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Recent Progress on Surface Water Quality Models Utilizing Machine Learning Techniques

Surface waterbodies are heavily exposed to pollutants caused by natural disasters and human activities. Empowering sensor technologies in water quality monitoring, sufficient measurements have become available to develop machine learning (ML) models. Numerous ML models have quickly been adopted to predict water quality indicators in various surface waterbodies. This paper reviews 78 recent articles from 2022 to October 2024, categorizing water quality models utilizing ML into three groups: Point-to-Point (P2P), which estimates the current target value based on other measurements at the same time point; Sequence-to-Point (S2P), which utilizes previous time series data to predict the target value at one time point ahead; and Sequence-to-Sequence (S2S), which uses previous time series data to forecast sequential target values in the future. The ML models used in each group are classified and compared according to water quality indicators, data availability, and model performance. Widely used strategies for improving performance, including feature engineering, hyperparameter tuning, and transfer learning, are recognized and described to enhance model effectiveness. The interpretability limitations of ML applications are discussed. This review provides a perspective on emerging ML for surface water quality models.

machine learning (ML)↗
DOE OSTI2026

Xanthos-Lake Model Source Code

This repository contains the source code for Xanthos-Lake, a lake-modeling extension of the Xanthos framework that introduces a coupled lake component comprising the Xanthos-Lake Snow and Ice Model (xLSIM) and the Xanthos-Lake Water Balance Model (xLWBM). xLSIM is a basin-aware machine-learning model for lake snow, ice, and thermal conditions. It predicts monthly lake ice thickness, snow depth, snow-cover fraction, mixing-layer temperature, and lake ice fraction from meteorological forcing and lake surface-area information. It uses sequence-based deep-learning architectures, including Transformer and hybrid Long Short-Term Memory–Transformer (LSTM–Transformer) models, together with seasonal encoding, multi-lake learning, physical masking, and basin-level cryospheric and non-cryospheric classification. The training workflow uses Ray for scalable execution and includes optional Ray Tune hyperparameter optimization. Model predictions, observations, diagnostics, and feature-importance outputs are written in NetCDF. xLWBM is the water-balance component of the new lake framework. It simulates monthly lake storage, surface area, evaporation, inflow, outflow, and lake–groundwater exchange. It combines physical water-balance equations with calibrated bathymetric relationships, weir-based outlet flow, modified Penman open-water evaporation, groundwater head relaxation, Penman–Monteith snow and ice sublimation, and snow, ice, and thermal conditions supplied by xLSIM. The model calibrates lake parameters against satellite-derived surface-area data, using evaporation-based calibration where surface-area data are unavailable, and supports small, medium, and large lake classes. For large lakes, xLWBM is integrated with the managed-routing workflow so that lake storage and outflow interact directly with downstream river routing and reservoir operations. Together, xLSIM and xLWBM provide Xanthos with a coupled lake-modeling capability. xLSIM supplies the snow, ice, and thermal conditions that affect lake evaporation and snow- and ice-related water exchanges, while xLWBM translates those conditions into dynamic lake storage, surface area, evaporation, and discharge. In return, xLWBM supplies evolving lake surface area to xLSIM. This coupling enables Xanthos to represent lakes as active hydrologic components within basin-scale water-availability and routing simulations.

Machine Learning↗
DOE OSTI2021

Hybrid deep learning architecture for general disruption prediction across tokamaks

In this paper, we present a new deep learning disruption prediction algorithm based on important findings from explorative data analysis which effectively allows knowledge transfer from existing devices to new ones, thereby predicting disruptions using very limited disruptive data from the new devices. The explorative data analysis conducted via unsupervised clustering techniques confirms that time-sequence data are much better separators of disruptive and non-disruptive behavior than the instantaneous plasma state data with further advantageous implications for a sequence-based predictor. Based on such important findings, we have designed a new algorithm for multi-machine disruption prediction that achieves high predictive accuracy on the C-Mod (AUC=0.801), DIII-D (AUC=0.947) and EAST (AUC=0.973). tokamaks with limited hyperparameter tuning. Through numerical experiments, we show that boosted accuracy (AUC=0.959) is achieved on EAST predictions by including in the training only 20 disruptive discharges, thousands of non-disruptive discharges from EAST, and combining this with more than a thousand discharges from DIII-D and C-Mod. The improvement of predictive ability obtained by combining disruptive data from other devices is found to be true for all permutations of the three devices. Furthermore, by comparing the predictive performance of each individual numerical experiment, we find that non-disruptive data are machine-specific while disruptive data from multiple devices contain device-independent knowledge that can be used to inform predictions for disruptions occurring on a new device.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2013

Approximate l-fold cross-validation with Least Squares SVM and Kernel Ridge Regression

Kernel methods have difficulties scaling to large modern data sets. The scalability issues are based on computational and memory requirements for working with a large matrix. These requirements have been addressed over the years by using low-rank kernel approximations or by improving the solvers scalability. However, Least Squares Support VectorMachines (LS-SVM), a popular SVM variant, and Kernel Ridge Regression still have several scalability issues. In particular, the O(n^3) computational complexity for solving a single model, and the overall computational complexity associated with tuning hyperparameters are still major problems. We address these problems by introducing an O(n log n) approximate l-fold cross-validation method that uses a multi-level circulant matrix to approximate the kernel. In addition, we prove our algorithm s computational complexity and present empirical runtimes on data sets with approximately 1 million data points. We also validate our approximate method s effectiveness at selecting hyperparameters on real world and standard benchmark data sets. Lastly, we provide experimental results on using a multi-level circulant kernel approximation to solve LS-SVM problems with hyperparameters selected using our method.

Edwards, Richard E↗
DOE OSTI2021

Failure prediction and estimation of failure parameters

Machine-learning methods and apparatus are disclosed to determine frictional state or other parameters in an earthquake zone or other failing medium, using acoustic emission, seismic waves, or other detectable indicators of microscopic processes. Predictions of future failures are demonstrated in different regimes. A classifier is trained using time series of acoustic emission data along with historic data of frictional state or failure events. In disclosed examples, random forests and gradient boost trees are used, and grid-search or EGO procedures are used for hyperparameter tuning. Once trained, the classifier can be applied to testing or live data in order to assess a frictional state, assess seismic hazard, or make predictions regarding a future failure event. The technology has been developed in a double direct shear apparatus, but can be widely applied to seismic faults, other terrestrial failures, or failures in man-made structures. Variations are disclosed.

Johnson, Paul Allan↗
DOE OSTI2022

Subsurface stress criticality associated with fluid injection and determined using machine learning

Machine-learning methods and apparatus are disclosed to determine critical state or other parameters related to fluid-driven failure of a terrestrial locale impacted by anthropogenic activities such as hydraulic fracturing, hydrocarbon extraction, wastewater disposal, or geothermal harvesting. Acoustic emission, seismic waves, or other detectable indicators of microscopic processes are sensed. A classifier is trained using time series of microscopic data along with corresponding data of critical state or failure events. In disclosed examples, random forests and artificial neural networks are used, and grid-search or EGO procedures are used for hyperparameter tuning. Once trained, the classifier can be applied to live data from a fluid injection locale in order to assess a frictional state, assess seismic hazard, assess permeability, make predictions regarding a future fluid-driven failure event, or drive engineering solutions for mitigation or remediation. Variations are disclosed.

Johnson, Paul Allan↗
DOE OSTI2023

Deep Learning for Modeling Enhanced Geothermal Systems

Enhanced Geothermal Systems (EGS) offer a vast potential to expand the use of geothermal energy. Heat is extracted from this engineered system by injecting cold water into a subsurface fractures, which are in contact with the hot dry rock, and pulled through the production wells. Creating EGS requires improving the natural permeability of hot crystalline rocks. To develop economically viable EGS reservoirs, significant technical barriers (e.g., better stimulation technologies without adequate water and/or permeability) and non-technical barriers (e.g., land access and permitting) must be overcome. In this short conference paper, we present a workflow to address a part of this challenge – “How to develop economically viable EGS using existing technologies?”. Our workflow called the GeoThermalCloud (GTC) for EGS, leverages recent advances in machine learning, deep learning, and cloud computing. This GTC framework is open-source and available at https://github.com/SmartTensors/GeoThermalCloud.jl. The GTC framework provides trained deep learning (DL) models to estimate the net present value of a given EGS design scenario. The Geothermal Design Tool (https://github.com/GeoDesignTool/GeoDT.git), a fast and simplified multi-physics solver, is used to develop a database for training DL models. The database consists of EGS design parameters (inputs to DL model) and their net present value (output of DL model) in uncertain geologic systems. The EGS design parameters for constructing this training database are based on Utah FORGE but include the options of more wells and deeper depths. The DL models are trained by ingesting the EGS design parameters and estimating the corresponding net present value. Such an emulation allows us to screen various EGS designs quickly and identify good development strategies by coupling them with optimization techniques. Our preliminary results show promise in DL emulation of net present value. However, a lot more work is needed to improve the predictive capability of DL models (i.e., extensive hyperparameter tuning is necessary). This will be the primary focus of our future work.

artificial neural networks, geothermal↗
DOE OSTI2025

Neural Network Analysis of Nuclear Magnetic Resonance and Infrared Spectra

Nuclear magnetic resonance (NMR) spectroscopy and infrared (IR) spectroscopy are powerful chemical characterization techniques with broad general usage. However, the manual evaluation of the resulting spectra is time-consuming and requires significant expertise, preventing insights from being used in real-time applications. With recent advances in computation and artificial intelligence (AI), new tools are available for automating spectral interpretation. In this work, machine learning (ML) algorithms using 1-dimensional convolutional neural networks (CNNs) were applied to identify common functional groups from spectral information. Raw spectra were collected virtually from the Human Metabolome Database (HMDB) and National Institute of Standards and Technology (NIST) Chemistry WebBook and processed into a suitable standard. Algorithm design was tailored to best fit the nature of the problem, with built-in flexibility to accommodate relevant parameters beyond the raw spectral input, specifically solvent identity and magnetic frequency for NMR. The predictive capability of the algorithm in identifying functional groups is displayed in several examples. This methodology has been compiled into a code repository and could easily be modified to adapt alternative data sources, including other spectrum types. To mitigate overfitting, a common problem in mathematical modeling where overfamiliarity with training data produces trends that are not representative of the general data, a novel metric was developed, referred to as Accufit. Accufit includes a parameter that penalizes substantial differences in the training accuracy and the accuracy of an independent validation set. Examples are presented showing the effectiveness of Accufit in maintaining the model’s predictive capability while controlling the overfitting when used as a custom metric for hyperparameter tuning.

Sturgill, James↗
DOE OSTI2025

Predicting Initial Trans-Membrane Pressure for Optimized Operations in UF Unit Using Random Forest

With the growing scarcity of freshwater, innovative process design mechanisms like Reverse Osmosis (RO) are increasingly gaining attention among water treatment utilities to address the rising demand. Ensuring reliable water production necessitates efficient resource utilization, minimizing downtime in (ultra-filtration) UF systems. Recent advancements in machine learning (ML) have enabled the development of accurate data-driven models for Model Predictive Control (MPC), often requiring minimal prior knowledge of underlying physical processes. In this study, we present predictive regression models based on Random Forest (RF) and Auto-Regressive (AR) approaches to forecast the initial Trans-Membrane Pressure (TMP) for each filtration cycle in data generated by Direct Potable Reuse (DPR) systems. The proposed RF-based model demonstrates superior performance compared to baseline methods, including historical mean, Last Observation Carried Forward (LOCF), and naïve AR models, across various forecasting horizons in terms of root mean square error (RMSE) metric. To evaluate how different classes of process variables contribute to TMP dynamics over time, we examine the feature importance of independent covariates across multiple forecast horizons. This analysis provides insight into the temporal relevance of operational and sensor-derived features, guiding control and monitoring strategies. Additionally, the impact of hyperparameter tuning on TMP prediction performance is studied for both direct and recursive RF modelling approaches across increasing forecast horizons. Accurate prediction of initial TMP is critical for optimizing RO operations, as it enables the development of robust modelling frameworks by accurately estimating membrane fouling trends, thereby enhancing process efficiency and long-term reliability. The demonstrated efficacy of the RF-based approach highlights its potential as a tool for real-time decision-making in water treatment systems, paving the way for advanced process optimization and sustainable water resource management.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗
DOE OSTI2025

Investigation of the Performance and Explainability Tradeoffs for Machine-Learning Models for Predictive Maintenance of Circulating Water Systems in Nuclear Power Plants

Predictive maintenance (PdM) has shown great potential for achieving substantial cost savings and enhancing the economic competitiveness of nuclear power plants (NPPs) in today's energy market. Among the different modeling approaches that exist, machine learning (ML) tools in particular have a demonstrated ability to handle high dimensional and multivariate data and to extract hidden relationships within data in industrial environments. While ML methods show great potential, their lack of explainability---especially for black-box models---is a major hurdle to their adoption. Moreover, considering the supposed trade-off between explainability and performance challenges, careful consideration must be made as to which of these quality aspects takes precedence in light of multiple modeling options, resource availability, and domain characteristics. The present work evaluates the performance of six ML models, each with a different degree of explainability, in classifying the conditions of circulating water pumps (CWPs) by utilizing sensor data from nuclear power plants. To determine the drivers behind the trade-offs presented by this array of models, this work also tests different combinations of CWP units as the training and testing data, degrees of data imbalance, and objective functions for hyperparameter tuning. It was found that black-box models tend to afford superior performance in cases where there are far more instances of one type of labeled data than of any other type. It is recommended that a guided procedure be followed for designing and delivering an ML system that is sufficiently explainable to all involved stakeholders.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2026

Real-Time Detection of Charge Jumps in Superconducting Qubits with a Convolutional Neural Network

Ionizing radiation from cosmic rays and gammas can induce discontinuous jumps in the environmental charge of superconducting qubits (charge jumps), causing correlated errors that challenge fault-tolerant quantum computing while simultaneously providing a detection signature for quantum sensing applications. Current detection methods operate offline, introducing latency incompatible with in-the-loop qubit control. In this paper, an online detector of charge jumps for superconducting qubits, based on a dilated causal convolutional neural network (DCCNN) designed for in-the-loop deployment on the Quantum Instrumentation Control Kit (QICK) platform, is presented. The network is trained on synthetic Ramsey tomography scans generated from qubit templates measured at the Northwestern Experimental Underground Site (NEXUS) at Fermilab, and translated to FPGA firmware via hls4ml with ap_fixed$\langle 16,6 \rangle$ quantization, reaching a per-inference latency of $6.19 μ$s on the Zynq UltraScale+ RFSoC ZCU216. At this operating point the DCCNN matches the detection efficiency of the established offline $χ^2$ algorithm ($0.843 \pm 0.022$ vs. $0.866 \pm 0.020$ on $|Δq| \in [0.1, 0.5] e$ at matched false-positive rate), while requiring no per-qubit hyperparameter tuning. This shifts charge-jump detection from a post-hoc diagnostic to a control-loop primitive, enabling adaptive protocols that respond to radiation-induced events in situ, with applications to quantum-computing error mitigation and to the use of superconducting qubits as particle detectors.

Gaytan-Villarreal, Daniel [Carnegie Mellon U.]↗
DOE OSTI2021

Multi-Objective Hyperparameter Optimization for Spiking Neural Network Neuroevolution

Neuroevolution has had significant success over recent years, but there has been relatively little work applying neuroevolution approaches to spiking neural networks (SNNs). SNNs are a type of neural network that includes temporal processing component, are not easily trained using other methods, and can be deployed into energy-efficient neuromorphic hardware. In this work, we investigate two evolutionary approaches for training SNNs. We explore the impact of the hyperparameters of the evolutionary approaches, including tournament size, population size, and representation type, on the performance of the algorithms. We present a multi-objective Bayesian-based hyperparameter optimization approach to tune the hyperparameters to produce the most accurate and smallest SNNs. We show that the hyperparameters can significantly affect the performance of these algorithms. We also perform sensitivity analysis and demonstrate that every hyperparameter value has the potential to perform well, assuming other hyperparameter values are set correctly.

Parsa, Maryam↗
DOE OSTI2021

A Software Framework for Comparing Training Approaches for Spiking Neuromorphic Systems

There are a wide variety of training approaches for spiking neural networks for neuromorphic deployment. However, it is often not clear how these training algorithms perform or compare when applied across multiple neuromorphic hardware platforms and multiple datasets. In this work, we present a software framework for comparing performance across four neuromorphic training algorithms across three neuromorphic simulators and four simple classification tasks. We introduce an approach for training a spiking neural network using a decision tree, and we compare this approach to training algorithms based on evolutionary algorithms, back-propagation, and reservoir computing. We present a hyperparameter optimization approach to tune the hyperparameters of the algorithm, and show that these optimized hyperparameters depend on the processor, algorithm, and classification task. Finally, we compare the performance of the optimized algorithms across multiple metrics, including accuracy, training time, and resulting network size, and we show that there is not one best training algorithm across all datasets and performance metrics.

Schuman, Catherine↗
DOE OSTI2021

HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization

We present a new software, HYPPO, that enables the automatic tuning of hyperparameters of various deep learning (DL) models. Unlike other hyperparameter optimization (HPO) methods, HYPPO uses adaptive surrogate models and directly accounts for uncertainty in model predictions to find accurate and reliable models that make robust predictions. Using asynchronous nested parallelism, we are able to significantly alleviate the computational burden of training complex architectures and quantifying the uncertainty. HYPPO is implemented in Python and can be used with both TensorFlow and PyTorch libraries. We demonstrate various software features on time-series prediction and image classification problems as well as a scientific application in computed tomography image reconstruction. Finally, we show that (1) we can reduce by an order of magnitude the number of evaluations necessary to find the most optimal region in the hyperparameter space and (2) we can reduce by two orders of magnitude the throughput for such HPO process to complete.

adaptation models↗
DOE OSTI2023

Surrogate Model Based Optimization for Finding Robust Deep Learning Model Architectures

Deep Learning (DL) models are increasingly used throughout the sciences. However, their performance and usefulness depend greatly on their architecture which is defined by hyperparameters such as the number of nodes, layers, the learning rate, etc. Tuning these hyperparameters is time-consuming because evaluating their performance requires a lengthy training step. Stochastic optimizers used in training lead to performance variability and potentially prediction reliability issues. In this talk, we will describe an automated optimization method based on surrogate models and active learning strategies for tuning DL model architectures. We take into account the prediction variability with the goal to identify architectures that make reliable and robust predictions. We demonstrate our developments on an application arising in particle physics.

deep learning↗
DOE OSTI2023

Multiobjective Hyperparameter Optimization for Deep Learning Interatomic Potential Training Using NSGA-II

Deep neural network (DNN) potentials are an emerging tool for simulation of dynamical atomistic systems, with the promise of quantum mechanical accuracy at speedups of 10000$\times$. As with other DNN methods, hyperparameters used during training can make a substantial difference in model accuracy, and optimal settings vary with dataset. To enable rapid tuning of hyperparameters for DNN potential training, we developed a scalable multiobjective optimization evolutionary algorithm for supercomputers and tested it on the Summit system at the Oak Ridge Leadership Computing Facility (OLCF). The multiobjective approach is required due to the coupling of two learned values defining the potential: the energy and force. Using a large-scale implementation of the NSGA-II algorithm adapted for training DNN potentials, we discovered several optimal multiobjective combinations, including best choices of activation functions, learning rate scaling scheme, and pairing of the two radial cutoffs used in the three dimensional descriptor function.

Coletti, Mark↗
DOE OSTI2022

An Accuracy-Maximization Approach for Claims Classifiers in Document Content Analytics for Cybersecurity

This paper presents our research approach and findings towards maximizing the accuracy of our classifier of feature claims for cybersecurity literature analytics, and introduces the resulting model ClaimsBERT. Its architecture, after extensive evaluations of different approaches, introduces a feature map concatenated with a Bidirectional Encoder Representation from Transformers (BERT) model. We discuss deployment of this new concept and the research insights that resulted in the selection of Convolution Neural Networks for its feature mapping aspects. We also present our results showing ClaimsBERT to outperform all other evaluated approaches. This new claims classifier represents an essential processing stage within our vetting framework aiming to improve the cybersecurity of industrial control systems (ICS). Furthermore, in order to maximize the accuracy of our new ClaimsBERT classifier, we propose an approach for optimal architecture selection and determination of optimized hyperparameters, in particular the best learning rate, number of convolutions, filter sizes, activation function, the number of dense layers, as well as the number of neurons and the drop-out rate for each layer. Fine-tuning these hyperparameters within our model led to an increase in classification accuracy from 76% obtained with BertForSequenceClassification’s original model to a 97% accuracy obtained with ClaimsBERT.

Ameri, Kimia (ORCID:0000000328791871)↗