SEARCH · Search NASA

Results for “overfitting”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Machine learning and ligand binding predictions: A review of data, methods, and obstacles

We report that computational predictions of ligand binding is a difficult problem, with more accurate methods being extremely computationally expensive. The use of machine learning for drug binding predictions could possibly leverage the use of biomedical big data in exchange for time-intensive simulations. This paper reviews current trends in the use of machine learning for drug binding predictions, data sources to develop machine learning algorithms, and potential problems that may lead to overfitting and ungeneralizable models. A few popular datasets that can be used to develop virtual high-throughput screening models are characterized using spatial statistics to quantify potential biases. We can see from evaluating some common benchmarks that good performance correlates with models with high-predicted bias scores and models with low bias scores do not have much predictive power. A better understanding of the limits of available data sources and how to fix them will lead to more generalizable models that will lead to novel drug discovery.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2020

Application of Systems Engineering Principles and Techniques in Biological Big Data Analytics: A Review

In the past few decades, we have witnessed tremendous advancements in biology, life sciences and healthcare. These advancements are due in no small part to the big data made available by various high-throughput technologies, the ever-advancing computing power, and the algorithmic advancements in machine learning. Specifically, big data analytics such as statistical and machine learning has become an essential tool in these rapidly developing fields. As a result, the subject has drawn increased attention and many review papers have been published in just the past few years on the subject. Different from all existing reviews, this work focuses on the application of systems, engineering principles and techniques in addressing some of the common challenges in big data analytics for biological, biomedical and healthcare applications. Specifically, this review focuses on the following three key areas in biological big data analytics where systems engineering principles and techniques have been playing important roles: the principle of parsimony in addressing overfitting, the dynamic analysis of biological data, and the role of domain knowledge in biological data analytics.

dynamic analysis↗

DOE OSTI2023

Convoluted filtering for process cycle modeling

Principles of materials science and engineering, physics, mathematics, and information science are used to extract knowledge and insights from the process-structure–property-performance relationships hidden in materials data. The process-structure modeling can be accelerated without loss of interpretability, with artificial intelligence tools that mimic the salient features of the process and process-structure relations. In this work, a novel convoluted model-filtering technique was exploited to build and successfully train the Convoluted Filter (CoFi) artifacts for Fe-based alloy heat treatment cycles. The artifacts were pre-trained to filter out deep models that change the surrogate microstructure state after the heat treatment at ambient conditions. Direct representation of the thermal cycle features within knowledge Graph facilitated development of meaningful data models for microstructure evolution, which reduce overfitting to limited datasets.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Machine Learning for First Principles Calculations of Material Properties for Ferromagnetic Materials

The investigation of finite temperature properties using Monte-Carlo (MC) methods requires a large number of evaluations of the system’s Hamiltonian to sample the phase space needed to obtain physical observables as function of temperature. DFT calculations can provide accurate evaluations of the energies, but they are too computationally expensive for routine simulations. To circumvent this problem, machine-learning (ML) based surrogate models have been developed and implemented on high-performance computing (HPC) architectures. In this paper, we describe two ML methods (linear mixing model and HydraGNN) as surrogates for first principles density functional theory (DFT) calculations with classical MC simulations. These two surrogate models are used to learn the dependence of target physical properties from complex compositions and interactions of their constituents. We present the predictive performance of these two surrogate models with respect to their complexity while avoiding the danger of overfitting the model. An important aspect of our approach is the periodic retraining with newly generated first principles data based on the progressive exploration of the system’s phase space by the MC simulation. The numerical results show that HydraGNN model attains superior predictive performance compared to the linear mixing model for magnetic alloy materials.

Eisenbach, Markus↗

DOE OSTI2021

Jet tagging in the Lund plane with graph networks

The identification of boosted heavy particles such as top quarks or vector bosons is one of the key problems arising in experimental studies at the Large Hadron Collider. In this article, we introduce LundNet, a novel jet tagging method which relies on graph neural networks and an efficient description of the radiation patterns within a jet to optimally disentangle signatures of boosted objects from background events. We apply this framework to a number of different benchmarks, showing significantly improved performance for top tagging compared to existing state-of-the-art algorithms. We study the robustness of the LundNet taggers to non-perturbative and detector effects, and show how kinematic cuts in the Lund plane can mitigate overfitting of the neural network to model-dependent contributions. Finally, we consider the computational complexity of this method and its scaling as a function of kinematic Lund plane cuts, showing an order of magnitude improvement in speed over previous graph-based taggers.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2021

Optimal decision trees for categorical data via integer programming

Decision trees have been a very popular class of predictive models for decades due to their interpretability and good performance on categorical features. However, they are not always robust and tend to overfit the data. Additionally, if allowed to grow large, they lose interpretability. In this paper, we present a mixed integer programming formulation to construct optimal decision trees of a prespecified size. We take the special structure of categorical features into account and allow combinatorial decisions (based on subsets of values of features) at each node. Our approach can also handle numerical features via thresholding. Here we show that very good accuracy can be achieved with small trees using moderately-sized training sets. The optimization problems we solve are tractable with modern solvers.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Modeling household online shopping demand in the U.S.: a machine learning approach and comparative investigation between 2009 and 2017

Despite the rapid growth of online shopping and research interest in the relationship between online and in-store shopping, national-level modeling and investigation of the demand for online shopping with a prediction focus remain limited in the literature. Here, this paper differs from prior work and leverages two recent releases of the U.S. National Household Travel Survey (NHTS) data for 2009 and 2017 to develop machine learning (ML) models, specifically gradient boosting machine (GBM), for predicting household-level online shopping purchases. The NHTS data allow for not only conducting nationwide investigation but also at the level of households, which is more appropriate than at the individual level given the connected consumption and shopping needs of members in a household. We follow a systematic procedure for model development including employing Recursive Feature Elimination algorithm to select input variables (features) in order to reduce the risk of model overfitting and increase model explainability. Among several ML models, GBM is found to yield the best prediction accuracy. Extensive post-modeling investigation is conducted in a comparative manner between 2009 and 2017, including quantifying the importance of each input variable in predicting online shopping demand, and characterizing value-dependent relationships between demand and the input variables. In doing so, two latest advances in machine learning techniques, namely Shapley value-based feature importance and Accumulated Local Effects plots, are adopted to overcome inherent drawbacks of the popular techniques in current ML modeling. The modeling and investigation are performed at the national level, with a number of findings obtained. The models developed and insights gained can be used for online shopping-related freight demand generation and may also be considered for evaluating the potential impact of relevant policies on online shopping demand.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2022

Assessment of Outliers in Alloy Datasets Using Unsupervised Techniques

We report advancements in data analytics techniques have enabled complex, disparate datasets to be leveraged for alloy design. Identifying outliers in a dataset can reduce noise, identify erroneous and/or anomalous records, prevent overfitting, and improve model assessment and optimization. In this work, two alloy datasets (9-12% Cr ferritic martensitic steels, and austenitic stainless steels) have been assessed for outliers using unsupervised techniques and supplemented with domain knowledge. Principal component analysis and k-means clustering were applied to the data, and points were assessed as outliers based on their distance away from other points in the cluster and from other points in the dataset. The outlier characteristics were investigated to determine both cluster-specific and overall trends in the properties of the outlier points. The approach demonstrated here is extensible to other alloy datasets for outlier identification and evaluation to improve the reliability of machine learning and modeling predictions for advanced alloy design.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Short-term apartment-level load forecasting using a modified neural network with selected auto-regressive features

Residential electricity load profiles and their diversity have become increasingly important to realize the benefits of Smart or Transactive Energy Networks (TENs). An important element of TENs will be practical, accurate, and implementable residential load forecasting techniques. While there have been many approaches to short-term load forecasting, few have included forecasting for individual households, partly because the high volatility and idiosyncrasies present in individual household load data can pose significant challenges. In this study, we develop a Convolutional Long Short-Term Memory-based neural network with Selected Autoregressive Features (termed a CLSAF model) to improve short-term household electricity load forecasting accuracy by employing three strategies: autoregressive features selection, exogenous features selection, and a “default” state to avoid overfitting at times of high load volatility. We include aggregations of apartments to floor and building level, because utilities may favor transactive approaches that rely on aggregator models, e.g., a cluster of consumers as opposed to an individual. We demonstrate that the CLSAF model, by virtue of its enhanced feature representation and modest computational resources, can accomplish load forecasting in a multi-family residential building across three spatial granularities (individual apartment/household, floor, and building levels), with an accuracy improvement of up to 25% compared to a persistence model. We propose a data screening technique to characterize time-series electricity-load data. This technique is suitable for integration into a TEN ecosystem and allows one to estimate confidence levels of the load forecasts to optimize computational resources and the risks associated with uncertain forecasts.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2026

Virtual refrigerant charge sensor for variable-speed heat pumps based on feature selection

The refrigerant charge level in heat pump systems significantly impacts their energy efficiency. Virtual refrigerant charge (VRC) sensing technology has been comprehensively investigated and well-established due to its lower cost compared to physical sensors. However, the previous VRC research often relied on expert judgment and physical reasoning for their variable selection, which can potentially select redundant (or highly correlated) or insignificant features, and it is also primarily focused on single-speed systems. To address these challenges, this study proposes a VRC algorithm for variable-speed heat pumps that selects features through a rigorous feature selection method in combination with physical insights. We also propose a piecewise linear model structure segmented by subcooling temperature to accurately predict charge levels, particularly when subcooling temperatures are substantially low. The proposed algorithm was evaluated using experimental data of a residential R410A heat pump, and the performance was compared with two baseline VRC algorithms. The results are: (1) The proposed algorithm outperforms for the case with subcooling temperature less than 1 °C. (2) The proposed algorithm achieves a tested mean absolute percentage error (MAPE) of 4.23%, and improves the overall accuracy for cooling conditions by approximately 60%, compared with the two baseline algorithms. (3) The proposed algorithm uses two fewer features and improves the accuracy for undercharge cooling conditions by 68.0%, compared with baseline algorithm 2. These improvements enhance prediction accuracy and prevent overfitting, providing a more reliable refrigerant charge level prediction and helping improve the heat pump energy efficiency.

Liang, Chenjiyu↗

DOE OSTI2026

Virtual refrigerant charge sensing algorithm for residential CO₂ heat pumps

Natural refrigerants are increasingly adopted in next-generation heat pump systems, among which CO₂ heat pumps have attracted significant attention. However, due to their high operating pressures, the leakage risk is higher, resulting in undercharge conditions and degraded heat pump performance. Thus, developing an accurate refrigerant charge level detection technique is necessary to guarantee safe and efficient operation. Although virtual refrigerant charge (VRC) level calculation algorithms for CO₂ heat pumps exist, they typically rely on empirically selected features without a systematic selection framework, leading to multicollinearity and potential overfitting, which limit their prediction accuracy and generalizability. To address these issues, this study proposes a VRC algorithm framework with a systematic feature selection method that identifies physically meaningful and statistically significant features, and is applied using a residential CO₂ heat pump as a case study. The method is extended from previous work on conventional refrigerants to account for charge behavior in CO₂ gas coolers. The selected features include gas cooler outlet density, evaporator pressure, and superheat temperature. The results demonstrate that the proposed feature selection method significantly improves prediction accuracy compared to existing VRC approaches. A relatively small training dataset (∼30 samples) is sufficient for feature identification and model development. The developed algorithm achieves less than 3% prediction error under both undercharge and overcharge conditions, representing reductions of 46.7% and 35.3% compared to two recent reference VRC algorithms for transcritical CO₂ heat pumps reported in the literature. The proposed algorithm and feature selection method enhance leakage detection capability, facilitate the deployment of CO₂ heat pump systems, and contribute to reduced energy waste and maintenance costs.

Guo, Fangzhou [Lawrence Berkeley National Laborato↗

DOE OSTI2024

Predicting RNA structure and dynamics with deep learning and solution scattering

Advanced deep learning and statistical methods can predict structural models for RNA molecules. However, RNAs are flexible, and it remains difficult to describe their macromolecular conformations in solutions where varying conditions can induce conformational changes. Small-angle x-ray scattering (SAXS) in solution is an efficient technique to validate structural predictions by comparing the experimental SAXS profile with those calculated from predicted structures. There are two main challenges in comparing SAXS profiles to RNA structures: the absence of cations essential for stability and charge neutralization in predicted structures and the inadequacy of a single structure to represent RNA’s conformational plasticity. We introduce a solution conformation predictor for RNA (SCOPER) to address these challenges. This pipeline integrates kinematics-based conformational sampling with the innovative deep learning model, IonNet, designed for predicting Mg 2+ ion binding sites. Validated through benchmarking against 14 experimental data sets, SCOPER significantly improved the quality of SAXS profile fits by including Mg 2+ ions and sampling of conformational plasticity. We observe that an increased content of monovalent and bivalent ions leads to decreased RNA plasticity. Therefore, carefully adjusting the plasticity and ion density is crucial to avoid overfitting experimental SAXS data. SCOPER is an efficient tool for accurately validating the solution state of RNAs given an initial, sufficiently accurate structure and provides the corrected atomistic model, including ions.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Predicting concrete compressive strength using hybrid ensembling of surrogate machine learning models

This study aims to implement a hybrid ensemble surrogate machine learning technique in predicting the compressive strength (CS) of concrete, an important parameter used for durability design and service life prediction of concrete structures in civil engineering projects. For this purpose, an experimental database consisting of 1030 records has been compiled from the machine learning repository of the University of California, Irvine. The database was used to train and validate four conventional machine learning (CML) models, namely Artificial Neural Network (ANN), Linear and Non-Linear Multivariate Adaptive Regression Splines (MARS-L and MARS-C), Gaussian Process Regression (GPR), and Minimax Probability Machine Regression (MPMR). Subsequently, the predicted outputs of CML models were combined and trained using ANN to construct the Hybrid Ensemble Model (HENSM). It is observed that the proposed HENSM produces higher predictive accuracy compared to the CML models used in the present study. The predictive performance of all models for CS prediction was compared using the testing dataset and it is found that the HENSM model attained the highest predictive accuracy in both phases. Based on the experimental results, the newly constructed HENSM model is very potential to be a new alternative in handling the overfitting issues of CML models and hence, can be used to predict the concrete CS, including the design of less polluting and more sustainable concrete constructions.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Multisublattice cluster expansion study of short-range ordering in iron-substituted strontium titanate

Owing to the challenges in obtaining realistic atomic configurations in large chemical phase spaces, it is not straightforward to describe structure–property relations in materials exhibiting configurational disorder. One example is iron-substituted strontium titanate (SrTi 1–x Fe x O 3–d , STF), a promising perovskite-derivative cathode material in solid oxide fuel cells that exhibits full solid solubility 0 ≤ x ≤ 1 and a tendency to exhibit short-range order. Here we demonstrate a multisublattice cluster expansion (CE) framework and apply it to STF across the full composition range. The CE approach is distinct from more traditional CE formulations in that clusters are defined explicitly by the chemical species distributed among multiple sublattices, rather than via cluster functions of occupation variables with decoration. The modified CE approach makes it easy to distinguish meaningful chemical interactions that are harder to extract from conventional CE, since for the latter chemical identity in a cluster is expressed as a product of site occupations. The least absolute shrinkage and selection operator (LASSO) is implemented as a regression analysis tool to select key clusters and avoid overfitting. We demonstrate this formulation on STF, and show that it can accurately predict configurational energies in comparison to conventional CE. From the key clusters, we identify that short-range ordering between substitutional Fe and oxygen vacancies (V O ) results in the formation of Fe–VO strings. In addition, we consider the stability of STF through CE-based Monte Carlo (MC) simulations and confirm the presence of superstructures that were previously observed in transmission electron microscopy. In this work, analysis of atomic configurations from MC samples reveals variations in the oxidation state of Fe atoms, which can be explained by the ordering tendency of Fe and V O . The cluster description and selection formalism described here may be applied to other disordered multisublattice systems for accurate and efficient material modeling.

36 MATERIALS SCIENCE↗

DOE OSTI2025

Challenges in predicting protein-protein interactions of understudied viruses: Arenavirus-human interactions

Understanding protein-protein interactions (PPIs) between viruses and host organisms is crucial for uncovering infection mechanisms and identifying potential therapeutic targets. The ability to generalize PPI predictive models across understudied viruses presents a significant challenge. In this work, we use arenavirus-human PPIs to illustrate the difficulties associated with model generalization, which are compounded by a lack of both positive and negative data. We employ a Transfer Learning approach to investigate arenavirus-human PPIs by utilizing models trained on better-studied virus-human and human-human PPIs. Additionally, we curate and assess four types of negative sampling datasets to evaluate their impact on model performance. Despite the overall high accuracies (93–99 %) and AUPRC scores (0.8–0.9) appearing promising, further analysis indicates that these performance metrics can be misleading due to data leakage, data bias, and overfitting, especially concerning under-represented viral proteins. We reveal these gaps and assess the impact of data imbalance using standard k-fold cross-validation and Independent Blind Testing with a Balanced Dataset, resulting in a drop in accuracy below 50 %. We propose a viral protein-specific evaluation framework that categorizes viral proteins into majority and minority classes based on their representation in the dataset, enabling comparison of model performance across these groups using balanced accuracies. This framework offers a more robust evaluation of model generalizability, addressing biases inherent in standard evaluation techniques and paving the way for more reliable PPI prediction models for understudied viruses.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2025

Nuclear microreactor transient and load-following control with deep reinforcement learning

The economic feasibility of nuclear microreactors will depend on minimizing operating costs through advancements in autonomous control, especially when these microreactors are operating alongside other types of energy systems (e.g., renewable energy). This study explores the application of deep reinforcement learning (RL) for real-time drum control in microreactors, exploring performance in regard to load-following scenarios. By leveraging a point kinetics model with thermal and xenon feedback, we first establish a baseline using a single-output RL agent, then compare it against a traditional proportional–integral–derivative (PID) controller. This study demonstrates that RL controllers, including both single- and multi-agent RL (MARL) frameworks, can achieve similar or even superior load-following performance as traditional PID control across a range of load-following scenarios. In short transients, the RL agent was able to reduce the tracking error rate in comparison to PID by one half to one third. Over extended 300-minute load-following scenarios in which xenon feedback becomes a dominant factor, PID maintained better accuracy, but RL still remained within a 1% error margin despite being trained only on short-duration scenarios. This highlights RL’s strong ability to generalize and extrapolate to longer, more complex transients, affording substantial reductions in training costs and reduced overfitting. Furthermore, when control was extended to multiple drums, MARL enabled independent drum control as well as maintained reactor symmetry constraints without sacrificing performance---an objective that standard single-agent RL could not learn. We also found that, as increasing levels of Gaussian noise were added to the power measurements, the RL controllers were able to maintain lower error rates than PID, and to do so with at least 10% and upwards of 150% less control effort. These findings illustrate RL's potential for autonomous nuclear reactor control, laying the groundwork for future integration into high-fidelity simulations and experimental validation efforts.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2022

Logistic Regression in Clinical Studies

• A logistic regression model is used when the outcome of interest is binary. The term “logistic” refers to the underlying “logit” (log odds) function that is used to model the binary outcome. • Odds ratios are produced from a logistic regression model and have a useful interpretation. • Tips, tricks and concepts used to fit logistic regression models are similar to those used in linear regression models. • Modeling building that is knowledge-based rather than automatic is preferred in most applications of logistic regression. • A logistic regression model that is overparameterized (ie, too many variables for too few events) can result in odds ratios that are implausibly large and confidence intervals that are wide and uninterpretable. These types of “overfitted” models should be avoided. • Logistic regression models can be fit using most standard statistical software.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

DOE OSTI2020

DPM: A deep learning PDE augmentation method with application to large-eddy simulation

A framework is introduced that leverages known physics to reduce overfitting in machine learning for scientific applications. The partial differential equation (PDE) that expresses the physics is augmented with a neural network that uses available data to learn a description of the corresponding unknown or unrepresented physics. Training within this combined system corrects for missing, unknown, or erroneously represented physics, including discretization errors associated with the PDE's numerical solution. For optimization of the network within the PDE, an adjoint PDE is solved to provide high-dimensional gradients, and a stochastic adjoint method (SAM) further accelerates training. Additionally, the approach is demonstrated for large-eddy simulation (LES) of turbulence. High-fidelity direct numerical simulations (DNS) of decaying isotropic turbulence provide the training data used to learn sub-filter-scale closures for the filtered Navier–Stokes equations. Out-of-sample comparisons show that the deep learning PDE method outperforms widely-used models, even for filter sizes so large that they become qualitatively incorrect. It also significantly outperforms the same neural network when a priori trained based on simple data mismatch, not accounting for the full PDE. Measures of discretization errors, which are well-known to be consequential in LES, point to the importance of the unified training formulation's design, which without modification corrects for them. For comparable accuracy, simulation runtime is significantly reduced. A relaxation of the typical discrete enforcement of the divergence-free constraint in the solver is also successful, instead allowing the DPM to approximately enforce incompressibility physics. Since the training loss function is not restricted to correspond directly to the closure to be learned, training can incorporate diverse data, including experimental data.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗