Search NASA⌕ Search

SEARCH · Search NASA

Results for “model accuracy”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Combining pairwise structural similarity and deep learning interface contact prediction to estimate protein complex model accuracy in CASP15

Abstract Estimating the accuracy of quaternary structural models of protein complexes and assemblies (EMA) is important for predicting quaternary structures and applying them to studying protein function and interaction. The pairwise similarity between structural models is proven useful for estimating the quality of protein tertiary structural models, but it has been rarely applied to predicting the quality of quaternary structural models. Moreover, the pairwise similarity approach often fails when many structural models are of low quality and similar to each other. To address the gap, we developed a hybrid method (MULTICOM_qa) combining a pairwise similarity score (PSS) and an interface contact probability score (ICPS) based on the deep learning inter‐chain contact prediction for estimating protein complex model accuracy. It blindly participated in the 15th Critical Assessment of Techniques for Protein Structure Prediction (CASP15) in 2022 and performed very well in estimating the global structure accuracy of assembly models. The average per‐target correlation coefficient between the model quality scores predicted by MULTICOM_qa and the true quality scores of the models of CASP15 assembly targets is 0.66. The average per‐target ranking loss in using the predicted quality scores to rank the models is 0.14. It was able to select good models for most targets. Moreover, several key factors (i.e., target difficulty, model sampling difficulty, skewness of model quality, and similarity between good/bad models) for EMA are identified and analyzed. The results demonstrate that combining the multi‐model method (PSS) with the complementary single‐model method (ICPS) is a promising approach to EMA.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Prototype-Wise Sensitivity Analysis of Urban Building Energy Simulation Surrogate Modeling Accuracy

Urban Building Energy Modeling (UBEM) is an important reference for urban energy-related policymaking. Because of the significant impact of urban microclimates on the energy simulation, UBEM requires simulations of many microclimate-prototype pairs. Surrogate modeling is commonly used to reduce the cost of simulation computations. In UBEM surrogate modeling, it is important to determine the percentage of microclimates related to a prototype used for generating surrogate model training data. This study analyzes the prototype-wise variations and sensitivities of surrogate model estimation accuracy to the microclimate sampling ratios. The results of the study can help determine the number of simulations used for generating surrogate modeling data, avoid redundant simulations, and reduce the computational cost for UBEM surrogate modeling and its time.

Pan, Xiyu↗
DOE OSTI2021

Protein model accuracy estimation empowered by deep learning and inter-residue distance prediction in CASP14

Abstract The inter-residue contact prediction and deep learning showed the promise to improve the estimation of protein model accuracy (EMA) in the 13th Critical Assessment of Protein Structure Prediction (CASP13). To further leverage the improved inter-residue distance predictions to enhance EMA, during the 2020 CASP14 experiment, we integrated several new inter-residue distance features with the existing model quality assessment features in several deep learning methods to predict the quality of protein structural models. According to the evaluation of performance in selecting the best model from the models of CASP14 targets, our three multi-model predictors of estimating model accuracy (MULTICOM-CONSTRUCT, MULTICOM-AI, and MULTICOM-CLUSTER) achieve the averaged loss of 0.073, 0.079, and 0.081, respectively, in terms of the global distance test score (GDT-TS). The three methods are ranked first, second, and third out of all 68 CASP14 predictors. MULTICOM-DEEP, the single-model predictor of estimating model accuracy (EMA), is ranked within top 10 among all the single-model EMA methods according to GDT-TS score loss. The results demonstrate that inter-residue distance features are valuable inputs for deep learning to predict the quality of protein structural models. However, larger training datasets and better ways of leveraging inter-residue distance information are needed to fully explore its potentials.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Evaluating E. coli genome‐scale metabolic model accuracy with high‐throughput mutant fitness data

Abstract The Escherichia coli genome‐scale metabolic model (GEM) is an exemplar systems biology model for the simulation of cellular metabolism. Experimental validation of model predictions is essential to pinpoint uncertainty and ensure continued development of accurate models. Here, we quantified the accuracy of four subsequent E. coli GEMs using published mutant fitness data across thousands of genes and 25 different carbon sources. This evaluation demonstrated the utility of the area under a precision–recall curve relative to alternative accuracy metrics. An analysis of errors in the latest (iML1515) model identified several vitamins/cofactors that are likely available to mutants despite being absent from the experimental growth medium and highlighted isoenzyme gene‐protein‐reaction mapping as a key source of inaccurate predictions. A machine learning approach further identified metabolic fluxes through hydrogen ion exchange and specific central metabolism branch points as important determinants of model accuracy. This work outlines improved practices for the assessment of GEM accuracy with high‐throughput mutant fitness data and highlights promising areas for future model refinement in E. coli and beyond.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

DISTEMA: distance map-based estimation of single protein model accuracy with attentive 2D convolutional neural network

Abstract Background Estimation of the accuracy (quality) of protein structural models is important for both prediction and use of protein structural models. Deep learning methods have been used to integrate protein structure features to predict the quality of protein models. Inter-residue distances are key information for predicting protein’s tertiary structures and therefore have good potentials to predict the quality of protein structural models. However, few methods have been developed to fully take advantage of predicted inter-residue distance maps to estimate the accuracy of a single protein structural model. Result We developed an attentive 2D convolutional neural network (CNN) with channel-wise attention to take only a raw difference map between the inter-residue distance map calculated from a single protein model and the distance map predicted from the protein sequence as input to predict the quality of the model. The network comprises multiple convolutional layers, batch normalization layers, dense layers, and Squeeze-and-Excitation blocks with attention to automatically extract features relevant to protein model quality from the raw input without using any expert-curated features. We evaluated DISTEMA’s capability of selecting the best models for CASP13 targets in terms of ranking loss of GDT-TS score. The ranking loss of DISTEMA is 0.079, lower than several state-of-the-art single-model quality assessment methods. Conclusion This work demonstrates that using raw inter-residue distance information with deep learning can predict the quality of protein structural models reasonably well. DISTEMA is freely at https://github.com/jianlin-cheng/DISTEMA

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Evaluating Property Model Accuracy for Cost Optimization of Desalination Technologies

Conference presentation conveying work conducted on property modeling and implications on cost optimization of seawater desalination technology. The authors assess the accuracy of four different property modeling approaches to represent thermodynamic properties of seawater. These properties are used with process flowsheets that include either reverse osmosis or mechanical vapor compression. The accuracy of property estimation is related to deviations in process design and levelized cost.

Sakhai, Savannah↗
DOE OSTI2025

Refining NEAMS MARVEL Reactor Model Accuracy

This report outlines the progress of Idaho National Laboratory (INL) in improving the fidelity of the multiphysics model of the Microreactor Applications Research Validation and Evaluation (MARVEL) reactor developed in FY24. The multiphysics model firstly developed in FY24 leverages three single-physics models coupled via the MOOSE’s MultiApps and Transfer systems. The first single-physics model, that functions as the driver application, leverages Griffin to model the neutron transport in the core through the discontinuous finite element (DFEM) discrete ordinates solver (SN). The second single-physics model uses BISON to compute the solid temperature in the reactor. Finally, the System Analysis Module (SAM) was used to model the flow of the sodium-potassium eutectic in the primary loop. In this report we detail the following improvement and enhancements: • An updated mesh leveraging the latest feature in the reactor module (see Section 3.1.).

42 - ENGINEERING↗
DOE OSTI2024

Effects of machine learning errors on human decision-making: manipulations of model accuracy, error types, and error importance

Abstract This study addressed the cognitive impacts of providing correct and incorrect machine learning (ML) outputs in support of an object detection task. The study consisted of five experiments that manipulated the accuracy and importance of mock ML outputs. In each of the experiments, participants were given the T and L task with T-shaped targets and L-shaped distractors. They were tasked with categorizing each image as target present or target absent. In Experiment 1, they performed this task without the aid of ML outputs. In Experiments 2–5, they were shown images with bounding boxes, representing the output of an ML model. The outputs could be correct (hits and correct rejections), or they could be erroneous (false alarms and misses). Experiment 2 manipulated the overall accuracy of these mock ML outputs. Experiment 3 manipulated the proportion of different types of errors. Experiments 4 and 5 manipulated the importance of specific types of stimuli or model errors, as well as the framing of the task in terms of human or model performance. These experiments showed that model misses were consistently harder for participants to detect than model false alarms. In general, as the model’s performance increased, human performance increased as well, but in many cases the participants were more likely to overlook model errors when the model had high accuracy overall. Warning participants to be on the lookout for specific types of model errors had very little impact on their performance. Overall, our results emphasize the importance of considering human cognition when determining what level of model performance and types of model errors are acceptable for a given task.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Lens Model Accuracy in the Expected LSST Lensed AGN Sample

Strong gravitational lensing of active galactic nuclei (AGN) enables measurements of cosmological parameters through time-delay cosmography (TDC). With data from the upcoming LSST survey, we anticipate using a sample of O(1000) lensed AGN for TDC. To prepare for this dataset and enable this measurement, we construct and analyze a realistic mock sample of 1300 systems drawn from the OM10 (Oguri & Marshall 2010) catalog of simulated lenses with AGN sources at $z<3.1$ in order to test a key aspect of the analysis pipeline, that of the lens modeling. We realize the lenses as power law elliptical mass distributions and simulate 5-year LSST i-band coadd images. From every image, we infer the lens mass model parameters using neural posterior estimation (NPE). Focusing on the key model parameters, $θ_E$ (the Einstein Radius) and $γ_{lens}$ (the projected mass density profile slope), with consistent mass-light ellipticity correlations in test and training data, we recover $θ_E$ with less than 1% bias per lens, 6.5% precision per lens and $γ_{lens}$ with less than 3% bias per lens, 8% precision per lens. We find that lens light subtraction prior to modeling is only useful when applied to data sampled from the training prior. If emulated deconvolution is applied to the data prior to modeling, precision improves across all parameters by a factor of 2. Finally, we combine the inferred lens mass models using Bayesian Hierarchical Inference to recover the global properties of the lens sample with less than 1% bias.

Venkatraman, Padmavathi [Illinois U., Urbana; KIPA↗
DOE OSTI2021

Selecting durable building envelope systems with machine learning assisted hygrothermal simulations database

Hygrothermal simulations provide insight into the energy performance and moisture durability of building envelope components under dynamic conditions. The inputs required for hygrothermal simulations are extensive, and carrying out simulations and analyses requires expert knowledge. An expert system, the Building Science Advisor (BSA), has been developed to predict the performance and select the energy-efficient and durable building envelope systems for different climates. The BSA consists of decision rules based on expert opinions and thousands of parametric simulation results for selected wall systems. The number of potential wall systems results in millions, too many to simulate all of them. We present how machine learning can help predict durability data, such as mold growth, while minimizing the number of simulations needed to run. The simulation results are used for training and validation of machine learning tools for predicting wall durability. We tested Artificial Neural Network (ANN) and Gradient Boosted Decision Trees (GBDT) for their applicability and model accuracy. Models developed with both methods showed adequate prediction performance (root mean square error of 0.195 and 0.209, respectively). Finally, we introduce how the information supports guidance for envelope design via an easy-to-use web-based tool that does not require the end-user to run hygrothermal simulations.

Salonvaara, Mikael↗
DOE OSTI2021

Interpreting Write Performance of Supercomputer I/O Systems with Regression Models

This work seeks to advance the state of the art in HPC I/O performance analysis and interpretation. In particular, we demonstrate effective techniques to: (1) model output performance in the presence of I/O interference from production loads; (2) build features from write patterns and key parameters of the system architecture and configurations; (3) employ suitable machine learning algorithms to improve model accuracy. We train models with five popular regression algorithms and conduct experiments on two distinct production HPC platforms. We find that the lasso and random forest models predict output performance with high accuracy on both of the target systems. We also explore use of the models to guide adaptation in I/O middleware systems, and show potential for improvements of at least 15% from model-guided adaptation on 70% of samples, and improvements up to 10× on some samples for both of the target systems.

Xie, Bing↗
DOE OSTI2021

Active learning strategy for high fidelity short-term data-driven building energy forecasting

The quality of a data-driven model is heavily dependent on the quality of data. Data from building operation often have data bias problems, which means that the data sample is collected in a way that some members of the intended data population are less likely to be included than others. Data-driven energy forecasting models built on such data hence are biased and could lead to large forecasting errors. Active learning—an effective method to defying data bias—is rarely studied or applied in the area of data-driven building energy forecasting modeling. This paper attempts to fill this gap and explores the application of active learning in data-driven building energy forecasting. The developed strategy in this paper efficiently generate informative training data within a time budget and uses block design to passively consider weather disturbances. The developed active learning strategy is applied and evaluated in both virtual and real-building testbeds against traditional data-driven methods. Via these virtual and real-building evaluation cases, we have demonstrated that the data bias problem typically exists in building operation data is resolved by applying the developed active learning strategy. Furthermore, building energy forecasting models trained from data generated from the active learning strategy have shown improved performances in both model accuracy and model extendibility perspectives. The effectiveness of the block design module is also validated to effectively consider the impact of weather conditions on active learning design.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

High-resolution hourly surrogate modeling framework for physics-based large-scale building stock modeling

Surrogate modeling can play a key role in reducing high computational burdens for large-scale physics-based modeling and uncertainty quantification. With the rapid development of large-scale building stock energy modeling, surrogate modeling has also begun to be widely applied in this field; however, most existing surrogate models lack hourly time resolution for regional-scale modeling, which is essential for understanding building demand profiles and grid impacts. Further, there is generally a lack of necessary data and feature engineering frameworks specific to building modeling for efficiently managing large datasets and complex computations. Furthermore, this paper proposes a modeling framework for large-scale (city-/region-scale), high-resolution, high-fidelity surrogate building stock energy models. Our developed framework consists of six modules: (1) building stock energy modeling (ComStock TM and ResStock TM ), (2) data engineering for large simulation data, (3) high performance computing workflow, (4) feature engineering, (5) machine learning model development, and (6) model performance evaluation. Two case studies apply the developed framework in both residential and commercial building stock analysis to demonstrate its computational efficiency and surrogate modeling accuracies. Results show that surrogate models, when efficiently trained using the HPC workflow module, reach a high level of modeling accuracy for two case studies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Improving scalability of parallel CNN training by adaptively adjusting parameter update frequency

Synchronous SGD with data parallelism, the most popular parallelization strategy for CNN training, suffers from the expensive communication cost of averaging gradients among all workers. The iterative parameter updates of SGD cause frequent communications and it becomes the performance bottleneck. In this paper, we propose a lazy parameter update algorithm that adaptively adjusts the parameter update frequency to address the expensive communication cost issue. Our algorithm accumulates the gradients if the difference of the accumulated gradients and the latest gradients is sufficiently small. Here, the less frequent parameter updates reduce the per-iteration communication cost while maintaining the model accuracy. Our experimental results demonstrate that the lazy update method remarkably improves the scalability while maintaining the model accuracy. For ResNet50 training on ImageNet, the proposed algorithm achieves a significantly higher speedup (739.6 on 2048 Cori KNL nodes) as compared to the vanilla synchronous SGD (276.6) while the model accuracy is almost not affected (<0.2% difference).

97 MATHEMATICS AND COMPUTING↗