Search NASA⌕ Search

SEARCH · Search NASA

Results for “Neural Networks Research”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities

Although deep learning has made great progress in recent years, the exploding economic and environmental costs of training neural networks are becoming unsustainable. To address this problem, there has been a great deal of research on algorithmically-efficient deep learning, which seeks to reduce training costs not at the hardware or implementation level, but through changes in the semantics of the training program. In this paper, we present a structured and comprehensive overview of the research in this field. First, we formalize the algorithmic speedup problem, then we use fundamental building blocks of algorithmically efficient training to develop a taxonomy. Our taxonomy highlights commonalities of seemingly disparate methods and reveals current research gaps. Next, we present evaluation best practices to enable comprehensive, fair, and reliable comparisons of speedup techniques. To further aid research and applications, we discuss common bottlenecks in the training pipeline (illustrated via experiments) and offer taxonomic mitigation strategies for them. Finally, we highlight some unsolved research challenges and present promising future directions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A survey of techniques for optimizing transformer inference

Recent years have seen a phenomenal rise in the performance and applications of transformer neural networks. The family of transformer networks, including Bidirectional Encoder Representations from Transformer (BERT), Generative Pretrained Transformer (GPT) and Vision Transformer (ViT), have shown their effectiveness across Natural Language Processing (NLP) and Computer Vision (CV) domains. Transformer-based networks such as ChatGPT have impacted the lives of common men. However, the quest for high predictive performance has led to an exponential increase in transformers' memory and compute footprint. Researchers have proposed techniques to optimize transformer inference at all levels of abstraction. Further, this paper presents a comprehensive survey of techniques for optimizing the inference phase of transformer networks. We survey techniques such as knowledge distillation, pruning, quantization, neural architecture search and lightweight network design at the algorithmic level. We further review hardware-level optimization techniques and the design of novel hardware accelerators for transformers. We summarize the quantitative results on the number of parameters/FLOPs and the accuracy of several models/techniques to showcase the tradeoff exercised by them. We also outline future directions in this rapidly evolving field of research. We believe that this survey will educate both novice and seasoned researchers and also spark a plethora of research efforts in this field.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Harnessing large language models’ zero-shot and few-shot learning capabilities for regulatory research

Abstract Large language models (LLMs) are sophisticated AI-driven models trained on vast sources of natural language data. They are adept at generating responses that closely mimic human conversational patterns. One of the most notable examples is OpenAI's ChatGPT, which has been extensively used across diverse sectors. Despite their flexibility, a significant challenge arises as most users must transmit their data to the servers of companies operating these models. Utilizing ChatGPT or similar models online may inadvertently expose sensitive information to the risk of data breaches. Therefore, implementing LLMs that are open source and smaller in scale within a secure local network becomes a crucial step for organizations where ensuring data privacy and protection has the highest priority, such as regulatory agencies. As a feasibility evaluation, we implemented a series of open-source LLMs within a regulatory agency’s local network and assessed their performance on specific tasks involving extracting relevant clinical pharmacology information from regulatory drug labels. Our research shows that some models work well in the context of few- or zero-shot learning, achieving performance comparable, or even better than, neural network models that needed thousands of training samples. One of the models was selected to address a real-world issue of finding intrinsic factors that affect drugs' clinical exposure without any training or fine-tuning. In a dataset of over 700 000 sentences, the model showed a 78.5% accuracy rate. Our work pointed to the possibility of implementing open-source LLMs within a secure local network and using these models to perform various natural language processing tasks when large numbers of training examples are unavailable.

Biochemistry & Molecular Biology↗
DOE OSTI2025

Quantum optimal control of superconducting qubits based on machine-learning characterization

Implementing fast and high-fidelity quantum operations using open-loop quantum optimal control relies on having an accurate model of the quantum dynamics. Any deviations between this model and the complete dynamics of the device, such as the presence of spurious modes or pulse distortions, can degrade the performance of optimal controls in practice. Here, we propose an experimentally simple approach to realize optimal quantum controls tailored to the device parameters and environment while specifically characterizing this quantum system. Concretely, we use physics-inspired machine learning to infer an accurate model of the dynamics from experimentally available data and then optimize our experimental controls on this trained model. We show the power and feasibility of this approach by optimizing arbitrary single-qubit operations in detailed numerical simulations of a superconducting transmon qubit. Furthermore, we demonstrate that this framework produces an accurate description of the device dynamics under arbitrary controls, together with the precise pulses achieving arbitrary single-qubit gates with a high fidelity of ∼99.99%.

Artificial neural networks↗
DOE OSTI2024

Leveraging Hydropower Multi-Sensor Data for Inference and Age-Informed Modeling

Increased demand of operational flexibility such as faster ramp up/down in generation, and more frequent start/stops are putting hydropower plants and their associated components in unprecedented stress. Consequently, these plants are at the high risk of extended and more frequent outage to accommodate unscheduled, and unexpected maintenance. Therefore, hydropower plants are in critical need of data driven and age-informed analysis for their regular and unscheduled operation. Yet not all hydropower plants are exhaustively equipped with sensors and/or measurement streams for their respective components – demanding solutions on how to detect, identify, and locate the cause of any event from the unobservable. Idaho National Laboratory (INL) analyzed the anonymized measurements and event records from the Hydropower Research Institute (HRI) to address this issue, as part of the Water Power Technologies Office (WPTO) funded one year multi-lab project. First, we investigated how time series of multiple sensor measurements can be leveraged to identify an event “root cause” as well as to develop an inference (i.e., estimate the unobservable) problem. INL also investigated how individual hydropower components’ reaction or response times vary across the pre-event, during event, and post-event conditions – enabling the hydropower dynamic models to be age-informed. Finally, the impact of clustering multi-sensor time series on short-term vibration prediction is analyzed. INL will present key findings from these analyses and recommend next steps for stakeholder adoption.

13 HYDRO ENERGY↗
DOE OSTI2023

Invertible neural networks for real-time control of extrusion additive manufacturing

Material extrusion additive manufacturing (AM) has enabled an elegant fabrication pathway for a vast material library. Nonetheless, each material requires optimization of printing parameters generally determined through significant trial-and-error testing. To eliminate arduous, iteration-based optimization approaches, many researchers have used machine learning (ML) algorithms which provide opportunities for automated process optimization. Here, in this work, we demonstrate the use of an ML-driven approach for real-time material extrusion print-parameter optimization through in-situ monitoring of printed line geometry. To do this, we use deep invertible neural networks (INNs) which can solve both forward and inverse, or optimization, problems using a single network. By combining in-situ computer vision and deep INNs, the printing parameters can be autonomously optimized to print a target line width in 1.2 s. Furthermore, defects that occur during printing can be rapidly identified and corrected autonomously. The methods developed and presented in this work eliminate user-intensive, time-consuming, and iterative parameter discovery approaches that currently limit accelerated implementation of extrusion-based AM processes. Furthermore, the presented approach can be generalized to provide real-time monitoring and optimization pathways for increasingly complex AM environments.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Federated Quantum Machine Learning

Distributed training across several quantum computers could significantly improve the training time and if we could share the learned model, not the data, it could potentially improve the data privacy as the training would happen where the data is located. One of the potential schemes to achieve this property is the federated learning (FL), which consists of several clients or local nodes learning on their own data and a central node to aggregate the models collected from those local nodes. However, to the best of our knowledge, no work has been done in quantum machine learning (QML) in federation setting yet. In this work, we present the federated training on hybrid quantum-classical machine learning models although our framework could be generalized to pure quantum machine learning model. Specifically, we consider the quantum neural network (QNN) coupled with classical pre-trained convolutional model. Our distributed federated learning scheme demonstrated almost the same level of trained model accuracies and yet significantly faster distributed training. It demonstrates a promising future research direction for scaling and privacy aspects.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

A workflow for segmenting soil and plant X-ray computed tomography images with deep learning in Google’s Colaboratory

X-ray micro-computed tomography (X-ray μCT) has enabled the characterization of the properties and processes that take place in plants and soils at the micron scale. Despite the widespread use of this advanced technique, major limitations in both hardware and software limit the speed and accuracy of image processing and data analysis. Recent advances in machine learning, specifically the application of convolutional neural networks to image analysis, have enabled rapid and accurate segmentation of image data. Yet, challenges remain in applying convolutional neural networks to the analysis of environmentally and agriculturally relevant images. Specifically, there is a disconnect between the computer scientists and engineers, who build these AI/ML tools, and the potential end users in agricultural research, who may be unsure of how to apply these tools in their work. Additionally, the computing resources required for training and applying deep learning models are unique, more common to computer gaming systems or graphics design work, than to traditional computational systems. To navigate these challenges, we developed a modular workflow for applying convolutional neural networks to X-ray μCT images, using low-cost resources in Google’s Colaboratory web application. Here we present the results of the workflow, illustrating how parameters can be optimized to achieve best results using example scans from walnut leaves, almond flower buds, and a soil aggregate. We expect that this framework will accelerate the adoption and use of emerging deep learning techniques within the plant and soil sciences.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Next-Level Energy Management in Manufacturing: Facility-Level Energy Digital Twin Framework Based on Machine Learning and Automated Data Collection

This research introduces an energy prediction framework at the facility level supported by automated data collection and machine learning models. It investigates whether reducing the prediction time scale allows for applying more complex machine learning techniques and if those techniques improve the prediction accuracy. The primary advantages of this framework lie in its automation of the energy prediction process and its provision of real-time energy data suitable for use in energy dashboards or digital twins. A sitewide dataset was created by combining 15 min energy and daily production data of five shops—assembly, battery, body (electric), body (gas), and paint—from a globally recognized electric vehicle manufacturer. Various machine learning models were evaluated on daily, weekly, and monthly datasets, including, in increasingly complex order: naïve, simple linear regression, net regularized generalized linear regression, principal component regression, k-nearest neighbor, random forest, and Bayesian regularized neural network. Compared to the current state-of-the-art energy consumption prediction for the industrial facility level, this research investigates more complex models and smaller time intervals for higher accuracy. The findings revealed that the more complex monthly models require a minimum of a year and a half of data to operate, while weekly models demand a year of data to achieve improved accuracy. Daily models can operate with only six months of data but exhibit poor performance due to reduced prediction accuracy of production. Key challenges identified include access to reliable, high-quality energy and production data and the initial demand for human labor.

digital twin↗
DOE OSTI2023

Grey-box and ANN-based building models for multistep-ahead prediction of indoor temperature to implement model predictive control

Model-based predictive control (MPC) strategies for heating, ventilation, and air-conditioning (HVAC) systems present an opportunity to lower building energy consumption and operational costs. Such approaches rely on the development of a model to precisely forecast building thermal dynamics, such as room air temperature or heating/cooling rate, and make control-related decisions. The control-oriented modeling of building energy systems should be accurate in predicting indoor conditions and present low computational complexity. These features are the key challenge of implementing advanced control methods such as MPC. Extant studies on building modeling for MPC have focused on step-ahead forecasting techniques to forecast building thermal dynamics, while multistep-ahead forecasting is essential. Moreover, machine learning model suitable in case of the domain-based engineering expertise are also not available. To this aim, we perform a comparative analysis of the grey-box model based on a resistance-capacitance (RC) thermal network and a machine learning model composed of an artificial neural network (ANN) for multistep-ahead prediction of building thermal dynamics using current and historical data. Actual experimental data obtained from the Flexible Research Platform (FRP) in Oak Ridge National Laboratory (US) are used for estimation and validation purposes. The average root mean squared error (RMSE) of the grey-box and ANN models are 0.89 °C and 1.02°C, respectively. Finally, the results indicate that the grey-box model outperforms the ANN model in the considered validation periods in terms of accuracy and prediction stability.

42 ENGINEERING↗
DOE OSTI2023

ZeoNet: 3D convolutional neural networks for predicting adsorption in nanoporous zeolites

Zeolites are one of the most widely used materials in the chemical industry due to their nanometer-sized pores that can adsorb and react upon molecules selectively. With hundreds of known framework topologies and hundreds of thousands of computationally predicted structures, the ability to rapidly predict zeolite performance allows researchers to prioritize their efforts on the most promising structures for a given application. Although the accuracy of forcefield-based atomistic simulations has advanced significantly in the past two decades, these simulations can be computationally expensive, especially for long-chain, complex molecules. Here, we present ZeoNet, a representation learning framework using convolutional neural networks (ConvNets) and 3D volumetric representations for predicting adsorption in zeolites. ZeoNet was trained on the task of predicting Henry's constants for adsorption, k H , of n-octadecane in more than 330 000 known and predicted zeolite materials. Employing a 3D grid based on the distances to solvent-accessible surfaces, a volumetric representation that can be generated efficiently, the best-performing ZeoNet achieved a correlation coefficient r 2 = 0.977 and a mean-squared error MSE = 3.8 in ln k H , which corresponds to an error of 9.3 kJ mol -1 in adsorption free energy. In comparison, a model based on hand-designed geometric features has values of r 2 = 0.783 and MSE = 35.7. ZeoNet is also relatively efficient and can process ≈8 structures per second on an Nvidia RTX 2080TI GPU, orders of magnitude faster than forcefield-based simulations. A systematic analysis was conducted to investigate how the choice of ConvNet architectures, the linear dimension (L) and spatial resolution (Δd) of the distance grids, batch size, optimizer, and learning rate impact the model performance. We found that ConvNets based on the ResNet architecture offer the best tradeoff between expressiveness and efficiency. The performance for all models reaches a plateau at L = 30–45 Å and depends less sensitively on grid resolution, with a small benefit around Δd = 0.30–0.45 Å. Finally, saliency maps were visualized to identify which regions of the materials contributed the most to model predictions. It was found, interestingly, that the predictions are driven primarily by the accessible pore volume rather than the region occupied by the framework atoms.

36 MATERIALS SCIENCE↗
DOE OSTI2024

A comparative study on deep learning models for condition monitoring of advanced reactor piping systems

Advanced nuclear reactors offer innovative applications due to their portability, reliability, resiliency, and high capacity factors. To operate them on a wider scale, reducing maintenance life-cycle costs while ensuring their integrity is essential. Autonomous operations in advanced nuclear reactors using augmented Digital Twin (DT) technology can serve as a cost-effective solution by increasing awareness about the system’s health. A key component of nuclear DT frameworks is the condition monitoring of safety systems, such as piping-equipment systems, which involves acquiring and monitoring the plant’s sensor data. Here, this research proposes a condition monitoring methodology utilizing deep learning algorithms, such as multilayer perceptions (MLP) and convolutional neural networks (CNNs), to detect degradation and its severity in nuclear piping-equipment systems. Sensor signals are processed to obtain the power spectral density and the Short-Time Fourier transform, and feature extraction methodologies are proposed to develop degradation-sensitive data repositories. The performance of MLP, one-dimensional (1D) CNN, and 2D CNN within the proposed condition monitoring framework is compared using a finite element model of a 3D piping system subjected to seismic loads as the application case study. Various approaches, such as dropout, k-Fold validation, regularization, and early stopping of training the network, are investigated to avoid overfitting the models to the input sensor data. The predictive capability and computational capacity of the deep learning algorithms are also compared to detect degradation in the Z-pipe system of the Experimental Breeder Reactor II (EBRII). The Z-pipe system is subjected to harmonic excitations that represent normal operating loads, such as pump-induced vibrations. The findings of the study indicate that the proposed artificial intelligence (AI)-driven condition monitoring framework demonstrates superior prediction accuracies with a 2D CNN, whereas the MLP exhibits higher computational efficiency.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2021

A Benchmark to Test Generalization Capabilities of Deep Learning Methods to Classify Severe Convective Storms in a Changing Climate

Abstract This is a test case study assessing the ability of deep learning methods to generalize to a future climate (end of 21st century) when trained to classify thunderstorms in model output representative of the present‐day climate. A convolutional neural network (CNN) was trained to classify strongly rotating thunderstorms from a current climate created using the Weather Research and Forecasting model at high‐resolution, then evaluated against thunderstorms from a future climate and found to perform with skill and comparatively in both climates. Despite training with labels derived from a threshold value of a severe thunderstorm diagnostic (updraft helicity), which was not used as an input attribute, the CNN learned physical characteristics of organized convection and environments that are not captured by the diagnostic heuristic. Physical features were not prescribed but rather learned from the data, such as the importance of dry air at mid‐levels for intense thunderstorm development when low‐level moisture is present (i.e., convective available potential energy). Explanation techniques also revealed that thunderstorms classified as strongly rotating are associated with learned rotation signatures. Results show that the creation of synthetic data with ground truth is a viable alternative to human‐labeled data and that a CNN is able to generalize a target using learned features that would be difficult to encode due to spatial complexity. Most importantly, results from this study show that deep learning is capable of generalizing to future climate extremes and can exhibit out‐of‐sample robustness with hyperparameter tuning in certain applications.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Supervised learning and the finite-temperature string method for computing committor functions and reaction rates

A central object in the computational studies of rare events is the committor function. Though costly to compute, the committor function encodes complete mechanistic information of the processes involving rare events, including reaction rates and transition-state ensembles. Under the framework of transition path theory, Rotskoff et al. [Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, Proceedings of Machine Learning Research (PLMR, 2022), Vol. 145, pp. 757–780] proposes an algorithm where a feedback loop couples a neural network that models the committor function with importance sampling, mainly umbrella sampling, which collects data needed for adaptive training. Here, in this work, we show additional modifications are needed to improve the accuracy of the algorithm. The first modification adds elements of supervised learning, which allows the neural network to improve its prediction by fitting to sample-mean estimates of committor values obtained from short molecular dynamics trajectories. The second modification replaces the committor-based umbrella sampling with the finite-temperature string (FTS) method, which enables homogeneous sampling in regions where transition pathways are located. We test our modifications on low-dimensional systems with non-convex potential energy where reference solutions can be found via analytical or finite element methods, and show how combining supervised learning and the FTS method yields accurate computation of committor functions and reaction rates. We also provide an error analysis for algorithms that use the FTS method, using which reaction rates can be accurately estimated during training with a small number of samples. The methods are then applied to a molecular system in which no reference solution is known, where accurate computations of committor functions and reaction rates can still be obtained.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Experimental Observations of the Topology of Convolutional Neural Network Activations

Topological data analysis (TDA) is a branch of computational mathematics, bridging algebraic topology and data science, that provides compact, noise-robust representations of complex structures. Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture resulting in high-dimensional, difficult to interpret internal representations of input data. As DNNs become more ubiquitous across multiple sectors of our society, there is increasing recognition that mathematical methods are needed to aid analysts, researchers, and practitioners in understanding and interpreting how these models' internal representations relate to the final classification. In this paper we apply cutting edge techniques from TDA with the goal of gaining insight towards interpretability of convolutional neural networks used for image classification. We use two common TDA approaches to explore several methods for modeling hidden layer activations as high-dimensional point clouds, and provide experimental evidence that these point clouds capture valuable structural information about the model's process. First, we demonstrate that a distance metric based on persistent homology can be used to quantify meaningful differences between layers and discuss these distances in the broader context of existing representational similarity metrics for neural network interpretability. Second, we show that a mapper graph can provide semantic insight as to how these models organize hierarchical class knowledge at each layer. These observations demonstrate that TDA is a useful tool to help deep learning practitioners unlock the hidden structures of their models.

topological data analysis, deep learning↗
DOE OSTI2025

Spatio-spectral optical fission in time-varying subwavelength layers

Transparent conducting oxides are highly doped semiconductors that exhibit favourable optical features compared with metals, including reduced material losses, tuneable electronic and optical properties, and enhanced damage thresholds. Recently, the photonic community has renewed its attention towards these materials, recognizing their remarkable nonlinear optical properties in the near-infrared spectrum. The exceptionally large and ultrafast change in the refractive index, which can be optically induced in these compounds, extends beyond the boundaries of conventional perturbative analysis and makes this class of materials the closest approximation to a time-varying system. Here we report the spatio-spectral fission of an ultrafast pulse trespassing a thin film of aluminium zinc oxide with a non-stationary refractive index. By applying phase conservation to this time-varying layer, our model can account for both space and time refraction and explain, in quantitative terms, the spatial separation of both spectrum and energy. Our findings represent an example of extreme nonlinear phenomena on subwavelength propagation distances, which provides new insights into transparent conducting oxides’ transient optical properties. This can be critical for the ongoing research on photonic time crystals, on-chip generation of non-classical states of light, integrated optical neural networks, ultrafast beam steering and frequency-division multiplexing.

integrated optics↗
DOE OSTI2023

Journey Over Destination: Differentiable Sensor Placement Enhances Generalization [Poster]

The challenge of reconstructing spatial fields that change over time from limited sensor data has been a focal point for many research studies. Various machine learning methods have been used in attempts to address this complex issue, including convolutional neural networks. All of the proposed methods share a common requirement that the user needs to manually determine the sensor positions. This requirement remains a limiting factor in the ongoing quest for efficient learning and accurate field reconstruction. This study aims to present a method that enables a model to optimize sensor positions via backpropagation, thereby facilitating the model’s exploration of the spatial domain and enhancing sensor positioning effectively. Indexing naturally incorporates discrete decisions. This operation is nondifferentiable which is a requirement for the application of gradient-based optimization methods. We showcased its effectiveness by training an attention-based neural network, which achieved top-tier performance on two separate datasets. To our knowledge, this represents the first fully end-to-end differentiable workflow for enhancing sensor placement within a neural network model.

58 GEOSCIENCES↗
DOE OSTI2020

Real-time Intrusion Detection for High-bandwidth Research Networks using Unsupervised Deep Learning

Nowadays, data generated by large-scale scientific experiments are on the scale of petabytes per month. These data are transferred through dedicated high-bandwidth networks across distributed collaboration sites for processing, storage, and analysis. Like general purpose networks, research networks experience attacks. However, monitoring traffic in such high-bandwidth network environments is challenging with conventional deep packet inspection (DPI)- based network intrusion detection systems (NIDS). Recently, machine learning (ML) algorithms have shown potential in detecting anomalous patterns with better generalizability and faster speed. Compared to the general purpose networks, traffic movement in research network is less volatile and more likely to benefit from the ML-based anomaly analysis. We present ThunderSecure, an efficient AI-powered system for detecting anomalies in 100G research networks. ThunderSecure implements a high-throughput packet processing infrastructure using multi-cores and GPUs. It takes time-series statistics from a variety of network measurements and uses 2D Convolutional Neural Network (CNN) to explore both spatial and temporal dependencies in network data stream. Patterns of science data traffic are learned with a one-class neural network, which blends an Adversarial Autoencoder (AAE) with Gaussian mixture density estimation. Testing traffic flows exhibiting significant deviations from the learned baseline of normality are marked as anomalies. We trained ThunderSecure on hundreds of billions of science data packets recorded from a 100G research network at Fermi National Accelerator Laboratory. The detection performance was evaluated on traffic captured from the same research network days and weeks after the training with different types of attack flows injected. To the best of our knowledge, this is the first ML work in the area of network anomaly detection that has been validated on such extreme scale datasets. Results show that ThunderSecure can recognize science data traffic that are captured long after the training and make nearly certain detection on those with anomalous flows injected, even in the case when the anomaly-to-normal mixing ratio is 0.1%.

Gong, Qian↗