Search NASA⌕ Search

SEARCH · Search NASA

Results for “Shallow neural networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Variation-Resilient FeFET-Based In-Memory Computing Leveraging Probabilistic Deep Learning

Reliability issues stemming from device level nonidealities of nonvolatile emerging technologies like ferroelectric field-effect transistors (FeFETs), especially at scaled dimensions, cause substantial degradation in the accuracy of in-memory crossbar-based AI systems. Here, in this work, we present a variation-aware design technique to characterize the device level variations and to mitigate their impact on hardware accuracy employing a Bayesian neural network (BNN) approach. An effective conductance variation model is derived from the experimental measurements of cycle-to-cycle (C2C) and device-to-device (D2D) variations performed on FeFET devices fabricated using 28 nm high-k metal gate technology. The variations were found to be a function of different conductance states within the given programming range, which sharply contrasts earlier efforts where a fixed variation dispersion was considered for all conductance values. Such variation characteristics formulated for three different device sizes at different read voltages were provided as prior variation information to the BNN to yield a more exact and reliable inference. Near-ideal accuracy for shallow networks (MLP5 and LeNet models) on the MNIST dataset and limited accuracy decline by ~3.8%–16.1% for deeper AlexNet models on CIFAR10 dataset under a wide range of variations corresponding to different device sizes and read voltages, demonstrates the efficacy of our proposed device-algorithm co-design technique.

97 MATHEMATICS AND COMPUTING↗

Facial Named Entity Recognition by Attention-Based Graph Convolutional Neural Network

In the realm of facial recognition and analysis, the ability to accurately cluster large datasets of facial images stands as a cornerstone for various applications, ranging from security surveillance to user biometric identification. This project evolves a novel approach to facial data clustering by embedding facial images into a high-dimensional vector space using an advanced embedding model trained on separate data and assumes a graph-like structure on the high-dimensional vectors. We find our method works significantly better than common shallow methods.

97 MATHEMATICS AND COMPUTING↗

Pre-trained network-based transfer learning: A small-sample machine learning approach to nuclear power plant classification problem

Some research topics belonging to classification problems in the nuclear industry, such as fault diagnosis and accident identification, can be solved by feature extraction and subsequent application of statistical machine learning classifiers. Recently, deep neural network-based methods with automatic feature extraction and high accuracy have gained wide attention. They usually require large-scale training data, however, plant fault or accident data are scarce or difficult to obtain. Here this paper proposes a convolutional network (CNN)-based transfer learning method to solve this problem. The network's shallow layer is derived from a pre-trained CNN based on the ImageNet database to automatically extract features, and the deep layer is customized to match the classification problem. Data in non-image formats are converted to image formats and subsequently used to train the network. Case studies of rotating machines fault diagnosis show that the proposed method requires only limited training data to achieve high accuracy.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Modeling Multi-View Impedance-Based Cross-Geometry SOH Estimator for Li-ion Batteries

Abstract: Accurately estimating battery’s State of Health (SOH) remains challenging when models must generalize across cell designs and operating conditions. Most Electrochemical Impedance Spectroscopy (EIS)-based approaches either (i) hand-engineer a few Nyquist-plot features for shallow models—fast but does not generalize across geometries—or (ii) learn directly from Nyquist plots with deep networks, which removes manual feature extraction, yet still limited to a single plot type. As a result, cross-geometry robustness and deployability on constrained Internet of Things (IoT) devices remain open problems. We propose a compact Convolutional Neural Network (CNN) (∼ 10k parameters) that takes multi-representation EIS inputs—Nyquist (real/imaginary) and phase–magnitude (|Z|/ϕ) stacked as four channels, so the model can learn complementary degradation signatures while remaining small enough for fast inference. We build a dataset from cyclic aging of two geometries (LG INR18650MJ1 cylindrical cells and LIR2032 coin cells), acquire EIS every ten cycles from 10 kHz to 10 mHz (10 points/decade), and evaluate with leave-one-cell-out testing strategy. We further study fusion vs. single-representation inputs and assess feasibility for on-device deployment (e.g., NVIDIA Jetson device). The results show that training on multiple EIS representations improves SOH estimation accuracy and cross-geometry generalization compared to single-representation models, which uses only Nyquist or phase–magnitude plots. This design targets accurate, generalizable SOH prediction without manual feature engineering while enabling practical real-time use.

Bakr, Ahmed [The University of Alabama (UA)]↗

Triangle Method for Dense ReLU Layers [SWR-25-72]

This software is an implementation of the methods for initializing and training neural networks to be more efficient per parameter, described more fully below and in the related publication: In theory, depth should make a ReLU network EXPONENTIALLY more efficient by enabling it to produce an exponential number of piecewise linear sections in its output. This reasoning is largely based on the work of mathematicians that have hand-constructed networks that make good use of depth. In practice however, even very deep ReLU networks that have been randomly initialized will behave identically to their shallow counterparts - missing an entire exponential dimension of efficiency. The triangle method is a first attempt at realizing the exponential potential of deep networks. Instead of randomly setting weights, we force pairs of neurons in each layer learn to build triangles (i.e. functions from [0,1] -> [0,1] that look like triangles). This is a very efficient pattern for generating lots of linear pieces because composing two triangular functions doubles the number of pieces with each composition. The triangle method is more than just a different initialization, it is a new paradigm of training. Instead of making direct updates to the matrix weights, we do an extra step of backpropagation to collect the derivatives of the loss function with respect to the shapes of the triangles, training them to tilt left or right. This process essentially holds the networks hand throughout the loss landscape and forces it to always use depth effectively by producing triangular shapes internally. This can produce several orders of magnitude of improvement on convex one-dimensional regression problems. Much more theoretical work is needed to realize its full potential beyond this context, but the implementation in this repository will still work in arbitrary numbers of dimensions. The file Triangle_Method.py is a generalized form of the method that will build each neuron its own custom 1-d convex activation function (with exponential efficiency). Example usage on one dimensional problems can be found in Example_Usage.ipynb and an example of using this in a real neural network can be found in Example_VGG16_CIFAR10.ipynb.

Milkert, Max [National Renewable Energy Laboratory↗

Inferring safety critical events from vehicle kinematics in naturalistic driving environment: Application of deep learning Algorithms

Advances in sensing technology has enabled the collection of countless terabytes of second-by-second kinematics data. Such data provides opportunities for real-time monitoring of driving behavior and identification of safety critical events (SCEs) including crashes and near crashes. The concept of volatility is relevant in this context, which identifies instability and erratic variations in driving behavior prior to involvement in SCEs. This study utilized vehicle kinematics from a large-scale naturalistic driving data to develop a deep learning approach based on 1D convolutional neural networks (CNN) for inferring SCEs. The data are unique in the sense that such accurate pre-crash data at high fidelity are not available in traditional crash repositories. This study contributes to the literature by providing a first attempt at predicting responses to SCEs by developing deep learning-based CNN architectures using novel driving volatility based kinematic thresholds for a sample of 9553 events. The key contribution lies in developing a volatility-based CNN input layout that is acceptable to CNN schemes and represents the motion kinematics such as speed, acceleration and volatility measures. Several 1D-CNN architectures were developed using layers, numbers of convolutions, layer patterns, and kernels. Shallow and deep architectures were tested, revealing higher accuracy of shallow architectures in detecting SCEs. The optimal number of epochs were identified using an early stopping method while the CNN performance was improved by increasing the number of epochs. The ensemble CNN had the highest predictive accuracy of 95.6% for detection of crashes and near crashes, which was 2.5% higher than the optimal CNN using 20% hold out test data. The ensemble CNN also outperformed classical machine learning models and model performance reported in past studies on detection of SCEs. Finally, these results have implications for identification of safety hotspots and providing real-time alerts and warnings in connected and highly automated vehicle environment including society of automotive engineers levels 3–5.

42 ENGINEERING↗

Do Machine Learning Approaches Offer Skill Improvement for Short-Term Forecasting of Wind Gust Occurrence and Magnitude?

Abstract Wind gusts, and in particular intense gusts, are societally relevant but extremely challenging to forecast. This study systematically assesses the skill enhancement that can be achieved using artificial neural networks (ANNs) for forecasting of wind gust occurrence and magnitude. Geophysical predictors from the ERA5 reanalysis are used in conjunction with an autoregressive term in regression and ANN models with different predictors, and varying model complexity. Models are derived and assessed for the warm (April–September) and cold (October–March) seasons for three high passenger volume airports in the United States. Model uncertainty is assessed by deriving models for 1000 different randomly selected training (70%) and testing (30%) subsets. Gust prediction fidelity in independent test samples is critically dependent on inclusion of an autoregressive term. Gust occurrence probabilities derived using five-layer ANNs exhibit consistently higher fidelity than those from regression models and shallower ANNs. Inclusion of the autoregressive term and increasing the number of hidden layers in ANNs from 1 to 5 also improve the model performance for gust magnitudes (lower RMSE, increased correlation, and model standard deviations that more closely approximate observed values). Deeper ANNs (e.g., 20 hidden layers) exhibit higher skill in forecasting strong (17–25.7 m s −1 ) and damaging (≥25.7 m s −1 ) wind gusts. However, such deep networks exhibit evidence of overfitting and still substantially underestimate (by 50%) the frequency of strong and damaging wind gusts at the three airports considered herein. Significance Statement Improved short-term forecasting of wind gusts will enhance aviation safety and logistics and may offer other societal benefits. Here we present a rigorous investigation of the relative skill of models of wind gust occurrence and magnitude that employ different statistical methods. It is shown that artificial neural networks (ANNs) offer considerable skill enhancement over regression methods, particularly for strong and damaging wind gusts. For wind gust magnitudes in particular, application of deeper learning networks (e.g., five or more hidden layers) offers tangible improvements in forecast accuracy. However, deeper networks are vulnerable to overfitting and exhibit substantial variability with the specific training and testing data subset used. Also, even deep ANNs reproduce only half of strong and damaging wind gusts. These results indicate the need for future work to elucidate the dynamical mechanisms of intense wind gusts and advance solutions to their prediction.

54 ENVIRONMENTAL SCIENCES↗

Multiple Lattice Instabilities and Complex Ground State in Cs 2 Ag Bi Br 6

Metal-halide perovskites (MHPs) are attracting considerable interest for optoelectronic applications, with Cs 2 Ag Bi Br 6 one of the main contenders among lead-free systems. Cs 2 Ag Bi Br 6 crystallizes in a nominally double-perovskite structure, but exhibits a soft lattice with large atomic fluctuations characteristic of MHPs. While crucial to understand electron-phonon and phonon-phonon couplings, the spatiotemporal correlations of these fluctuations remain largely unknown. Here, we reveal these correlations using comprehensive neutron and x-ray scattering measurements on Cs 2 Ag Bi Br 6 single crystals, complemented with first-principles simulations augmented with machine-learned neural-network potentials. We report the discovery of an unexpected complex modulated ground-state structure containing several hundred atoms, arising from a soft-phonon instability of the low-temperature tetragonal phase. Further, our experiments and simulations both reveal extensive correlated two-dimensional fluctuations of Br octahedra at finite temperature, arising from soft optic phonons that are strongly broadened by anhamonicity, reflecting very shallow potential wells. These results provide new insights into the atomic structure and fluctuations in MHPs, critical to understand and control their thermal and optoelectronic properties. Published by the American Physical Society 2024

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Mesoscale Cellular Convection Detection and Classification Using Convolutional Neural Networks: Insights From Long-Term Observations at ARM Eastern North Atlantic Site

Marine boundary layer clouds are crucial in Earth's climate system. They frequently manifest as closed or open cell mesoscale cellular convection (MCC). MCC clouds are challenging to represent accurately in current climate models, highlighting the need for detailed observational data sets and in-depth analyses. This study utilizes over 8 years of observations from the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) User Facility Eastern North Atlantic (ENA) site at Graciosa Island, Azores, to investigate these clouds. We first apply a convolutional neural network with a U-Net architecture to classify open and closed cells, marking the first application of such an approach for automatically detecting MCC patterns from ground-based radar measurements. This method addresses some observational gaps in satellite data related to low temporal resolution, nighttime challenges, and limited vertical structure capture. The analysis of the MCC cases shows clear differences between closed and open MCCs: Closed MCC clouds are characterized by lower cloud tops and bases, shallower cloud geometrical depth, weaker horizontal wind speeds, stronger atmospheric stability, and a more homogeneous liquid water path than open MCCs. Finally, we demonstrate two potential applications of our radar-based MCC classifications: (a) facilitating the investigation of aerosol-cloud interactions and (b) exploring meteorological factors along with MCC's evolution by integrating satellite imagery and back-trajectory analysis. The identified MCC cases offer a valuable resource for the scientific community to study MCC processes further and improve climate model accuracy.

54 ENVIRONMENTAL SCIENCES↗

Identifying atmospheric rivers and their poleward latent heat transport with generalizable neural networks: ARCNNv1

Abstract. Atmospheric rivers (ARs) are extreme weather events that can alleviate drought or cause billions of US dollars in flood damage. By transporting significant amounts of latent energy towards the poles, they are crucial to maintaining the climate system's energy balance. Since there is no first-principle definition of an AR grounded in geophysical fluid mechanics, AR identification is currently performed by a multitude of expert-defined, threshold-based algorithms. The variety of AR detection algorithms has introduced uncertainty into the study of ARs, and the thresholds of the algorithms may not generalize to new climate datasets and resolutions. We train convolutional neural networks (CNNs) to detect ARs while representing this uncertainty; we name these models ARCNNs. To detect ARs without requiring new labeled data and labor-intensive AR detection campaigns, we present a semi-supervised learning framework based on image style transfer. This framework generalizes ARCNNs across climate datasets and input fields. Using idealized and realistic numerical models, together with observations, we assess the performance of the ARCNNs. We test the ARCNNs in an idealized simulation of a shallow-water fluid in which nearly all the tracer transport can be attributed to AR-like filamentary structures. In reanalysis and a high-resolution climate model, we use ARCNNs to calculate the contribution of ARs to meridional latent heat transport, and we demonstrate that this quantity varies considerably due to AR detection uncertainty.

54 ENVIRONMENTAL SCIENCES↗

On the Relationship Between Shallow Cumulus Cloud Field Properties and Surface Solar Irradiance

Abstract Shallow cumulus clouds exhibit highly three‐dimensional (3‐D) spatial structure leading to complex variability in the surface solar irradiance (SSI) beneath. This variability is captured by the typically bimodal shape of the SSI probability density function (PDF). Using large eddy simulation to generate well‐resolved cloud fields and Monte Carlo 3‐D radiative transfer to reproduce realistic associated SSI PDFs, we seek direct relationships between the cloud field properties and the SSI PDF shape. Applying both random forest and artificial neural network algorithms, we find variations in the two modes of the SSI PDF are well predicted by just a handful of cloud field properties. The two algorithms utilize cloud properties similarly, with indistinguishable performance despite their different architectures. These results offer a marked improvement in realism relative to one‐dimensional radiative transfer while bypassing computationally expensive 3‐D radiative transfer, with immediate application to renewable energy assessments, and potential for several other geophysical applications.

58 GEOSCIENCES↗

Convolutional neural networks for intra-hour solar forecasting based on sky image sequences

Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%-39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 x 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.

14 SOLAR ENERGY↗

Machine Learning for Daily Forecasts of Arctic Sea Ice Motion: An Attribution Assessment of Model Predictive Skill

Physics-based simulations of Arctic sea ice are highly complex, involving transport between different phases, length scales, and time scales. Resultantly, numerical simulations of sea ice dynamics have a high computational cost and model uncertainty. We employ data-driven machine learning (ML) to make predictions of sea ice motion. The ML models are built to predict present-day sea ice velocity given present-day wind velocity and previous-day sea ice concentration and velocity. Models are trained using reanalysis winds and satellite-derived sea ice properties. We compare the predictions of three different models: persistence (PS), linear regression (LR), and a convolutional neural network (CNN). We quantify the spatiotemporal variability of the correlation between observations and the statistical model predictions. Additionally, we analyze model performance in comparison to variability in properties related to ice motion (wind velocity, ice velocity, ice concentration, distance from coast, bathymetric depth) to understand the processes related to decreases in model performance. Results indicate that a CNN makes skillful predictions of daily sea ice velocity with a correlation up to 0.81 between predicted and observed sea ice velocity, while the LR and PS implementations exhibit correlations of 0.78 and 0.69, respectively. The correlation varies spatially and seasonally: lower values occur in shallow coastal regions and during times of minimum sea ice extent. LR parameter analysis indicates that wind velocity plays the largest role in predicting sea ice velocity on 1-day time scales, particularly in the central Arctic. Regions where wind velocity has the largest LR parameter are regions where the CNN has higher predictive skill than the LR.

54 ENVIRONMENTAL SCIENCES↗

Deep material network via a quilting strategy: visualization for explainability and recursive training for improved accuracy

Recent developments integrating micromechanics and neural networks offer promising paths for rapid predictions of the response of heterogeneous materials with similar accuracy as direct numerical simulations. The deep material network is one such approaches, featuring a multi-layer network and micromechanics building blocks trained on anisotropic linear elastic properties. Once trained, the network acts as a reduced-order model, which can extrapolate the material’s behavior to more general constitutive laws, including nonlinear behaviors, without the need to be retrained. However, current training methods initialize network parameters randomly, incurring inevitable training and calibration errors. Here, we introduce a way to visualize the network parameters as an analogous unit cell and use this visualization to “quilt” patches of shallower networks to initialize deeper networks for a recursive training strategy. The result is an improvement in the accuracy and calibration performance of the network and an intuitive visual representation of the network for better explainability.

97 MATHEMATICS AND COMPUTING↗

Development of a Full-Scale Connected U-Net for Reflectivity Inpainting in Spaceborne Radar Blind Zones

CloudSat’s Cloud Profiling Radar is a valuable tool for remotely monitoring high-latitude snowfall, but its ability to observe hydrometeor activity near the Earth’s surface is limited by a radar blind zone caused by ground clutter contamination. This study presents the development of a deeply supervised U-Net-style convolutional neural network to predict cold season reflectivity profiles within the blind zone at two Arctic locations. The network learns to predict the presence and intensity of near-surface hydrometeors by coupling latent features encoded in blind zone-aloft clouds with additional context from collocated atmospheric state variables (i.e., temperature, specific humidity, and wind speed). Results show that the U-Net predictions outperform traditional linear extrapolation methods, with low mean absolute error, a 38% higher Sørensen–Dice coefficient, and vertical reflectivity distributions 60% closer to observed values. The U-Net is also able to detect the presence of near-surface cloud with a critical success index (CSI) of 72% and cases of shallow cumuliform snowfall and virga with 18% higher CSI values compared to linear methods. An explainability analysis shows that reflectivity information throughout the scene, especially at cloud edges and at the 1.2-km blind zone threshold, along with atmospheric state variables near the tropopause, are the most significant contributors to model skill. This surface-trained generative inpainting technique has the potential to enhance current and future remote sensing precipitation missions by providing a better understanding of the nonlinear relationship between blind zone reflectivity values and the surrounding atmospheric state.

54 ENVIRONMENTAL SCIENCES↗

Cost function dependent barren plateaus in shallow parametrized quantum circuits

Variational quantum algorithms (VQAs) optimize the parameters θ of a parametrized quantum circuit V(θ) to minimize a cost function C. While VQAs may enable practical applications of noisy quantum computers, they are nevertheless heuristic methods with unproven scaling. Here, we rigorously prove two results, assuming V(θ) is an alternating layered ansatz composed of blocks forming local 2-designs. Our first result states that defining C in terms of global observables leads to exponentially vanishing gradients (i.e., barren plateaus) even when V(θ) is shallow. Hence, several VQAs in the literature must revise their proposed costs. On the other hand, our second result states that defining C with local observables leads to at worst a polynomially vanishing gradient, so long as the depth of V(θ) is Ο(logn). Our results establish a connection between locality and trainability. We illustrate these ideas with large-scale simulations, up to 100 qubits, of a quantum autoencoder implementation.

97 MATHEMATICS AND COMPUTING↗

Using Machine Learning to Generate a GISS ModelE Calibrated Physics Ensemble (CPE)

A neural network (NN) surrogate of the NASA GISS ModelE atmosphere (version E3) is trained on a perturbed parameter ensemble (PPE) spanning 45 physics parameters and 36 outputs. The NN is leveraged in a Markov Chain Monte Carlo (MCMC) Bayesian parameter inference framework to generate a second posterior constrained ensemble coined a “calibrated physics ensemble,” or CPE. The CPE members are characterized by diverse parameter combinations and are, by definition, close to top-of-atmosphere radiative balance, and must broadly agree with numerous hydrologic, energy cycle and radiative forcing metrics simultaneously. Global observations of numerous cloud, environment, and radiation properties (provided by global satellite products) are crucial for CPE generation. The inference framework explicitly accounts for discrepancies (or biases) in satellite products during CPE generation. We demonstrate that product discrepancies strongly impact calibration of important model parameter settings (e.g., convective plume entrainment rates; fall speed for cloud ice). Structural improvements new to E3 are retained across CPE members (e.g., stratocumulus simulation). Notably, the framework improved the simulation of shallow cumulus and Amazon rainfall while not degrading radiation fields, an upgrade that neither default parameters nor Latin Hypercube parameter searching achieved. Analyses of the initial PPE suggested several parameters were unimportant for output variation. However, many “unimportant” parameters were needed for CPE generation, a result that brings to the forefront how parameter importance should be determined in PPEs. From the CPE, two diverse 45-dimensional parameter configurations are retained to generate radiatively-balanced, auto-tuned atmospheres that were used in two E3 submissions to CMIP6.

54 ENVIRONMENTAL SCIENCES↗

Model Evaluation and Intercomparison of Marine Warm Low Cloud Fractions With Neural Network Ensembles

Abstract Low cloud fractions (LCFs) and meteorological factors (MFs) over an oceanic region containing multiple cloud regimes are examined for three data sets: one Energy Exascale Earth System Model (E3SM) simulation with the default 72‐layer vertical grid (E3SM72), another one with 8‐times vertical resolution via the Framework for Improvement by Vertical Enhancement (E3SM 8), and one with MFs from ERA5 reanalysis and LCFs from the CERES SSF product (ERA5‐SSF). Neural networks (NNs) are trained to capture the relationship between MFs and LCF and to select the best‐performing MF subsets for predicting LCF. NN ensembles are used to (a) confirm the performance of selected MF subsets, (b) to serve as proxy models for each data set to predict LCFs for MFs from all data sets, and (c) to classify MFs into those in shared and uniquely occupied MF subspaces. Overall, E3SM72 and E3SM 8 have large fractions of MFs in shared MF subspace, but less so near the Californian and Peruvian stratocumulus decks. E3SM 8 and ERA5 have small fractions of MFs in shared MF subspace but greater than E3SM72 and ERA5, especially in the Southeast Pacific. The differences in LCFs between three pairs of data sets are decomposed into those associated with the differences in the LCF‐MF relationship and those involving different MFs. Given the same MFs, LCFs produced by E3SM 8 are greater than those produced by E3SM72 but are still different from those in ERA5‐SSF. In general, the shift in MFs dominates the difference in the LCFs.

54 ENVIRONMENTAL SCIENCES↗