Search NASA⌕ Search

SEARCH · Search NASA

Results for “convolutional networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRS2022

A Comprehensive Machine Learning Study to Classify Precipitation Type over Land from Global Precipitation Measurement Microwave Imager (GPM-GMI) Measurements

Precipitation type is a key parameter used for better retrieval of precipitation characteristics as well as to understand the cloud–convection–precipitation coupling processes. Ice crystals and water droplets inherently exhibit different characteristics in different precipitation regimes (e.g., convection, stratiform), which reflect on satellite remote sensing measurements that help us distinguish them. The Global Precipitation Measurement (GPM) Core Observatory’s microwave imager (GMI) and dual-frequency precipitation radar (DPR) together provide ample information on global precipitation characteristics. As an active sensor, the DPR provides an accurate precipitation type assignment, while passive sensors such as the GMI are traditionally only used for empirical understanding of precipitation regimes. Using collocated precipitation type flags from the DPR as the “truth”, this paper employs machine learning (ML) models to train and test the predictability and accuracy of using passive GMI-only observations together with ancillary information from a reanalysis and GMI surface emissivity retrieval products. Out of six ML models, four simple ones (support vector machine, neural network, random forest, and gradient boosting) and the 1-D convolutional neural network (CNN) model are identified to produce 90–94% prediction accuracy globally for five types of precipitation (convective, stratiform, mixture, no precipitation, and other precipitation), which is much more robust than previous similar effort. One novelty of this work is to introduce data augmentation (subsampling and bootstrapping) to handle extremely unbalanced samples in each category. A careful evaluation of the impact matrices demonstrates that the polarization difference (PD), brightness temperature (Tc) and surface emissivity at high-frequency channels dominate the decision process, which is consistent with the physical understanding of polarized microwave radiative transfer over different surface types, as well as in snow and liquid clouds with different microphysical properties. Furthermore, the view-angle dependency artifact that the DPR’s precipitation flag bears with does not propagate into the conical-viewing GMI retrievals. This work provides a new and promising way for future physics-based ML retrieval algorithm development.

machine learning/artificial intelligence↗
NASA NTRS2018

A 3D Citizen Science Video Game for NeMO-Net, the NASA Neural Multi-Modal Observation and Training Network for Global Coral Reef Assessment

NeMO-Net, the NASA neural multi-modal observation and training network for global coral reef assessment, is an open-source deep convolutional neural network aimed at accurately assessing the present and past dynamics of coral reef ecosystems through determination of percent living cover and morphology. We present here the active learning component of the project, which consists of an interactive video game prototype for tablet and mobile devices where players are able to intuitively label morphology classifications over mm-scale 3D coral reef imagery. Active learning applications present a novel methodology for engaging the public while efficiently providing large-scale training and test data for increasingly complex and data-intensive machine learning algorithms. NeMO-Net trains players on domain-specific knowledge through interactive tutorials and periodically checks players' input against pre-classified coral imagery to gauge their accuracy and utilize in-game mechanics to provide personalized classification training. Players can rate the classifications of other players, unlock rewards and join a global community as they explore and classify coral reefs and other shallow marine environments.

Citizen Science↗
NASA NTRSDate not supplied

Interpretable Convolutional Learning Classifier System (C-LCS) for Higher Dimensional Datasets

The purpose of this paper is to devise an interpretable hybrid classification model for Convolutional Neural Networks (CNN) and a Learning Classifier System (LCS). The presented hybrid system integrates the fundamental attributes from both types of these classifiers. In the proposed hybrid model CNN works as an automatic feature extractor, and LCS works to provide interpretable rule-based classification results. Although LCS has limitations working on higher dimensional datasets, we resolve this limitation by using CNN as a feature extractor. The other concept of the non-interpretability of CNN is addressed by using the LCS rule. Furthermore, our experiment with higher dimensional datasets like CIFAR-10 and Fashion-MNIST shows that extended LCS provides comparable performance to the standard neural network model while also providing interpretable results. We named this extended LCS method Convolutional Learning Classifier Cystem (C-LCS).

Jelani Owens↗
NASA NTRS2023

Statistical Decomposition and Machine Learning to Clean In Situ Spaceflight Magnetic Field Measurements

Robust in situ magnetic field measurements are critical to understanding the various mechanisms that couple mass, momentum, and energy throughout our solar system. However, the spacecraft on which magnetometers are often deployed contaminate the magnetic field measurements via onboard subsystems including reaction wheels and magnetorquers. Two magnetometers can be deployed at different distances from the spacecraft to determine an approximation of the interfering field for subsequent removal, but constant data streams from both magnetometers can be impractical due to power and telemetry limitations. Here we propose a method to identify and remove time-varying magnetic interference from sources such as reaction wheels using statistical decomposition and convolutional neural networks, providing high-fidelity magnetic field data even in cases where dual-sensor measurements are not constantly available. For example, a measurement interval from the Parker Solar Probe outboard magnetometer experienced a 95.1% reduction in reaction wheel interference following application of the proposed technique.

magnetic field data↗
NASA NTRS2018

NeMO-Net & Fluid Lensing: The Neural Multi-Modal Observation & Training Network for Global Coral Reef Assessment Using Fluid Lensing Augmentation of NASA EOS Data

We present preliminary results from NASA NeMO-Net, the first neural multi-modal observation and training network for global coral reef assessment. NeMO-Net is an open-source deep convolutional neural network (CNN) and interactive active learning training software in development which will assess the present and past dynamics of coral reef ecosystems. NeMO-Net exploits active learning and data fusion of mm-scale remotely sensed 3D images of coral reefs captured using fluid lensing with the NASA FluidCam instrument, presently the highest-resolution remote sensing benthic imaging technology capable of removing ocean wave distortion, as well as hyperspectral airborne remote sensing data from the ongoing NASA CORAL mission and lower-resolution satellite data to determine coral reef ecosystem makeup globally at unprecedented spatial and temporal scales. Aquatic ecosystems, particularly coral reefs, remain quantitatively misrepresented by low-resolution remote sensing as a result of refractive distortion from ocean waves, optical attenuation, and remoteness. Machine learning classification of coral reefs using FluidCam mm-scale 3D data show that present satellite and airborne remote sensing techniques poorly characterize coral reef percent living cover, morphology type, and species breakdown at the mm, cm, and meter scales. Indeed, current global assessments of coral reef cover and morphology classification based on km-scale satellite data alone can suffer from segmentation errors greater than 40%, capable of change detection only on yearly temporal scales and decameter spatial scales, significantly hindering our understanding of patterns and processes in marine biodiversity at a time when these ecosystems are experiencing unprecedented anthropogenic pressures, ocean acidification, and sea surface temperature rise. NeMO-Net leverages our augmented machine learning algorithm that demonstrates data fusion of regional FluidCam (mm, cm-scale) airborne remote sensing with global low-resolution (m, km-scale) airborne and spaceborne imagery to reduce classification errors up to 80% over regional scales. Such technologies can substantially enhance our ability to assess coral reef ecosystems dynamics.

satellite data↗
NASA NTRS2018

NASA NeMO-Net

We present preliminary results from NASA NeMO-Net, the first neural multi-modal observation and training network for global coral reef assessment. NeMO-Net is an open-source deep convolutional neural network (CNN) and interactive active learning training software in development which will assess the present and past dynamics of coral reef ecosystems. NeMO-Net exploits active learning and data fusion of mm-scale remotely sensed 3D images of coral reefs captured using fluid lensing with the NASA FluidCam instrument, presently the highest-resolution remote sensing benthic imaging technology capable of removing ocean wave distortion, as well as hyperspectral airborne remote sensing data from the ongoing NASA CORAL mission and lower-resolution satellite data to determine coral reef ecosystem makeup globally at unprecedented spatial and temporal scales. Aquatic ecosystems, particularly coral reefs, remain quantitatively misrepresented by low- resolution remote sensing as a result of refractive distortion from ocean waves, optical attenuation, and remoteness. Machine learning classification of coral reefs using FluidCam mm-scale 3D data show that present satellite and airborne remote sensing techniques poorly characterize coral reef percent living cover, morphology type, and species breakdown at the mm, cm, and meter scales. Indeed, current global assessments of coral reef cover and morphology classification based on km-scale satellite data alone can suffer from segmentation errors greater than 40%, capable of change detection only on yearly temporal scales and decameter spatial scales, significantly hindering our understanding of patterns and processes in marine biodiversity at a time when these ecosystems are experiencing unprecedented anthropogenic pressures, ocean acidification, and sea surface temperature rise. NeMO-Net leverages our augmented machine learning algorithm that demonstrates data fusion of regional FluidCam (mm, cm-scale) airborne remote sensing with global low-resolution (m, km-scale) airborne and spaceborne imagery to reduce classification errors up to 80% over regional scales. Such technologies can substantially enhance our ability to assess coral reef ecosystems dynamics.

NASA↗
NASA NTRS2017

Detection of Hail Storms in Radar Imagery Using Deep Learning

In 2016, hail was responsible for 3.5 billion and 23 million dollars in damage to property and crops, respectively, making it the second costliest weather phenomenon in the United States. In an effort to improve hail-prediction techniques and reduce the societal impacts associated with hail storms, we propose a deep learning technique that leverages radar imagery for automatic detection of hail storms. The technique is applied to radar imagery from 2011 to 2016 for the contiguous United States and achieved a precision of 0.848. Hail storms are primarily detected through the visual interpretation of radar imagery (Mrozet al., 2017). With radars providing data every two minutes, the detection of hail storms has become a big data task. As a result, scientists have turned to neural networks that employ computer vision to identify hail-bearing storms (Marzbanet al., 2001). In this study, we propose a deep Convolutional Neural Network (ConvNet) to understand the spatial features and patterns of radar echoes for detecting hailstorms.

natural hazard↗
NASA NTRS2019

Using Deep Learning to Automate Inference of Meteoroid Pre-Entry Properties

Properly assessing the asteroid threat depends on the knowledge of asteroid pre-entry parameters, such as size, velocity, mass, density, and strength. Although a vast number of possible bodies to study exist, such characterization of asteroid populations is currently limited by substantial costs associated with space rendezvous missions and rare meteorite findings. As asteroids fragment, ablate, and decelerate in the atmosphere, they emit light detectable by ground-based and space-borne instruments. Earth’s atmosphere, thus, becomes an accessible laboratory that enables impactor risk assessments by facilitating inference of the pre-entry parameters. These asteroid pre-entry conditions are typically deduced by modeling the entry and breakup physics that best reproduce the observed light or energy deposition curve. However, this process requires extensive manual trial-and-error of uncertain modeling parameters. Automating meteor modeling and inference would improve property distributions used in risk assessments and enable population characterization as more light curves become more readily available through the presence of space assets and ground-based camera networks. We previously developed a genetic algorithm to automate meteor modeling by using the fragment-cloud model (FCM) to search for the values of the FCM input parameters (e.g., diameter) that generate energy deposition profiles that match the observed one. Now, we apply deep learning to infer asteroid diameter, velocity, and density from observed energy deposition curves. We trained and tested our neural network models with synthetic energy deposition curves modeled using the FCM rubble pile implementation. We present an application of a 1D convolutional neural network and compare its performance to other attempted regressors and machine learning techniques, such as a fully connected neural network and Random Forest regression, to demonstrate its capabilities. We validate our model weights and approach using the Chelyabinsk, Tagish Lake, Benešov, Košice, and Lost City meteors.

Tarano, Ana Maria↗
NASA NTRS1979

Design of a 24-channel transmultiplexer

The design of a transmultiplexer capable of performing the bilateral conversion between one 1544 kbit/s digital signal (which represents 24 PCM coded voice channels) and two analog group signals (each one containing 12 voice channels in the 60-108 kHz band) is investigated. It is shown that an FIR filter bank required as part of such a transmultiplexer can be realized efficiently by cascading a discrete cosine transform processor and a weighting network. Fast convolution algorithms are derived for evaluating the cosine transform. A method of using the symmetry conditions to reduce the computation rate in the weighting network and an elegant hardware configuration for implementing it are also discussed.

Narasimha, M. J.↗
NASA NTRS2022

A Novel Machine Learning-Based Gap-Filling of Fine-Resolution Remotely Sensed Snow Cover Fraction Data By Combining Downscaling and Regression

Satellite-based remotely sensed observations of snow cover fraction (SCF) can have data gaps in spatially distributed coverage from sensor and orbital limitations. We mitigate these limitations in the example fine-resolution Moderate Resolution Imaging Spectroradiometer (MODIS) data by gap-filling using auxiliary 1-km datasets that either aid in downscaling from coarser-resolution (5 km) MODIS SCF wherever not fully covered by clouds, or else by themselves via regression wherever fully cloud covered. This study’s prototype predicts a 1-km version of the 500-m MOD10A1 SCF target. Due to noncollocatedness of spatial gaps even across input and auxiliary datasets, we consider a recent gap-agnostic advancement of partial convolution in computer vision for both training and predictive gap-filling. Partial convolution accommodates spatially consistent gaps across the input images, effectively implementing a two-dimensional masking. To overcome reduced usable data from noncollocated spatial gaps across inputs, we innovate a fully generalized three-dimensional masking in this partial convolution. This enables a valid output value at a pixel even if only a single valid input variable and its value exist in the neighborhood covered by the convolutional filter zone centered around that pixel. Thus, our gap-agnostic technique can use significantly more examples for training (∼67%) and prediction (∼100%), instead of only less than 10% for the previous partial convolution. We train an example simple three-layer legacy super-resolution convolutional neural network (SRCNN) to obtain downscaling and regression component performances that are better than baseline values of either climatology or MOD10C1 SCF as relevant. Our generalized partial convolution can enable multiple Earth science applications like downscaling, regression, classification, and segmentation that were hindered by data gaps.

Soni Yatheendradas↗
NASA NTRS2022

Identification of Flux Rope Orientation via Neural Networks

Geomagnetic disturbance forecasting is based on the identification of solar wind structures and accurate determination of their magnetic field orientation. For nowcasting activities, this is currently a tedious and manual process. Focusing on the main driver of geomagnetic disturbances, the twisted internal magnetic field of interplanetary coronal mass ejections (ICMEs), we explore a convolutional neural network’s (CNN) ability to predict the embedded magnetic flux rope’s orientation once it has been identified from in situ solar wind observations. Our work uses CNNs trained with magnetic field vectors from analytical flux rope data. The simulated flux ropes span many possible spacecraft trajectories and flux rope orientations. We train CNNs first with full duration flux ropes and then again with partial duration flux ropes. The former provides us with a baseline of how well CNNs can predict flux rope orientation while the latter provides insights into real-time forecasting by exploring how accuracy is affected by percentage of flux rope observed. The process of casting the physics problem as a machine learning problem is discussed as well as the impacts of different factors on prediction accuracy such as flux rope fluctuations and different neural network topologies. Finally, results from evaluating the trained network against observed ICMEs from Wind during 1995–2015 are presented.

Thomas Narock↗
NASA NTRS2018

Enriching the Twitter Stream Increasing Data Mining Yield and Quality Using Machine Learning

Social media data streams are important sources of real-time and historical global information for science applications. At the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we are exploring the Twitter data stream for its potential in augmenting the validation program of NASA Earth science missions, specifically the Global Precipitation Measurement (GPM) mission. We have implemented a tweet processing infrastructure that outputs classified precipitation tweets. Inputs are "passive" tweets, along with a smaller number of tweets from "active" participants, i.e., those knowingly contributing to our effort. The "active" tweets, presumably of higher quality, enrich the Twitter stream. "Active" sources include data scraped from other social media (e.g., public Facebook posts) and data from existing crowdsourcing programs (e.g., mPING reports). In addition, there is likely relevant precipitation information in images and documents that are the end points of links often included in tweets. Information derived from these "active" sources could then be tweeted into the Twitter stream, thus enriching its quality. The objective of our current work is to mine these tweet­ linked images and documents, using neural networks, to increase the information content and quality related to precipitation. For images, we classified them as either precipitation-related or not. For training and validation, we used images obtained via the Google custom search API. We created two models: (1) by training a simple Convolutional Neural Network and (2) by using transfer learning principles to adapt a pre-trained object recognition model. For documents, both those linked to tweets and the tweet contents, we trained Hierarchical Attention Networks to determine precipitation occurrence, type, and intensity. For training and validation, we used a keyword-filtered tweet data set labelled with ground truth data from Dark Sky (an API to retrieve weather-related labels) and the National Severe Storms Laboratory's Multi­ Radar/Multi-Sensor (MRMS) system. Our results demonstrated the efficacy of our machine learning approaches for enriching the Twitter stream, to derive information potentially useful for validation of earth science satellite data.

Albayrak, Arif↗
NASA NTRS2026

PERSIANN-Unet: A Global Deep Learning Framework for Near-Real-Time Precipitation Estimation Using Infrared Data

Access to high-quality, high-resolution, near-real-time precipitation data is essential for hydrological and meteorological research and disaster mitigation. Traditional tools such as rain gauges and radar networks, though effective, have limitations, including sparse coverage in remote areas and high operational costs. Satellite data, with its global coverage and high spatial and temporal resolutions, mitigates limitations in coverage. Satellite precipitation products like Hydro Estimator (HE), Integrated Multi-satellitE Retrievals for Global Precipitation Measurement (IMERG), and Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks (PERSIANN) utilize both geosynchronous thermal infrared (IR) and passive microwave (PMW) data in their operation. PMW sensors offer detailed atmospheric profiles but suffer from higher latency, whereas IR sensors provide lower latency but only capture cloud-top information. Despite this constraint, IR data remains attractive for low-latency precipitation estimation. Recent advances in deep learning, particularly convolutional neural networks (CNNs), have further improved satellite precipitation retrievals. This study introduces PERSIANN-Unet (PUnet or PERSIANN V3), a quasi-global algorithm covering 60°N–60°S that combines IR data, monthly climatology, and the UNet architecture to produce half-hourly precipitation estimates at 0.04° resolution. The product is evaluated against HE, IMERG, and PDIR-Now for 2022–2023. Results show that PUnet closely matches its training target, IMERG V07 Final, at the global scale, and performance is further evaluated against Stage IV as a reference over CONUS. Training PUnet on IMERG (2016–2021) leverages a high-quality, integrated PMW IR-gauge precipitation product while developing an IR-based framework not reliant on PMW availability. By operating on a single global image, PUnet avoids tile partitioning and blending steps, reducing edge discontinuities, and produces more spatially consistent precipitation fields across hemispheres.

Phu Nguyen↗
NASA NTRS2020

Improving Trust in Deep Neural Networks with Nearest Neighbors

Deep neural networks are used increasingly for perception and decision-making in UAVs. For example, they can be used to recognize objects from images and decide what actions the vehicle should take. While deep neural networks can perform very well at complex tasks, their decisions may be unintuitive to a human operator. When a human disagrees with a neural network prediction, due to the black box nature of deep neural networks, it can be unclear whether the system knows something the human does not or whether the system is malfunctioning. This uncertainty is problematic when it comes to ensuring safety. As a result, it is important to develop technologies for explaining neural network decisions for trust and safety. This paper explores a modification to the deep neural network classification layer to produce both a predicted label and an explanation to support its prediction. Specifically, at test time, we replace the final output layer of the neural network classifier by a k-nearest neighbor classifier. The nearest neighbor classifier produces 1) a predicted label through voting and 2) the nearest neighbors involved in the prediction, which represent the most similar examples from the training dataset. Because prediction and explanation are derived from the same underlying process, this approach guarantees that the explanations are always relevant to the predictions. We demonstrate the approach on a convolutional neural network for a UAV image classification task. We perform experiments using a forest trail image dataset and show empirically that the hybrid classifier can produce intuitive explanations without loss of predictive performance compared to the original neural network. We also show how the approach can be used to help identify potential issues in the network and training process.

Lee, Ritchie↗
NASA NTRS2022

Machine learning for Deep Space Network antenna motions detection

Highly stable frequency and timing standards are essential for deep-space missions and radio science. At the NASA Deep Space Network (DSN), these standards are distributed through a network of underground fiber cables to support several Goldstone antennas. Independently developed frequency-measuring instruments generate tremendous quantities of data to monitor and validate the antennas’ stringent frequency requirements. In this paper, we propose a lightweight processing tool capable of detecting disturbances on the frequency signal caused by DSN antenna motions. Our training data is sampled from the movement log of the antenna of interest and the generated data from the fiber optic metrology instrument linked to the antenna. We demonstrate that a convolutional neural network (CNN) model can achieve high accuracies on classifying instances of antenna movements and is an effective predictor when used iteratively on longer, variable stretches of metrology data. The simplicity, low training cost, and high accuracies of our model strongly suggest its efficacy in identifying and troubleshooting frequency disturbances caused by the antenna.

Yi, Lin↗
NASA NTRS2017

NeMO-Net The Neural Multi-Modal Observation Training Network for Global Coral Reef Assessment

In the past decade, coral reefs worldwide have experienced unprecedented stresses due to climate change, ocean acidification, and anthropomorphic pressures, instigating massive bleaching and die-off of these fragile and diverse ecosystems. Furthermore, remote sensing of these shallow marine habitats is hindered by ocean wave distortion, refraction and optical attenuation, leading invariably to data products that are often of low resolution and signal-to-noise (SNR) ratio. However, recent advances in UAV and Fluid Lensing technology have allowed us to capture multispectral 3D imagery of these systems at sub-cm scales from above the water surface, giving us an unprecedented view of their growth and decay. Exploiting the fine-scaled features of these datasets, machine learning methods such as MAP, PCA, and SVM can not only accurately classify the living cover and morphology of these reef systems (below 8 error), but are also able to map the spectral space between airborne and satellite imagery, augmenting and improving the classification accuracy of previously low-resolution datasets.We are currently implementing NeMO-Net, the first open-source deep convolutional neural network (CNN) and interactive active learning and training software to accurately assess the present and past dynamics of coral reef ecosystems through determination of percent living cover and morphology. NeMO-Net will be built upon the QGIS platform to ingest UAV, airborne and satellite datasets from various sources and sensor capabilities, and through data-fusion determine the coral reef ecosystem makeup globally at unprecedented spatial and temporal scales. To achieve this, we will exploit virtual data augmentation, the use of semi-supervised learning, and active learning through a tablet platform allowing for users to manually train uncertain or difficult to classify datasets. The project will make use of Pythons extensive libraries for machine learning, as well as extending integration to GPU and High-End Computing Capability (HECC) on the Pleiades supercomputing cluster, located at NASA Ames. The project is being supported by NASAs Earth Science Technology Office (ESTO) Advanced Information Systems Technology (AIST-16) Program.

Remote Sensin↗
NASA NTRS2017

NeMO-Net: The Neural Multi-Modal Observation and Training Network for Global Coral Reef Assessment

In the past decade, coral reefs worldwide have experienced unprecedented stresses due to climate change, ocean acidification, and anthropomorphic pressures, instigating massive bleaching and die-off of these fragile and diverse ecosystems. Furthermore, remote sensing of these shallow marine habitats is hindered by ocean wave distortion, refraction and optical attenuation, leading invariably to data products that are often of low resolution and signal-to-noise (SNR) ratio. However, recent advances in UAV and Fluid Lensing technology have allowed us to capture multispectral 3D imagery of these systems at sub-cm scales from above the water surface, giving us an unprecedented view of their growth and decay. Exploiting the fine-scaled features of these datasets, machine learning methods such as MAP, PCA, and SVM can not only accurately classify the living cover and morphology of these reef systems (below 8 percent error), but are also able to map the spectral space between airborne and satellite imagery, augmenting and improving the classification accuracy of previously low-resolution datasets. We are currently implementing NeMO-Net, the first open-source deep convolutional neural network (CNN) and interactive active learning and training software to accurately assess the present and past dynamics of coral reef ecosystems through determination of percent living cover and morphology. NeMO-Net will be built upon the QGIS platform to ingest UAV, airborne and satellite datasets from various sources and sensor capabilities, and through data-fusion determine the coral reef ecosystem makeup globally at unprecedented spatial and temporal scales. To achieve this, we will exploit virtual data augmentation, the use of semi-supervised learning, and active learning through a tablet platform allowing for users to manually train uncertain or difficult to classify datasets. The project will make use of Pythons extensive libraries for machine learning, as well as extending integration to GPU and High-End Computing Capability (HECC) on the Pleiades supercomputing cluster, located at NASA Ames. The project is being supported by NASAs Earth Science Technology Office (ESTO) Advanced Information Systems Technology (AIST-16) Program.

NeMO-Net↗
NASA NTRSDate not supplied

From Simulation to Reality With Random Noise

The challenging environment of autonomous vehicle (AV) navigation necessitates certain functions be performed by deep neural networks. Optimizing these models involves collecting vast quantities of domain-specific training data and ensuring that the dataset is representative of expected conditions. High-fidelity simulation plays a vital role in making this process feasible, allowing a wide range of scenarios to be explored at low cost. However, learning from simulation introduces subtle biases into models, which can degrade real-world performance in unpredictable ways. This effect can be mitigated with learning schemes specialized to bridge distributional shifts (transfer learning). Given the complex nature of these methods, the underlying models, and their environments, meaningfully evaluating performance is notstraight forward. Many unrelated factors can effect an improvement in generalization accuracy, but a full ablation analysis is often difficult. To tease out signal from noise, it is necessary to understand how transfer learning performance is affected by noise itself. The goals of this paper are (i) to establish a domain randomization baseline for a simple classification transfer learning task and (ii) to validate the RRAV testbed as a platform for further research in sim-to-real learning. We generate imagery from a simulation of NASA Ames Research Center and train a small convolutional neural network (ConvNet) to classify position relative to a centerline. Further models are trained with different types of noise progressively added to the data. The models are deployed aboard the on-site test vehicle to test real-world performance. In our experiments, we find that such naive domain randomization raises sim-to-real accuracy from 64% to 79%, while training directly on real data yields an 89% accuracy ceiling. These results suggest that the isolated mechanism of domain randomization can significantly improve generalization.

simulation↗