Understanding and Verifying Neural Networks

Divya Gopinath

NASA NTRS · 20210024509

Understanding and Verifying Neural Networks

Abstract

Deep Neural Networks (DNNs) have gained immense popularity in recent times and have widespread use in applications such as image classification, sentiment analysis, speech recognition and also in safety-critical applications such as autonomous driving. However, they suffer limitations such as lack of explainability and robustness which raise safety and security concerns in their usage. Further, the complex structure and large input spaces of DNNs act as an impediment to thorough verification and testing. The SafeDNN project at the Robust Software Engineering (RSE) group at NASA aims at exploring techniques to ensure that systems that use deep neural networks are safe, robust and interpretable. In this talk, I will be presenting our technique Prophecy that automatically infers formal properties of deep neural network models. The tool extracts patterns based on neuron activations as preconditions that imply certain desirable output properties of the model. I would be highlighting case studies that use Prophecy in obtaining explanations for network decisions, understanding correct and incorrect behavior, providing formal guarantees wrt safety and robustness, and debugging neural network models. We have applied the tool on image classification networks, neural network controllers providing turn advisories in unmanned aircrafts, regression models used for autonomous center-line tracking in aircrafts and neural network object detectors

Explore related subjects

Deep Neural Networks Explanability Robustness Repair Debugging

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Divya Gopinath. Understanding and Verifying Neural Networks. https://ntrs.nasa.gov/citations/20210024509

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

NASA NTRSDate not supplied

QuantifyML: How good is my machine learning model?

This paper presents an approach, QuantifyML, which employs model counting to assess the learnability and robustness of machine learning models. Typically the efficacy of machine learning models is determined by computing their accuracy statistically on test data sets. However, this may be misleading, if the test data is not representative of the problem that is being studied. Further, two different models may have the same accuracy on a given data set, measured statistically, but may be very different in their behavior on unseen data. Also, models with high accuracy could have poor adversarial robustness. In QuantifyML, our goal is to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into a C program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise counts w.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, expressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the robustness of trained models around given inputs. Our evaluation demonstrates these applications of QuantifyML on decision trees and neural networks trained to learn relational properties of graphs, for which we know the ground truth, and to perform image classification, for which we do not have the ground truth, but we can quantify local robustness.

Deep Neural Networks↗

NASA NTRSDate not supplied

Analysis of TaxiNet Neural Network

Explore the source record for details and available documents.

Deep Neural Networks↗

NASA NTRSDate not supplied

QuantifyML: How Good is My Machine Learning Model?

We present QuantifyML, which applies model counting to assess the learn ability, safety, and robustness of machine learning models. Typically, the efficacy of machine learning models is determined by computing their accuracy statistically on test datasets. However, this may be misleading, if the test data is not representative of the problem that is being studied. With QuantifyML we aim to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into aC program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise countsw.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, ex-pressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the safety and robustness of trained models.

Deep Neural Networks↗