SEARCH · Search NASA

Results for “adversarial evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

NASA NTRS2013

AGATE: Adversarial Game Analysis for Tactical Evaluation

AGATE generates a set of ranked strategies that enables an autonomous vehicle to track/trail another vehicle that is trying to break the contact using evasive tactics. The software is efficient (can be run on a laptop), scales well with environmental complexity, and is suitable for use onboard an autonomous vehicle. The software will run in near-real-time (2 Hz) on most commercial laptops. Existing software is usually run offline in a planning mode, and is not used to control an unmanned vehicle actively. JPL has developed a system for AGATE that uses adversarial game theory (AGT) methods (in particular, leader-follower and pursuit-evasion) to enable an autonomous vehicle (AV) to maintain tracking/ trailing operations on a target that is employing evasive tactics. The AV trailing, tracking, and reacquisition operations are characterized by imperfect information, and are an example of a non-zero sum game (a positive payoff for the AV is not necessarily an equal loss for the target being tracked and, potentially, additional adversarial boats). Previously, JPL successfully applied the Nash equilibrium method for onboard control of an autonomous ground vehicle (AGV) travelling over hazardous terrain.

Huntsberger, Terrance L.↗

NASA NTRSDate not supplied

NNREPAIR: Constraint-based Repair of Neural Network Classifiers

We present NNREPAIR, a constraint-based technique for repairing neural network classifiers by i) fixing the logic of the network at an intermediate layer, and ii) fixing the decision conditions of the network at the last layer. NNREPAIR first uses fault localization to find potentially faulty network parameters (specifically weights) and then performs repair using constraint solving to apply small modifications to remedy the defects. We present novel strategies to enable precise yet efficient repair such as inferring correctness specifications to act as oracles for intermediate layer repair, and generation of experts wrt each class. We demonstrate the technique in the context of three different scenarios: (1) Improving the over all accuracy of a model, (2) Fixing security vulnerabilities caused by poisoning of training data and (3) Improving the robustness of the network against adversarial attacks. Our evaluation on MNIST and CIFAR-10 models shows that NNREPAIR can improve the accuracy by 45.56% on poisoned data and 11.11% on adversarial data. NNREPAIR also provides improvement, albeit modest, in the overall accuracy of models, without requiring new data or re-training.

Repair↗

NASA NTRS2013

NASA Tech Briefs, Februrary 2013

Topics covered include: Measurements of Ultra-Stable Oscillator (USO) Allan Deviations in Space; Gaseous Nitrogen Orifice Mass Flow Calculator; Validation of Proposed Metrics for Two-Body Abrasion Scratch Test Analysis Standards; Rover Low Gain Antenna Qualification for Deep Space Thermal Environments; Automated, Ultra-Sterile Solid Sample Handling and Analysis on a Chip; Measuring and Estimating Normalized Contrast in Infrared Flash Thermography; Spectrally and Radiometrically Stable, Wideband, Onboard Calibration Source; High-Reliability Waveguide Vacuum/Pressure Window; Methods of Fabricating Scintillators With Radioisotopes for Beta Battery Applications; Magnetic Shield for Adiabatic Demagnetization Refrigerators (ADR); CMOS-Compatible SOI MESFETS for Radiation-Hardened DC-to-DC Converters; Silicon Heat Pipe Array; Adaptive Phase Delay Generator; High-Temperature, Lightweight, Self-Healing Ceramic Composites for Aircraft Engine Applications; Treatment to Control Adhesion of Silicone-Based Elastomers; High-Temperature Adhesives for Thermally Stable Aero-Assist Technologies; Rockballer Sample Acquisition Tool; Rock Gripper for Sampling, Mobility, Anchoring, and Manipulation; Advanced Magnetic Materials Methods and Numerical Models for Fluidization in Microgravity and Hypogravity; Data Transfer for Multiple Sensor Networks Over a Broad Temperature Range; Using Combustion Synthesis to Reinforce Berms and Other Regolith Structures; Visible-Infrared Hyperspectral Image Projector; Three-Axis Attitude Estimation With a High-Bandwidth Angular Rate Sensor Change_Detection.m; AGATE: Adversarial Game Analysis for Tactical Evaluation; Ionospheric Simulation System for Satellite Observations and Global Assimilative; Modeling Experiments (ISOGAME); An Extensible, User- Modifiable Framework for Planning Activities; Mission Operations Center (MOC) - Precipitation Processing System (PPS) Interface Software System (MPISS); Automated 3D Damaged Cavity Model Builder for Lower Surface Acreage Tile on Orbiter; Mixed Linear/Square-Root Encoded Single-Slope Ramp Provides Low-Noise ADC with High Linearity for Focal Plane Arrays; RUSHMAPS: Real-Time Uploadable Spherical Harmonic Moment Analysis for Particle Spectrometers; Powered Descent Guidance with General Thrust-Pointing Constraints; X-Ray Detection and Processing Models for Spacecraft Navigation and Timing; and Extreme Ionizing-Radiation-Resistant Bacterium

Source record↗

NASA NTRS2020

Partially Observable Games for Secure Autonomy

Technology development efforts in autonomy and cyber-defense have been evolving independently of each other, over the past decade. In this paper, we report our ongoing effort to integrate these two presently distinct areas into a single framework. To this end, we propose the two-player partially observable stochastic game formalism to capture both high-level autonomous mission planning under uncertainty and adversarial decision making subject to imperfect information. We show that synthesizing sub-optimal strategies for such games is possible under finite-memory assumptions for both the autonomous decision maker and the cyber-adversary. We then describe an experimental testbed to evaluate the efficacy of the proposed framework.

Ames, Aaron D.↗

NASA NTRS2021

Semantic Segmentation of High-Resolution Satellite Imagery using Generative Adversarial Networks with Progressive Growing

With increase in urbanization and Earth Sciences research into urban areas, the need to quickly and accurately segment urban rooftop maps has never been greater. Cur-rent machine learning techniques struggle to produce high accuracy maps in dense urban zones where there is high image noise and foot print overlap. In this paper, we evaluate a training methodology for pixel-wise segmentation for high resolution satellite imagery using progressive growing of generative adversarial networks as a solution. We apply our model to segmenting building rooftops and compare these results to conventional methods for rooftop segmentation. We evaluate our approach using the SpaceNet version 2 and xView datasets. Our experiments show that for SpaceNet, progressive Generative Adversarial Network (GAN) training achieved a test accuracy of 93% compared to 89% for traditional GAN training and 87% for U-Net architecture, while for xView, we achieved 71% accuracy using progressive GAN training compared to 69% through traditional GAN training and 65% using U-Net.

Semantic↗

NASA NTRSDate not supplied

Data Augmentation for Intelligent Contingency Management Using Generative Adversarial Neural Networks

Artificial intelligence (AI)-based techniques for intelligent contingency management (ICM) require that intelligent agents learn various aspects of system dynamics to create and execute contingencies. For high assurance contingency management, agents achieve the most compelling results through supervised or semi-supervised machine learning, for which agents require large datasets to learn the dynamics of the system. Unfortunately, data collection in aerospace applications can be costly, due to both time and resources. Presented work describes a framework for data augmentation of ICM databases containing training data for machine learning models. This framework populates the database with the outputs of generative adversarial network (GAN) models that were trained on flight data. Methods for evaluating the suitability of these models based on the equations of motion, as well as other physical constraints, are discussed. The paper demonstrates the utility of this database for training intelligent agents on the NASA T2 generic transport aircraft model and experimental vertical takeoff and landing (VTOL) simulation model.

Generative Machine Learning↗

NASA NTRSDate not supplied

Data Augmentation for Intelligent Contingency Management Using Generative Adversarial Neural Networks

Generative Machine Learning↗

NASA NTRS2020

CyberGAN: Generating High-fidelity Cybersecurity Data With Generative Adversarial Networks

Machine learning for cyber defense offers the promise of detecting adversarial activity against the ground data systems managing critical space assets. A fundamental challenge facing machine learning research in cybersecurity is the lack of high-fidelity, shareable datasets for robust evaluation and testing of machine learning-based solutions. High-fidelity, real-world datasets are necessary for reliable benchmarking of nominal system behavior and malicious activity. Unfortunately, such realistic datasets of both nominal and adversarial activity are rarely shared publicly by data owners due to security and privacy concerns. Besides, the available adversarial data is sparse, which makes training models on malicious activity much harder. This situation has impeded and continues to impede the research and successful adoption of machine learning methods for cyber defense. Researchers have dealt with this problem by generating data within a low-fidelity lab environment, using classified and thus unshareable datasets, or downloading low-fidelity public datasets made available by others. We propose an innovative solution to the problem by employing machine learning methods to generate high-fidelity data. Specifically, we propose the use of Generative Adversarial Networks (GANs) to generate high-fidelity data for cybersecurity purposes. GANs have found successful image processing and natural language applications, but have not yet been investigated for cyber data generation. Our proposed approach first involves training the `discriminator' network of the GAN with a sample of real-world data consisting of malicious and nominal samples. We then use the `generator' network to generate new high-fidelity data samples consisting of an appropriate mix of malicious and nominal activity. We demonstrate applications of our architecture by generating high-fidelity cybersecurity data containing both malicious and nominal samples. We thoroughly evaluate the fidelity of our generated data using heuristics and evaluate its usefulness for machine learning applications using three different datasets. Overall, our approach results in high-fidelity, shareable datasets.

Zhang, Yuening↗

NASA NTRS2018

Towards Informing an Intuitive Mission Planning Interface for Autonomous Multi-Asset Teams via Image Descriptions

Establishing a basis for certification of autonomous systems using trust and trustworthiness is the focus of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR). The Human-Machine Interface (HMI) team is working to capture and utilize the multitude of ways in which humans are already comfortable communicating mission goals and translate that into an intuitive mission planning interface. Several input/output modalities (speech/audio, typing/text, touch, and gesture) are being considered and investigated in the context human-machine teaming for the ATTRACTOR design reference mission (DRM) of Search and Rescue or (more generally) intelligence, surveillance, and reconnaissance (ISR). The first of these investigations, the Human Informed Natural-language GANs Evaluation (HINGE) data collection effort, is aimed at building an image description database to train a Generative Adversarial Network (GAN). In addition to building an image description database, the HMI team was interested if, and how, modality (spoken vs. written) affects different aspects of the image description given. The results will be analyzed to better inform the designing of an interface for mission planning.

Generative Adversarial Network (GAN)↗

NASA NTRSDate not supplied

Adaptive Stress Testing of Trajectory Predictions in Flight Management Systems

To find failure events and their likelihoods in flight-critical systems, we investigate the use of an advanced black-box stress testing approach called adaptive stress testing. We analyze a trajectory predictor from a developmental commercial flight management system which takes as input a collection of lateral waypoints and en-route environmental conditions. Our aim is to search for failure events relating to inconsistencies in the predicted lateral trajectories. The intention of this work is to find likely failures and report them back to the developers so they can address and potentially resolve shortcomings of the system before deployment. To improve search performance, this work extends the adaptive stress testing formulation to be applied more generally to sequential decision-making problems with episodic reward by collecting the state transitions during the search and evaluating at the end of the simulated rollout. We use a modified Monte Carlo tree search algorithm with progressive widening as our adversarial reinforcement learner. The performance is compared to direct Monte Carlo simulations and to the cross-entropy method as an alternative importance sampling baseline. The goal is to find potential problems otherwise not found by traditional requirements-based testing. Results indicate that our adaptive stress testing approach finds more failures and finds failures with higher likelihood relative to the baseline approaches.

adaptive stress testing↗

NASA NTRS1994

DTS: Building custom, intelligent schedulers

DTS is a decision-theoretic scheduler, built on top of a flexible toolkit -- this paper focuses on how the toolkit might be reused in future NASA mission schedulers. The toolkit includes a user-customizable scheduling interface, and a 'Just-For-You' optimization engine. The customizable interface is built on two metaphors: objects and dynamic graphs. Objects help to structure problem specifications and related data, while dynamic graphs simplify the specification of graphical schedule editors (such as Gantt charts). The interface can be used with any 'back-end' scheduler, through dynamically-loaded code, interprocess communication, or a shared database. The 'Just-For-You' optimization engine includes user-specific utility functions, automatically compiled heuristic evaluations, and a postprocessing facility for enforcing scheduling policies. The optimization engine is based on BPS, the Bayesian Problem-Solver (1,2), which introduced a similar approach to solving single-agent and adversarial graph search problems.

Hansson, Othar↗

NASA NTRS1975

Advocacy and technology assessment

A highly structured treatment is presented of adversarial systems as they apply to technology assessment. One approach to the problem of adequate criteria of assessment focuses upon the internal operations of assessment entities; operations include problem perception, problem formulation, selection, utilization, determination, and evaluation. Potential contributions of advocacy as a mode of inquiry in technology are discussed; advocacy is evaluated by representative sets of criteria of adequate assessment which include participant criteria, perspectives criteria, situations criteria, base values criteria, and strategies criteria.

Jones, E. M.↗

NASA NTRS1990

Optimal helicopter trajectory planning for terrain following flight

Helicopters operating in high threat areas have to fly close to the earth surface to minimize the risk of being detected by the adversaries. Techniques are presented for low altitude helicopter trajectory planning. These methods are based on optimal control theory and appear to be implementable onboard in realtime. Second order necessary conditions are obtained to provide a criterion for finding the optimal trajectory when more than one extremal passes through a given point. A second trajectory planning method incorporating a quadratic performance index is also discussed. Trajectory planning problem is formulated as a differential game. The objective is to synthesize optimal trajectories in the presence of an actively maneuvering adversary. Numerical methods for obtaining solutions to these problems are outlined. As an alternative to numerical method, feedback linearizing transformations are combined with the linear quadratic game results to synthesize explicit nonlinear feedback strategies for helicopter pursuit-evasion. Some of the trajectories generated from this research are evaluated on a six-degree-of-freedom helicopter simulation incorporating an advanced autopilot. The optimal trajectory planning methods presented are also useful for autonomous land vehicle guidance.

Menon, P. K. A.↗

NASA NTRS1993

Assessment of Russian VSTOL technology evaluating the YAK-38 'FORGER' and YAK-141 'FREESTYLE'

The dissolution of the Former Soviet Union (FSU) created new relationships between the world superpowers. Overnight, the Commonwealth of Independent States (CIS), formed from the remnants of the FSU, began the difficult transformation to a free market society. Military hardware that had once been highly classified and the basis for our own defense planning was now openly marketed at airshows around the world. 'Test' flights were available for potential customers and cooperative partnerships were explored between former adversaries. This environment permitted a visit to the Yakovlev Design Bureau, (YAK) for a vertical/short takeoff and landing (VSTOL) technology assessment. Yakovlev is the FSU's sole Design Bureau with experience in VSTOL aircraft and has developed two flying examples, the YAK-38 'FORGER' and YAK-141 'FREESTYLE'. This article reviews the performance of the YAK-38 'FORGER' and the YAK-141 'FREESTYLE'.

Nalls, Art↗

NASA NTRS2019

A Communication Channel Density Estimating Generative Adversarial Network

Autoencoder-based communication systems use neural network channel models to backwardly propagate message reconstruction error gradients across an approximation of the physical communication channel. In this work, we develop and test a new generative adversarial network (GAN) architecture for the purpose of training a stochastic channel approximating neural network. In previous research, investigators have focused on additive white Gaussian noise (AWGN) channels and/or simplified Rayleigh fading channels, both of which are linear and have well defined analytic solutions. Given that training a neural network is computationally expensive, channel approximation networks— and more generally the autoencoder systems—should be evaluated in communication environments that are traditionally difficult. To that end, our investigation focuses on channels that contain a combination of non-linear amplifier distortion, pulse shape filtering, intersymbol interference, frequency-dependent group delay, multipath, and non-Gaussian statistics. Each of our models are trained without any prior knowledge of the channel. We show that the trained models have learned to generalize over an arbitrary amplifier drive level and constellation alphabet. We demonstrate the versatility of our GAN architecture by comparing the marginal probability density function of several channel simulations with that of their corresponding neural network approximations

Smith, Aaron↗

NASA NTRS1995

Integrated helmet mounted display concepts for air combat

A piloted simulation study was conducted in a dome simulator to evaluate several Helmet Mounted Display (HMD) formats developed as part of the NASA High Alpha Technology Program (HATP). The display formats conveyed energy management, spatial orientation, and weapons management information. The HMD format was compared to a generic Heads Up Display (HUD) typical of current operational fighter aircraft. Pilots were tasked to spend as much time in a weapon solution as possible, to have the correct weapon selected for the envelope they were in, and to avoid the adversary's weapon envelope as much as possible. Several different displays were tested individually and simultaneously to see how separate display concepts coexisted. Objective results showed that the ability for the pilot to select the correct weapon for the envelope he was in increased by 50% in a moderate workload condition and 90% in a high workload condition with the HMD format. In the post-test comments pilots generally favored the helmet display formats over the HUD formats with a few instances where pilots preferred a simple numeric readout of the parameter. Short term exposure effects of the HMD on visual acuity were also measured and showed no advers results.

Clark, Joseph W.↗

NASA NTRSDate not supplied

QuantifyML: How good is my machine learning model?

This paper presents an approach, QuantifyML, which employs model counting to assess the learnability and robustness of machine learning models. Typically the efficacy of machine learning models is determined by computing their accuracy statistically on test data sets. However, this may be misleading, if the test data is not representative of the problem that is being studied. Further, two different models may have the same accuracy on a given data set, measured statistically, but may be very different in their behavior on unseen data. Also, models with high accuracy could have poor adversarial robustness. In QuantifyML, our goal is to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into a C program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise counts w.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, expressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the robustness of trained models around given inputs. Our evaluation demonstrates these applications of QuantifyML on decision trees and neural networks trained to learn relational properties of graphs, for which we know the ground truth, and to perform image classification, for which we do not have the ground truth, but we can quantify local robustness.

Deep Neural Networks↗

NASA NTRS1991

Real-time simulation of helicopter air-to-air combat

The AUTOMAN computer program develops automated maneuvering decisions for helicopters during air-to-air combat over hilly terrain. Recently, the capabilities of this program have been extended and enhanced significantly. The revised program was installed at the NASA Ames manned flight-simulation facility to drive a computer-generated image of an enemy helicopter, thereby providing an adversary for the human pilot. Maneuvers are selected by employing game theory. Enhancements include a guidance law for target acquisition when a firing opportunity arises; fire-control sequence logic; improved low-flying capabilities; line-of-sight computations for the cockpit field-of-view, terrain obstructions, and visual range limits; use of terrain for masking; air-to-air collision-avoidance maneuvers; decision on dispensing flares and chaff; and adjustable levels of pilot experience. The program was found to be extremely useful for both rotorcraft handling-quality evaluations and air-to-air combat training.

Austin, Fred↗