Search NASA⌕ Search

SEARCH · Search NASA

Results for “adversarial evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Community detection robustness of graph neural networks

Graph neural networks (GNNs) are increasingly widely used for community detection in attributed networks. They combine structural topology with node attributes through message passing and pooling. However, their robustness or lack thereof with respect to different perturbations and targeted attacks in conjunction with community detection tasks is not well understood. To shed light on latent mechanisms behind GNN sensitivity on community detection tasks, we conduct a systematic computational evaluation of six widely adopted GNN architectures graph convolutional network, graph attention network, graph sample and aggregate (GraphSAGE), differentiable pooling (DiffPool), minimum cut pooling (MinCUT), and deep modularity networks (DMoN). The analysis covers three perturbation categories: node attribute manipulations, edge topology distortions, and adversarial attacks. We use element-centric similarity as the evaluation metric on synthetic benchmarks and real-world citation networks. Our findings indicate that supervised GNNs tend to achieve higher baseline accuracy, while unsupervised methods, particularly DMoN, maintain stronger resilience under targeted and adversarial perturbations. Furthermore, robustness appears to be strongly influenced by community strength, with well-defined communities reducing performance loss. Across all models, node attribute perturbations associated with targeted edge deletions and shifts in attribute distributions tend to cause the largest degradation in community recovery. These findings highlight important trade-offs between accuracy and robustness in GNN-based community detection and offer insights into selecting architectures resilient to noise and adversarial attacks.

Goel, Jaidev [Virginia Polytechnic Inst. and State↗

Partially Observable Games for Secure Autonomy

Technology development efforts in autonomy and cyber-defense have been evolving independently of each other, over the past decade. In this paper, we report our ongoing effort to integrate these two presently distinct areas into a single framework. To this end, we propose the two-player partially observable stochastic game formalism to capture both high-level autonomous mission planning under uncertainty and adversarial decision making subject to imperfect information. We show that synthesizing sub-optimal strategies for such games is possible under finite-memory assumptions for both the autonomous decision maker and the cyber-adversary. We then describe an experimental testbed to evaluate the efficacy of the proposed framework.

Ames, Aaron D.↗

Dante

SAND2022-12905 O Dante is a physical security simulation package used in evaluating protective staffing and physical security elements against adversarial threats. Dante simulates force-on-force scenarios in a batch Monte Carlo process. Analysts use the included Scenario Editor to define the scenarios for their site and then execute and analyze the results using the Batch Manager tool. Dante provides insight into how well a security system performs against threats and lets analysts mine the results for key indicators of system performance. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Hart, Brian↗

Cyber/Physical Security Vulnerability Assessment Integration

Securing high value and critical assets is one of the biggest challenges facing this nation and others around the world. In modern integrated systems, there are four potential modes of attack available to an adversary: • physical only attack, • cyber only attack, • physical-enabled cyber attack, • cyber-enabled physical attack. Blended attacks involve an adversary working in one domain to reduce system effectiveness in another domain. This enables the attacker to penetrate further into the overall layered defenses. Existing vulnerability assessment (VA) processes and software tools which predict facility vulnerabilities typically evaluate the physical and cyber domains separately. Vulnerabilities which result from the integration of cyber-physical control systems are not well characterized and are often overlooked by existing assessment approaches. In this paper, we modified modification of the timely detection methodology, used for decades in physical security VAs, to include cyber components. The Physical and Cyber Risk Analysis Tool (PACRAT) prototype illustrates an integrated vulnerability assessment that includes cyber-physical interdependencies. Information about facility layout, network topology, and emplaced safeguards is used to evaluate how well suited a facility is to detect, delay, and respond to attacks, to identify the pathways most vulnerable to attack, and to evaluate how often safeguards are compromised for a given threat or adversary type. We have tested the PACRAT prototype on critical infrastructure facilities and the results are promising. Future work includes extending the model to prescribe the recommended security improvements via an automated cost-benefit analysis.

MacDonald, Douglas G.↗

Cycle-Consistent Adversarial Networks for Realistic Pervasive Change Generation in Remote Sensing Imagery

This paper introduces a new method of generating realistic pervasive changes in the context of evaluating the effectiveness of change detection algorithms in controlled settings. The method - a cycle-consistent adversarial network (CycleGAN) - requires low quantities of training data to generate realistic changes. Here we show an application of CycleGAN in creating realistic snow-covered scenes of multispectral Sentinel-2 imagery, and demonstrate how these images can be used as a test bed for anomalous change detection algorithms.

97 MATHEMATICS AND COMPUTING↗

GRUMDN: A Multi-Task Model for Predicting Human Patterns-of-Life from Stay Transition Data

Understanding human patterns-of-life (PoL) is essential towards ensuring safe and secure indoor facility environment as well as outdoor urban environment. Prediction of human movement in between places of interest is vital in understanding human PoL. Movement between spaces maybe represented and detected in one of the two forms: 1) trajectories: locations measured at regular time intervals by mobile sensors, bluetooth or GPS sensors; or 2) stay transitions: semantic PoI (points of interest) and stay duration data measurable by eventbased sensors that collect data when a check-in or check-out event is detected. Stay transition data provides a more compressed data format compared to trajectories data, especially in situations with longer stay durations, while preserving the information necessary for PoL analysis. Now as introduced briefly in the paper, our deployed end application (Digital Twin of a facility with non-player characters, besides the interactive user in virtual reality) needed a well-performing and validated AI/ML model for simulating high quality stay transitions behavior. In this study we thus primarily present our findings with developing and validating that model, which is a multi-task neural network for stay transition prediction. The neural network consists of two heads, for corresponding two tasks of stay category prediction and stay duration prediction. We evaluated gated recurrent units and multi-layer perceptrons of varying network sizes for stay category prediction; while mixture density networks, noisy generator-only networks, and generative adversarial networks of varying network sizes for stay duration prediction. We have then evaluated four multi-task models, constructed by combining these specialized models, on their ability to predict stay transition data. We tested our models on datasets from two different cases: 1) a simulation-generated dataset of indoor movement within the HFIR (high flux isotope reactor) nuclear reactor facility at Oak Ridge National Laboratory (ORNL); and 2) the GeoLife human mobility dataset of outdoor urban movement available in literature. Our results indicate that GRUMDN, which combines gated recurrent units (GRU) for stay category prediction task, and mixture density networks (MDN) for stay duration prediction task, did overall outperform other multitask models and the current state-of-the-art.

Gunaratne, Chathika [ORNL] (ORCID:0000000225088745↗

Security-by-Design: Light Water Small Modular Reactor

The growing demand for nuclear power is increasing pressure to find solutions to cost prohibitive requirements of both construction and security. Offsite response has been proposed as an option to reduce costs associated with training and maintaining an onsite response force. A previous report explored this option and revealed that security could be provided at the required level, but cost savings was not a result of this methodology. An offsite response strategy required costly active and passive delay barriers to provide sufficient time for responders to muster and deploy to a site in time to interrupt a determined and well-equipped adversary. Also, contrary to the hypothesis, the number of responders required for this strategy exceeded that needed for an onsite response force, as the adversaries could avail themselves of advantageous positions within the facility to repel arriving responders. This report builds upon the previous evaluation by using the same hypothetical light water small modular reactor (LWSMR) facility model, but this time an onsite response strategy was assessed. The goal of this analysis was to show that an onsite response strategy could be implemented effectively at a cost point that removes barriers within the industry at this critical time of growth and development. The assessment of the facility design and response strategy was completed through modeling using Scribe3D© and subsequent scenario analysis over the course of a two-day tabletop exercise. Subject matter experts in nuclear security, nuclear facility design, and response strategy and tactics contributed to the effort to ensure accurate representation of hypothetical scenarios. Several adaptations were made to the layout of the LWSMR based on lessons learned during the first day of scenario analysis. The subsequent design evaluated on the second day proved to provide a robust response posture against a large and well-trained adversary force. This report details the process of the analysis and compares the cost of the final facility design with that of the LWSMR model used for evaluation of offsite response. Ultimately, the results of this effort indicate that, when implemented correctly, an onsite response strategy is the best option from a security and cost perspective.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition

Over the past decade, the machine learning security community has developed a myriad of defenses for evasion attacks. An understudied question in that community is: for whom do these defenses defend? This work considers common approaches to defending learned systems and how security defenses result in performance inequities across different sub-populations. We outline appropriate parity metrics for analysis and begin to answer this question through empirical results of the fairness implications of machine learning security methods. We find that many methods that have been proposed can cause direct harm, like false rejection and unequal benefits from robustness training. The framework we propose for measuring defense equality can be applied to robustly trained models, preprocessing-based defenses, and rejection methods. We identify a set of datasets with a user-centered application and a reasonable computational cost suitable for case studies in measuring the equality of defenses. In our case study of speech command recognition, we show how such adversarial training and augmentation have non-equal but complex protections for social subgroups across gender, accent, and age in relation to user coverage. We present a comparison of equality between two rejection-based defenses: randomized smoothing and neural rejection, finding randomized smoothing more equitable due to the sampling mechanism for minority groups. This represents the first work examining the disparity in the adversarial robustness in the speech domain and the fairness evaluation of rejection-based defenses.

• Artificial intelligence (AI) / machine learning ↗

Probabilistic Risk Assessment of Spent Fuel Pool Cooling System Failure Considering Safety and Security Initiating Events

In order to quantify the probable risk associated with a safety system, nuclear safety risk evaluation process is applied which is generally performed in three steps. The first step is to calculate the damage frequency of a critical system. This step is followed by the radioactive source term estimation due to the potential system damage. The final step is the assessment of off-site consequences. The safety risk formula is shown by Eq. (1). P{sub F} is the probability of safety system failure (the probability used describes a random, independent event), and C is the consequence value. R{sub SAF} = P{sub F} * C (1) The security risk associated with an attack on critical infrastructure and/or special nuclear materials theft is usually evaluated using three- term risk equation (probability of attack, probability of adversary success, and consequence). A security threat and the related initiating events are not considered in a pure safety risk analysis. The vulnerability of the physical protection system is performed independently even though the consequence resulting from a security initiating event can be similar to a safety related initiating event and thus gets double counted. The security risk can be calculated using Eq. (2) R{sub SEC} = P{sub A} * P{sub S} * C (2) PA is the probability of attack, P{sub S} is the probability of adversary success (P{sub S} = 1 - P{sub I} P{sub N}). P{sub I} being the probability of interruption which accounts for the probabilities of detection of all timely detection elements along an adversary path. PN being the probability of neutralization: the result of engagement of the response force with the adversaries after the interruption has occurred. C is the consequence magnitude from a successful attack. The two security probabilities do not describe random, independent events. As a result, because of the differences in the randomness and independence of the respective safety and security probabilities, the safety and the security risks are currently calculated separately. The consequence term in nuclear safety and security risk evaluation have the same definition. An effort is made here to study the effect on the overall risk by considering security initiating events along with the subsequent failure of the intermediate safety element. The target selected is a spent fuel storage pool for analyzing the public risk arising from the interplay between nuclear security and safety initiating events. EASI (Estimate of Adversary Sequence Interruption) model is used to calculate the probability of interruption related to a security system that leads to the calculation of the security initiating event frequency. Further progression of this initiating security event to a top event is analyzed using a Probabilistic Safety Assessment (PSA) tool, SAPHIRE (version 8) to obtain a joint pool damage frequency. For the progression of events leading to a top event, the failure of the intermediate safety elements is assumed to be external random events such as heavy load drop and fire events. The objectives of this study is to (1) determine the double accounting of frequency of failure, risk and allocated resources in the scenario of a non-random security initiating event leading to a safety random event chain (2) compare the damage frequency from pure security with the frequency of combined security-safety failure event and pure safety failure events. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Human reliability-based MC&A models for detecting insider theft.

Material control and accounting (MC&A) safeguards operations that track and account for critical assets at nuclear facilities provide a key protection approach for defeating insider adversaries. These activities, however, have been difficult to characterize in ways that are compatible with the probabilistic path analysis methods that are used to systematically evaluate the effectiveness of a site's physical protection (security) system (PPS). MC&A activities have many similar characteristics to operator procedures performed in a nuclear power plant (NPP) to check for anomalous conditions. This work applies human reliability analysis (HRA) methods and models for human performance of NPP operations to develop detection probabilities for MC&A activities. This has enabled the development of an extended probabilistic path analysis methodology in which MC&A protections can be combined with traditional sensor data in the calculation of PPS effectiveness. The extended path analysis methodology provides an integrated evaluation of a safeguards and security system that addresses its effectiveness for attacks by both outside and inside adversaries.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Semantic Segmentation of High-Resolution Satellite Imagery using Generative Adversarial Networks with Progressive Growing

With increase in urbanization and Earth Sciences research into urban areas, the need to quickly and accurately segment urban rooftop maps has never been greater. Cur-rent machine learning techniques struggle to produce high accuracy maps in dense urban zones where there is high image noise and foot print overlap. In this paper, we evaluate a training methodology for pixel-wise segmentation for high resolution satellite imagery using progressive growing of generative adversarial networks as a solution. We apply our model to segmenting building rooftops and compare these results to conventional methods for rooftop segmentation. We evaluate our approach using the SpaceNet version 2 and xView datasets. Our experiments show that for SpaceNet, progressive Generative Adversarial Network (GAN) training achieved a test accuracy of 93% compared to 89% for traditional GAN training and 87% for U-Net architecture, while for xView, we achieved 71% accuracy using progressive GAN training compared to 69% through traditional GAN training and 65% using U-Net.

Semantic↗

Deep reinforcement learning with online data augmentation to improve sample efficiency for intelligent HVAC control

Deep Reinforcement Learning (DRL) has started showing success in real-world applications such as building energy optimization. Much of the research in this space utilized simulated environments to train RL-agent in an offline mode. Very few research have used DRL-based control in real-world systems due to two main reasons: 1) sample efficiency challenge---DRL approaches need to perform a lot of interactions with the environment to collect sufficient experiences to learn from, which is difficult in real systems, and 2) comfort or safety related constraints---user's comfort must never or at least rarely be violated. In this work, we propose a novel deep Reinforcement Learning framework with online Data Augmentation (RLDA) to address the sample efficiency challenge of real-world RL. We used a time series Generative Adversarial Network (TimeGAN) architecture as a data generator. We further evaluated the proposed RLDA framework using a case study of an intelligent HVAC control. With a ≈28% improvement in the sample efficiency, RLDA framework lays the way towards increased adoption of DRL-based intelligent control in real-world building energy management systems.

Kurte, Kuldeep↗

Temporal Cyber Attack Detection.

Rigorous characterization of the performance and generalization ability of cyber defense systems is extremely difficult, making it hard to gauge uncertainty, and thus, confidence. This difficulty largely stems from a lack of labeled attack data that fully explores the potential adversarial space. Currently, performance of cyber defense systems is typically evaluated in a qualitative manner by manually inspecting the results of the system on live data and adjusting as needed. Additionally, machine learning has shown promise in deriving models that automatically learn indicators of compromise that are more robust than analyst-derived detectors. However, to generate these models, most algorithms require large amounts of labeled data (i.e., examples of attacks). Algorithms that do not require annotated data to derive models are similarly at a disadvantage, because labeled data is still necessary when evaluating performance. In this work, we explore the use of temporal generative models to learn cyber attack graph representations and automatically generate data for experimentation and evaluation. Training and evaluating cyber systems and machine learning models requires significant, annotated data, which is typically collected and labeled by hand for one-off experiments. Automatically generating such data helps derive/evaluate detection models and ensures reproducibility of results. Experimentally, we demonstrate the efficacy of generative sequence analysis techniques on learning the structure of attack graphs, based on a realistic example. These derived models can then be used to generate more data. Additionally, we provide a roadmap for future research efforts in this area.

97 MATHEMATICS AND COMPUTING↗

Modularization of Small Modular Reactor Facilities: Physical Protection Recommendations

U.S. nuclear power facilities face increasing challenges in meeting dynamic security requirements caused by evolving and expanding threats while keeping costs reasonable to make nuclear energy competitive. In consideration of the economic viability of small modular reactor (SMR) facilities, many designers and utilities are considering a modular approach to the construction and operation of these facilities. This modular approach considers building and operating a first unit; once the first reactor is in operation, construction will begin on a second reactor. This process would allow the vendor or utility to ensure production of energy and financial income while the second reactor is under construction. This project evaluates the feasibility of modular construction in terms of physical protection and identifies several recommendations for vendors and utilities considering this approach. To conduct this evaluation, a hypothetical three-unit SMR facility was developed, a physical protection system (PPS) was considered for the design, and a hypothetical design basis threat (DBT) was used to evaluate this PPS. Multiple outsider sabotage scenarios were examined, with adversary team sizes ranging from 4-to-8 to determine security system effectiveness. The results of this work will influence PPS designs and facility designs for U.S. domestic SMRs. This work will also demonstrate how a series of experimental and modeling capabilities across the Department of Energy (DOE) complex can impact the design and completion of security-by-design (SeBD) for SMRs considering modular construction. The conclusions and recommendations in this document may be applicable to all SMR designs.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

The Grey Zone Test Range Integrated Urban Simulation Environment

Sandia National Laboratories is part of the government test and evaluation team for the Defense Advanced Research Projects Agency Collection and Monitoring via Planning for Active Situational Scenarios program. The program is designed to better understand competition in the area between peace and conventional conflict when adversary actions are subtle and difficult to detect. For the purposes of test and evaluation, Sandia conducted a range of activities for the program: creation of the Grey Zone Test Range; design of the data stream for a user experiment conducted with U.S. Indo-Pacific Command; design, implementation, and execution of the formal evaluation; and analysis and summary of the evaluation results. This report details Sandia's activities and provides additional information on the Grey Zone Test Range urban simulation environment developed to evaluate the performer technologies.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Utilization of Synthetic Near-Infrared Spectra via Generative Adversarial Network to Improve Wood Stiffness Prediction

Near-infrared (NIR) spectroscopy is widely used as a nondestructive evaluation (NDE) tool for predicting wood properties. When deploying NIR models, one faces challenges in ensuring representative training data, which large datasets can mitigate but often at a significant cost. Machine learning and deep learning NIR models are at an even greater disadvantage because they typically require higher sample sizes for training. In this study, NIR spectra were collected to predict the modulus of elasticity (MOE) of southern pine lumber (training set = 573 samples, testing set = 145 samples). To account for the limited size of the training data, this study employed a generative adversarial network (GAN) to generate synthetic NIR spectra. The training dataset was fed into a GAN to generate 313, 573, and 1000 synthetic spectra. The original and enhanced datasets were used to train artificial neural networks (ANNs), convolutional neural networks (CNNs), and light gradient boosting machines (LGBMs) for MOE prediction. Overall, results showed that data augmentation using GAN improved the coefficient of determination (R 2 ) by up to 7.02% and reduced the error of predictions by up to 4.29%. ANNs and CNNs benefited more from synthetic spectra than LGBMs, which only yielded slight improvement. All models showed optimal performance when 313 synthetic spectra were added to the original training data; further additions did not improve model performance because the quality of the datapoints generated by GAN beyond a certain threshold is poor, and one of the main reasons for this can be the size of the initial training data fed into the GAN. LGBMs showed superior performances than ANNs and CNNs on both the original and enhanced training datasets, which highlights the significance of selecting an appropriate machine learning or deep learning model for NIR spectral-data analysis. The results highlighted the positive impact of GAN on the predictive performance of models utilizing NIR spectroscopy as an NDE technique and monitoring tool for wood mechanical-property evaluation. Further studies should investigate the impact of the initial size of training data, the optimal number of generated synthetic spectra, and machine learning or deep learning models that could benefit more from data augmentation using GANs.

59 BASIC BIOLOGICAL SCIENCES↗

Attack Surface of Wind Energy Technologies in the United States

Low cost, reliable electrical energy production from wind relies upon automation and control systems, arguably more so than traditional thermal generation. These same systems, however, can serve as the target of adversaries’ cyber-attacks. Idaho National Laboratory (INL), at the request of the Department of Energy’s (DOE’s) Office of Cybersecurity, Energy Security, and Emergency Response (CESER) and Energy Efficiency and Renewable Energy’s (EERE’s) Wind Energy Technologies Office (WETO), evaluated a generalized wind plant architecture to understand the classes of potential threat actors and the vectors that could enable a cyber-attack. This evaluation explores the attack surface of a representative wind plant, identifying potential methods and vectors that an adversary could leverage to conduct a cyber-attack. Included in this assessment are some recommended mitigations and approaches. Each recommendation requires a full security evaluation, cost/benefit analysis, and risk analysis by each owner and operator.

17 WIND ENERGY↗

Generation IV PR and PP Methods and Applications

This paper presents an evaluation methodology for proliferation resistance and physical protection (PR&PP) of Generation IV nuclear energy systems (NESs). For a proposed NES design, the methodology defines a set of challenges, analyzes system response to these challenges, and assesses outcomes. The challenges to the NES are the threats posed by potential actors (proliferant States or sub-national adversaries). The characteristics of Generation IV systems, both technical and institutional, are used to evaluate the response of the system and determine its resistance against proliferation threats and robustness against sabotage and terrorism threats. The outcomes of the system response are expressed in terms of six measures for PR and three measures for PP, which are the high-level PR&PP characteristics of the NES. The methodology is organized to allow evaluations to be performed at the earliest stages of system design and to become more detailed and more representative as design progresses. Uncertainty of results are recognized and incorporated into the evaluation at all stages. The results are intended for three types of users: system designers, program policy makers, and external stakeholders. Particular current relevant activities will be discussed in this regard. The methodology has been illustrated in a series of demonstration and case studies and these will be summarized in the paper.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗