Search NASA⌕ Search

SEARCH · Search NASA

Results for “adversarial evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗

Testing and Evaluation of Security Response at High-Consequence Nuclear Facilities

High-consequence nuclear facilities are required to implement a physical protection system (PPS) capable of preventing a design basis threat (DBT) adversary from completing radiological sabotage or theft of nuclear material. Response forces are a core component of the PPS. A rigorous testing and evaluation program is required to ensure that the response forces are appropriately trained and equipped and are capable of meeting the regulatory and mission requirements.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Methodology Development and Applications of Proliferation Resistance and Physical Protection Evaluation.

We present an overview of the program on the evaluation methodology for proliferation resistance and physical protection (PR&PP) of advanced nuclear energy systems (NESs) sponsored by the Generation IV International Forum (GIF). For a proposed NES design, the methodology defines a set of challenges, analyzes system response to these challenges, and assesses outcomes. The challenges to the NES are the threats posed by potential actors (proliferant States or sub-national adversaries). The characteristics of Generation IV systems, both technical and institutional, are used to evaluate the response of the system and to determine its resistance against proliferation threats and robustness against sabotage and terrorism threats. The outcomes of the system response are expressed in terms of a set of measures, which are the high-level PR&PP characteristics of the NES. The methodology is organized to allow evaluations to be performed at the earliest stages of system design and to become more detailed and more representative as the design progresses. It can thus be used to enable a program in safeguards by design or to enhance the conceptual design process of an NES with regard to intrinsic features for PR&PP.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Development of computer-based analytical tool for assessing physical protection system

Assessment of physical protection system effectiveness is the priority for ensuring the optimum protection caused by unlawful acts against a nuclear facility, such as unauthorized removal of nuclear materials and sabotage of the facility itself. Since an assessment based on real exercise scenarios is costly and time-consuming, the computer-based analytical tool can offer the solution for approaching the likelihood threat scenario. There are several currently available tools that can be used instantly such as EASI and SAPE, however for our research purpose it is more suitable to have the tool that can be customized and enhanced further. In this work, we have developed a computer–based analytical tool by utilizing the network methodological approach for modelling the adversary paths. The inputs are multi-elements in security used for evaluate the effectiveness of the system’s detection, delay, and response. The tool has capability to analyze the most critical path and quantify the probability of effectiveness of the system as performance measure.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Data Augmentation for Intelligent Contingency Management Using Generative Adversarial Neural Networks

Artificial intelligence (AI)-based techniques for intelligent contingency management (ICM) require that intelligent agents learn various aspects of system dynamics to create and execute contingencies. For high assurance contingency management, agents achieve the most compelling results through supervised or semi-supervised machine learning, for which agents require large datasets to learn the dynamics of the system. Unfortunately, data collection in aerospace applications can be costly, due to both time and resources. Presented work describes a framework for data augmentation of ICM databases containing training data for machine learning models. This framework populates the database with the outputs of generative adversarial network (GAN) models that were trained on flight data. Methods for evaluating the suitability of these models based on the equations of motion, as well as other physical constraints, are discussed. The paper demonstrates the utility of this database for training intelligent agents on the NASA T2 generic transport aircraft model and experimental vertical takeoff and landing (VTOL) simulation model.

Generative Machine Learning↗

Data Augmentation for Intelligent Contingency Management Using Generative Adversarial Neural Networks

Artificial intelligence (AI)-based techniques for intelligent contingency management (ICM) require that intelligent agents learn various aspects of system dynamics to create and execute contingencies. For high assurance contingency management, agents achieve the most compelling results through supervised or semi-supervised machine learning, for which agents require large datasets to learn the dynamics of the system. Unfortunately, data collection in aerospace applications can be costly, due to both time and resources. Presented work describes a framework for data augmentation of ICM databases containing training data for machine learning models. This framework populates the database with the outputs of generative adversarial network (GAN) models that were trained on flight data. Methods for evaluating the suitability of these models based on the equations of motion, as well as other physical constraints, are discussed. The paper demonstrates the utility of this database for training intelligent agents on the NASA T2 generic transport aircraft model and experimental vertical takeoff and landing (VTOL) simulation model.

Generative Machine Learning↗

Use of Multi-attribute Utility Functions in Evaluating Security Systems

In analyzing security systems, we are concerned with protecting a building or facility from an attack by an adversary. Typically, we address the possibility that an adversary could enter a building and cause damage resulting in an immediate loss of life, or at least substantial disruption in the operations of the facility. In response to this setting, we implement security systems including devices, procedures, and facility upgrades designed to (a) prevent the adversary from entering, (b) detect and neutralize him if he does enter, and (c) harden the facility to minimize damage if an attack is carried out successfully. Although we have cast this in terms of physical protection of a building, the same general approach can be applied to non-physical attacks such as cyber attacks on a computer system. A rigorous analytic process is valuable for quantitatively evaluating an existing system, identifying its weaknesses, and proposing useful upgrades. As such, in this paper we describe an approach to assess the degree of overall protection provided by security measures. Our approach evaluates the effectiveness of the individual components of the system, describes how the components work together, and finally assesses the degree of overall protection achieved. This model can then be used to quantify the amount of protection provided by existing security measures, as well as to address proposed upgrades to the system and help identify a robust and cost effective set of improvements. Within the model, we use multiattribute utility functions to perform the overall evaluations of the system.

99 GENERAL AND MISCELLANEOUS↗

Report on the AAPM grand challenge on deep generative modeling for learning medical image statistics

Abstract Background The findings of the 2023 AAPM Grand Challenge on Deep Generative Modeling for Learning Medical Image Statistics are reported in this Special Report. Purpose The goal of this challenge was to promote the development of deep generative models for medical imaging and to emphasize the need for their domain‐relevant assessments via the analysis of relevant image statistics. Methods As part of this Grand Challenge, a common training dataset and an evaluation procedure was developed for benchmarking deep generative models for medical image synthesis. To create the training dataset, an established 3D virtual breast phantom was adapted. The resulting dataset comprised about 108 000 images of size 512 512. For the evaluation of submissions to the Challenge, an ensemble of 10 000 DGM‐generated images from each submission was employed. The evaluation procedure consisted of two stages. In the first stage, a preliminary check for memorization and image quality (via the Fréchet Inception Distance [FID]) was performed. Submissions that passed the first stage were then evaluated for the reproducibility of image statistics corresponding to several feature families including texture, morphology, image moments, fractal statistics, and skeleton statistics. A summary measure in this feature space was employed to rank the submissions. Additional analyses of submissions was performed to assess DGM performance specific to individual feature families, the four classes in the training data, and also to identify various artifacts. Results Fifty‐eight submissions from 12 unique users were received for this Challenge. Out of these 12 submissions, 9 submissions passed the first stage of evaluation and were eligible for ranking. The top‐ranked submission employed a conditional latent diffusion model, whereas the joint runners‐up employed a generative adversarial network, followed by another network for image superresolution. In general, we observed that the overall ranking of the top 9 submissions according to our evaluation method (i) did not match the FID‐based ranking, and (ii) differed with respect to individual feature families. Another important finding from our additional analyses was that different DGMs demonstrated similar kinds of artifacts. Conclusions This Grand Challenge highlighted the need for domain‐specific evaluation to further DGM design as well as deployment. It also demonstrated that the specification of a DGM may differ depending on its intended use.

Radiology, Nuclear Medicine & Medical Imaging↗

Side-channel Leakage Assessment Metrics: A Case Study of GIFT Block Ciphers

Determination of an adequate level of security and providing subsequent mechanisms to achieve it, is one of the most pressing problems regarding embedded computing devices. While there are some solutions available for resource-rich computer systems, direct application of these solutions to resource-constrained environments are often unfeasible. The fundamental problem for such resource-constrained systems is the fact that current cryptographic algorithms utilize significant energy consumption and storage overhead. Both the cryptographic algorithm and its physical implementation affect the resilience of a cryptosystem against side-channel attacks. A side-channel attack represents a process that exploits leakages in order to extract sensitive information such as the key. This paper focuses on Correlation Power Analysis (CPA) which is side-channel attack based on the power consumption leakage. In 2016 the U.S. Commerce Department’s National Institute of Standards and Technology (NIST) initiated the call for proposals of new cryptographic algorithms to strengthen the cryptographic defense of networked devices against cyberattacks and to protect the data created by those innumerable device. This work evaluates S-boxes used by NIST candidates PICCOLO, GIFT, and PRESENT, as well as several S-box variants that demonstrated sufficient weaknesses against classical cryptanalysis, for a quantitative comparison in terms of resiliency to CPA attack. Three well-known theoretical metrics are evaluated: transparency order (TO and RTO), nonlinearity, and signal-to-noise (SNR) ratio, aiming to characterize the resistance of these S-boxes against adversaries exploiting physical leakages. Experimental results from attacks on an 8- bit XMEGA were obtained via the ChipWhisperer platform and of all the S-boxes evaluated, GIFT64 with a PICCOLO S-box was found to be the most susceptible to CPA. Results showed that variations in TO and RTO were not sufficient to ensure practical CPA resistance and that among S-boxes with equal non-linearity there were no significant differences in the TO and SNR variants.

97 MATHEMATICS AND COMPUTING↗

Side-channel Leakage Assessment Metrics: A Case Study of GIFT Block Ciphers

Determination of an adequate level of security and providing subsequent mechanisms to achieve it, is one of the most pressing problems regarding embedded computing devices. While there are some solutions available for resource-rich computer systems, direct application of these solutions to resource-constrained environments are often unfeasible. The fundamental problem for such resource-constrained systems is the fact that current cryptographic algorithms utilize significant energy consumption and storage overhead. Both the cryptographic algorithm and its physical implementation affect the resilience of a cryptosystem against side-channel attacks. A side-channel attack represents a process that exploits leakages in order to extract sensitive information such as the key. This paper focuses on Correlation Power Analysis (CPA) which is side-channel attack based on the power consumption leakage. In 2016 the U.S. Commerce Department’s National Institute of Standards and Technology (NIST) initiated the call for proposals of new cryptographic algorithms to strengthen the cryptographic defense of networked devices against cyberattacks and to protect the data created by those innumerable device. This work evaluates S-boxes used by NIST candidates PICCOLO, GIFT, and PRESENT, as well as several S-box variants that demonstrated sufficient weaknesses against classical cryptanalysis, for a quantitative comparison in terms of resiliency to CPA attack. Three well-known theoretical metrics are evaluated: transparency order (TO and RTO), nonlinearity, and signal-to-noise (SNR) ratio, aiming to characterize the resistance of these S-boxes against adversaries exploiting physical leakages. Experimental results from attacks on an 8- bit XMEGA were obtained via the ChipWhisperer platform and of all the S-boxes evaluated, GIFT64 with a PICCOLO S-box was found to be the most susceptible to CPA. Results showed that variations in TO and RTO were not sufficient to ensure practical CPA resistance and that among S-boxes with equal non-linearity there were no significant differences in the TO and SNR variants.

97 MATHEMATICS AND COMPUTING↗

Cybersecurity Assessments on Emulated DER Communication Networks

An increasing number of public utility commissions are adopting Distributed Energy Resource (DER) interconnection standards which require photovoltaic (PV) inverters, energy storage systems, and other DER to include interoperable grid-support functionality. The recently updated national standard, IEEE 1547-2018, requires all DER to include a Sun Spec Modbus, IEEE 2030.5, or IEEE 1815 communication interface in order to provide local and bulk power system services. Those communication protocols and associated information models will ensure system interoperability for PV and storage systems, but these new utility-to-DER communication networks must be deployed with sufficient cybersecurity to protect the U.S. power system and other critical infrastructure reliant on dependable power. Unlike bulk generators, DER are commonly connected to grid operators via public internet channels. These DER networks are exposed to a large attack surface that may leverage sophisticated techniques and infrastructure developed on IT systems, including remote exploits and distributed attacks. Although DER make up a growing portion of the national generation mix, they have limited processing capabilities and do not typically support modern security features such as encryption or authentication. In this work, Sandia National Laboratories constructed simulated DER communication net- works with a range of security features in order to study the security posture of different communication approaches. The experimental test environment was created in a Sandia-developed co-simulation platform, called SCEPTRE, which emulated Sun Spec-compliant DER equipment, the utility DER management system, communication network, and distribution power system. Adversary-based assessments were conducted and a quantitative scoring criteria was applied to evaluate the resilience of various architectures against cyber attacks and to measure the systemic impact during such attacks. The team found that network segmentation, encryption, and moving target defense improved the security of these networks and would be recommended for utility, aggregator, and local DER networks.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Assessment of Physical Security Modeling and Simulation in the Vulnerability Assessment Process

This report provides a comprehensive assessment of physical security modeling and simulation tools available for use in the vulnerability assessment (VA) process for nuclear facilities. It outlines the historical evolution of VA methodologies, emphasizing the transition from traditional layer-based approaches to a more holistic framework that integrates detection probabilities directly into combat simulations. The document details the critical components of the VA process, including the characterization of targets, threats, and protective measures, as well as the development of adversary scenarios that reflect both insider and outsider threats. It highlights the importance of performance assurance programs, emphasizing the need for continuous evaluation and testing of security systems to ensure their effectiveness against evolving threats. Additionally, the report discusses the significance of utilizing accredited modeling and simulation tools in accredited areas to accurately represent adversary actions and the corresponding responses of protective forces. By establishing a systematic approach to VA, this document aims to enhance the overall security posture of nuclear facilities, ensuring compliance with regulatory standards while effectively mitigating risks associated with potential adversarial actions.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Quantum-assisted associative adversarial network: applying quantum annealing in deep learning

Abstract Generative models have the capacity to model and generate new examples from a dataset and have an increasingly diverse set of applications driven by commercial and academic interest. In this work, we present an algorithm for learning a latent variable generative model via generative adversarial learning where the canonical uniform noise input is replaced by samples from a graphical model. This graphical model is learned by a Boltzmann machine which learns low-dimensional feature representation of data extracted by the discriminator. A quantum processor can be used to sample from the model to train the Boltzmann machine. This novel hybrid quantum-classical algorithm joins a growing family of algorithms that use a quantum processor sampling subroutine in deep learning, and provides a scalable framework to test the advantages of quantum-assisted learning. For the latent space model, fully connected, symmetric bipartite and Chimera graph topologies are compared on a reduced stochastically binarized MNIST dataset, for both classical and quantum sampling methods. The quantum-assisted associative adversarial network successfully learns a generative model of the MNIST dataset for all topologies. Evaluated using the Fréchet inception distance and inception score, the quantum and classical versions of the algorithm are found to have equivalent performance for learning an implicit generative model of the MNIST dataset. Classical sampling is used to demonstrate the algorithm on the LSUN bedrooms dataset, indicating scalability to larger and color datasets. Though the quantum processor used here is a quantum annealer, the algorithm is general enough such that any quantum processor, such as gate model quantum computers, may be substituted as a sampler.

Wilson, Max (ORCID:0000000207983391)↗

A Technical Evaluation of Self-Protection Dose Rates in U.S. Domestic Nuclear Security Policy

U.S. nuclear security policy uses radiation self-protection as a basis for reducing material attractiveness, but existing dose-rate criteria may not reflect the time scales of theft or sabotage. This report evaluates whether the commonly cited 1 Gy/h at 1 m criterion can plausibly cause adversary task failure during short-duration malicious acts.

Fritchie, Jacob Wesley [Sandia National Laboratori↗

TomoGAN: low-dose synchrotron x-ray tomography with generative adversarial networks: discussion

Synchrotron-based x-ray tomography is a noninvasive imaging technique that allows for reconstructing the internal structure of materials at high spatial resolutions from tens of micrometers to a few nanometers. In order to resolve sample features at smaller length scales, however, a higher radiation dose is required. Therefore, the limitation on the achievable resolution is set primarily by noise at these length scales. We present TomoGAN, a denoising technique based on generative adversarial networks, for improving the quality of reconstructed images for low-dose imaging conditions. We evaluate our approach in two photon-budget-limited experimental conditions: (1) sufficient number of low-dose projections (based on Nyquist sampling), and (2) insufficient or limited number of high-dose projections. In both cases, the angular sampling is assumed to be isotropic, and the photon budget throughout the experiment is fixed based on the maximum allowable radiation dose on the sample. Evaluation with both simulated and experimental datasets shows that our approach can significantly reduce noise in reconstructed images, improving the structural similarity score of simulation and experimental data from 0.18 to 0.9 and from 0.18 to 0.41, respectively. Furthermore, the quality of the reconstructed images with filtered back projection followed by our denoising approach exceeds that of reconstructions with the simultaneous iterative reconstruction technique, showing the computational superiority of our approach.

42 ENGINEERING↗

Clean Energy Cybersecurity Accelerator Cohort 1: Authentication and Authorization

In the 2023 National Cybersecurity Strategy, the Biden-Harris Administration defines the need for a "defensible, resilient digital ecosystem where it is costlier to attack systems than defend them." The strategy cites the Clean Energy Cybersecurity Accelerator (CECA) as an exemplary effort to bolster the security and resilience of clean energy generation. These efforts help "secure the clean energy grid of the future and [generate] security best practices that extend to other critical infrastructure sectors" and promise broad and far-reaching impacts to bridge the capabilities of private industry and the needs of energy production. Cohort 1 of CECA launched in the fall of 2022 with a focus on solutions that provide strong authentication and authorization for industrial control systems to mitigate attacks on the energy grid. Authentication and authorization verify that the identity (authentication) and permissions (authorization) of a user or device are aligned with their assigned roles. Weaknesses in either can have serious repercussions. To assess the strength of Cohort 1's solutions, CECA devised threat scenarios grounded in historical precedents: the CECA team reviewed exploits from real-world case studies of state-sponsored actors to match the assessment's attack paths and targets. Cohort 1 results provided the energy industry, product vendors, and related agencies valuable insights into the efficacy and applicability of solutions in common system configurations under realistic threat scenarios. The results of the assessment highlight points for interrogation and improvement in subsequent technology iterations. CECA's evaluations are part of an ongoing conversation and collaboration to bolster U.S. cyber resilience against adversaries today and in the future.

24 POWER TRANSMISSION AND DISTRIBUTION↗

CCE and Resilience Strategies

Consequence-driven Cyber-informed Engineering (CCE) is a methodology focused on securing the nation’s critical infrastructure systems. Developed at Idaho National Laboratory, CCE starts with the assumption that if a critical infrastructure system is targeted by a skilled and determined adversary, the targeted network can and will be penetrated. This think like the adversary approach provides critical infrastructure owners, operators, vendors and manufacturers with a disciplined methodology to: evaluate complex systems; determine what must be fully safeguarded; and apply proven engineering strategies to isolate and protect an industry’s most critical assets.

42 ENGINEERING↗

Enhancing Operational Safety via Agentic Dialogue Hazard Identification Analysis

Operational safety in high-stakes domains such as industrial process control, autonomous, and safety-critical systems demand reliable hazard identification. While large language models (LLMs) have shown promise in automating safety analysis tasks, single-turn, monolithic inference is brittle: it lacks the self-correction, deliberation, and contextual refinement that safety engineers apply iteratively. In this paper, we introduce HAZDIAL, a framework that investigates whether structured agentic dialogue (multi-agent, multi-turn interactions) improves the quality of NLP-based hazard identification over single-pass baselines. We systematically compare two dialogue modalities: adversarial debate and constructive discussion, and propose an genetic algorithm-based agentic interaction optimization. We evaluate all configurations against a curated golden dataset using standard classification metrics (accuracy, precision, recall, F1) and a novel dialogue metrics. This work advances the intersection of dialogue systems, multi-agent reasoning, and AI safety, providing empirical evidence for dialogue-driven hazard analysis.

Das, Sanjay [ORNL] (ORCID:0009000542591915)↗