SEARCH · Search NASA
Results for “Explanability”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A human-in-the-loop explanation framework for morphologically transparent AI predictions from whole-slide images
Deep learning models enable the prediction of clinical endpoints from whole-slide images (WSIs), but many such models function as “black boxes”, lacking transparency about whether and which histomorphological patterns drive their predictions, hindering interpretability and clinical adoption. Here we propose a human-in-the-loop explanation framework, MorphoXAI, which provides both local and global interpretability for deep learning models by incorporating human-expert interpretations. At the global level, it reveals the histomorphological patterns on which the model consistently relies to distinguish between classes of WSIs, as well as the patterns associated with confusion between classes. At the local level, it indicates which of these patterns are used in the prediction of an individual WSI and which regions within the slide correspond to such patterns. We validated our method across multiple deep learning–based WSI analysis tasks spanning different tissue types. The results show that our framework generates explanations that accurately reflect the histomorphology underlying the model’s predictions at both global and local levels. For interpretability and clinical utility in diagnostic contexts, human evaluation results showed that our explanations were easy to interpret, rich in diagnostic features, and directly helpful for diagnostic decision-making, thereby enhancing pathologist-AI collaboration. Our work highlights that unifying global and local explanations and grounding them in expert-interpreted morphology enhances the interpretability and verifiability of deep learning models, thereby facilitating the transparent deployment of such models in clinical practice.
Robust Explanations using Diverse Adversarially Trained Ensembles, Multi-Modal Contrastive Learning, and Attribution-based Confidence Metrics
The primary objective of this project is to strengthen the trustworthiness of AI systems by designing algorithms that make their internal decision-making processes more understandable to human users. This involves creating clear, interpretable explanations for AI decisions and developing metrics to assess these explanations' validity and reliability. Significant progress has been achieved through (i) developing symbolic explanations, (ii) generating meaningful interpretive insights, (iii) establishing accuracy and confidence metrics, and (iv) devising methods to evaluate the knowledge boundaries of AI models. To date, the research findings have been shared in peer-reviewed publications, with accompanying scientific and technical information (STI) detailed below.
Explanation of the seasonal variation of cosmic multiple muon events observed with the NOvA Near Detector
The flux of cosmic ray muons at the Earth’s surface exhibits seasonal variations due to changes in the temperature of the atmosphere affecting the production and decay of mesons in the upper atmosphere. Using 1473 live days of data collected by the NuMI Off-axis 𝜈 𝑒 Appearance (NOvA) Near Detector during 2018–2022, we studied the seasonal pattern in the multiple-muon event rate. The data confirm an anticorrelation between the multiple-muon event rate and effective atmospheric temperature, consistent across all the years of data. Previous analyses from MINOS and NOvA saw a similar anticorrelation but did not include an explanation. We find that this anticorrelation is driven by altitude–geometry effects as the average muon production height changes with the season. This has been studied with a CORSIKA cosmic ray simulation package by varying atmospheric parameters, and provides an explanation to a longstanding discrepancy between the seasonal phases of single and multiple-muon events.
Integrating adaptive learning with post hoc model explanation and symbolic regression to build interpretable surrogate models
Abstract We develop a materials informatics workflow to build an interpretable surrogate model for micromagnetic simulations. Our goal is to predict the energy barrier of a moving isolated skyrmion in rare-earth-free $$\hbox {Mn}_4$$ Mn 4 N. Our approach integrates adaptive learning with post hoc model explanation and symbolic regression methods. We discuss an unexplored acquisition function (information condensing active learning) within the adaptive learning loop and compare it with the known standard deviation function for efficient navigation of the search space. Model-agnostic post hoc explanation techniques then uncover trends learned by the trained model, which we then leverage to constrain the expressions used for symbolic regression. Graphical abstract
Testing Explanations of Short Baseline Neutrino Anomalies
The experimental observation of neutrino oscillations profoundly impacted the physics of neutrinos, from being well understood theoretically to requiring new physics beyond the standard model of particle physics. Indeed, the mystery of neutrino masses implies the presence of new particles never observed before, often called sterile neutrinos, as they would not undergo standard weak interactions. And while neutrino oscillation measurements entered the precision era, reaching percent-level precision, many experimental results show significant discrepancies with the standard model, at baselines much shorter than typical oscillation baselines, like LSND, MiniBooNE, gallium experiments, and reactor antineutrino measurements. These short baseline anomalies could be explained by the addition of a light sterile neutrino, with mass in the $1-10~\text{eV}$ range, however, in strong tension with many null experimental observations. Other explanations that rely on sterile neutrinos with masses in the $1-500~\text{MeV}$ could resolve the tension. Here we test both classes of models. On the one hand, we look for datasets collected at a short baseline which can constrain heavy sterile neutrino models. We find that the minimal model is fully constrained, but several extensions of this model could weaken the current constraint and be tested with current and future datasets. On the other hand, we test the presence of neutrino oscillations at short baselines, induced by a light sterile state, with the data collected by the MicroBooNE experiment, a liquid argon time projection chamber specifically designed to resolve the details of each neutrino interaction. We report null results from both analyses, further constraining the space of possible explanations for the short baseline anomalies. If new physics lies behind the short baseline anomaly puzzle, it is definitely not described by a simple model.
Machine learning model explanation apparatus and methods
Explanation apparatus and methods are described. In one aspect, an explanation apparatus includes processing circuity configured to access a source instance which has been classified by a machine learning model; create associations of the source instance with a plurality of training instances; and process the associations of the source instance and the training instances to identify a first subset of the training instances which have less relevance to the classification decision of the source instance by the machine learning model compared with a second subset of the training instances; and an interface configured to communicate information to a user, and wherein the processing circuitry is configured to control the user interface to communicate the second subset of the training instances to the user as evidence to explain the classification of the source instance by the machine learning model.
Turbulent drifts of impurity ions as an explanation for anomalous radial transport in the far-SOL of DIII-D
Abstract Successful fusion reactor operation relies on minimal core contamination by impurities, otherwise too much power may be radiated and harm performance. This requires reliable predictions of impurity transport from the scrape-off layer (SOL) into the core, beyond the traditional ‘anomalous’ diffusion approach. We report a set of far-SOL tungsten transport simulations that demonstrate the role of turbulent drifts on radial impurity transport. A turbulent plasma background is simulated using the gyrokinetic SOL code Gkeyll. Tungsten ions are followed within the plasma background using only their drifts. We find that tungsten tends to travel radially outwards with velocities between v r = 300–1200 m s −1 primarily due to polarization drift. We also extract an anomalous radial diffusion coefficient that varies from D r anom = 5–20 m 2 s −1 . These results are compared to and agree with previous interpretive modeling results. We also show how the turbulent polarization drift can transport some tungsten ions from the wall inwards with effective pinch velocities up to 10 000 m s −1 . We conclude that turbulent drifts are a likely explanation for historically anomalous radial impurity transport.
NuGraph2 with explainability: post-hoc explanations for geometric neural network predictions
With the growing popularity of artificial intelligence (AI) used for scientific applications, the ability of attribute a result to a reasoning process from the network is in high demand for robust scientific generalizations to hold. In this work we aim to motivate the need for and demonstrate the use of post-hoc explainability methods when applied to AI methods used in scientific applications. To this end, we introduce explainability add-ons to the existing graph neural network (GNN) for neutrino tagging, NuGraph2. The explanations take the form of a suite of techniques examining the output of the network (node classifications) and the edge connections between them, and probing of the latent space using novel general-purpose tools applied to this network. We show how none of these methods are singularly sufficient to show network ‘understanding’, but together can give insights into the processes used in classification. While these methods are tested on the NuGraph2 application, they can be applied to a broad range of networks, not limited to GNNs. The code for this work is publicly available on GitHub at https://github.com/voetberg/XNuGraph.
First Search for Dark Sector 𝑒 + 𝑒 − Explanations of the MiniBooNE Anomaly at MicroBooNE
We present MicroBooNE’s first search for dark sector 𝑒 + 𝑒 − explanations of the long-standing MiniBooNE anomaly. The MiniBooNE anomaly has garnered significant attention over the past 20 years including previous MicroBooNE investigations into both anomalous electron and photon excesses, but its origin still remains unclear. In this Letter, we provide the first direct test of dark sector models in which dark neutrinos, produced through neutrino-induced scattering, decay into missing energy and visible 𝑒 + 𝑒 − pairs comprising the MiniBooNE anomaly. Many such models have recently gained traction as a viable solution to the anomaly while evading past bounds. Using an exposure of 6.87 × 10 20 protons-on-target in the Booster Neutrino Beam, we implement a selection targeting forward-going, coherently produced 𝑒 + 𝑒 − events. After unblinding, we observe 95 events, which we compare with the constrained background-only prediction of 69.7 ±17.3. This analysis sets the world’s first direct limits on these dark sector models and, at the 95% confidence level, excludes the entirety of the single dark neutrino and majority of the dual dark neutrino, parameter space that is viable as a solution to the MiniBooNE anomaly.
Experimental Search for Neutron to Mirror Neutron Oscillations as an Explanation of the Neutron Lifetime Anomaly
We report an unexplained >4σ discrepancy persists between “beam” and “bottle” measurements of the neutron lifetime. A new model proposed that conversions of neutrons n into mirror neutrons n', part of a dark mirror sector, can increase the apparent neutron lifetime by 1% via a small mass splitting Δm between n and n' inside the 4.6 T magnetic field of the National Institute of Standards and Technology Beam Lifetime experiment. A search for neutron conversions in a 6.6 T magnetic field was performed at the Spallation Neutron Source which excludes this explanation for the neutron lifetime discrepancy.
Probing the Pulsar Explanation of the Galactic-Center GeV Excess Using Continuous Gravitational-Wave Searches
Over 10 years ago, Fermi observed an excess of GeV gamma rays from the Galactic Center whose origin is still under debate. One explanation for this excess involves annihilating dark matter, another requires an unresolved population of millisecond pulsars concentrated at the Galactic Center. In this work, we use the results from LIGO and Virgo’s most recent all-sky search for quasimonochromatic, persistent gravitational-wave signals from isolated neutron stars, which is estimated to be about 20%–50% of the population, to determine whether unresolved millisecond pulsars could actually explain this excess. First, we choose a luminosity function that determines the number of millisecond pulsars required to explain the observed excess. Then, we consider two models for deformations on millisecond pulsars to determine their ellipticity distributions, which are directly related to their gravitational-wave radiation. Lastly, based on null results from the O3 frequency-Hough all-sky search for continuous gravitational waves, we find that a large set of the parameter space in the pulsar luminosity function can be excluded. We also evaluate how these exclusion regions may change with respect to various model choices. Further, our results are the first of their kind and represent a bridge between gamma-ray astrophysics, gravitational-wave astronomy, and dark-matter physics.
XSub: Explanation-Driven Adversarial Attack against Blackbox Classifiers via Feature Substitution
Despite its significant benefits in enhancing the transparency and trustworthiness of artificial intelligence (AI) systems, explainable AI (XAI) can unintentionally provide adversaries with insights into blackbox models, increasing their vulnerability to various attacks. In this paper, we develop a novel explanation-driven adversarial attack against blackbox classifiers based on feature substitution, called XSub. The key idea of XSub is to strategically replace important features (identified via XAI) in the original sample with corresponding important features of a different label, thereby increasing the likelihood of the model misclassifying the perturbed sample. XSub only requires a minimal number of queries and can be easily extended to launch backdoor attacks in case the attacker has access to the model's training data. Our evaluation shows that XSub is not only effective and stealthy but also low-cost, showcasing its feasibility across a wide range of AI applications.
Gamma-ray Spectrum Explanations (GRSE) v0.1
This code repository provides tools for producing various types of machine learning model explanations trained on gamma-ray spectra. It includes implementations of saliency mapping, Grad-CAM, LIME, and Kernel SHAP, including improvements and clarifications to some of the approaches that are specific to their use with gamma-ray spectral data. These tools are in support of the manuscript M.S. Bandstra et al., "Explaining machine-learning models for gamma-ray detection and identification," under review at PLOS ONE.
Explaining the Explanation: In Search of Understanding for Saliency Maps [Slides]
Pixel attribution methods are the most popular way to present explanations for image classifiers.
First Search for Dark Sector e+e- Explanations of the MiniBooNE Anomaly at MicroBooNE
We present MicroBooNE’s first search for dark sector e⁺e⁻ explanations of the long-standing MiniBooNE anomaly. The MiniBooNE anomaly has garnered significant attention over the past 20 years including previous MicroBooNE investigations into both anomalous electron and photon excesses, but its origin still remains unclear. In this talk we present the first direct test of dark sector models in which dark neutrinos, produced through neutrino-induced scattering, decay into missing energy and visible e⁺e⁻ pairs that could comprise the MiniBooNE anomaly. Many such models have recently gained traction as a viable solution to the anomaly while evading past bounds. Using an exposure of 6.87×$10^20$ protons-on-target in the Booster Neutrino Beam, we implement a selection targeting forward-going, coherently produced e⁺e⁻ events to study this possibility. This talk will present the results of this targeted dark sector search and summarize the latest three recently presented MicroBooNE photon results, along with discussing their connection to this result.
Testing and Characterization to Develop a Mechanistic Explanation for Unsaturated Drift of Fiber Optic Sensors during High-Dose Irradiation
The primary limitation for any optical fiber-based sensor for nuclear reactor applications is radiation-induced attenuation (RIA) of the transmitted and/or reflected signals. Based on several recent studies, RIA is tolerable for some fused silica optical fibers with the proper choice of sensing wavelength and fiber dopants. For extreme temperature applications (> 1000°C), sapphire optical fibers have been proposed; however, recent optical transmission measurements performed on bulk sapphire samples showed prohibitively large RIA. For some sensors, radiation-induced dimensional changes in the fiber materials can also cause significant drift. Moreover, the drift that was observed in numerous experiments performed in the High Flux Isotope Reactor (HFIR), the Advanced Test Reactor, the Massachusetts Institute of Technology Reactor, and other international facilities far exceeded what would be expected based on compaction of fused silica glass. Clearly, additional work is needed to better understand the origins of both RIA and radiation-induced drift in both silica and sapphire optical fiber-based sensors before these sensors can be reliably deployed for nuclear applications. This work evaluated the underlying mechanisms that may be responsible for RIA and drift in silica and sapphire materials. First, detailed characterization was performed on bulk fused silica glass samples that were previously irradiated to different neutron fluences at different temperatures to better understand the structural changes that drive radiation-induced drift in the absence of coating effects that are discussed later. Results show that the non-monotonic compaction that occurs with increasing neutron fluence continues up to fast neutron fluences approaching 10 22 n/cm 2 , which has important implications for physics-based models that may be used to compensate for the sensor drift. Initial Raman spectroscopy and synchrotron x-ray diffraction provide insights into the nature of the structural changes. Next, detailed characterizations were performed on silica fibers with various coatings that were subjected to several different thermal treatments. The hypothesis is that the coatings convert to carbon-rich materials that compact under irradiation, putting a large compressive strain on the fiber. Out-of-pile testing confirms that both polyimide and acrylate fiber coatings convert to glassy carbon (GC) materials when heated under inert conditions, and the degree of order (i.e., graphitization) increases with increasing temperature. The results provide increasingly strong evidence that the combination of polymeric coatings and inert (or vacuum) conditions render fiber optic sensors susceptible to significant radiation-induced drift that would not otherwise exist in uncoated fibers. Finally, transmission electron microscopy was performed on bulk sapphire samples that were irradiated to two neutron fluences at different temperatures to gain insights into the potential mechanisms driving the prohibitive RIA at higher neutron fluences and temperatures. Contrary to previous hypotheses, results show that scattering from radiation-induced voids cannot explain the observed RIA. Similarly, models for scattering losses from dislocation loops also do not agree with the experimental results. Instead, fitting to the experimental data shows that increased absorption from aluminum vacancy centers is the most likely explanation for the the prohibitively large RIA that was observed at high irradiation temperature and dose. In addition, the voids that formed in these single-crystal samples were found to align along the basal plane (a-axis) as opposed to that seen in previous observations of c-axis alignment in polycrystalline samples, which could have important implications for anisotropic swelling and other phenomena that could affect sensor performance at high neutron fluence.
Testing and Characterization to Develop a Mechanistic Explanation for Unsaturated Drift of Fiber Optic Sensors during High-Dose Irradiation
The primary limitation for any optical fiber-based sensor for nuclear reactor applications is radiation-induced attenuation (RIA) of the transmitted and/or reflected signals. Based on several recent studies, RIA is tolerable for some fused silica optical fibers with the proper choice of sensing wavelength and fiber dopants. For extreme temperature applications (> 1000°C), sapphire optical fibers have been proposed; however, recent optical transmission measurements performed on bulk sapphire samples showed prohibitively large RIA. For some sensors, radiation-induced dimensional changes in the fiber materials can also cause significant drift. Moreover, the drift that was observed in numerous experiments performed in the High Flux Isotope Reactor (HFIR), the Advanced Test Reactor, the Massachusetts Institute of Technology Reactor, and other international facilities far exceeded what would be expected based on compaction of fused silica glass. Clearly, additional work is needed to better understand the origins of both RIA and radiation-induced drift in both silica and sapphire optical fiber-based sensors before these sensors can be reliably deployed for nuclear applications. This work evaluated the underlying mechanisms that may be responsible for RIA and drift in silica and sapphire materials. First, detailed characterization was performed on bulk fused silica glass samples that were previously irradiated to different neutron fluences at different temperatures to better understand the structural changes that drive radiation-induced drift in the absence of coating effects that are discussed later. Results show that the non-monotonic compaction that occurs with increasing neutron fluence continues up to fast neutron fluences approaching 10 22 n/cm 2 , which has important implications for physics-based models that may be used to compensate for the sensor drift. Initial Raman spectroscopy and synchrotron x-ray diffraction provide insights into the nature of the structural changes. Next, detailed characterizations were performed on silica fibers with various coatings that were subjected to several different thermal treatments. The hypothesis is that the coatings convert to carbon-rich materials that compact under irradiation, putting a large compressive strain on the fiber. Out-of-pile testing confirms that both polyimide and acrylate fiber coatings convert to glassy carbon (GC) materials when heated under inert conditions, and the degree of order (i.e., graphitization) increases with increasing temperature. The results provide increasingly strong evidence that the combination of polymeric coatings and inert (or vacuum) conditions render fiber optic sensors susceptible to significant radiation-induced drift that would not otherwise exist in uncoated fibers. Finally, transmission electron microscopy was performed on bulk sapphire samples that were irradiated to two neutron fluences at different temperatures to gain insights into the potential mechanisms driving the prohibitive RIA at higher neutron fluences and temperatures. Contrary to previous hypotheses, results show that scattering from radiation-induced voids cannot explain the observed RIA. Similarly, models for scattering losses from dislocation loops also do not agree with the experimental results. Instead, fitting to the experimental data shows that increased absorption from aluminum vacancy centers is the most likely explanation for the the prohibitively large RIA that was observed at high irradiation temperature and dose. In addition, the voids that formed in these single-crystal samples were found to align along the basal plane (a-axis) as opposed to that seen in previous observations of c-axis alignment in polycrystalline samples, which could have important implications for anisotropic swelling and other phenomena that could affect sensor performance at high neutron fluence.