Search NASA⌕ Search

SEARCH · Search NASA

Results for “Model interpretation and visualization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

SAFE-OCC: A novelty detection framework for Convolutional Neural Network sensors and its application in process control

Herein we present a novelty detection framework for Convolutional Neural Network (CNN) sensors that we call Sensor-Activated Feature Extraction One-Class Classification (SAFE-OCC). We show that this framework enables the safe use of computer vision sensors in process control architectures. Emergent control applications use CNN models to map visual data to a state signal that can be interpreted by the controller. Incorporating such sensors introduces a significant system operation vulnerability because CNN sensors can exhibit high prediction errors when exposed to novel (abnormal) visual data. Unfortunately, identifying such novelties in real-time is nontrivial. To address this issue, the SAFE-OCC framework leverages the convolutional blocks of the CNN to create an effective feature space to conduct novelty detection using a desired one-class classification technique. This approach engenders a feature space that directly corresponds to that used by the CNN sensor and avoids the need to derive an independent latent space. We demonstrate the effectiveness of SAFE-OCC via simulated control environments.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Volumetric imaging of the 3D orientation of cellular structures with a polarized fluorescence light-sheet microscope

Polarized fluorescence microscopy is a valuable tool for measuring molecular orientations in biological samples, but techniques for recovering three-dimensional orientations and positions of fluorescent ensembles are limited. We report a polarized dual-view light-sheet system for determining the diffraction-limited three-dimensional distribution of the orientations and positions of ensembles of fluorescent dipoles that label biological structures. We share a set of visualization, histogram, and profiling tools for interpreting these positions and orientations. We model the distributions based on the polarization-dependent efficiency of excitation and detection of emitted fluorescence, using coarse-grained representations we call orientation distribution functions (ODFs). We apply ODFs to create physics-informed models of image formation with spatio-angular point-spread and transfer functions. We use theory and experiment to conclude that light-sheet tilting is a necessary part of our design for recovering all three-dimensional orientations. We use our system to extend known two-dimensional results to three dimensions in FM1-43-labeled giant unilamellar vesicles, fast-scarlet-labeled cellulose in xylem cells, and phalloidin-labeled actin in U2OS cells. Additionally, we observe phalloidin-labeled actin in mouse fibroblasts grown on grids of labeled nanowires and identify correlations between local actin alignment and global cell-scale orientation, indicating cellular coordination across length scales.

Science & Technology - Other Topics↗
DOE OSTI2019

Quantifying Climate and Catchment Control on Hydrological Drought in the Continental United States

The evolution of hydrological drought events is a result of complex (nonlinear) interactions between climate and catchment processes. To investigate such nonlinear relationship, we integrated a machine learning modeling framework based on the random forest (RF) algorithms with an interpretation framework to quantify the role of climate and catchment controls on hydrological drought. More particularly, our framework interprets a built RF machine-learning model to identify dominant variables and visualize their functional dependence and interaction effects on hydrological drought characteristics utilizing concepts of minimal depth, interactive depth, and partial dependence. We test our proposed modeling framework based on a set of 652 continental United States catchments with minimal human interference for a period of 1979–2010. Application of this framework indicated presence of three distinct drought regimes, which includes, Regime 1: droughts with longer duration, less frequent and lesser intensity; Regime 2: droughts with moderate duration, moderate frequency, and moderate intensity; and Regime 3: droughts with shorter duration, more frequent, and more intense. RF algorithm was able to accurately model the drought characteristics (intensity, duration, and number of events) for all the three drought regimes as a function of selected variables. It was observed that the type of dominant variables as well as their nonlinear functional relationship with hydrological droughts characteristics can vary between three selected regimes. Finally, our interpretation framework indicated that catchment characteristics have a significant role in controlling the hydrologic drought for catchments (regime 1), whereas both climate and catchment characteristics control hydrological drought in regimes 2 and 3.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

The weakly nonlinear response and nonaffine interpretation of the Johnson–Segalman/Gordon–Schowalter model

We derive new analytical solutions for the non-affine Johnson-Segalman/Gordon-Schowalter (JS/GS) constitutive equation with a general relaxation kernel in medium-amplitude oscillatory shear (MAOS) deformation. The results show time-strain separable (TSS) nonlinearity, therefore providing new physically-meaningful interpretation to the heuristic TSS nonlinear parameter in MAOS. The upper-convected, lower-convected, and corotational Maxwell models are all subsets of the results presented here. The model assumes that the microscale elements causing stress in the material slip compared to the continuum deformation. We introduce a visualization of the non-affine deformation field that acts on stress-generating elements to reinforce the hysical interpretation of the JS/GS class of models. Lastly, a case study is presented where previously published results, from fitting TSS models to MAOS data, can be re-interpreted based on the concept of non-affine motion of the JS/GS framework.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Unified Language Frontend for Physic-Informed AI/ML

Artificial intelligence and machine learning (AI/ML) are becoming important tools for scientific modeling and simulation as in several other fields such as image analysis and natural language processing. ML techniques can leverage the computing power available in modern systems and reduce the human effort needed to configure experiments, interpret and visualize results, draw conclusions from huge quantities of raw data, and build surrogates for physics based models. Domain scientists in fields like fluid dynamics, microelectronics and chemistry can automate many of their most difficult and repetitive tasks or improve the design times by use of the faster ML-surrogates. However, modern ML and traditional scientific highperformance computing (HPC) tend to use completely different software ecosystems. While ML frameworks like PyTorch and TensorFlow provide Python APIs, most HPC applications and libraries are written in C++. Direct interoperability between the two languages is possible but is tedious and error-prone. In this work, we show that a compiler-based approach can bridge the gap between ML frameworks and scientific software with less developer effort and better efficiency. We use the MLIR (multi-level intermediate representation) ecosystem to compile a pre-trained convolutional neural network (CNN) in PyTorch to freestanding C++ source code in the Kokkos programming model. Kokkos is a programming model widely used in HPC to write portable, shared-memory parallel code that can natively target a variety of CPU and GPU architectures. Our compiler-generated source code can be directly integrated into any Kokkosbased application with no dependencies on Python or cross-language interfaces.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Rheo-Structural Spectroscopy: Fingerprinting the In Situ Response of Fluids to Arbitrary Flow Fields

The objectives of this project were to develop new sample environments, measurement methodologies and associated modeling tools for characterizing the structural response to arbitrarily complex processing flows using small angle scattering, and to apply these new tools for understanding the fundamental physics governing the structuring of anisotropic particulate and polymeric materials under flow histories and conditions relevant to industrial processing flows. The research resulted in the development and implementation of a new sample environment, the fluidic four roll mill (FFoRM), for in situ small angle neutron and X-ray scattering (SANS/SAXS) measurements. These measurements are capable of generating large data sets that “fingerprint” how a complex fluid responds to a wide range of flow histories involving time variations in deformation type and rate. New modeling tools were developed to extract detailed microstructural information from such data sets, including orientation distribution functions and interparticle correlation functions, as well as reduced-order parametric descriptors of these high-dimensional functions that can be used to readily map, visualize and interpret a fluid’s structural response to its flow history. These new tools were applied to a range of model materials involving elongated particle suspensions in order to provide new insights into the physics of how flow couples with orientational and structural order in complex flows, particularly under non-dilute conditions for which no accurate theories currently exist. Using these investigations, we elucidated a number of new insights into the fundamental phenomena driving such process-structure-property relationships. These findings provide guidance for the further development of rheological models, and ultimately can inform the rational and model-based design of flow processes to achieve optimized orientational ordering that is key to the properties and function of a wide range of energy-relevant materials.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Multiscale Patterning from Competing Interactions and Length Scales

We live in an era of complexity marked by impressive new tools powering the scientific method to accelerate discovery, prediction, and control of increasingly complex systems. In common with many disciplines and societal challenges and opportunities, materials and condensed matter sciences are beneficiaries. The volume and fidelity of experimental, computational, and visualization data available, and tools to rapidly interpret them, are remarkable. Conceptual frameworks, including multiscale, multiphysics modeling of this complexity, are fueled by the data and, in turn, guide directions for future experimental and computational strategies. In this spirit, we discuss the importance of competing interactions, length scales, and constraints as pervasive sources of spatio-temporal complexity. We use representative examples drawn from materials and condensed matter, including the important role of elasticity in some technologically important quantum materials.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Edge ML for CAN bus intrusion detection in AVs

Autonomous Vehicles (AVs) are revolutionizing transportation, but their reliance on interconnected cyber-physical systems exposes them to unprecedented cybersecurity risks. This study addresses the critical challenge of detecting real-time cyber intrusions in self-driving vehicles by leveraging a dataset from the Udacity self-driving car project. We simulate four high-impact attack vectors, Denial of Service (DoS), spoofing, replay, and fuzzy attacks, by injecting noise into spatial features (e.g., bounding box coordinates) to replicate adversarial scenarios. We develop and evaluate two lightweight neural network architectures (NN-1 and NN-2) alongside a logistic regression baseline (LG-1) for intrusion detection. The models achieve exceptional performance, with NN-2 attaining an AUC score of 93.15% and 93.15% accuracy, demonstrating their suitability for edge deployment in AV environments. Through explainable AI techniques, we uncover unique forensic fingerprints of each attack type, such as spatial corruption in fuzzy attacks and temporal anomalies in replay attacks, offering actionable insights for feature engineering and proactive defense. Visual analytics, including confusion matrices, ROC curves, and feature importance plots, validate the models' robustness and interpretability. This research sets a new benchmark for AV cybersecurity, delivering a scalable, field-ready toolkit for Original Equipment Manufacturers (OEMs) and policymakers. By aligning intrusion fingerprints with SAE J3061 automotive security standards, we provide a pathway for integrating machine learning into safety-critical AV systems. Our findings underscore the urgent need for security-by-design AI, ensuring that AVs not only drive autonomously but also defend autonomously. This work bridges the gap between theoretical cybersecurity and life-preserving engineering, offering a leap toward safer, more secure autonomous transportation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Conceptual Model Update of Mercury Sources and Flux at Y-12 and Upper East Fork Poplar Creek, Oak Ridge,Tennessee

The interactions between the key processes that control the fate and transport of mercury (Hg) in the sediments, groundwater, and stream water within the Y-12 National Security Complex (Y-12) in Oak Ridge, Tennessee are complex, and many aspects are not well understood. In contaminated environments such as Y-12, conceptual models are frequently developed to aid in visualizing and understanding the dynamic nature of the hydrologic, geochemical, and physical environment. These conceptual models integrate data in an internally consistent manner to understand processes that control the fate and transport of contaminants. Over the past few decades of environmental investigation at Y-12, a number of conceptual models have been developed to identify and define various technical processes at various scales. The main purpose of the activities described in this report is to update the 2011 conceptual model for Upper East Fork Poplar Creek using the most recent Hg concentration and flux data and informed scientific interpretation. Emphasis was placed on improving the previous model by implementing a more detailed spatial and temporal approach to visualize transport pathways in the watershed and trends in flux and concentration over time, and to compare baseflow and stormflow system dynamics. Detailed descriptions of historical and current Hg sources and transport pathways are also provided for the east and west ends of the facility. This conceptual model will allow the US Department of Energy to evaluate past and present remedial activities and provide a strong technical basis for prioritizing and optimizing remedial responses in a cost-effective and efficient manner. Twenty years ago, Hg flux from Outfall (OF) 200 at the headwaters of East Fork Poplar Creek represented approximately 20% of the overall flux leaving Y-12. By the time of the 2011 conceptual model report, that percentage had increased to 70%–80%. Flux estimates compiled for this report using data from 2009 to 2018 show that OF200 accounted for roughly 60% of the flux leaving Y-12. The relative role of the four storm drain conduits to OF200 appeared similar to 2011, with OF163 being the greatest contributor to downstream flux. An increase in Hg concentrations and flux occurred throughout and downstream of west end storm drains in 2011 in response to a storm drain cleanout. Leading up to the 2011 storm drain cleanout, annual baseflow flux was higher at OF200A6 than at Station 17. From 2011 to 2018, the trend reversed with concentrations and flux being higher at Station 17 than at OF200A6. A spike in concentrations occurred in July 2018, potentially due to a one-time influx of Hg that occurred during COLEX decontamination and decommissioning activities at Y-12. This report includes recommendations for improving our conceptual understanding of Hg sources, transport pathways, and flux at Y-12. Important recommendations include further evaluation of transport pathways, concurrent measurement of Hg concentration and flux under baseflow and stormflow conditions at multiple sites, and research to understand the connections among periphyton and methylmercury concentration, bioavailability, and bioaccumulation. Ongoing facility decontamination and decommissioning efforts, site characterization, remedial actions, and research are facilitating the collection of new Hg data at Y-12. The conceptual model update provided herein takes advantage of the extensive recent Hg sampling to provide an updated assessment of Hg mass balance and support a refined understanding of Hg behavior at or near Y-12. This assessment should assist in future environmental management decisions and in mitigating the impacts of Hg on the surrounding environment.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2023

Making Corgis Important for Honeycomb Classification: Adversarial Attacks on Concept-based Explainability Tools

Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. Concept-based interpretability techniques, which use a small set of human-interpretable concept exemplars in order to measure the influence of a concept on a model's internal representation of input, are an important thread in this line of research. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based interpretability methods: TCAV and faceted feature visualization. We show that by leveraging the geometry of the problem and carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method. The attacks that we propose can either induce positive interpretations (polka dots are an important concept for a model when classifying zebras) or negative interpretations (stripes are not an important factor in identifying images of a zebra). Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.

Brown, Davis R.↗
DOE OSTI2021

DeepGRN: prediction of transcription factor binding site across cell-types using attention-based deep neural networks

Abstract Background Due to the complexity of the biological systems, the prediction of the potential DNA binding sites for transcription factors remains a difficult problem in computational biology. Genomic DNA sequences and experimental results from parallel sequencing provide available information about the affinity and accessibility of genome and are commonly used features in binding sites prediction. The attention mechanism in deep learning has shown its capability to learn long-range dependencies from sequential data, such as sentences and voices. Until now, no study has applied this approach in binding site inference from massively parallel sequencing data. The successful applications of attention mechanism in similar input contexts motivate us to build and test new methods that can accurately determine the binding sites of transcription factors. Results In this study, we propose a novel tool (named DeepGRN) for transcription factors binding site prediction based on the combination of two components: single attention module and pairwise attention module. The performance of our methods is evaluated on the ENCODE-DREAM in vivo Transcription Factor Binding Site Prediction Challenge datasets. The results show that DeepGRN achieves higher unified scores in 6 of 13 targets than any of the top four methods in the DREAM challenge. We also demonstrate that the attention weights learned by the model are correlated with potential informative inputs, such as DNase-Seq coverage and motifs, which provide possible explanations for the predictive improvements in DeepGRN. Conclusions DeepGRN can automatically and effectively predict transcription factor binding sites from DNA sequences and DNase-Seq coverage. Furthermore, the visualization techniques we developed for the attention modules help to interpret how critical patterns from different types of input features are recognized by our model.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics With Temporal Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis

Detecting and analyzing complex patterns in multivariate time-series data is crucial for decision-making in urban and environmental system operations. However, challenges arise from the high dimensionality, intricate complexity, and interconnected nature of complex patterns, which hinder the understanding of their underlying physical processes. Existing AI methods often face limitations in interpretability, computational efficiency, and scalability, reducing their applicability in real-world scenarios. This paper proposes a novel visual analytics framework that integrates two generative AI models, Temporal Fusion Transformer (TFT) and Variational Autoencoders (VAEs), to reduce complex patterns into lower-dimensional latent spaces and visualize them in 2D using dimensionality reduction techniques such as PCA, t-SNE, and UMAP with DBSCAN. These visualizations, presented through coordinated and interactive views and tailored glyphs, enable intuitive exploration of complex multivariate temporal patterns, identifying patterns’ similarities and uncover their potential correlations for a better interpretability of the AI outputs. The framework is demonstrated through a case study on power grid signal data, where it identifies multi-label grid event signatures, including faults and anomalies with diverse root causes. Additionally, novel metrics and visualizations are introduced to validate the models and assess the performance, efficiency, and consistency of latent maps generated by VAE, which have been utilized in prior studies for latent space cartography and used as a benchmark in this study, and the emerging TFT architecture under various configurations. These analyses provide actionable insights for model parameter tuning and reliability improvements. Comparative results highlight that TFT achieves shorter run times and superior scalability to diverse time-series data shapes compared to VAE. This work advances fault diagnosis in multivariate time series, fostering explainable AI to support critical system operations.

Explainable AI↗
DOE OSTI2026

A procedure for rule extraction from a Self-Organising plasma disruption predictor for JET

In a previous paper, a Self-Organizing Map had proven to be able to identify the regions of the plasma operative space characterizing the pre-disruptive phase at JET without relying on any a priori information. One of the strengths of this disruption predictor lies in its inherent self-organization capability. The Self-Organizing Map discovers non-trivial relationships and captures the complicated interplay of device diagnostics on the internal plasma states directly from the experimental data. Moreover, the provided model allows the visualization of high-dimensional plasma parameters and facilitates easy interrogation of the model to understand the reasons behind its correlations. In this paper, an additional step is taken towards the interpretability of models for predicting disruptions by training a Decision Tree to classify the plasma states according to the interpretation provided by the Self-Organizing Map (stable or at high risk of disruptions). The Decision tree provides a set of rules which describe the transition of the plasma towards the pre-disruptive phase as visualized in the Self-Organizing Map. The obtained rules for the database explored in the study identify four regions in the map, two of which are at risk of disruption. These regions correspond to partitions of a 3D space based on the peaking factors of the core and divertor radiation, as well as the Locked Mode. The agreement between the Self-Organizing Map answers and the rules supplied by the Decision Tree is confirmed by the comparison of the performance exhibited by the two models in the prediction of disruptions.

Setzu, Samuele [Univ. of Cagliari, Monserrato, Cag↗
DOE OSTI2019

Scalable Topological Data Analysis and Visualization for Evaluating Data-Driven Models in Scientific Applications

With the rapid adoption of machine learning techniques for large-scale applications in science and engineering comes the convergence of two grand challenges in visualization. First, the utilization of black box models (e.g., deep neural networks) calls for advanced techniques in exploring and interpreting model behaviors. Second, the rapid growth in computing has produced enormous datasets that require techniques that can handle millions or more samples. Although some solutions to these interpretability challenges have been proposed, they typically do not scale beyond thousands of samples, nor do they provide the high-level intuition scientists are looking for. Here, we present the first scalable solution to explore and analyze high-dimensional functions often encountered in the scientific data analysis pipeline. By combining a new streaming neighborhood graph construction, the corresponding topology computation, and a novel data aggregation scheme, namely topology aware datacubes, we enable interactive exploration of both the topological and the geometric aspect of high-dimensional data. Following two use cases from high-energy-density (HED) physics and computational biology, we demonstrate how these capabilities have led to crucial new insights in both applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Toward Smart, Automated Junctional Tourniquets—AI Models to Interpret Vessel Occlusion at Physiological Pressure Points

Hemorrhage is the leading cause of preventable death in both civilian and military medicine. Junctional hemorrhages are especially difficult to manage since traditional tourniquet placement is often not possible. Ultrasound can be used to visualize and guide the caretaker to apply pressure at physiological pressure points to stop hemorrhage. However, this process is technically challenging, requiring the vessel to be properly positioned over rigid boney surfaces and applying sufficient pressure to maintain proper occlusion. As a first step toward automating this life-saving intervention, we demonstrate an artificial intelligence algorithm that classifies a vessel as patent or occluded, which can guide a user to apply the appropriate pressure required to stop flow. Neural network models were trained using images captured from a custom tissue-mimicking phantom and an ex vivo swine model of the inguinal region, as pressure was applied using an ultrasound probe with and without color Doppler overlays. Using these images, we developed an image classification algorithm suitable for the determination of patency or occlusion in an ultrasound image containing color Doppler overlay. Separate AI models for both test platforms were able to accurately detect occlusion status in test-image sets to more than 93% accuracy. In conclusion, this methodology can be utilized for guiding and monitoring proper vessel occlusion, which, when combined with automated actuation and other AI models, can allow for automated junctional tourniquet application.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

I Can’t Read All That! Improving the Usability of Semantic Models Using Concise, Ontology-Agnostic, Building-Specific Schemas

Semantic ontologies have enabled the creation of formalized, machine-readable descriptions of heterogenous building systems by providing dictionaries of well defined concepts that can be applied to model them. Within a semantic model of a particular building, a subset of an ontology's concepts may be applied in different ways to represent a particular perspective of the building's systems. How the concepts were applied can only be understood by examining the large amount of instance data within a semantic model, which leads to usability challenges. We propose a concise, ontology-agnostic method for defining building-specific schema (b-schema) graphs that summarize the structure and content of a semantic model. This approach provides a queryable and concise representation of the model's contents, separate from the instance data within a model, that can mitigate the challenges posed by the size and complexity of semantic models in processes such as visualization, querying, validation, and the use of large language models (LLMs). We validate our approach on semantic models based on the Brick and ASHRAE S223 ontologies. Results demonstrate that b-schemas significantly reduce the complexity of visual interpretation, accelerate SPARQL queries and SHACL validation, and improve LLM-based knowledge graph question answering.

Paul, Lazlo [Lawrence Berkeley National Laboratory↗
DOE OSTI2024

Investigating Mountain Watershed Headwater‐To‐Groundwater Connections, Water Sources, and Storage Selection Behavior With Dynamic‐Flux Particle Tracking

Abstract Climate change will impact mountain watershed streamflow both directly—with changing precipitation amounts and variability—and indirectly—through temperature shifts altering snowpack, melt, and evapotranspiration. To understand how these complex processes will affect ecosystem functioning and water resources, we need tools to distinguish connections between water sources (rain/snowmelt), groundwater storage, and exit fluxes (streamflow/evapotranspiration), and to determine how these connections change seasonally and as climate shifts. Here, we develop novel watershed‐scale approaches to understand water source, storage, and exit flux connections using a dynamic‐flux particle tracking model (EcoSLIM) applied in California's Cosumnes Watershed, which connects the Sierra Nevada and Central Valley. This work develops new visualizations and applications to provide mechanistic understanding that underpins the interpretation of isotopic field data at watershed scales to distinguish sources, flow paths, residence times, and storage selection. In our simulations, streamflow comes primarily from snow‐derived water while evapotranspiration generally comes from rain. Most streamflow starts above 1,000 m while evapotranspiration is sourced relatively evenly across the watershed and is generally younger than streamflow. Modeled streamflow consists primarily of water sourced from precipitation in the previous 5 years but before the current water year, while ET consists primarily of water from precipitation in the current water year. ET, and to a lesser extent streamflow, are both younger than water in groundwater storage. However, snowmelt‐derived streamflow preferentially discharges older water from snow‐derived storage. Dynamic‐flux particle tracking and new approaches presented here enable novel model‐tracer comparisons in large‐scale watersheds to better understand watershed behavior in a changing climate.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding

Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain’s high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding.

Wei, Yuxiang [Georgia Institute of Technology]↗