Search NASA⌕ Search

SEARCH · Search NASA

Results for “multimodal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Pretraining Billion-Scale Geospatial Foundational Models on Frontier

As AI workloads increase in scope, generalization capability becomes challenging for small task-specific models and their demand for large amounts of labeled training samples increases. On the contrary, Foundation Models (FMs) are trained with internet-scale unlabeled data via self-supervised learning and have been shown to adapt to various tasks with minimal fine-tuning. Although large FMs have demonstrated significant impact in natural language processing and computer vision, efforts toward FMs for geospatial applications have been restricted to smaller size models, as pretraining larger models requires very large computing resources equipped with state-of-the-art hardware accelerators. Current satellite constellations collect 100+TBs of data a day, resulting in images that are billions of pixels and multimodal in nature. Such geospatial data poses unique challenges opening up new opportunities to develop FMs. We investigate billion scale FMs and HPC training profiles for geospatial applications by pretraining on publicly available data. We studied from end-to-end the performance and impact in the solution by scaling the model size. Our larger 3B parameter size model achieves up to 30% improvement in top1 scene classification accuracy when comparing a 100M parameter model. Moreover, we detail performance experiments on the Frontier supercomputer, America's first exascale system, where we study different model and data parallel approaches using PyTorch's Fully Sharded Data Parallel library. Specifically, we study variants of the Vision Transformer architecture (ViT), conducting performance analysis for ViT models with size up to 15B parameters. By discussing throughput and performance bottlenecks under different parallelism configurations, we offer insights on how to leverage such leadership-class HPC resources when developing large models for geospatial imagery applications.

Tsaris, Aristeidis (aris)↗

Connecting Femtosecond Transient Absorption Microscopy with Spatially Coregistered Time Averaged Optical Imaging Modalities

Multimodal all-optical imaging involving coregistered femtosecond transient absorption microscopy (TAM), time-integrated photoluminescence (PL), and steady-state modalities such as confocal reflectance and transmission offers an appealing approach to gain a comprehensive understanding of complex electronic excited-state phenomena in spatially heterogeneous systems. A unique combination of these modalities allows us to unravel not only the competing electronic excited-state dynamical processes but also the underlying morphological information with simultaneous high temporal and spatial resolution. However, correlating the various images obtained from time-resolved and time-independent modalities is generally nontrivial and particularly challenging when the electronic dynamics under study evolve in both time and space. Here, we demonstrate a new approach for rationally correlating time-resolved microscopy with coregistered time-integrated or steady-state modalities. Specifically, our approach involves an extended global lifetime analysis of the time-resolved microscopic data set to separate distinct dynamical processes taking place on commensurate time scales, and the resulting decay-associated amplitude maps (DAAMs) were applied to explore correlations with the images acquired using time-independent modalities. The feasibility of our approach was validated through analyzing a multimodal data set acquired from a thin film of chloride-containing mixed lead halide perovskites (CH 3 NH 3 PbI 3–x Cl x ) using femtosecond transient absorption, time-integrated PL, and confocal reflectance microscopies. Analysis of the results obtained enable us to gain new insight into the complex ultrafast relaxation dynamics in this highly heterogeneous system.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Multimodal X-ray nano-spectromicroscopy analysis of chemically heterogeneous systems

Abstract Understanding the nanoscale chemical speciation of heterogeneous systems in their native environment is critical for several disciplines such as life and environmental sciences, biogeochemistry, and materials science. Synchrotron-based X-ray spectromicroscopy tools are widely used to understand the chemistry and morphology of complex material systems owing to their high penetration depth and sensitivity. The multidimensional (4D+) structure of spectromicroscopy data poses visualization and data-reduction challenges. This paper reports the strategies for the visualization and analysis of spectromicroscopy data. We created a new graphical user interface and data analysis platform named XMIDAS (X-ray multimodal image data analysis software) to visualize spectromicroscopy data from both image and spectrum representations. The interactive data analysis toolkit combined conventional analysis methods with well-established machine learning classification algorithms (e.g. nonnegative matrix factorization) for data reduction. The data visualization and analysis methodologies were then defined and optimized using a model particle aggregate with known chemical composition. Nanoprobe-based X-ray fluorescence (nano-XRF) and X-ray absorption near edge structure (nano-XANES) spectromicroscopy techniques were used to probe elemental and chemical state information of the aggregate sample. We illustrated the complete chemical speciation methodology of the model particle by using XMIDAS. Next, we demonstrated the application of this approach in detecting and characterizing nanoparticles associated with alveolar macrophages. Our multimodal approach combining nano-XRF, nano-XANES, and differential phase-contrast imaging efficiently visualizes the chemistry of localized nanostructure with the morphology. We believe that the optimized data-reduction strategies and tool development will facilitate the analysis of complex biological and environmental samples using X-ray spectromicroscopy techniques.

36 MATERIALS SCIENCE↗

Toward a Multimodal-Deep Learning Retrieval System for Monitoring Nuclear Proliferation Activities

As the amount of available data increases, the human ability to locate, process, and analyze it is strained and eventually overwhelmed. To address this challenge for nonproliferation analysts, the authors have been creating a large-scale multimodal retrieval system to help analysts triage and search open source science, technology, and news data for indicators of nuclear proliferation capabilities and activities. The system relies on a set of deep neural networks (DNNs) trained to evaluate conceptual similarities across data modalities, such as text, image, and video. These DNNs can be used to search and prioritize data, according to a nuclear fuel cycle (NFC) process template, that are conceptually closest to the seed query items regardless of data modality. The DNNs have been trained to map conceptually related words, sentences, and images to nearby points in a multimodal feature space, enabling intra- and intermodal retrieval via nearest-neighbor calculations to seed query points. The authors evaluate the system’s ability to retrieve NFC-related data that have been purposely hidden in collections of unrelated background data. They demonstrate quantitative and qualitative results for text-to-image, image-to-image, and image-to-video retrievals. This paper discusses data challenges confronting nonproliferation analysts, describes the DNNs designed to retrieve multimodal data that are proximal in the semantic space, and demonstrates their effectiveness by applying the DNNs to nonproliferation-specific multimodal data sets.

97 MATHEMATICS AND COMPUTING↗

Challenges at the Frontiers of Matter and Energy: Transformative Opportunities for Discovery Science

FIVE TRANSFORMATIVE OPPORTUNITIES FOR DISCOVERY SCIENCE As a result of this effort, it has become clear that the progress made to date on the five Grand Challenges has created a springboard for seizing five new Transformative Opportunities that have the potential to further transform key technologies involving matter and energy. These five new Transformative Opportunities and the evidence supporting them are discussed in this new report, “Challenges at the Frontiers of Matter and Energy: Transformative Opportunities for Discovery Science.” Mastering Hierarchical Architectures and Beyond-Equilibrium Matter Complex materials and chemical processes transmute matter and energy, for example from CO2 and water to chemical fuel in photosynthesis, from visible light to electricity in solar cells and from electricity to light in light emitting diodes (LEDs) Such functionality requires complex assemblies of heterogeneous materials in hierarchical architectures that display time-dependent away-from-equilibrium behaviors. Much of the foundation of our understanding of such transformations however, is based on monolithic single- phase materials operating at or near thermodynamic equilibrium. The emergent functionalities enabling next-generation disruptive energy technologies require mastering the design, synthesis, and control of complex hierarchical materials employing dynamic far-from-equilibrium behavior. A key guide in this pursuit is nature, for biological systems prove the power of hierarchical assembly and far- from-equilibrium behavior. The challenges here are many: a description of the functionality of hierarchical assemblies in terms of their constituent parts, a blueprint of atomic and molecular positions for each constituent part, and a synthesis strategy for (a) placing the atoms and molecules in the proper positions for the component parts and (b) arranging the component parts into the required hierarchical structure. Targeted functionality will open the door to significant advances in the harvesting, transforming (e.g., reducing CO2, splitting water, and fixing nitrogen), storing, and use of energy to create new materials, manufacturing processes, and technologies—the lifeblood of human societies and economic growth. Beyond Ideal Materials and Systems: Understanding the Critical Roles of Heterogeneity, Interfaces, and Disorder Real materials, both natural ones and those we engineer, are usually a complex mixture of compositional and structural heterogeneities, interfaces, and disorder across all spatial and temporal scales. It is the fluctuations and disorderly states of these heterogeneities and interfaces that often determine the system’s properties and functionality. Much of our fundamental scientific knowledge is based on “ideal” systems, meaning materials that are observed in “frozen” states or represented by spatially or temporally averaged states. Too often, this approach has yielded overly simplistic models that hide important nuances and do not capture the complex behaviors of materials under realistic conditions. These behaviors drive vital chemical transformations such as catalysis, which initiates most industrial manufacturing processes, and friction and corrosion, the parasitic effects of which cost the U.S. economy billions of dollars annually. Expanding our scientific knowledge from the relative simplicity of ideal, perfectly ordered, or structurally averaged materials to the true complexity of real-world heterogeneities, interfaces, and disorder should enable us to realize enormous benefits in the materials and chemical sciences, which translates to the energy sciences, including solar and nuclear power, hydraulic fracturing, power conversion, airframes, and batteries. Harnessing Coherence in Light and Matter Quantum coherence in light and matter is a measure of the extent to which a wave field vibrates in unison with itself at neighboring points in space and time. Although this phenomenon is expressed at the atomic and electronic scales, it can dominate the macroscopic properties of materials and chemical reactions such as superconductivity and efficient photosynthesis. In recent years, enormous progress has been made in recognizing, manipulating, and exploiting quantum coherence. This progress has already elucidated the role that symmetry plays in protecting coherence in key materials, taught us how to use light to manipulate atoms and molecules, and provided us with increasingly sophisticated techniques for controlling and probing the charges and spins of quantum coherent systems. With the arrival of new sources of coherent light and electron beams, thanks in large part to investments by the U.S. Department of Energy’s Office of Basic Energy Sciences (BES), there is now an opportunity to engineer coherence in heterostructures that incorporate multiple types of materials and to control complex, multistep chemical transformations. This approach will pave the way for quantum information processing and next-generation photovoltaic cells and sensors. Revolutionary Advances in Models, Mathematics, Algorithms, Data, and Computing Science today is benefiting from a convergence of theoretical, mathematical, computational, and experimental capabilities that put us on the brink of greatly accelerating our ability to predict, synthesize, and control new materials and chemical processes, and to understand the complexities of matter across a range of scales. Imagine being able to chart a path through a vast sea of possible new materials to find a select few with desired properties. Instead of the time-honored forward approach, in which materials with desired properties are found through either trial-and-error experiments or lucky accidents, we have the opportunity to inversely design and create new materials that possess the properties we desire. The traditional approach has allowed us to make only a tiny fraction of all the materials that are theoretically possible. The inverse design approach, through the harmonious convergence of theoretical, mathematical, computational, and experimental capabilities, could usher in a virtual cornucopia of new materials with functionalities far beyond what nature can provide. Similarly, enhanced mathematical and computational capabilities significantly enhance our ability to extract physical and chemical insights from vastly larger data streams gathered during multimodal and multidimensional experiments using advanced characterization facilities. Exploiting Transformative Advances in Imaging Capabilities across Multiple Scales Historically, improvements in imaging capabilities have always resulted in improved understanding of scientific phenomena. A prime challenge today is finding ways to reconstruct raw data, obtained by probing and mapping matter across multiple scales, into analyzable images. BES investments in new and improved imaging facilities, most notably synchrotron x-ray sources, free-electron lasers, electron microscopes, and neutron sources, have greatly advanced our powers of observation, as have substantial improvements in laboratory- scale technologies. Furthermore, BES is now planning or actively discussing exciting new capabilities. Taken together, these advances in imaging capabilities provide an opportunity to expand our ability to observe and study matter from the 3D spatial perspectives of today to true “4D” spatially and temporally resolved maps of dynamics that allow quantitative predictions of time-dependent material properties and chemical processes. The knowledge gained will impact data storage, catalyst design, drug delivery, structural materials, and medical implants, to name just a few key technologies. ENABLING SUCCESS Seizing each of these five Transformative Opportunities, as well as accelerating further progress on Grand Challenge research, will require specific, targeted investments from BES in the areas of synthesis, meaning the ability to make the materials and architectures that are envisioned; instrumentation and tools, a category that includes theory and computation; and human capital, the most important asset for advancing the Grand Challenges and Transformative Opportunities. While “Challenges at the Frontiers of Matter and Energy: Transformative Opportunities for Discovery Science” could be viewed as a sequel to the original Grand Challenges report, it breaks much new ground in its assessment of the scientific landscape today versus the scientific landscape just a few years ago. In the original Grand Challenges report, it was noted that if the five Grand Challenges were met, our ability to direct matter and energy would be measured only by the limits of human imagination. This new report shows that, prodded by those challenges, the scientific community is positioned today to seize new opportunities whose impacts promise to be transformative for science and society, as well as dramatically accelerate progress in the pursuit of the original Grand Challenges.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Could a neuroscientist understand a microprocessor?

There is a popular belief in neuroscience that we are primarily data limited, and that producing large, multimodal, and complex datasets will, with the help of advanced data analysis algorithms, lead to fundamental insights into the way the brain processes information. These datasets do not yet exist, and if they did we would have no way of evaluating whether or not the algorithmically-generated insights were sufficient or even correct. To address this, here we take a classical microprocessor as a model organism, and use our ability to perform arbitrary experiments on it to see if popular data analysis methods from neuroscience can elucidate the way it processes information. Microprocessors are among those artificial information processing systems that are both complex and that we understand at all levels, from the overall logical flow, via logical gates, to the dynamics of transistors. We show that the approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor. This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data. Furthermore, we argue for scientists using complex non-linear dynamical systems with known ground truth, such as the microprocessor as a validation platform for time-series and structure discovery methods.

60 APPLIED LIFE SCIENCES↗

Alternative Vegetation States in Tropical Forests and Savannas: The Search for Consistent Signals in Diverse Remote Sensing Data

Globally, the spatial distribution of vegetation is governed primarily by climatological factors (rainfall and temperature, seasonality, and inter-annual variability). The local distribution of vegetation, however, depends on local edaphic conditions (soils and topography) and disturbances (fire, herbivory, and anthropogenic activities). Abrupt spatial or temporal changes in vegetation distribution can occur if there are positive (i.e., amplifying) feedbacks favoring certain vegetation states under otherwise similar climatic and edaphic conditions. Previous studies in the tropical savannas of Africa and other continents using the MODerate Resolution Imaging Spectroradiometer (MODIS) vegetation continuous fields (VCF) satellite data product have focused on discontinuities in the distribution of tree cover at different rainfall levels, with bimodal distributions (e.g., concentrations of high and low tree cover) interpreted as alternative vegetation states. Such observed bimodalities over large spatial extents may not be evidence for alternate states, as they may include regions that have different edaphic conditions and disturbance histories. In this study, we conduct a systematic multi-scale analysis of diverse MODIS data streams to quantify the presence and spatial consistency of alternative vegetation states in Sub-Saharan Africa. The analysis is based on the premise that major discontinuities in vegetation structure should also manifest as consistent spatial patterns in a range of remote sensing data streams, including, for example, albedo and land surface temperature (LST). Our results confirm previous observations of bimodal and multimodal distributions of estimated tree cover in the MODIS VCF. However, strong disagreements in the location of multimodality between VCF and other data streams were observed at 1 km scale. Results suggest that the observed distribution of VCF over vast spatial extents are multimodal, not because of local-scale feedbacks and emergent bifurcations (the definition of alternative states), but likely because of other factors including regional scale differences in woody dynamics associated with edaphic, disturbance, and/or anthropogenic processes. These results suggest the need for more in-depth consideration of bifurcation mechanisms and thus the likely spatial and temporal scales at which alternative states driven by different positive feedback processes should manifest.

Savanna↗

Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)

Leveraging the chemical data available in legacy formats such as publications and patents is a significant challenge for the community. Automated reaction mining offers a promising solution to unleash this knowledge into a learnable digital form and therefore help expedite materials and reaction discovery. However, existing reaction mining toolkits are limited to single input modalities (text or images) and cannot effectively integrate heterogeneous data that is scattered across text, tables, and figures. In this work, we go beyond single input modalities and explore multimodal large language models (MLLMs) for the analysis of diverse data inputs for automated electrosynthesis reaction mining. We compiled a test dataset of 65 articles (MERMES-T24 set) and employed it to benchmark five prominent MLLMs against two critical tasks: (i) reaction diagram parsing and (ii) resolving cross-modality data interdependencies. The frontrunner MLLM achieved ≥96% accuracy in both tasks, with the strategic integration of single-shot visual prompts and image pre-processing techniques. We integrate this capability into a toolkit named MERMES (multimodal reaction mining pipeline for electrosynthesis). Our toolkit functions as an end-to-end MLLM-powered pipeline that integrates article retrieval, information extraction and multimodal analysis for streamlining and automating knowledge extraction. This work lays the groundwork for the increased utilization of MLLMs to accelerate the digitization of chemistry knowledge for data-driven research.

Leong, Shi Xuan↗

An Agenda for Multimodal Foundation Models for Earth Observation

Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

Ambrozio Dias, Philipe↗

Multimodal framework for the joint analysis of single-cell RNA and T cell receptor sequencing data predicts T cell response to cancer immunotherapy

T cell states are prognostic in different cancer types. Recent technologies enable joint profiling of T cell RNA and T cell receptor (TCR) sequences at single-cell resolution. Here we present the TCR-RNA Integrating Model (TRIM), a multi-modal variational autoencoder framework that integrates RNA-TCR data and predicts T cell clonality and transcriptional states. TRIM learns a shared representation of the data conditioned on patient, tissue source, and treatment timepoint. We applied TRIM to three independent datasets that included T cells collected before and after checkpoint inhibitor treatment, sourced either from blood and tumor biopsies in patients with head and neck squamous cell carcinoma and colorectal cancer, or from tumor and adjacent tissue in a pan-cancer dataset. In all settings, TRIM accurately predicted intra-tumor T cell clonal expansion and transcriptional status based on T cells from blood or normal tissue before treatment, demonstrating its utility in modeling multimodal T cell data and predicting T cell response to treatment and disease progression.

60 APPLIED LIFE SCIENCES↗

Robotics for HVAC applications: A critical review and future perspectives

Recent advances in artificial intelligence (AI), enhanced computational capabilities, and innovations in sensors and hardware have driven the increasing development and application of robots in heating, ventilation, and air conditioning (HVAC) systems. We selected and reviewed 101 studies published between 2005 and 2025, sourced from IEEE Xplore, Scopus, Web of Science, and the ACM Digital Library. To analyze these works, we developed a five-dimensional analytical framework (morphology, sensing, navigation, task execution, and system integration), inspired by the Springer Handbook of Robotics and tailored specifically for robotic applications in HVAC. Based on the reviewed studies, six distinct tasks spanning the entire HVAC lifecycle have been identified. Among the six tasks, inspection and maintenance dominate (59 %), followed by indoor monitoring and auditing (21 %), whereas leakage detection, comfort support, and installation/retrofit remain less explored. To address the identified gaps, this review proposes future research directions including investigating robot-aware HVAC design principles, developing multimodal HVAC sensing and data fusion techniques, enhancing robot training and hardware capabilities, and expanding robotic applications beyond Maintenance and Operations (M&O). The findings from this review inform future robotics research for HVAC applications and ultimately enhance system affordability, energy efficiency, resilience or reliability, and occupant environmental comfort. Moreover, it seeks to inspire researchers to explore the intersections of robotics, computer science, building science, and HVAC engineering fostering advancements in this multidisciplinary field.

AI↗

In situ spin coater for multimodal grazing incidence x-ray scattering studies

We present herein a custom-made, in situ, multimodal spin coater system with an integrated heating stage that can be programmed with spinning and heating recipes and that is coupled with synchrotron-based, grazing-incidence wide- and small-angle x-ray scattering. The spin coating system features an adaptable experimental chamber, with the ability to house multiple ancillary probes such as photoluminescence and visible optical cameras, to allow for true multimodal characterization and correlated data analysis. This system enables monitoring of structural evolutions such as perovskite crystallization and polymer self-assembly across a broad length scale (2 Å–150 nm) with millisecond temporal resolution throughout a complete thin film fabrication process. The use of this spin coating system allows scientists to gain a deeper understanding of temporal processes of a material system, to develop ideal conditions for thin film manufacturing.

47 OTHER INSTRUMENTATION↗

LLaMP v0.1.0

Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences, where reliability and reproducibility are crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to fine-tune them on domain-specific literature and data. LLaMP is a multimodal retrieval-augmented generation (RAG) framework of hierarchical reasoning and acting (ReAct) agents that can dynamically and recursively interact with Materials Project to ground large language models on high-fidelity materials informatics.

Riebesell, Janosh [Lawrence Berkeley National Labo↗

Application of the cubic polynomial strength criterion to the failure analysis of composite materials

A comparative failure analysis is presented based on the application of quadratic and cubic forms of the tensor polynomial lamina strength criterion to various composite structural configurations in a plane stress state. Failure loads have been predicted for off-angle laminates under simple loading conditions and for symmetric-balanced laminates subject to varying degrees of biaxial tension, including configurations subject to multimode failures. Some experimental data are also provided to support these calculations. From these results, the necessity of employing a cubic strength criterion to accurately predict the failure of composite laminae is demonstrated.

Tennyson, R. C.↗

Frequency-Bessel Transform Method for Effective Imaging of Higher-Mode Rayleigh Dispersion Curves From Ambient Seismic Noise Data

It has been widely recognized that the cross-correlation function of ambient seismic noise data recorded at two stations approximates to the part of Greens function between two stations. Therefore, the cross-correlation function should include higher modes, aside from the fundamental mode. However, the problem of measuring or extracting overtones from ambient seismic noise data remains. In this paper, we propose the frequency-Bessel transform method (F-J method) for extracting the dispersion curves of higher modes from ambient seismic noise data. We then assess the validity, accuracy, and applicability of the F-J method by conducting extensive numerical simulations and processing the observed ambient seismic noise data of the USArray. As demonstrated in this study, the F-J method is a convenient, practical, and accurate method for extracting the dispersion curves of multimodes from ambient seismic noise data and therefore has significant potentiality in the field of ambient seismic noise tomography.

58 GEOSCIENCES↗

Discriminative analysis of schizophrenia patients using graph convolutional networks: A combined multimodal MRI and connectomics analysis

Introduction Recent studies in human brain connectomics with multimodal magnetic resonance imaging (MRI) data have widely reported abnormalities in brain structure, function and connectivity associated with schizophrenia (SZ). However, most previous discriminative studies of SZ patients were based on MRI features of brain regions, ignoring the complex relationships within brain networks. Methods We applied a graph convolutional network (GCN) to discriminating SZ patients using the features of brain region and connectivity derived from a combined multimodal MRI and connectomics analysis. Structural magnetic resonance imaging (sMRI) and resting-state functional magnetic resonance imaging (rs-fMRI) data were acquired from 140 SZ patients and 205 normal controls. Eighteen types of brain graphs were constructed for each subject using 3 types of node features, 3 types of edge features, and 2 brain atlases. We investigated the performance of 18 brain graphs and used the TopK pooling layers to highlight salient brain regions (nodes in the graph). Results The GCN model, which used functional connectivity as edge features and multimodal features (sMRI + fMRI) of brain regions as node features, obtained the highest average accuracy of 95.8%, and outperformed other existing classification studies in SZ patients. In the explainability analysis, we reported that the top 10 salient brain regions, predominantly distributed in the prefrontal and occipital cortices, were mainly involved in the systems of emotion and visual processing. Discussion Our findings demonstrated that GCN with a combined multimodal MRI and connectomics analysis can effectively improve the classification of SZ at an individual level, indicating a promising direction for the diagnosis of SZ patients. The code is available at https://github.com/CXY-scut/GCN-SZ.git .

Chen, Xiaoyi↗