Search NASA⌕ Search

SEARCH · Search NASA

Results for “multimodal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Multimodal Event Detection in Twitter Hashtag Networks

In this study, event detection in a multimodal Twitter dataset is considered. We treat the hashtags in the dataset as instances with two modes: text and geolocation features. The text feature consists of a bag-of-words representation. The geolocation feature consists of geotags (i.e., geographical coordinates) of the tweets. Fusing the multimodal data we aim to detect, in terms of topic and geolocation, the interesting events and the associated hashtags. To this end, a generative latent variable model is assumed, and a generalized expectation-maximization (EM) algorithm is derived to learn the model parameters. The proposed method is computationally efficient, and lends itself to big datasets. Lastly, experimental results on a Twitter dataset from August 2014 show the efficacy of the proposed method.

97 MATHEMATICS AND COMPUTING↗

Evaluation of UAV-derived multimodal remote sensing data for biomass prediction and drought tolerance assessment in bioenergy sorghum

Screening for drought tolerance is critical to ensure high biomass production of bioenergy sorghum in arid or semi-arid environments. The bottleneck in drought tolerance selection is the challenge of accurately predicting biomass for a large number of genotypes. Although biomass prediction by low-altitude remote sensing has been widely investigated on various crops, the performance of the predictions are not consistent, especially when applied in a breeding context with hundreds of genotypes. In some cases, biomass prediction of a large group of genotypes benefited from multimodal remote sensing data; while in other cases, the benefits were not obvious. In this study, we evaluated the performance of single and multimodal data (thermal, RGB, and multispectral) derived from an unmanned aerial vehicle (UAV) for biomass prediction for drought tolerance assessments within a context of bioenergy sorghum breeding. The biomass of 360 sorghum genotypes grown under well-watered and water-stressed regimes was predicted with a series of UAV-derived canopy features, including canopy structure, spectral reflectance, and thermal radiation features. Biomass predictions using canopy features derived from the multimodal data showed comparable performance with the best results obtained with the single modal data with coefficients of determination (R 2 ) ranging from 0.40 to 0.53 under water-stressed environment and 0.11 to 0.35 under well-watered environment. The significance in biomass prediction was highest with multispectral followed by RGB and lowest with the thermal sensor. Finally, two well-recognized yield-based drought tolerance indices were calculated from ground truth biomass data and UAV predicted biomass, respectively. Results showed that the geometric mean productivity index outperformed the yield stability index in terms of the potential for reliable predictions by the remotely sensed data. Collectively, this study demonstrated a promising strategy for the use of different UAV-based imaging sensors to quantify yield-based drought tolerance.

09 BIOMASS FUELS↗

Towards verifiable cancer digital twins: tissue level modeling protocol for precision medicine

Cancer exhibits substantial heterogeneity, manifesting as distinct morphological and molecular variations across tumors, which frequently undermines the efficacy of conventional oncological treatments. Developments in multiomics and sequencing technologies have paved the way for unraveling this heterogeneity. Nevertheless, the complexity of the data gathered from these methods cannot be fully interpreted through multimodal data analysis alone. Mathematical modeling plays a crucial role in delineating the underlying mechanisms to explain sources of heterogeneity using patient-specific data. Intra-tumoral diversity necessitates the development of precision oncology therapies utilizing multiphysics, multiscale mathematical models for cancer. This review discusses recent advancements in computational methodologies for precision oncology, highlighting the potential of cancer digital twins to enhance patient-specific decision-making in clinical settings. We review computational efforts in building patient-informed cellular and tissue-level models for cancer and propose a computational framework that utilizes agent-based modeling as an effective conduit to integrate cancer systems models that encode signaling at the cellular scale with digital twin models that predict tissue-level response in a tumor microenvironment customized to patient information. Furthermore, we discuss machine learning approaches to building surrogates for these complex mathematical models. These surrogates can potentially be used to conduct sensitivity analysis, verification, validation, and uncertainty quantification, which is especially important for tumor studies due to their dynamic nature.

60 APPLIED LIFE SCIENCES↗

Postearthquake Damage Mapping via Remote Sensing: Lessons From the 2023 Türkiye Disaster

This review addresses the urgent need for scalable, accurate, and reproducible remote sensing solutions following the February 2023 Türkiye earthquakes. It synthesizes the contributions of five peer-reviewed studies published in the IEEE JSTARS Special Issue on postearthquake damage and risk assessment. These studies cover areas such as damage classification with deep learning, fusion of multisource remote sensing data, creation of benchmark datasets, detailed damage mapping, and analysis of geophysical signals using outgoing longwave radiation. The article summarizes the methodological approaches and the practical relevance of the reviewed studies for detecting, evaluating, and quantifying damage, and outlines key challenges, including model generalization, class ambiguity, and data integration. It also discusses emerging trends, including explainable artificial intelligence, multimodal data fusion, and open-data platforms. This synthesis provides a foundation for building robust, interpretable, and real-time disaster response systems and aims to guide future research in earthquake-related Earth observation and rapid damage assessment.

Taskin, Gulsen [Istanbul Technical University] (OR↗

Toward Intelligent Multimodal Holography for Real-Time Chemical Imaging of Dynamic Ion Separation

Molecular-level visualization of ion transport and separation dynamics in complex environments is crucial for advancing energy systems, water purification, and critical materials recovery. Achieving this requires imaging platforms that combine structural sensitivity, chemical specificity, and real-time operation. Digital off-axis holography (DOAH) provides high-throughput, label-free quantitative phase imaging but inherently lacks chemical selectivity. Integrating DOAH with complementary spectroscopic channels such as fluorescence or hyperspectral imaging introduces the needed molecular specificity, while also creating challenges in multimodal data fusion, synchronization, and computational throughput. Artificial intelligence offers a powerful route to address these limitations by uniting physics-based reconstruction with data-driven interpretation. In this Perspective, we outline a framework for intelligent multimodal holography and demonstrate its potential using a preliminary AI-driven test case. Raw DOAH holograms of lanthanide solutions subjected to magnetic field gradients were analyzed using multi-agent AI workflows that autonomously selected reconstruction tools, extracted NMF components, and generated scientific claims consistent with true paramagnetic and diamagnetic behavior. This demonstration shows how AI-enabled reasoning can deliver real-time chemical–structural interpretation directly from raw holograms. Together, these advances define a path toward adaptive, intelligent holography platforms capable of supporting in situ chemical separations, dynamic ion transport analysis, and next-generation interfacial science.

Ricchiuti, Giovanna↗

Learning instrument invariant characteristics for generating high-resolution global coral reef maps

Coral reefs are one of the most biologically complex and diverse ecosystems within the shallow marine environment. Unfortunately, these underwater ecosystems are threatened by a number of anthropogenic challenges, including ocean acidification and warming, overfishing, and the continued increase of marine debris in oceans. This requires a comprehensive assessment of the world's coastal environments, including a quantitative analysis on the health and extent of coral reefs and other associated marine species, as a vital Earth Science measurement. However, limitations in observational and technological capabilities inhibit global sustained imaging of the marine environment. Harmonizing multimodal data sets acquired using different remote sensing instruments presents additional challenges, thereby limiting the availability of good quality labeled data for analysis. In this work, we develop a deep learning model for extracting domain invariant features from multimodal remote sensing imagery and creating high-resolution global maps of coral reefs by combining various sources of imagery and limited hand-labeled data available for certain regions. This framework allows us to generate, for the first time, coral reef segmentation maps at 2-meter resolution, which is a significant improvement over the kilometer-scale state-of-the-art maps. Additionally, this framework doubles accuracy and IoU metrics over baselines that do not account for domain invariance.

Domain Adaptation↗

A Representation Fusion Framework for Decoupling Diagnostic Information in Multimodal Learning

Modern medicine increasingly relies on multimodal data, ranging from clinical notes to imaging and genomics, to guide diagnosis and treatment. However, integrating these heterogeneous data sources in a principled and interpretable manner remains a major challenge. We present MODES (Multi-mOdal Disentangled Embedding Space), a representation fusion framework that explicitly separates shared and modality-specific factors of variation, offering a structured latent space for multimodal information that improves both prediction and interpretability. By leveraging pre-trained unimodal foundation models, MODES mitigates the dependency on extensive paired datasets, crucial in data-scarce clinical settings. We introduce a masking strategy that optimizes representation dimensionality by eliminating low-information dimensions, to achieve compact, information-rich representations. Our framework demonstrates superior performance in predicting diagnoses and phenotypes compared to unimodal and conventional fusion models. MODES also enables robust diagnostic inference in missing data scenarios, offering an opportunity toward interpretable and efficient multimodal diagnostics in personalized healthcare.

60 APPLIED LIFE SCIENCES↗

Suppressing simulation bias in multi-modal data using transfer learning

Abstract Many problems in science and engineering require making predictions based on few observations. To build a robust predictive model, these sparse data may need to be augmented with simulated data, especially when the design space is multi-dimensional. Simulations, however, often suffer from an inherent bias. Estimation of this bias may be poorly constrained not only because of data sparsity, but also because traditional predictive models fit only one type of observed outputs, such as scalars or images, instead of all available output data modalities, which might have been acquired and simulated at great cost. To break this limitation and open up the path for multi-modal calibration, we propose to combine a novel, transfer learning technique for suppressing the bias with recent developments in deep learning, which allow building predictive models with multi-modal outputs. First, we train an initial neural network model on simulated data to learn important correlations between different output modalities and between simulation inputs and outputs. Then, the model is partially retrained, or transfer learned, to fit the experiments; a method that has never been implemented in this type of architecture. Using fewer than 10 inertial confinement fusion experiments for training, transfer learning systematically improves the simulation predictions while a simple output calibration, which we design as a baseline, makes the predictions worse. We also offer extensive cross-validation with real and carefully designed synthetic data. The method described in this paper can be applied to a wide range of problems that require transferring knowledge from simulations to the domain of experiments.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

CrossMP: Enabling Cross-Modality Translation between Single-Cell RNA-Seq and Single-Cell ATAC-Seq through Web-Based Portal

In recent years, there has been a growing interest in profiling multiomic modalities within individual cells simultaneously. One such example is integrating combined single-cell RNA sequencing (scRNA-seq) data and single-cell transposase-accessible chromatin sequencing (scATAC-seq) data. Integrated analysis of diverse modalities has helped researchers make more accurate predictions and gain a more comprehensive understanding than with single-modality analysis. However, generating such multimodal data is technically challenging and expensive, leading to limited availability of single-cell co-assay data. Here, we propose a model for cross-modal prediction between the transcriptome and chromatin profiles in single cells. Our model is based on a deep neural network architecture that learns the latent representations from the source modality and then predicts the target modality. It demonstrates reliable performance in accurately translating between these modalities across multiple paired human scATAC-seq and scRNA-seq datasets. Additionally, we developed CrossMP, a web-based portal allowing researchers to upload their single-cell modality data through an interactive web interface and predict the other type of modality data, using high-performance computing resources plugged at the backend.

59 BASIC BIOLOGICAL SCIENCES↗

Deformation Mechanisms at Grain Boundaries in Polycrystals (Final Report)

Despite its significance in polycrystalline materials, there have been few experimental investigations of the activity of grain boundary sliding (GBS) and the relationship between GBS and slip transmission at grain boundaries. This work addressed this knowledge gap by an experimental and analytical investigation into the interactions between grain boundary sliding (GBS) and slip transmission plastic deformation mechanisms, and how these are impacted by local microstructure. Polycrystalline columnar aluminum of 99.99% purity was examined as a highly modellable system for characterizing grain boundary-local deformation mechanism interaction. High-resolution, large field-of-view strain fields and grain orientation information were captured and statistically analyzed to identify deformation activity with respect to the local microstructure, using a combination of scanning electron microscopy and digital image correlation (SEM-DIC) with electron backscatter diffraction (EBSD). Alignment between the lower-resolution electron backscatter diffraction (EBSD) data and the higher-resolution strain fields was significantly improved by clustering of strain data within an EBSD-defined boundary mantle. Although this work specifically focuses on characterizing the relationship between grain boundary sliding (GBS) and slip transmission, the developed experimental approach, multimodal data alignment process, and automated displacement discontinuity measurement process can be used to study a variety of mechanism interactions across a broad range of materials. Regarding GBS and slip transmission interaction, significant findings include that (1) grain boundaries that slide easily do not facilitate slip transmission; (2) substantial gradients in GBS magnitude could be found along the length of individual grain boundaries, and these were caused by strain discontinuities across the boundary related to deformation anisotropy between adjacent grains; (3) direct transmission and GBS were anti-compatible and facilitated by opposing boundary types (low misorientation and high energy grain boundaries respectively); (4) increased GBS activity was correlated with decreased indirect transmission behavior; (5) GBS accommodation at triple junctions was enabled by intragranular plasticity; and (6) the local intragranular plastic strain discontinuity between grains determined the magnitude of GBS gradients. The experimental strain fields were compared to that of a tangent crystal plasticity finite element (CPFE) model developed by collaborators (Mr. Ajey Venkataraman and Professor Michael Sangid, Purdue University). Implication of this work for improving CPFE models and deformation prediction are discussed. This work provides insight into the nature of these mechanisms and can be used to identify strain transfer criteria that can lead to improved GBS-sensitive crystal plasticity models.

36 MATERIALS SCIENCE↗

Multiomics Data Collection, Visualization, and Utilization for Guiding Metabolic Engineering

Biology has changed radically in the past two decades, growing from a purely descriptive science into also a design science. The availability of tools that enable the precise modification of cells, as well as the ability to collect large amounts of multimodal data, open the possibility of sophisticated bioengineering to produce fuels, specialty and commodity chemicals, materials, and other renewable bioproducts. However, despite new tools and exponentially increasing data volumes, synthetic biology cannot yet fulfill its true potential due to our inability to predict the behavior of biological systems. Here, we showcase a set of computational tools that, combined, provide the ability to store, visualize, and leverage multiomics data to predict the outcome of bioengineering efforts. We show how to upload, visualize, and output multiomics data, as well as strain information, into online repositories for several isoprenol-producing strain designs. We then use these data to train machine learning algorithms that recommend new strain designs that are correctly predicted to improve isoprenol production by 23%. This demonstration is done by using synthetic data, as provided by a novel library, that can produce credible multiomics data for testing algorithms and computational tools. In short, this paper provides a step-by-step tutorial to leverage these computational tools to improve production in bioengineered strains.

09 BIOMASS FUELS↗

Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images

Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies. A primary challenge in modeling drug response prediction (DRP) with PDXs and neural networks (NNs) is the limited number of drug response samples. We investigate multimodal neural network (MM-Net) and data augmentation for DRP in PDXs. The MM-Net learns to predict response using drug descriptors, gene expressions (GE), and histology whole-slide images (WSIs). We explore whether combining WSIs with GE improves predictions as compared with models that use GE alone. We propose two data augmentation methods which allow us training multimodal and unimodal NNs without changing architectures with a single larger dataset: 1) combine single-drug and drug-pair treatments by homogenizing drug representations, and 2) augment drug-pairs which doubles the sample size of all drug-pair samples. Unimodal NNs which use GE are compared to assess the contribution of data augmentation. The NN that uses the original and the augmented drug-pair treatments as well as single-drug treatments outperforms NNs that ignore either the augmented drug-pairs or the single-drug treatments. In assessing the multimodal learning based on the MCC metric, MM-Net outperforms all the baselines. Our results show that data augmentation and integration of histology images with GE can improve prediction performance of drug response in PDXs.

60 APPLIED LIFE SCIENCES↗

Sensor Fusion Techniques for Phased-Array Eddy Current and Phased-Array Ultrasound Data

Sensor (or Data) fusion is the process of integrating multiple data sources to produce more consistent, accurate and comprehensive information than is provided by a single data source. Sensor fusion may also be used to combine multiple signals from a single modality to improve the performance of a particular inspection technique. Industrial nondestructive testing may utilize multiple sensors to acquire inspection data depending upon the object under inspection and the anticipated types of defects that can be identified. Sensor fusion can be performed at various levels of signal abstraction with each having its strengths and weaknesses. A multimodal data fusion strategy first proposed by Heideklang and Shokouhi that combines spatially scattered detection locations to improve detection performance of surface-breaking and near-surface cracks in ferromagnetic metals is shown using a surface inspection example and is then extended for volumetric inspections. Utilizing data acquired from an Olympus Omniscan MX2 from both phased array eddy current and ultrasound probes on test phantoms, single and multilevel fusion techniques are employed to integrate signals from the two modalities. Preliminary results demonstrate how confidence in defect identification and interpretation benefit from sensor fusion techniques. Lastly, techniques for integrating data into radiographic and volumetric imagery from computed tomography are described and results are presented.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗

Current and future directions in network biology

Network biology is an interdisciplinary field bridging computational and biological sciences that has proved pivotal in advancing the understanding of cellular functions and diseases across biological systems and scales. Although the field has been around for two decades, it remains nascent. It has witnessed rapid evolution, accompanied by emerging challenges. These stem from various factors, notably the growing complexity and volume of data together with the increased diversity of data types describing different tiers of biological organization. We discuss prevailing research directions in network biology, focusing on molecular/cellular networks but also on other biological network types such as biomedical knowledge graphs, patient similarity networks, brain networks, and social/contact networks relevant to disease spread. In more detail, we highlight areas of inference and comparison of biological networks, multimodal data integration and heterogeneous networks, higher-order network analysis, machine learning on networks, and network-based personalized medicine. Following the overview of recent breakthroughs across these five areas, we offer a perspective on future directions of network biology. Additionally, we discuss scientific communities, educational initiatives, and the importance of fostering diversity within the field. This article establishes a roadmap for an immediate and long-term vision for network biology.

59 BASIC BIOLOGICAL SCIENCES↗

Bayesian seismic inversion based on rock-physics prior modeling for the joint estimation of acoustic impedance, porosity and lithofacies

We propose a Bayesian approach for seismic inversion to estimate acoustic impedance, porosity and lithofacies within the reservoir conditioned to post-stack seismic and well data. The link between elastic and petrophysical properties is given by a joint prior distribution for the logarithm of impedance and porosity, based on a rock-physics model. The well conditioning is performed through a background model obtained by well log interpolation. Two different approaches are presented: in the first approach, the prior is defined by a single Gaussian distribution, whereas in the second approach it is defined by a Gaussian mixture to represent the well data multimodal distribution and link the Gaussian components to different geological lithofacies. The forward model is based on a linearized convolutional model. For the single Gaussian case, we obtain an analytical expression for the posterior distribution, resulting in a fast algorithm to compute the solution of the inverse problem, i.e. the posterior distribution of acoustic impedance and porosity as well as the facies probability given the observed data. For the Gaussian mixture prior, it is not possible to obtain the distributions analytically, hence we propose a Gibbs algorithm to perform the posterior sampling and obtain several reservoir model realizations, allowing an uncertainty analysis of the estimated properties and lithofacies. Both methodologies are applied to a real seismic dataset with three wells to obtain 3D models of acoustic impedance, porosity and lithofacies. The methodologies are validated through a blind well test and compared to a standard Bayesian inversion approach. Using the probability of the reservoir lithofacies, we also compute a 3D isosurface probability model of the main oil reservoir in the studied field.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

AI‐Driven Robot Enables Synthesis‐Property Relation Prediction for Metal Halide Perovskites in Humid Atmosphere

Materials Acceleration Platforms (MAPs) – also known as self-driving laboratories– present a new paradigm for materials science and promise an order of magnitude accelerated materials discovery compared to the traditional trial-and-error approach. Metal halide perovskites (MHPs) are an emerging class of materials for optoelectronic applications but are plagued by irreproducible optoelectronic quality, particularly for films fabricated in a humid atmosphere. Here, in this work, a machine learning (ML)-guided closed-loop platform is developed with a multimodal data fusion approach to predict synthesis–property relations for the optical quality of MHP thin films in relative humidities (RHs) ranging from 5–55%. The efficiency of this approach is confirmed by the fast-dropping learning rate to 2% after experimentally sampling less than 1% of the possible 5,000+ combinations. The prediction of synthesis–property relations is done by optical and imaging characterizations. In situ photoluminescence characterization revealed the origin of thin film quality variation at different RH. These insights provide an avenue for controlling the MHP crystallization by fine-tuning the synthesis parameters and RH for a given chemistry, thus lifting the need for stringent atmosphere control. The MAP enables an accelerated screening and understanding of the synthesis design space, facilitating rational synthesis recipe choice for a wide range of materials.

AI-driven robot↗

A Dual-Gated Structures for Lossless Ion Manipulations-Ion Mobility Orbitrap Mass Spectrometry Platform for Combined Ultra-High-Resolution Molecular Analysis

High-resolution ion mobility spectrometry-mass spectrometry (HR-IMS-MS) instruments have enormously advanced the ability to characterize complex biological mixtures. Unfortunately, HR-IMS and HR-MS measurements are typically performed independently due to mismatches in analysis time scales. Here we overcome this limitation by using a dual-gated ion injection approach to couple an 11-meter path length structures for lossless ion manipulations (SLIM) module to a Q-Exactive Plus Orbitrap MS. The dual-gate setup was implemented by placing one ion gate before the SLIM module and a second ion gate after. The dual-gated ion injection approach allowed the new SLIM-Orbitrap platform to simultaneously perform an 11-meter SLIM separation, Orbitrap mass analysis using the highest selectable mass resolution setting (up to 140k), and high-energy collision induced dissociation (HCD) in ~25 minutes over an $m/z$ range of ~1500 amu. The SLIM-Orbitrap was initially characterized using a mixture of standard phosphazene cations and demonstrated an average SLIM CCS resolving power (Rp CCS ) of ~218 and SLIM peak capacity of ~156 while simultaneously obtaining high mass resolutions. SLIM-Orbitrap analysis with fragmentation was then performed on mixtures of standard peptides and two reverse peptides (SDGRG 1+ , GRGDS 1+ , Rp CCS = 305) to demonstrate the utility of combined HR-IMS-MS/MS measurements for peptide identification. Our new HR-IMS-MS/MS capability was further demonstrated by analyzing a complex lipid mixture and showcasing SLIM separations on isobaric lipids. In conclusion, this new SLIM-Orbitrap platform demonstrates a critical new capability for proteomics and lipidomics applications, and the high-resolution multimodal data obtainable with this system establishes the foundation for reference-free identification of unknown ion structures.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Operating advanced scientific instruments with AI agents that learn on the job

Advanced scientific user facilities, such as next generation X-ray light sources and self-driving laboratories, are revolutionizing scientific discovery by automating routine tasks and enabling rapid experimentation and characterizations. However, these facilities must continuously evolve to support new experimental workflows, adapt to diverse user projects, and meet growing demands for more intricate instruments and experiments. This continuous development introduces significant operational complexity, necessitating a focus on usability, reproducibility, and intuitive human-instrument interaction. In this work, we explore the integration of agentic AI, powered by Large Language Models (LLMs), as a transformative tool to achieve this goal. We present our approach to developing a human-in-the-loop pipeline for operating advanced instruments including an X-ray nanoprobe beamline and an autonomous robotic station dedicated to the design and characterization of materials. Specifically, we evaluate the potential of various LLMs as trainable scientific assistants for orchestrating complex, multi-task workflows, which also include multimodal data, optimizing their performance through optional human input and iterative learning. We demonstrate the ability of AI agents to bridge the gap between advanced automation and user-friendly operation, paving the way for more adaptable and intelligent scientific facilities.

Large Language Models↗