Search NASA⌕ Search

SEARCH · Search NASA

Results for “Multimodal integration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Current and future directions in network biology

Network biology is an interdisciplinary field bridging computational and biological sciences that has proved pivotal in advancing the understanding of cellular functions and diseases across biological systems and scales. Although the field has been around for two decades, it remains nascent. It has witnessed rapid evolution, accompanied by emerging challenges. These stem from various factors, notably the growing complexity and volume of data together with the increased diversity of data types describing different tiers of biological organization. We discuss prevailing research directions in network biology, focusing on molecular/cellular networks but also on other biological network types such as biomedical knowledge graphs, patient similarity networks, brain networks, and social/contact networks relevant to disease spread. In more detail, we highlight areas of inference and comparison of biological networks, multimodal data integration and heterogeneous networks, higher-order network analysis, machine learning on networks, and network-based personalized medicine. Following the overview of recent breakthroughs across these five areas, we offer a perspective on future directions of network biology. Additionally, we discuss scientific communities, educational initiatives, and the importance of fostering diversity within the field. This article establishes a roadmap for an immediate and long-term vision for network biology.

59 BASIC BIOLOGICAL SCIENCES↗

Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data

Zero-shot and prompt-based models have excelled at visual reasoning tasks by leveraging large-scale natural image corpora, but they often fail on sparse and domain-specific scientific image data. We introduce Zenesis, a no-code interactive computer vision platform designed to reduce data readiness bottlenecks in scientific imaging workflows. Zenesis integrates lightweight multimodal adaptation for zero-shot inference on raw scientific data, human-in-the-loop refinement, and heuristic-based temporal enhancement. We validate our approach on Focused Ion Beam Scanning Electron Microscopy (FIB-SEM) datasets of catalyst-loaded membranes. Zenesis outperforms baselines, achieving an average accuracy of 0.947, Intersection over Union (IoU) of 0.858, and Dice score of 0.923 on amorphous catalyst samples; and 0.987 accuracy, 0.857 IoU, and 0.923 Dice on crystalline samples. These results represent a significant performance gain over conventional methods such as Otsu thresholding and standalone models like the Segment Anything Model (SAM). Zenesis enables effective image segmentation in domains where annotated datasets are limited, offering a scalable solution for scientific discovery.

Mukherjee, Shubhabrata↗

Basic Research Needs for Inverse Methods for Complex Systems under Uncertainty [Brochure]

The four priority research directions outlined in this brochure represent a cohesive vision for advancing the science of inverse problems for complex systems under uncertainty. Together, they address the critical challenges of: discovering, exploiting, and preserving physical and problem structure; overcoming model limitations; integrating disparate, multimodal, and/or dynamic data; and tailoring the solution of inverse problems to downstream tasks. While each PRD focuses on a distinct aspect of inverse-problem research, their interconnected nature highlights the importance of a holistic approach that leverages progress across all areas to achieve transformative solutions. This agenda calls for research across mathematics, statistics, and computer science disciplines, which are guided and complemented by rapid advances in artificial intelligence, high-performance computing, and experimental facilities, to unlock new capabilities, maximize scientific impact, and meet the growing demands of inverse problems that arise across applications that are critical to DOE's mission.

97 MATHEMATICS AND COMPUTING↗

Seeing is Believing: Autonomous Microscopy and the Data Revolution in Materials Science

This presentation explores the transformative potential of autonomous electron microscopy and artificial intelligence (AI) in accelerating materials science discovery, particularly for energy applications and materials operating in extreme environments. We discuss pioneering self-driving laboratories at NREL designed to intelligently probe material synthesis and degradation across multiple scales, aiming to rapidly bridge the gap between atomic-level understanding and the development of high-performance, reliable materials. Utilizing advanced machine learning techniques, such as few-shot learning and multimodal analysis integrating imaging and spectroscopy, we demonstrate methods to extract actionable descriptors for material behavior, quantify complex microstructural evolution, and statistically link synthesis parameters to defect populations. This AI-driven approach promises to accelerate the creation of predictive materials tailored for specific missions, enabling faster development cycles and enhanced material assurance.

36 MATERIALS SCIENCE↗

Seeing is Believing: Autonomous Microscopy and the Data Revolution in Materials Science

This presentation explores the transformative potential of autonomous electron microscopy and artificial intelligence (AI) in accelerating materials science discovery, particularly for energy applications and materials operating in extreme environments. We discuss pioneering self-driving laboratories at NREL designed to intelligently probe material synthesis and degradation across multiple scales, aiming to rapidly bridge the gap between atomic-level understanding and the development of high-performance, reliable materials. Utilizing advanced machine learning techniques, such as few-shot learning and multimodal analysis integrating imaging and spectroscopy, we demonstrate methods to extract actionable descriptors for material behavior, quantify complex microstructural evolution, and statistically link synthesis parameters to defect populations. This AI-driven approach promises to accelerate the creation of predictive materials tailored for specific missions, enabling faster development cycles and enhanced material assurance.

97 MATHEMATICS AND COMPUTING↗

Progress in integrated-circuit horn antennas for receiver applications. Part 1: Antenna design

The purpose of this work is to present a systematic method for the design of multimode quasi-integrated horn antennas. The design methodology is based on the Gaussian beam approach and the structures are optimized for achieving maximum fundamental Gaussian coupling efficiency. For this purpose, a hybrid technique is employed in which the integrated part of the antennas is treated using full-wave analysis, whereas the machined part is treated using an approximate method. This results in a simple and efficient design process. The developed design procedure has been applied for the design of a 20, a 23, and a 25 dB quasi-integrated horn antennas, all with a Gaussian coupling efficiency exceeding 97 percent. The designed antennas have been tested and characterized using both full-wave analysis and 90 GHz/370 GHz measurements.

Eleftheriades, George V.↗

Visual-vestibular integration motion perception reporting

Self-orientation and self/surround-motion perception derive from a multimodal sensory process that integrates information from the eyes, vestibular apparatus, proprioceptive and somatosensory receptors. Results from short and long duration spaceflight investigations indicate that: (1) perceptual and sensorimotor function was disrupted during the initial exposure to microgravity and gradually improved over hours to days (individuals adapt), (2) the presence and/or absence of information from different sensory modalities differentially affected the perception of orientation, self-motion and surround-motion, (3) perceptual and sensorimotor function was initially disrupted upon return to Earth-normal gravity and gradually recovered to preflight levels (individuals readapt), and (4) the longer the exposure to microgravity, the more complete the adaptation, the more profound the postflight disturbances, and the longer the recovery period to preflight levels. While much has been learned about perceptual and sensorimotor reactions and adaptation to microgravity, there is much remaining to be learned about the mechanisms underlying the adaptive changes, and about how intersensory interactions affect perceptual and sensorimotor function during voluntary movements. During space flight, SMS and perceptual disturbances have led to reductions in performance efficiency and sense of well-being. During entry and immediately after landing, such disturbances could have a serious impact on the ability of the commander to land the Orbiter and on the ability of all crew members to egress from the Orbiter, particularly in a non-nominal condition or following extended stays in microgravity. An understanding of spatial orientation and motion perception is essential for developing countermeasures for Space Motion Sickness (SMS) and perceptual disturbances during spaceflight and upon return to Earth. Countermeasures for optimal performance in flight and a successful return to Earth require the development of preflight and in-flight training to help astronauts acquire and maintain a dual adaptive state. Despite the considerable experience with, and use of, an extensive set of countermeasures in the Russian space program, SMS and perceptual disturbances remain an unresolved problem on long-term flights. Reliable, valid perceptual reports are required to develop and refine stimulus rearrangements presented in the PAT devices currently being developed as countermeasures for the prevention of motion sickness and perceptual disturbances during spaceflight, and to ensure a less hazardous return to Earth. Prior to STS-8, crew member descriptions of their perceptual experiences were, at best, anecdotal. Crew members were not schooled in the physiology or psychology of sensory perception, nor were they exposed to the appropriate professional vocabulary. However, beginning with the STS-8 Shuttle flight, a serious effort was initiated to teach astronauts a systematic method to classify and quantify their perceptual responses in space, during entry, and after flight. Understanding, categorizing, and characterizing perceptual responses to spaceflight has been greatly enhanced by implementation of that training system.

Harm, Deborah L.↗

Development of an Integrated Nozzle for a Symmetric, RBCC Launch Vehicle Configuration

The development of rocket based combined cycle (RBCC) engines is highly dependent upon integrating several different modes of operation into a single system. One of the key components to develop acceptable performance levels through each mode of operation is the nozzle. It must be highly integrated to serve the expansion processes of both rocket and air-breathing modes without undue weight, drag, or complexity. The NASA GTX configuration requires a fixed geometry, altitude-compensating nozzle configuration. The initial configuration, used mainly to estimate weight and cooling requirements was a 1 So half-angle cone, which cuts a concave surface from a point within the flowpath to the vehicle trailing edge. Results of 3-D CFD calculations on this geometry are presented. To address the critical issues associated with integrated, fixed geometry, multimode nozzle development, the GTX team has initiated a series of tasks to evolve the nozzle design, and validate performance levels. An overview of these tasks is given. The first element is a design activity to develop tools for integration of efficient expansion surfaces With the existing flowpath and vehicle aft-body, and to develop a second-generation nozzle design. A preliminary result using a "streamline-tracing" technique is presented. As the nozzle design evolves, a combination of 3-D CFD analysis and experimental evaluation will be used to validate the design procedure and determine the installed performance for propulsion cycle modeling. The initial experimental effort will consist of cold-flow experiments designed to validate the general trends of the streamline-tracing methodology and anchor the CFD analysis. Experiments will also be conducted to simulate nozzle performance during each mode of operation. As the design matures, hot-fire tests will be conducted to refine performance estimates and anchor more sophisticated reacting-flow analysis.

Smith, Timothy D.↗

Knowledge Oriented Graph Unified Transformer (KOGUT) v0.1

KOGUT — Knowledge Oriented Graph Unified Transformer KOGUT implements the Relational Graph Transformer (RelGT) architecture for knowledge graph link prediction in biological domains, with a primary focus on microbial growth media prediction. While the original RelGT (arXiv:2505.10960) targets relational tables, time series, and multi-table databases, KOGUT adapts this architecture for heterogeneous biological knowledge graphs, providing first-in-class AI predictive models for microbial cultivation. Key Adaptations Beyond Original RelGT: - Knowledge Graph Focus: Applied to biological KGs with semantic node types (taxa, chemicals, media, phenotypes, environments) versus generic relational database tables, trained on the KG-Microbe knowledge graph (1.3M entities, 2.9M edges, 24 relation types). - Multimodal Node Encoding: Integrates node labels, categories, descriptions, and synonyms from KG metadata through learned embedding layers—adapting relational column features to graph node attributes with textual semantics. - Extended K-Hop Subgraph Strategy: Optimized neighborhood sampling (3-hop default, configurable up to 200 nodes) tuned for sparse biological networks, building on the original local-global attention framework with biological relation preservation. - Biolink Predicate Preservation: Type-specific transformations for 24 biological edge semantics (occurs_in, consumes, produces, has_phenotype, subclass_of) beyond standard relational foreign keys, enabling multi-relation link prediction. - Inductive Learning Support: Enables zero-shot predictions for novel taxa through feature-based embeddings (temperature, oxygen requirements, gram stain, cell shape), extending the original transductive relational benchmark scope to uncultured microorganisms. CheapSOTA Performance Optimizations (This Distribution): - VQ-EMA Centroid Attention: Vector quantization with exponential moving average for improved global context modeling (+5-10% MRR improvement). - HDF5 Precomputed Data Loading: One-time preprocessing of k-hop subgraphs to eliminate redundant graph traversals (2-5× training speedup). - Distributed Data Parallel Training: Multi-GPU support for scaling to larger knowledge graphs (tested on 4× NVIDIA A100 GPUs at NERSC Perlmutter). - Mixed Precision Training: Automatic mixed precision (AMP) for memory efficiency and faster training. Advantages Over Standard Knowledge Graph Embedding Models: Combines RelGT's proven multi-element tokenization (features, type, hop, structure) with graph-native biological representations, enabling interpretable link prediction across heterogeneous entities that standard embedding models (TransE, RotatE, ComplEx) and table-based transformers cannot directly model. Achieves near-perfect performance on microbial growth media prediction (MRR: 0.9966, Precision@1: 0.9932, Hit@10: 1.0000) while maintaining explainability through attention-based reasoning over biological pathways. Training Data: - KG-Microbe merged knowledge graph: 1,379,337 nodes, 2,960,472 edges - 24 biological relation types including taxonomic hierarchies, metabolic interactions, phenotype associations, and environmental relationships - Primary prediction task: Growth media suitability for microbial taxa (biolink:occurs_in, 50K edges) - Multi-relation capability: Predicts links for any of the 24 relation types, including chemical consumption/production, phenotype associations, and taxonomic classification Citation: Original RelGT Architecture: Dwivedi et al., "Relational Graph Transformer", arXiv:2505.10960, 2025 KOGUT Implementation: Knowledge Oriented Graph Unified Transformer for Microbial Growth Media Prediction Developed at Lawrence Berkeley National Laboratory (LBNL) Trained on NERSC Perlmutter supercomputer

Joachimiak, Marcin [Lawrence Berkeley National Lab↗

An Agenda for Multimodal Foundation Models for Earth Observation

Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.

Ambrozio Dias, Philipe↗

CRCNS21 Computational Models of Multisensory Integration by Upper Limb in Humanoids and Amputees

This international collaborative research project between Johns Hopkins University (JHU) and the Technical University of Munich (TUM) investigated how the human brain processes and integrates multiple types of sensory information, such as touch and force, with the goal of improving prosthetic limbs for amputees and advancing sensory capabilities in humanoid robots. The research advanced our understanding of how the brain responds to sensory feedback in upper-limb amputees. Through experiments in which amputees received electrical stimulation while performing phantom hand movements, we demonstrated that sensory feedback activates the cortical sensorimotor and multisensory regions, and that these regions communicate dynamically during stimulation. Experiments with intact-limb participants explored the integration of visual, haptic, and force feedback, as well as in virtual reality motor training, further showing how the brain processes multimodal sensory information. In addition, this research inspired work on examining the reliability of where amputees perceive sensations over time, which contributed to a successful doctoral fellowship for continued investigation. Our collaborators at TUM improved multimodal sensor technology combining tactile and thermal feedback for humanoid robots, demonstrating the feasibility of integrating multiple sensor types into a unified system for detecting and responding to environmental stimuli. The experimental methods and analysis techniques developed across both teams, including functional network analysis and multimodal sensor integration, provide a foundation for future research in prosthetics and robotics. This research benefits the public by generating knowledge about how amputees process restored sensory information. Advances in humanoid sensing contribute to safer human-robot interaction. The project also fostered international collaboration and cross-disciplinary training: one TUM doctoral student spent a summer at JHU working on multimodal sensor integration, while two JHU students traveled to TUM to host workshops on neuromorphic sensory encoding and sensory integration.

42 ENGINEERING↗

Capturing Travel Mode Adoption in Designing On-Demand Multimodal Transit Systems

This paper studies how to integrate rider mode preferences into the design of on-demand multimodal transit systems (ODMTSs). It is motivated by a common worry in transit agencies that an ODMTS may be poorly designed if the latent demand, that is, new riders adopting the system, is not captured. This paper proposes a bilevel optimization model to address this challenge, in which the leader problem determines the ODMTS design, and the follower problems identify the most cost efficient and convenient route for riders under the chosen design. The leader model contains a choice model for every potential rider that determines whether the rider adopts the ODMTS given her proposed route. To solve the bilevel optimization model, the paper proposes an exact decomposition method that includes Benders optimal cuts and no-good cuts to ensure the consistency of the rider choices in the leader and follower problems. Moreover, to improve computational efficiency, the paper proposes upper and lower bounds on trip durations for the follower problems, valid inequalities that strengthen the no-good cuts, and approaches to reduce the problem size with problem-specific preprocessing techniques. The proposed method is validated using an extensive computational study on a real data set from the Ann Arbor Area Transportation Authority, the transit agency for the broader Ann Arbor and Ypsilanti region in Michigan. The study considers the impact of a number of factors, including the price of on-demand shuttles, the number of hubs, and access to transit systems criteria. The designed ODMTSs feature high adoption rates and significantly shorter trip durations compared with the existing transit system and highlight the benefits of ensuring access for low-income riders. Finally, the computational study demonstrates the efficiency of the decomposition method for the case study and the benefits of computational enhancements that improve the baseline method by several orders of magnitude. Funding: This research was partly supported by National Science Foundation [Leap HI Proposal NSF-1854684] and the Department of Energy [Research Award 7F-30154].

Operations Research & Management Science↗

Advanced Facility Design and AI/ML Enabled Safeguards to Establish Secure, Economical Recycling of Fast Reactor Fuels (Final Scientific/Technical Report)

The project, "Advanced Facility Design and AI/ML Enabled Safeguards to Establish Secure, Economical Recycling of Fast Reactor Fuels," represents a significant advancement in nuclear fuel recycling technology. It integrates cutting-edge multimodal sensor fusion, machine learning (ML), and digital twin (DT) technologies to address challenges in material safeguarding, process optimization, and regulatory compliance for pyroprocessing facilities. This research has significantly enhanced the understanding of pyrochemical fuel recycling processes by developing innovative tools and methodologies. The Multimodal Safeguards Monitoring Unit (MSMU) combines electroanalytical techniques, Raman spectroscopy, and differential thermal analysis (DTA) to enable high-fidelity, near-real-time material accountancy measurements. Machine learning techniques, such as Long Short-Term Memory (LSTM) autoencoders, are utilized to detect anomalies in material balances and sensor data, improving the reliability of safeguards monitoring. Additionally, digital twin technology has been established to provide real-time system-level monitoring and diagnostics, integrating physics-based models with sensor data to optimize process safety and efficiency.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Improving streamflow predictions across CONUS by integrating advanced machine learning models and diverse data

Accurate streamflow prediction is crucial to understand climate impacts on water resources and develop effective adaption strategies. A global long short-term memory (LSTM) model, using data from multiple basins, can enhance streamflow prediction, yet acquiring detailed basin attributes remains a challenge. To overcome this, we introduce the Geo-vision transformer (ViT)-LSTM model, a novel approach that enriches LSTM predictions by integrating basin attributes derived from remote sensing with a ViT architecture. Applied to 531 basins across the Contiguous United States, our method demonstrated superior prediction accuracy in both temporal and spatiotemporal extrapolation scenarios. Geo-ViT-LSTM marks a significant advancement in land surface modeling, providing a more comprehensive and effective tool for better understanding the environment responses to climate change.

Tayal, Kshitij↗

Predicting and Controlling Corrosion (Abbreviated Final Report)

Corrosion accumulates significant replacement costs in the transportation, utility, manufacturing, and infrastructure sectors. It also underpins several of LLNL’s core missions in stockpile stewardship, defense, and energy security. Whereas most corrosion and aging models are empirically parameterized to describe well-defined conditions late in the reaction, the factors that determine the early stages of corrosion—during which mitigation could be most impactful— are poorly understood. This project addressed the critical need for new approaches to predict the kinetics of corrosion initiation based on firm physical and chemical understanding. The activities encompassed degradation of relevant metals in both hydrogen-rich and environmental corrosion scenarios. The team integrated state-of-the-art multiscale simulation, in situ characterization, and data science within three technical thrusts: hydriding of Ti alloys; aqueous corrosion of Al and Ni-Cr alloys; and degradation of additively manufactured 316L stainless steel. In each case, novel capabilities were developed to identify and track the impacts of key atomistic, compositional, and microstructural features on the metal systems. For hydriding, protocols were developed to tightly integrate multiscale models, advanced multimodal characterization, and machine learning to determine how hydrogen interacts with native passivating surface oxides and nucleates new undesired phases, shedding new light on the critical role of grain boundaries, interfaces, and atomically disordered regions. For aqueous corrosion, the project demonstrated methods to predict dissolution rates of metal surfaces in corrosive solutions, to measure and understand microstructural and grain orientation effects on corrosion susceptibility, and to investigate competing growth and dissolution kinetics of surface oxides. For additively manufactured metals, analysis using state-of-the-art microscopy techniques revealed the role of specific heterogeneities invoked during laser processing, including cellular structure, dislocations, and precipitates, on corrosion susceptibility. In addition to new capabilities and understanding, the project provided an avenue for workforce development, as well as key partnerships with stakeholders in corrosion science.

08 HYDROGEN↗

Autonomous continuous flow reactor synthesis for scalable atom-precision

With new instrumentation design, robotics, and in-operando hyphenated analytical tool automation, the intelligent discovery of synthesis pathways is becoming feasible. It can potentially bridge the gap for the scale-up of new materials. In this article, we review current progress and describe a new system that uses an autonomous continuous flow chemistry framework to translate high-quality lead molecules and materials to quantities that can meet scalability demands. At the core is a continuous flow synthesis platform that can design its viable synthesis pathway to a particular molecule or material and then autonomously carry it out. This is realized by integrating: (1) A workflow/architecture for multimode chemical/materials characterization in-line. The in-line characterization modes are NMR, ESR, IR, Raman, UV-Vis, GC-MS, and HPLC, along with ex-situ modes for X-Ray and neutron scattering; (2) Integration for feedback/analysis/data storage of the control variables; (3) A core software stack that includes deep learning and reinforcement learning alongside quantum chemistry and molecular dynamics; (4) On-demand compute architectures that parse calculations to compute resources needed which include light-weight edge, mid-level edge (NVIDA DGX-2), and high-performance computing. We demonstrate preliminary results on how this autonomous reactor system can enhance our ability to deliver deuterated materials, copolymers, and site-substituted molecules.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

A comparative study of multimodal data fusion strategies for planetary spectroscopy

Integrating heterogeneous data sources can improve scientific inference when different modalities capture complementary information, but doing so is challenging in high-dimensional, small-sample settings. In spectroscopy for planetary exploration, Laser-Induced Breakdown Spectroscopy (LIBS), Raman Spectroscopy (Raman), Visible Infrared Spectroscopy (VISIR), and Mid-Infrared Spectroscopy (MIR) each examine different aspects of composition and mineralogy, raising fundamental questions about when and how data fusion improves predictive performance. Using a Mars-relevant set of geologic standards with measurements from all four modalities, we present a rigorous systematic evaluation of four data fusion strategies: low-level (data) fusion, mid-level (feature) fusion, high-level (decision) fusion, and residual-boosting (sequential) fusion. We assess performance in predicting oxide composition via nested cross-validation and corrected significance testing to evaluate whether data fusion improves upon single-modality baselines. We show that data fusion does not uniformly improve accuracy, and that observed gains are modest, oxide-dependent, and sensitive to modality and model structure. To move beyond aggregate accuracy metrics, we use model coefficients, permutation importance, and residual gain analysis to examine how the fusion models weight individual modalities and to identify patterns of apparent complementarity or redundancy. Though focused on spectroscopy for planetary exploration, our framework for data fusion evaluation and interpretation extends to other scientific domains with heterogeneous and scarce data and provides a principled approach evaluating data fusion strategies, interpreting modality contributions, and understanding tradeoffs among data fusion strategies.

97 MATHEMATICS AND COMPUTING↗