Search NASA⌕ Search

SEARCH · Search NASA

Results for “Multimodal integration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Parallel measurement of transcriptomes and proteomes from same single cells using nanodroplet splitting

Single-cell multiomics provides comprehensive insights into gene regulatory networks, cellular diversity, and temporal dynamics. Here, we introduce nanoSPLITS (nanodroplet SPlitting for Linked-multimodal Investigations of Trace Samples), an integrated platform that enables global profiling of the transcriptome and proteome from same single cells via RNA sequencing and mass spectrometry-based proteomics, respectively. Benchmarking of nanoSPLITS demonstrates high measurement precision with deep proteomic and transcriptomic profiling of single-cells. We apply nanoSPLITS to cyclin-dependent kinase 1 inhibited cells and found phospho-signaling events could be quantified alongside global protein and mRNA measurements, providing insights into cell cycle regulation. We extend nanoSPLITS to primary cells isolated from human pancreatic islets, introducing an efficient approach for facile identification of unknown cell types and their protein markers by mapping transcriptomic data to existing large-scale single-cell RNA sequencing reference databases. Accordingly, we establish nanoSPLITS as a multiomic technology incorporating global proteomics and anticipate the approach will be critical to furthering our understanding of biological systems.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Parallel measurement of transcriptomes and proteomes from same single cells using nanodroplet splitting

Single-cell multiomics provides comprehensive insights into gene regulatory networks, cellular diversity, and temporal dynamics. While tools for co-profiling single-cell genomes, transcriptomes, and epigenomes are available, accessing proteomes in parallel is more challenging. We developed nanoSPLITS (nanodroplet SPlitting for Linked-multimodal Investigations of Trace Samples), an integrated platform that enables global profiling of the transcriptome and proteome from same single cells using RNA sequencing and mass spectrometry-based proteomics, respectively. nanoSPLITS can precisely quantify over 5000 genes, 2000 proteins, and 140 phosphopeptides per single cell and identify candidate cell markers from these modalities. By exploring Cdk1-mediated cell cycle arrest, we demonstrate how nanoSPLITS single-cell multiomics can provide comprehensive cellular characterization with insights into covarying protein/gene clusters, unique phosphorylation events, and mitotic pathways.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Unveiling the Structure and Dynamics of Water Confined in Colloidal Boehmite Suspensions

Thin fluid layers confined between nanoparticles play an important role in several natural and industrial systems, including radioactive wastes stored in tanks at the U.S. Department of Energy’s Hanford site. Multimodal neutron and computational approaches have been integrated to examine the properties of one, two, and four layers of water (H 2 O or D 2 O) on nanoparticulate, hydrous, or deuterated boehmite (γ-AlOOH or γ-AlOOD). Exposure of deuterated boehmite to H 2 O at 90 °C yielded rapid H/D isotopic exchange likely driven by a Grotthus-like proton-hopping mechanism. The in-plane and out-of-plane vibrations of the structural hydroxyls involved in this exchange were observed for both the hydrated and deuterated boehmite. These observations were confirmed by molecular dynamics simulations that also showed that single water molecules on the (010) surface bond to the structure via four bonds: two from hydrogens (deuteriums) on the water to surrounding oxygens and two from the water’s oxygen to surrounding OD/OH. Bulk water-like properties began to appear once four monolayers of water had been added, but steric crowding limited water diffusion rates once two monolayers had been added. The super-Arrhenius temperature dependence observed at four monolayers indicated glass-like behavior in a well-formed hydrogen-bonding network. Such a network is not sufficiently developed, however, when the surface water coverage is less than four layers. In conclusion, the unique nature of these layers can provide critical information for understanding forces between particles in proximity, and resultant effects on suspension rheology.

Anovitz, Lawrence M. [Oak Ridge National Laborator↗
DOE OSTI2023

Single-cell multimodal imaging uncovers energy conversion pathways in biohybrids

Microbe–semiconductor biohybrids, which integrate microbial enzymatic synthesis with the light-harvesting capabilities of inorganic semiconductors, have emerged as promising solar-to-chemical conversion systems. Improving the electron transport at the nano–bio interface and inside cells is important for boosting conversion efficiencies, yet the underlying mechanism is challenging to study by bulk measurements owing to the heterogeneities of both constituents. Here we develop a generalizable, quantitative multimodal microscopy platform that combines multi-channel optical imaging and photocurrent mapping to probe such biohybrids down to single- to sub-cell/particle levels. We uncover and differentiate the critical roles of different hydrogenases in the lithoautotrophic bacterium Ralstonia eutropha for bioplastic formation, discover this bacterium’s surprisingly large nanoampere-level electron-uptake capability, and dissect the cross-membrane electron-transport pathways. Furthermore, this imaging platform, and the associated analytical framework, can uncover electron-transport mechanisms in various types of biohybrid, and potentially offers a means to use and engineer R. eutropha for efficient chemical production coupled with photocatalytic materials.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Possibility of an integrated transmission electron microscope: enabling complex in-situ experiments

Abstract Multimodal in-situ experiments are the wave of the future, as this approach will permit multispectral data collection and analysis during real-time nanoscale observation. In contrast, the evolution of technique development in the electron microscopy field has generally trended toward specialization and subsequent bifurcation into more and more niche instruments, creating a challenge for reintegration and backward compatibility for in-situ experiments on state-of-the-art microscopes. We do not believe this to be a requirement in the field; therefore, we propose an adaptive instrument that is designed to allow nearly simultaneous collection of data from aberration-corrected transmission electron microscopy (TEM), probe-corrected scanning transmission electron microscopy, ultrafast TEM, and dynamic TEM with a flexible in-situ testing chamber, where the entire instrument can be modified as future technologies are developed. The value would be to obtain a holistic understanding of the underlying physics and chemistry of the process-structure–property relationships in materials exposed to controlled extreme environments. Such a tool would permit the ability to explore, in-situ, the active reaction mechanisms in a controlled manner emulating those of real-world applications with nanometer and nanosecond resolution. If such a powerful tool is developed, it has the potential to revolutionize our materials understanding of nanoscale mechanisms and transients. Graphical Abstract

36 MATERIALS SCIENCE↗
DOE OSTI2024

Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)

Leveraging the chemical data available in legacy formats such as publications and patents is a significant challenge for the community. Automated reaction mining offers a promising solution to unleash this knowledge into a learnable digital form and therefore help expedite materials and reaction discovery. However, existing reaction mining toolkits are limited to single input modalities (text or images) and cannot effectively integrate heterogeneous data that is scattered across text, tables, and figures. In this work, we go beyond single input modalities and explore multimodal large language models (MLLMs) for the analysis of diverse data inputs for automated electrosynthesis reaction mining. We compiled a test dataset of 65 articles (MERMES-T24 set) and employed it to benchmark five prominent MLLMs against two critical tasks: (i) reaction diagram parsing and (ii) resolving cross-modality data interdependencies. The frontrunner MLLM achieved ≥96% accuracy in both tasks, with the strategic integration of single-shot visual prompts and image pre-processing techniques. We integrate this capability into a toolkit named MERMES (multimodal reaction mining pipeline for electrosynthesis). Our toolkit functions as an end-to-end MLLM-powered pipeline that integrates article retrieval, information extraction and multimodal analysis for streamlining and automating knowledge extraction. This work lays the groundwork for the increased utilization of MLLMs to accelerate the digitization of chemistry knowledge for data-driven research.

Leong, Shi Xuan↗
DOE OSTI2024

Mode-multiplexed photonic integrated vector dot-product core from inverse design

Photonic computing has the potential to harness the full degrees of freedom (DOFs) of the light field, including the wavelength, spatial mode, spatial location, phase quadrature, and polarization, to achieve a higher level of computing parallelism and scalability than digital electronic processors. While multiplexing using the wavelength and other DOFs can be readily integrated on silicon photonics platforms with compact footprints, conventional mode-division multiplexed (MDM) photonic designs occupy areas exceeding tens to hundreds of microns for a few spatial modes, significantly limiting their scalability. Here, we utilize inverse design to demonstrate an ultracompact photonic computing core that calculates vector dot products based on MDM coherent mixing. Our dot-product core integrates the functionalities of two-mode multiplexers and one multimode coherent mixer within a nominal footprint of 5 μm x 3 μm . We have experimentally demonstrated computing examples on the fabricated dot-product core, including complex number multiplication and motion estimation using optical flow. The compact dot-product core design enables large-scale on-chip integration in a parallel photonic computing primitive cluster for high-throughput scientific computing and computer vision tasks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Gate-Tunable Short-Wave Infrared Polycrystalline GeSn Phototransistors on Noncrystalline Substrates

GeSn is a group-IV alloy with immense potential to advance microelectronics technology due to its intrinsic compatibility with existing Si CMOS processes. With a sufficiently high Sn composition, GeSn is classified as a direct bandgap semiconductor. Polycrystalline GeSn holds several additional advantages, including its significantly lower synthesis cost compared to its epitaxial counterpart, as well as the versatility to grow these films on a variety of substrates. Here, in this work, we present a polycrystalline thin-film GeSn phototransistor on a fused silica substrate with a Sn composition of ~10%, showing a photoresponse in the short-wave infrared wavelength range, critical for emerging sensing applications. This device shows a gate-tunable response, with responsivities approaching up to 1.7 mA/W with only a 30 nm-thick GeSn layer. Furthermore, phototransistors offer additional adaptability through gating, which allows for the reduction of dark current. This not only enhances the signal-to-noise ratio but also offers more flexible integration with various image sensor readout implementations using different substrates. The specific detectivity of this phototransistor is within an order of magnitude of those of previously reported GeSn photodetectors grown by molecular beam epitaxy and chemical vapor deposition, even though the absorber is 3 to 20× thinner while the electrode spacing for photocarrier transport is approximately 15× longer than the carrier diffusion length in this work, showing great potential benefits of extending similar device structures to epitaxial GeSn layers. As these GeSn phototransistors utilize a noncrystalline substrate, our work establishes a fundamentally more versatile path toward monolithically integrated GeSn-based photodetectors for next-generation multimodal sensors.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Joint Optimization of Multimodal Transit Frequency and Shared Autonomous Vehicle Fleet Size with Hybrid Metaheuristic and Nonlinear Programming

Shared autonomous vehicles (SAVs) bring competition to traditional transit services but redesigning multimodal transit network can utilize SAVs as feeders to enhance service efficiency and coverage. This paper presents an optimization framework for the joint multimodal transit frequency and SAV fleet size problem, a variant of the transit network frequency setting problem. The objective is to maximize total transit ridership (including SAV-fed trips and subtracting boarding rejections) across multiple time periods under budget constraints, considering endogenous mode choice (transit, point-to-point SAVs, driving) and route selection, while allowing for strategic route removal by setting frequencies to zero. Due to the problem’s non-linear, non-convex nature and the computational challenges of large-scale networks, we develop a hybrid solution approach that combines a metaheuristic approach (particle swarm optimization) with nonlinear programming for local solution refinement. To ensure computational tractability, the framework integrates analytical approximation models for SAV waiting times based on fleet utilization, multimodal network assignment for route choice, and multinomial logit mode choice behavior, bypassing the need for computationally intensive simulations within the main optimization loop. Applied to the Chicago metropolitan area’s multimodal network, our method illustrates a 33.3% increase in transit ridership through optimized transit route frequencies and SAV integration, particularly enhancing off-peak service accessibility and strategically reallocating resources.

Ng, Max↗
DOE OSTI2025

Predicting High‐Resolution Spatial and Spectral Features in Mass Spectrometry Imaging with Machine Learning and Multimodal Data Fusion

Recent advancements in molecular Mass Spectrometry Imaging have sparked interest in integrating high spatial resolution methods with molecular mass-spectrometry-based chemical imaging. Fusion-based algorithms have proven effective in generating high spatial-resolution molecular mass spectra. However, a significant challenge stems from the differing physical mechanisms underlying image generation and data upsampling techniques, potentially leading to discrepancies in integrated information channels. Integrating physical constraints into data processing workflows is essential to tackle this issue. In this study, we propose an innovative approach that merges data from Fourier transform ion cyclotron resonance (FTICR), time-of-flight matrix-assisted laser desorption/ionization, and time-of-flight secondary ion mass spectrometry imaging techniques. By leveraging FT-ICR's unparalleled spectral resolution and ToF-SIMS's exceptional spatial resolution, we achieve submicron spatial resolution, enabling the observation of intact molecular species with remarkable spectral precision. Canonical correlation analysis is employed to incorporate physical constraints. Through sophisticated image processing and machine learning techniques, the results of this fusion hold significant promise for advancing our comprehension of complex systems and unveiling concealed molecular intricacies.

canonical correlation analysis↗
DOE OSTI2020

Robust Group Subspace Recovery: A New Approach for Multi-Modality Data Fusion

Robust Subspace Recovery (RoSuRe) algorithm was recently introduced as a principled and numerically efficient algorithm that unfolds underlying Unions of Subspaces (UoS) structure, present in the data. The union of Subspaces (UoS) is capable of identifying more complex trends in data sets than simple linear models. In this work, we build on and extend RoSuRe to prospect the structure of different data modalities individually. We propose a novel multi-modal data fusion approach based on group sparsity which we refer to as Robust Group Subspace Recovery (RoGSuRe). Relying on a bi-sparsity pursuit paradigm and non-smooth optimization techniques, the introduced framework learns a new joint representation of the time series from different data modalities, respecting an underlying UoS model. We subsequently integrate the obtained structures to form a unified subspace structure. The proposed approach exploits the structural dependencies between the different modalities data to cluster the associated target objects. The resulting fusion of the unlabeled sensors’ data from experiments on audio and magnetic data has shown that our method is competitive with other state of the art subspace clustering methods. The resulting UoS structure is employed to classify newly observed data points, highlighting the abstraction capacity of the proposed method.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Postearthquake Damage Mapping via Remote Sensing: Lessons From the 2023 Türkiye Disaster

This review addresses the urgent need for scalable, accurate, and reproducible remote sensing solutions following the February 2023 Türkiye earthquakes. It synthesizes the contributions of five peer-reviewed studies published in the IEEE JSTARS Special Issue on postearthquake damage and risk assessment. These studies cover areas such as damage classification with deep learning, fusion of multisource remote sensing data, creation of benchmark datasets, detailed damage mapping, and analysis of geophysical signals using outgoing longwave radiation. The article summarizes the methodological approaches and the practical relevance of the reviewed studies for detecting, evaluating, and quantifying damage, and outlines key challenges, including model generalization, class ambiguity, and data integration. It also discusses emerging trends, including explainable artificial intelligence, multimodal data fusion, and open-data platforms. This synthesis provides a foundation for building robust, interpretable, and real-time disaster response systems and aims to guide future research in earthquake-related Earth observation and rapid damage assessment.

Taskin, Gulsen [Istanbul Technical University] (OR↗
DOE OSTI2025

In situ Visible Light and Thermal Imaging Data from a Laser Powder Bed Fusion Additive Manufacturing Process Co-Registered to X-ray Computed Tomography and Fatigue Data

This dataset is comprised of in situ sensing data collected during a laser-based powder bed fusion additive manufacturing process, as well as rasterized scan path information, post-build X-ray computed tomography (XCT), and fatigue test results. A total of 64 cylinders, approximately 15 mm in diameter and 102 mm tall, were printed out of stainless steel 316H on a Colibrium Additive Concept Laser M2 Series 5 machine. Parameters known to produce dense material were used to construct 56 of these cylinders, while the remaining 8 cylinders were printed with relatively high energy density parameters prone to producing keyhole pores. In addition, two spatter generation blocks were constructed upstream of the 64 cylinders such that ejecta produced during the melting of the spatter generators were stochastically seeded onto the 64 cylinders. Based on previous experiments, these spatter particles were theorized to produce stochastic lack-of-fusion pores. During the construction of the build, high-resolution images of reflected light in the visible spectrum were captured both before and after recoating for each print layer. Additionally, temporally integrated thermal imaging in the near infrared spectrum produced integrated sum and max images on a layerwise basis. The multimodal in situ data has been co-registered to the build plate coordinate system, allowing for identification of process anomalies (e.g., spatter particles) apparent in the two sensors. Following construction of the build, the cylinders were subjected to XCT to identify internal flaws, and the resulting data have also been registered to the build plate coordinate system. Finally, 60 of the 64 cylinders were machined into fatigue coupons conforming to ASTM E466 and subsequently subjected to either high- or -low-cycle fatigue testing. The results of the fatigue tests have also been included in the dataset, and the XCT data corresponded to the approximate location of the gauge sections of the machine fatigue specimen geometry.

42 ENGINEERING↗
DOE OSTI2025

Multimodal super-resolution: discovering hidden physics and its application to fusion plasmas

Understanding complex physical systems often requires integrating data from multiple diagnostics, each with limited resolution or coverage. We present a machine learning framework that reconstructs synthetic high-temporal-resolution data for a target diagnostic using information from other diagnostics, without direct target measurements during the inference. This multimodal super-resolution technique improves diagnostic robustness and enables monitoring even in case of measurement failures or degradation. Applied to fusion plasmas, our method targets edge-localized modes (ELMs), which can damage plasma-facing materials. By reconstructing super-resolution Thomson Scattering data from complementary diagnostics, we uncover fine-scale plasma dynamics and validate the role of resonant magnetic perturbations (RMPs) in ELM suppression through magnetic island formation. The approach provides new observation supporting the plasma profile flattening due to these islands. Our results demonstrate the framework’s ability to generate high-fidelity synthetic diagnostics, offering a powerful tool for ELM control development in future reactors like ITER. The approach is broadly transferable to other domains facing sparse, incomplete, or degraded diagnostic data, opening new avenues for discovery.

Jalalvand, Azarakhsh [Princeton Univ., NJ (United ↗
DOE OSTI2020

Connecting Femtosecond Transient Absorption Microscopy with Spatially Coregistered Time Averaged Optical Imaging Modalities

Multimodal all-optical imaging involving coregistered femtosecond transient absorption microscopy (TAM), time-integrated photoluminescence (PL), and steady-state modalities such as confocal reflectance and transmission offers an appealing approach to gain a comprehensive understanding of complex electronic excited-state phenomena in spatially heterogeneous systems. A unique combination of these modalities allows us to unravel not only the competing electronic excited-state dynamical processes but also the underlying morphological information with simultaneous high temporal and spatial resolution. However, correlating the various images obtained from time-resolved and time-independent modalities is generally nontrivial and particularly challenging when the electronic dynamics under study evolve in both time and space. Here, we demonstrate a new approach for rationally correlating time-resolved microscopy with coregistered time-integrated or steady-state modalities. Specifically, our approach involves an extended global lifetime analysis of the time-resolved microscopic data set to separate distinct dynamical processes taking place on commensurate time scales, and the resulting decay-associated amplitude maps (DAAMs) were applied to explore correlations with the images acquired using time-independent modalities. The feasibility of our approach was validated through analyzing a multimodal data set acquired from a thin film of chloride-containing mixed lead halide perovskites (CH 3 NH 3 PbI 3–x Cl x ) using femtosecond transient absorption, time-integrated PL, and confocal reflectance microscopies. Analysis of the results obtained enable us to gain new insight into the complex ultrafast relaxation dynamics in this highly heterogeneous system.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

Rolling Root Mean Square Based Multimodal Anomaly Detection for Real Time Monitoring of Smart Grid

Reliable real-time monitoring is valuable for maintaining the operational integrity of modern electrical smart grids. Deployment of heterogeneous sensing technologies in substations has enabled high-resolution, multichannel waveform monitoring, but also introduces challenges for anomaly detection due to noise, baseline drift, and modality-dependent signal characteristics. In this work, we present a computationally efficient unsupervised method for multimodal event detection based on Rolling Root Mean Square based Event Detection (RRMSED). The method is developed using in-house, field deployed sensors collecting data at a utility substation. The sensing system comprises voltage and current sensors, triaxial accelerometers, and magnetometers, collectively capturing electrical, vibrational, and magnetic waveform measurements at high temporal resolution. RRMSED operates by extracting rolling RMS energy features and their first-order temporal differences from consecutive waveform segments for each channel and then applying channel-specific statistical thresholds learned from historical data. A persistence-based exceedance logic is employed to robustly identify transient events while suppressing impulsive noise, and to provide precise temporal localization with high resolution. The framework is designed for continuous server-side operation and can be deployed in real time without requiring complex models. Experiments on simulated waveform data with known ground truth demonstrate low false positive (FP) and false negative (FN) rates. Application to real substation data shows RRMSED to identify events that are not captured by conventional monitoring indicators including fast transient detection algorithm currently deployed in the system. These results indicate that rolling RMS based features provide an effective and practical basis for real-time multimodal event detection in smart-grid substations.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗