Search NASA⌕ Search

SEARCH · Search NASA

Results for “multimodal processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)

Leveraging the chemical data available in legacy formats such as publications and patents is a significant challenge for the community. Automated reaction mining offers a promising solution to unleash this knowledge into a learnable digital form and therefore help expedite materials and reaction discovery. However, existing reaction mining toolkits are limited to single input modalities (text or images) and cannot effectively integrate heterogeneous data that is scattered across text, tables, and figures. In this work, we go beyond single input modalities and explore multimodal large language models (MLLMs) for the analysis of diverse data inputs for automated electrosynthesis reaction mining. We compiled a test dataset of 65 articles (MERMES-T24 set) and employed it to benchmark five prominent MLLMs against two critical tasks: (i) reaction diagram parsing and (ii) resolving cross-modality data interdependencies. The frontrunner MLLM achieved ≥96% accuracy in both tasks, with the strategic integration of single-shot visual prompts and image pre-processing techniques. We integrate this capability into a toolkit named MERMES (multimodal reaction mining pipeline for electrosynthesis). Our toolkit functions as an end-to-end MLLM-powered pipeline that integrates article retrieval, information extraction and multimodal analysis for streamlining and automating knowledge extraction. This work lays the groundwork for the increased utilization of MLLMs to accelerate the digitization of chemistry knowledge for data-driven research.

Leong, Shi Xuan↗
DOE OSTI2025

Fast Sideband Control of a Weakly Coupled Multimode Bosonic Memory

Circuit quantum electrodynamics (cQED) with superconducting cavities coupled to nonlinear circuits like transmons offers a promising platform for hardware-efficient quantum information processing. We address critical challenges in realizing this architecture by weakening the dispersive coupling while also demonstrating fast, high-fidelity multimode control by dynamically amplifying gate speeds through transmon-mediated sideband interactions. This approach enables transmon-cavity SWAP gates, for which we achieve speeds up to 30 times larger than the bare dispersive coupling. Combined with transmon rotations, this allows for efficient, universal state preparation in a single cavity mode, though achieving unitary gates and extending control to multiple modes remains a challenge. In this work, we overcome this by introducing two sideband control strategies: (1) a shelving technique that prevents unwanted transitions by temporarily storing populations in sideband-transparent transmon states and (2) a method that exploits the dispersive shift to synchronize sideband transition rates across chosen photon-number pairs to implement transmon-cavity SWAP gates that are selective on photon number. We leverage these protocols to prepare Fock and binomial code states across any of ten modes of a multimode cavity with millisecond cavity coherence times. We demonstrate the encoding of a qubit from a transmon into arbitrary vacuum and Fock state superpositions, as well as entangled NOON states of cavity mode pairs— a scheme extendable to arbitrary multimode Fock encodings. Furthermore, we implement a new binomial encoding gate that converts arbitrary transmon superpositions into binomial code states in $\qty{4}{\micro\second}$ (less than $1/\chi$), achieving an average post-selected final state fidelity of $\qty{96.3}{\percent}$ across different fiducial input states.

Huang, Jordan [Rutgers U., Piscataway]↗
DOE OSTI2022

A robust and interpretable machine learning approach using multimodal biological data to predict future pathological tau accumulation

The early stages of Alzheimer’s disease (AD) involve interactions between multiple pathophysiological processes. Although these processes are well studied, we still lack robust tools to predict individualised trajectories of disease progression. Here, we employ a robust and interpretable machine learning approach to combine multimodal biological data and predict future pathological tau accumulation. In particular, we use machine learning to quantify interactions between key pathological markers (β-amyloid, medial temporal lobe atrophy, tau and APOE 4) at mildly impaired and asymptomatic stages of AD. Using baseline non-tau markers we derive a prognostic index that: (a) stratifies patients based on future pathological tau accumulation, (b) predicts individualised regional future rate of tau accumulation, and (c) translates predictions from deep phenotyping patient cohorts to cognitively normal individuals. Our results propose a robust approach for fine scale stratification and prognostication with translation impact for clinical trial design targeting the earliest stages of AD.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

Coherent Pulse Echo in Hybrid Magnonics with Multimode Phonons

We report the hybridization of magnons and phonons is playing a critical role in the emerging field of hybrid magnonics because it combines the high tunability of magnetism with the long lifetime of mechanics for comprehensive coherent information processing. Recently there has been increasing interest in thin-film bulk acoustic waves because of their long lifetimes at high frequencies. However, the unique multimode nature of such phonon modes has not been exploited as an important resource for coherent information processing. In this Letter we study the simultaneous hybridization of multiple high-overtone bulk acoustic resonances with a magnon and a microwave mode. The demonstrated multimode hybridization allows us to observe coherent pulse echoes, opening opportunities for both fundamental studies and practical applications of hybrid magnonics.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2023

Revealing the Sodium Storage Mechanisms in Hard Carbon Pores

Abstract Hard carbon (HC) is the most promising anode for the commercialization of sodium‐ion batteries (NIBs); however, a general mechanism for sodium storage in HC remains unclear, obstructing the development of highly efficient anodes for NIBs. To elucidate the mechanism of sodium storage in the pores, operando synchrotron small‐angle X‐ray scattering, wide‐angle X‐ray scattering, X‐ray absorption near edge structure, Raman spectroscopy, and galvanostatic measurements are combined. The multimodal approach provides mechanistic insights into the sodium pore‐filling process for different HC microstructures including the pore sizes that are preferentially filled, the extent to which different pore sizes are filled, and how the defect concentration influences pore filling. It is observed that sodium in the larger pores has an increased pseudo‐metallic sodium character consistent with larger sodium clusters. Furthermore, it is shown that the HCs prepared at higher pyrolysis temperatures have a larger capacity from sodium stored in the pores and that sodium intercalation between graphene layers occurs simultaneously with the pore filling in the plateau region. Opportunities are outlined to improve the performance of HC anodes by fully utilizing the pores for sodium storage, helping to pave the way for the commercialization of sodium ion batteries.

Kitsu Iglesias, Luis↗
DOE OSTI2024

Artificial intelligence to unlock real-world evidence in clinical oncology: A primer on recent advances

Purpose: Real world evidence is crucial to understanding the diffusion of new oncologic therapies, monitoring cancer outcomes, and detecting unexpected toxicities. In practice, real world evidence is challenging to collect rapidly and comprehensively, often requiring expensive and time-consuming manual case-finding and annotation of clinical text. In this Review, we summarise recent developments in the use of artificial intelligence to collect and analyze real world evidence in oncology. Methods: We performed a narrative review of the major current trends and recent literature in artificial intelligence applications in oncology. Results: Artificial intelligence (AI) approaches are increasingly used to efficiently phenotype patients and tumors at large scale. These tools also may provide novel biological insights and improve risk prediction through multimodal integration of radiographic, pathological, and genomic datasets. Custom language processing pipelines and large language models hold great promise for clinical prediction and phenotyping. Conclusions: Despite rapid advances, continued progress in computation, generalizability, interpretability, and reliability as well as prospective validation are needed to integrate AI approaches into routine clinical care and real-time monitoring of novel therapies.

60 APPLIED LIFE SCIENCES↗
NASA NTRS1984

Development of long wavelength semiconductor diode lasers near 28 microns for use in infrared heterodyne spectrometers

The development of tunable diode lasers operating in the 28 micrometers spectral region for use in infrared heterodyne spectrometers is reported. A process capable of yielding lasers emitting 500 micron W of multimode power, 112 micron W in a true single mode and true single mode operation at laser currents of up to 35% above threshold was developed. Results were obtained from narrow mesastripe (20 micrometer wide) short cavity (120 micrometer length) laser configurations. Six stripe geometry lasers, with a variety of cavity widths and lengths were delivered. The techniques to fabricate such devices was obtained and the long term reliability of such lasers by reproducible electrical and optical output characteristics fabrication from lasers are demonstrated.

Linden, K. J.↗
DOE OSTI2022

Development of Multimodal Few-Shot Analytics for Electron Micrographs

Recent advances in materials data analytics have provided new avenues for determining process-structure-property (PSP) linkages in a variety of materials. Machine learning techniques including few-shot learning have increased the efficiency of classifying microscopy images for the purposes of material characterization. Attempts at creating a multimodal approach can provide further improvements to current models and help extract more salient features from data. In this vein, raw spectrum data was taken to provide an additional modality to our current pyCHIP classifier. Modifications in segmentation also show potential in improving the accuracy of the pyCHIP classifier. Classifier output was analyzed using network graphs and unsupervised clustering algorithms such as spectral clustering to detect better segmentation methods than the current “chipping” approach. We suggest that the chip selection process can be automated in the future using a combination of these techniques to enable high-throughput analyses.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Droplet breakup by multimodal nonlinear Rayleigh Taylor instability

A droplet impacted by a shock wave will undergo a process of fragmentation due to the development of interfacial hydrodynamic instabilities. The interface experiences variable acceleration and shear that result in the development of both inertial (Rayleigh–Taylor) and shear (Kelvin–Helmholtz) instabilities. These perturbations grow in time and drive the fragmentation and breakup of the deformed droplet. Experiments are performed on nominally 0.86 mm water droplet subjected to a Mach 7.6 detonation wave, resulting in a high Weber number (~36,000) breakup event. Perturbation growth is measured from a series of high-speed (> MHz) shadowgraph images. It is proposed that, given the size of the large-scale perturbations observed in experiments, these instabilities are growing in the non-linear regime and can be described by bubble-merger models for nonlinear mixing. Calculations are performed for the growth rates and size of these instabilities using deformation and external flow models to establish the time-dependent boundary conditions. The concurrence of the measured perturbation widths and the predictions of the simple model lend credence to the theory. This novel approach serves to open a new avenue in the characterization of droplet breakup via hydrodynamic instabilities.

Droplet breakup↗
DOE OSTI2026

Thermal cycling-driven microstructural changes of eutectic Al–Si phase change materials in SS304 containers revealed by multi-modal imaging

Aluminum-based Al–Si alloys are widely used as phase change materials (PCMs) in thermal energy storage (TES) systems owing to their high volumetric latent heat and superior thermal conductivity. However, their long-term reliability is limited by degradation processes that remain insufficiently understood. In this work, we employ a multimodal, correlative characterization framework to systematically resolve the degradation behavior of eutectic Al–Si PCMs in contact with SS304 containers under repeated thermal cycling. By integrating high-resolution electron microscopy, three-dimensional X-ray fluorescence (3D XRF) imaging, and differential scanning calorimetry (DSC), we directly link spatially resolved compositional and microstructural evolution to changes in thermophysical properties. The correlative analysis reveals that elemental leaching of Fe, Cr, and Ni from the stainless-steel container into the PCM drives the formation of intermetallic compounds (IMCs) both at the interface and within the bulk PCM, leading to pronounced compositional heterogeneity. These interfacial reactions and diffusion-induced transformations progressively destabilize the Al–Si eutectic, reducing the effective phase-transforming fraction. Consistent with these observations, DSC measurements show a decrease in melting temperature and latent heat of fusion with thermal cycling. These results underscore the critical influence of interfacial reactions and material compatibility on the stability, durability, and overall performance of Al–Si-based TES systems.

25 ENERGY STORAGE↗
DOE OSTI2025

Kelvin probe force microscopy under ambient conditions

Kelvin probe force microscopy (KPFM) is a technique derived from atomic force microscopy that provides maps of surface potential or work function differences across material systems, with nanometre-scale resolution. KPFM is a useful tool for investigating electrical phenomena such as dipole orientation, interfacial charge transfer, charge accumulation, band bending and doping levels. This Primer aims to provide an overview of typical ambient-condition KPFM measurements, covering their underlying principles, experimental implementations and wide-ranging applications. Key KPFM variants, including amplitude and frequency modulation, heterodyne detection schemes and innovative open loop and pulsed force techniques, are discussed, with practical guidance on optimizing signal acquisition and reducing errors. Specialized approaches, such as time-resolved KPFM and multimodal KPFM, are discussed for their ability to capture dynamic charge processes and chemical information, respectively. Here, we highlight recent advances in KPFM applications, spanning metal alloys, soft matter, ferroelectrics, photovoltaics and 2D materials, showcasing its versatility across research domains. By addressing current limitations and identifying future opportunities, this Primer underscores the transformative potential of KPFM in advancing the understanding of nanoscale electrical phenomena.

Zahmatkeshsaredorahi, Amirhossein [Lehigh Univ., B↗
DOE OSTI2021

Predictability as a probe of manifest and latent physics: The case of atomic scale structural, chemical, and polarization behaviors in multiferroic Sm-doped BiFeO 3

The predictability of a certain effect or phenomenon is often equated with the knowledge of relevant physical laws, typically understood as a functional or numerically derived relationship between the observations and known states of the system. Correspondingly, observations inconsistent with prior knowledge can be used to derive new knowledge on the nature of the system or indicate the presence of yet unknown mechanisms. In this work, we explore the applicability of Gaussian processes (GP) to establish predictability and uncertainty of local behaviors from multimodal observations, providing an alternative to this classical paradigm. Using atomic resolution scanning transmission electron microscopy (STEM) of multiferroic Sm-doped BiFeO 3 across a broad composition range, we directly visualize the atomic structure and structural, physical, and chemical order parameter fields for the material. GP regression is used to establish the predictability of the local polarization field from different groups of parameters, including the adjacent polarization values and several combinations of physical and chemical descriptors, including lattice parameters, column intensities, etc. We observe that certain elements of microstructure, including charged and uncharged domain walls and interfaces with the substrate, are best predicted with specific combinations of descriptors, and this predictability and associated uncertainties are consistent across the composition series. The associated generative physical mechanisms are discussed. It is also found that certain parameter combinations tend to predict the orthorhombic phase in the cases where rhombohedral phase is observed, suggesting a potential role of clamping and confinement phenomena in phase equilibrium in Sm-BiFeO 3 system close to morphotropic phase boundary. We argue that predictability and uncertainty in observational data offer a new pathway to probe the physics of condensed matter systems from multimodal local observations.

74 ATOMIC AND MOLECULAR PHYSICS↗
NASA NTRS1995

A three-dimensional orthogonal laser velocimeter for the NASA Ames 7- by 10-foot wind tunnel

A three-component dual-beam laser-velocimeter system has been designed, fabricated, and implemented in the 7-by 10-Foot Wind Tunnel at NASA Ames Research Center. The instrument utilizes optical access from both sides and the top of the test section, and is configured for uncoupled orthogonal measurements of the three Cartesian coordinates of velocity. Bragg cell optics are used to provide fringe velocity bias. Modular system design provides great flexibility in the location of sending and receiving optics to adapt to specific experimental requirements. Near-focus Schmidt-Cassegrain optic modules may be positioned for collection of forward or backward scattered light over a large solid angle, and may be clustered to further increase collection solid angle. Multimode fiber optics transmit collected light to the photomultiplier tubes for processing. Counters are used to process the photomultiplier signals and transfer the processed data digitally via buffered interface controller to the host MS-DOS computer. Considerable data reduction and graphical display programming permit on-line control of data acquisition and evaluation of the incoming data. This paper describes this system in detail and presents sample data illustrating the system's capability.

Dunagan, Stephen E.↗
DOE OSTI2024

Crosstalk-robust quantum control in multimode bosonic systems

High-coherence superconducting cavities offer a hardware-efficient platform for quantum information processing. To achieve universal operations of these bosonic modes, the requisite nonlinearity is realized by coupling them to a transmon ancilla. However, this configuration is susceptible to crosstalk errors in the dispersive regime, where the ancilla frequency is Stark shifted by the state of each coupled bosonic mode. This leads to a frequency mismatch of the ancilla drive, lowering the gate fidelities. To mitigate such coherent errors, we employ quantum optimal control to engineer ancilla pulses that are robust to the frequency shifts. These optimized pulses are subsequently integrated into a recently developed echoed conditional displacement protocol for executing single- and two-mode operations. Through numerical simulations, we examine two representative scenarios: the preparation of single-mode Fock states in the presence of spectator modes and the generation of two-mode entangled Bell-cat states. Our approach markedly suppresses crosstalk errors, outperforming conventional ancilla control methods by orders of magnitude. These results provide guidance for experimentally achieving high-fidelity multimode operations and pave the way for developing high-performance bosonic quantum information processors.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

Multimodal imaging and machine learning to enhance microscope images of shale

A machine learning based image processing workflow is presented to enhance shale source rock microscopic images obtained using diverse imaging platforms. Images were acquired from a 30 μm diameter cylindrical Vaca Muerta shale sample using both nondestructive Transmission X-Ray Microscopy (TXM, alternately referred to as nano computed tomography) and destructive Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM). Output cross-sectional images from each modality were aligned using a combination of manual and automated registration techniques to create a registered image dataset. We then apply this dataset for two image processing tasks: prediction of image cross sections with SEM-like resolution from nondestructive TXM data and repair of charged region artifacts (localized accumulation of electrons) within SEM images. The image processing algorithms for both tasks use deep learning models, specifically image-to-image Convolutional Neural Networks (CNNs) and conditional Generative Adversarial Networks (cGANs). In the image enhancement tasks, we are able to achieve significant qualitative and quantitative improvement in TXM images. Here, the best model reaches an average Peak Signal to Noise Ratio (PSNR) of 15.8 dB. Conditioning on TXM data is also shown to reduce artifacts from SEM charging, achieving an average PSNR of 25.8 dB. Furthermore, our results suggest that properly trained and validated networks are capable of significant enhancement of images obtained using nondestructive techniques, thereby improving interpretation of two- and three-dimensional images while preserving samples for future use.

58 GEOSCIENCES↗
DOE OSTI2021

Multimodal Bayesian registration of noisy functions using Hamiltonian Monte Carlo

Functional data registration is a necessary processing step for many applications. The observed data can be inherently noisy, often due to measurement error or natural process uncertainty; which most functional alignment methods cannot handle. A pair of functions can also have multiple optimal alignment solutions, which is not addressed in current literature. In this paper, a flexible Bayesian approach to functional alignment is presented, which appropriately accounts for noise in the data without any pre-smoothing required. Additionally, by running parallel MCMC chains, the method can account for multiple optimal alignments via the multi-modal posterior distribution of the warping functions. To most efficiently sample the warping functions, the approach relies on a modification of the standard Hamiltonian Monte Carlo to be well-defined on the infinite-dimensional Hilbert space. In this work, this flexible Bayesian alignment method is applied to both simulated data and real data sets to show its efficiency in handling noisy functions and successfully accounting for multiple optimal alignments in the posterior; characterizing the uncertainty surrounding the warping functions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

A Deep Multimodal Representation Learning Framework for Accurate Molecular Properties Prediction

Drug discovery is a complex and challenging process, requiring the optimization of candidate compounds to identify those with the potential to become safe and effective drugs. Predicting molecular properties is an indispensable step in the drug discovery pipeline. Traditionally, this process is costly and time-intensive, involving multiple rounds of experiments and clinical trials, rendering it impractical for every candidate compound. Deep learning techniques have emerged as a promising approach to drug discovery to reduce the cost and time required to identify novel drugs. However, prevalent research in deep learning models focused on predicting molecular properties has primarily fixated on single-modal models, which utilize a single modality of data, neglecting the potential benefits of combining different data modalities. To overcome this limitation, we introduce MRL-Mol: a deep \textbf{M}ultimodal \textbf{R}epresentation \textbf{L}earning framework for accurate \textbf{Mol}ecular properties prediction. MRL-Mol harnesses three data modalities: sequence, graph, and image, augmenting the depth of comprehension. Leveraging a large-scale unlabeled dataset~($\sim$1M unique molecules), we pretrain MRL-Mol to extract inter- and intra-modal information. Our study demonstrates the superior performance of MRL-Mol in predicting molecular properties across six benchmark datasets, including both classification and regression tasks. Notably, MRL-Mol outperforms other state-of-the-art molecular properties prediction models. These findings suggest that by combining information from multiple data modalities, MRL-Mol can comprehend molecules better than single-modal deep learning models and identify molecular properties with better accuracy.

Yang, Yuxin↗
DOE OSTI2022

Multimode Characterization Approach for Understanding Cell-Level PV Performance and Degradation

Cell-level degradation processes impact the economic viability and large-scale deployment prospects for both established and emerging photovoltaic (PV) technologies. This project addresses the need to develop experimental and device-modeling approaches for studying cell-level degradation processes in photovoltaic (PV) devices using a variety of characterization techniques that provide access to complementary material and device properties. Our results demonstrate that by coupling characterization results with device modeling it is possible to develop comprehensive understanding of processes leading to performance limitations and degradation. This project developed a suite of novel measurement techniques including pulsed-light-bias operando X-ray and photoelectron spectroscopy (popXPS), light-biased scanning microwave impedance microscopy (sMIM), and near-field transport imaging (TI). In addition, operando characterization methodologies and in situ stressing capabilities have been developed and applied for techniques including electron-beam-induced current (EBIC), cathodoluminescence (CL), and Kelvin probe force microscopy (KPFM). Device-physics models were developed and applied to simulate correlative, multi-mode measurements to extract material and device parameters that control performance degradation. These characterization and modeling techniques were applied in a multi-mode approach to probe cell-level degradation mechanisms in Cd(Se,Te) and hybrid perovskite PV devices. Together these efforts contribute to foundational PV degradation science by establishing a framework for understanding PV performance degradation at the cell level and benefit the U.S. PV industry by providing resources in the form of novel experimental capabilities, knowledge gained, and available expertise that can accelerate research and development of improved PV device materials and architectures. The project provided a comprehensive understanding of degradation in baseline Cd(Se,Te) solar cells provided by our collaborators at Colorado State University (CSU). EBIC and CL-based measurements and revealed unusual collection and recombination profiles in these devices, which underwent significant changes with during stressing. KPFM and operando XPS measurements showed that device stressing permanently alters energy-band alignments at the (Mg,Zn)O/Cd(Se,Te) interface, which in turn account for an observed loss in fill factor. Studies on hybrid perovskite devices were hampered to a significant extent by delays related to the pandemic. Nevertheless, a set of hybrid perovskite devices (supplied through an NREL-industry partnership) were stress tested and characterized with techniques including EBIC, sMIM, popXPS/popUPS and optically excited TI. Available results from these measurements informed the device modeling effort and suggest that defects and related band offsets at the C60/LiF/hybrid perovskite interface are the primary source of degradation in these devices.

14 SOLAR ENERGY↗