Search NASA⌕ Search

SEARCH · Search NASA

Results for “multimodal dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

87 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Multimodal imaging and machine learning to enhance microscope images of shale

A machine learning based image processing workflow is presented to enhance shale source rock microscopic images obtained using diverse imaging platforms. Images were acquired from a 30 μm diameter cylindrical Vaca Muerta shale sample using both nondestructive Transmission X-Ray Microscopy (TXM, alternately referred to as nano computed tomography) and destructive Focused Ion Beam-Scanning Electron Microscopy (FIB-SEM). Output cross-sectional images from each modality were aligned using a combination of manual and automated registration techniques to create a registered image dataset. We then apply this dataset for two image processing tasks: prediction of image cross sections with SEM-like resolution from nondestructive TXM data and repair of charged region artifacts (localized accumulation of electrons) within SEM images. The image processing algorithms for both tasks use deep learning models, specifically image-to-image Convolutional Neural Networks (CNNs) and conditional Generative Adversarial Networks (cGANs). In the image enhancement tasks, we are able to achieve significant qualitative and quantitative improvement in TXM images. Here, the best model reaches an average Peak Signal to Noise Ratio (PSNR) of 15.8 dB. Conditioning on TXM data is also shown to reduce artifacts from SEM charging, achieving an average PSNR of 25.8 dB. Furthermore, our results suggest that properly trained and validated networks are capable of significant enhancement of images obtained using nondestructive techniques, thereby improving interpretation of two- and three-dimensional images while preserving samples for future use.

58 GEOSCIENCES↗
DOE OSTI2022

Fractal Characterization of Multimodal, Multiscale Images of Shale Rock Fracture Networks

An array of multimodal and multiscale images of fractured shale rock samples and analogs was collected with the aim of improving the numerical representation of fracture networks. 2D and 3D reconstructions of fractures and matrices span from 10 –6 to 100 m. The origin of the fracture networks ranged from natural to thermal maturation to hydraulically induced maturation. Images were segmented to improve fracture identification. Then, the fractal dimension and length distribution of the fracture networks were calculated for each image dataset. The resulting network connectivity and scaling associations are discussed at length on the basis of scale, sample and order of magnitude. Fracture network origin plays an important role in the resulting fracture systems and their scaling. Rock analogs are also evaluated using these descriptive tools and are found to be faithful depictions of maturation-induced fracture networks.

58 GEOSCIENCES↗
DOE OSTI2024

CrossMP: Enabling Cross-Modality Translation between Single-Cell RNA-Seq and Single-Cell ATAC-Seq through Web-Based Portal

In recent years, there has been a growing interest in profiling multiomic modalities within individual cells simultaneously. One such example is integrating combined single-cell RNA sequencing (scRNA-seq) data and single-cell transposase-accessible chromatin sequencing (scATAC-seq) data. Integrated analysis of diverse modalities has helped researchers make more accurate predictions and gain a more comprehensive understanding than with single-modality analysis. However, generating such multimodal data is technically challenging and expensive, leading to limited availability of single-cell co-assay data. Here, we propose a model for cross-modal prediction between the transcriptome and chromatin profiles in single cells. Our model is based on a deep neural network architecture that learns the latent representations from the source modality and then predicts the target modality. It demonstrates reliable performance in accurately translating between these modalities across multiple paired human scATAC-seq and scRNA-seq datasets. Additionally, we developed CrossMP, a web-based portal allowing researchers to upload their single-cell modality data through an interactive web interface and predict the other type of modality data, using high-performance computing resources plugged at the backend.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Hybrid Quantum–Classical Graph Transformers for Efficient Sentiment Analysis

Quantum Machine Learning (QML) offers a promising paradigm that leverages quantum computing principles to develop efficient and expressive models for learning from complex and structured data. Recent advances in natural language processing (NLP) and artificial intelligence (AI) have demonstrated capabilities in understanding, generating, and reasoning over linguistic and multimodal information. In this work, we present the Quantum Graph Transformer (QGT), a hybrid quantum–classical architecture that extends graph transformer capabilities through quantum self-attention. The QGT models variable-length sentences as token graphs, where both the embedding encoding and the self-attention mechanisms are implemented using parameterized quantum circuits (PQCs), enabling efficient contextual learning with significantly fewer trainable parameters. We train QGT using both fully connected and 𝑘 -nearest-neighbor graph structures and evaluate it on five benchmark sentiment-classification datasets. Experimental results show that QGT consistently achieves higher or comparable accuracy to existing quantum NLP models and outperforms a Classical Graph Transformer (CGT) baseline with identical architecture, achieving 29.4 × fewer parameters while requiring 3–5 × fewer samples to reach comparable performance. These findings highlight the potential of graph-based quantum models as scalable and data-efficient architectures for natural language understanding.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Machine-learning-accelerated multimodal characterization and multiobjective design optimization of natural porous materials

Natural porous materials such as nanoporous clays are used as green and low-cost adsorbents and catalysts. The key factors determining their performance in these applications are the pore morphology and surface activity, which are typically represented by properties such as specific surface area, pore volume, micropore content and pH. The latter may be modified and tuned to specific applications through material processing and/or chemical treatment. Characterization of the material, raw or processed, is typically performed experimentally, which can become costly especially in the context of tuning of the properties towards specific application requirements and needing numerous experiments. In this work, we present an application of tree-based machine learning methods trained on experimental datasets to accelerate the characterization of natural porous materials. The resulting models allow reliable prediction of the outcomes of experimental characterization of processed materials (R2 from 0.78 to 0.99) as well as identification of key factors contributing to those properties through feature importance analysis. Furthermore, the high throughput of the models enables exploration of processing parameter–property correlations and multiobjective optimization of prototype materials towards specific applications. We have applied these methodologies to pinpoint and rationalize optimal processing conditions for clays exploitable in acid catalysis. One of such identified materials was synthesized and tested revealing appreciable acid character improvement with respect to the pristine material. Specifically, it achieved 79% removal of chlorophyll-a in acid catalyzed degradation.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

A unified large language model–based framework for heterogeneous PV image diagnosis

With advances in imaging technologies, modern photovoltaic (PV) systems generate large volumes of heterogeneous image data, including visible, electroluminescence (EL), and infrared (IR) images. Existing PV image analysis models, particularly deep learning approaches, are typically task-specific and lack cross-modality generalization. To address this limitation, this paper proposes an open-source large language model (LLM)–based unified framework for heterogeneous PV image diagnostics. Through task-aware diagnostic prompting, the framework enables analysis of visible, EL, and IR images within a single pipeline, supporting both zero-shot and few-shot inference and binary and multiclass classification. It is compatible with state-of-the-art multimodal LLMs, including ChatGPT, Gemini, Claude, Qwen, and CLIP. The framework is evaluated on PV module condition classification (clean, soiling, snow, hail, and bird droppings) using visible images, cell crack detection using EL images, and hotspot detection using IR images. GPT-5.1 in few-shot mode achieves the best performance, with classification accuracy exceeding 97.3%. Open-source models such as Qwen and CLIP also deliver competitive results on visible images (around 90% accuracy), though their performance is more limited on EL and IR modalities. On the full ELPV dataset, the framework achieves 83.5% zero-shot accuracy, within 2.8% of the supervised CNN baseline, confirming scalability to larger benchmarks. Practical aspects such as reproducibility, response latency, and confidence estimation are systematically analyzed. The framework operates across PV image modalities without modality- or task-specific training, making it well suited as a rapid pre-screening tool to support downstream detailed diagnostics. A benchmark dataset of diverse labeled PV images is also released.

Li, Baojie↗
DOE OSTI2020

Machine Learning-Enabled Correlation and Modeling of Multimodal Response of Thin Film to Environment on Macro and Nanoscale Using “Lab-on-a-Crystal”

To close the feedback loop between artificial intellegence-controlled materials synthesis and characterization, material functionality must be rapidly tested. A platform for high-throughput multifunctional materials characterization is developed using a quartz crystal microbalance with auxiliary in-plane electrodes and a custom gas/vapor flow cell, enabling simultaneous scanning probe microscopy and electrical, optical, gravimetric, and viscoelastic characterization on the same film under controlled environment. The lab-on-a-crystal in situ multifunctional output allows direct correlations between the gravimetric/viscoelastic, electrical, and optical responses of polymer film in response to environment. Finally, when multiple film properties are used to augment the training set for machine learning regression, prediction of material response to the environment improves by a factor of 13 when <5% of the total dataset is used for model training.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Evaluating the Use of Foundational Chemical Language Models in Multimodal Graph Fusion

Rapid and accurate prediction of the physicochemical properties of molecules given their structures remains a key challenge in cheminformatics. Machine learning approaches offer high-throughput options, but the optimality of inductive biases and data representations are up for debate. For example, BERT-based masked language models (MLMs) can be trained in a self-supervised way on hundreds of millions to billions of readily available SMILES strings. Another option is graph neural networks (GNNs), which can operate directly on molecular structures. Yet, generating accurate molecular geometry is computationally expensive, leading to a relative scarcity in data compared to SMILES strings. It is attractive to combine these two paradigms by pre-training an LM on a large corpus of SMILES strings and embedding these representation into a geometric graph neural network. Despite the promise of such an approach, and contrary to previous studies, we find mixed results with the combination of the LMs and GNNs on several molecule datasets. In particular, we found evidence for improvement on the FreeSolv and QM7 benchmarks, but degraded performance on the ESOL, LIPO and QM9 datasets compared to a GNN baseline.

Francel, Collin [University of Alabama]↗
DOE OSTI2022

Light-Duty Vehicle Trip Classification Using One-Class Novelty Detection and Exhaustive Feature Extraction

Travel mode classification within travel survey data sets, especially light-duty vehicle (LDV) trips, is foundational, though nontrivial, to emerging mobility systems, travel behavior analysis, and fuel consumption estimation. Current travel mode detection approaches require well-sampled and balanced data sets with ground truth travel mode labels. The detection approaches are rarely applied and validated on large-scale, real-world data sets, which may not satisfy the dataset requirements. This work proposes an LDV trip detection model as a supplement to current travel mode detection methods, for the case when the training set is highly (and/or completely) unbalanced, to the extent that classical machine-learning approaches become difficult or impossible to deploy. The proposed model uses a novelty detection technique - one-class support vector machines (OCSVMs) - and a novel exhaustive feature extraction (EFE) technique on continuous time series data (i.e., Global Positioning System [GPS] speed profiles) for single-mode trip trajectories. Training and validation of the model are conducted on a large-scale, real-world data set. The proposed method accurately identifies LDV trips from a broad set of multimodal trips by leveraging a wealth of preexisting in-vehicle GPS travel data. Additional sensitivity analysis sheds light on the optimal training size and feature selection, which will benefit applications limited by highly imbalanced data. The paper also discusses performance comparison with regular machine-learning approaches, the model's robustness, and the potential to extend the proposed model to multi-modal trip prediction.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

A Deep Multimodal Representation Learning Framework for Accurate Molecular Properties Prediction

Drug discovery is a complex and challenging process, requiring the optimization of candidate compounds to identify those with the potential to become safe and effective drugs. Predicting molecular properties is an indispensable step in the drug discovery pipeline. Traditionally, this process is costly and time-intensive, involving multiple rounds of experiments and clinical trials, rendering it impractical for every candidate compound. Deep learning techniques have emerged as a promising approach to drug discovery to reduce the cost and time required to identify novel drugs. However, prevalent research in deep learning models focused on predicting molecular properties has primarily fixated on single-modal models, which utilize a single modality of data, neglecting the potential benefits of combining different data modalities. To overcome this limitation, we introduce MRL-Mol: a deep \textbf{M}ultimodal \textbf{R}epresentation \textbf{L}earning framework for accurate \textbf{Mol}ecular properties prediction. MRL-Mol harnesses three data modalities: sequence, graph, and image, augmenting the depth of comprehension. Leveraging a large-scale unlabeled dataset~($\sim$1M unique molecules), we pretrain MRL-Mol to extract inter- and intra-modal information. Our study demonstrates the superior performance of MRL-Mol in predicting molecular properties across six benchmark datasets, including both classification and regression tasks. Notably, MRL-Mol outperforms other state-of-the-art molecular properties prediction models. These findings suggest that by combining information from multiple data modalities, MRL-Mol can comprehend molecules better than single-modal deep learning models and identify molecular properties with better accuracy.

Yang, Yuxin↗
DOE OSTI2020

Multimodal Data Fusion via Entropy Minimization

The use of gradient-based data-driven models to solve a range of real-world remote sensing problems can in practice be limited by the uniformity of available data. Use of data from disparate sensor types, resolutions, and qualities typically requires compromises based on assumptions that are made prior to model training and may not necessarily be optimal given over-arching objectives. For example, while deep neural networks (NNs) are state-of-the-art in a variety of target detection problems, training them typically requires either limiting the training data to a subset over which uniformity can be enforced or training independent models which subsequently require additional score fusion. The method we introduce here seeks to leverage the benefits of both approaches by allowing correlated inputs from different data sources to co-influence preferred model solutions, while maintaining flexibility over missing and mismatching data. In this work we propose a new data fusion technique for gradient updated models based on entropy minimization and experimentally validate it on a hyperspectral target detection dataset. We demonstrate superior performance compared to currently available techniques using a range of realistic data scenarios, where available data has limited spacial overlap and resolution.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Multimodal Mass Spectrometry Imaging (MSI) of Archean and Jurassic Geologic Samples

Current geochemical analytical techniques such as gas chromatography-mass spectrometry (GC-MS and $G\acute{C}GC-MS$) can effectively determine the composition and structure of the organic biosignatures within a sample but cannot resolve the spatial distribution of organic biosignatures within rocks or sediments. The spatial distribution can be used to determine the character of the organic biosignatures [1], which can be indigenous (deposited with host rocks), non-indigenous (incorporated after deposition via fluid migration), or contaminant. Determining the character of organic biosignatures is critical for ancient samples [2] and will be critical if organics are observed in samples returned from Mars [3].Fs-LDPI-MS has the ability to map organic compounds across the surface of samples at 2 - 10 μm lateral resolution. Furthermore, fs-LDPI-MS can be used to carry out multiple analyses in the same location for micron-scale analysis of previously buried material [4]. We used fs-LDPI-MS to examine a ~164 million year old organic-rich mudstone from SW England (14.2 wt. % total organic carbon) [5] and determined the spatial distribution of likely indigenous components buried below potentially contaminated surface layers.We then used ToF-SIMS to re-analyze the mudstone to directly compare ToF-SIMS to fs-LDPI-MS datasets, and to analyze a series of ~2.7 billion year old geologic samples from Timmins, ON, CA [6]. Ref. [6] previously analyzed the Archean samples and observed a series of archaeal biomarkers, as well as hopanes and steranes. Using ToF-SIMS analysis, we were able to detect hopanes and steranes in the mudstone samples, but not in the Archean samples indicating that the hopanes and steranes are most likely contaminants in the Archean samples. We also did not observe the archaeal biomarkers; however, this may be from the lack of molecular ion preservation during ToF-SIMS.

Pasterski, Michael↗
DOE OSTI2025

miss-SNF: a multimodal patient similarity network integration approach to handle completely missing data sources

Abstract Motivation Precision medicine leverages patient-specific multimodal data to improve prevention, diagnosis, prognosis, and treatment of diseases. Advancing precision medicine requires the non-trivial integration of complex, heterogeneous, and potentially high-dimensional data sources, such as multi-omics and clinical data. In the literature, several approaches have been proposed to manage missing data, but are usually limited to the recovery of subsets of features for a subset of patients. A largely overlooked problem is the integration of multiple sources of data when one or more of them are completely missing for a subset of patients, a relatively common condition in clinical practice. Results We propose miss-Similarity Network Fusion (miss-SNF), a novel general-purpose data integration approach designed to manage completely missing data in the context of patient similarity networks. miss-SNF integrates incomplete unimodal patient similarity networks by leveraging a non-linear message-passing strategy borrowed from the SNF algorithm. miss-SNF is able to recover missing patient similarities and is “task agnostic”, in the sense that can integrate partial data for both unsupervised and supervised prediction tasks. Experimental analyses on nine cancer datasets from The Cancer Genome Atlas (TCGA) demonstrate that miss-SNF achieves state-of-the-art results in recovering similarities and in identifying patients subgroups enriched in clinically relevant variables and having differential survival. Moreover, amputation experiments show that miss-SNF supervised prediction of cancer clinical outcomes and Alzheimer’s disease diagnosis with completely missing data achieves results comparable to those obtained when all the data are available. Availability and implementation miss-SNF code, implemented in R, is available at https://github.com/AnacletoLAB/missSNF.

Biochemistry & Molecular Biology↗
DOE OSTI2021

KS4A-Omics1.0_FspDS682

Soil fungi facilitate the translocation of inorganic nutrients from soil minerals to other microorganisms and plants. This ability is particularly advantageous in impoverished soils, because fungal mycelial networks can bridge otherwise spatially disconnected and inaccessible nutrient hotspots. However, the molecular mechanisms underlying fungal mineral weathering and transport through soil remains poorly understood. Here, we addressed this knowledge gap by directly visualizing nutrient acquisition and transport through fungal hyphae in a mineral doped soil micromodel using a multimodal imaging approach. Here, we observed how a representative of common saprotrophic soil fungi, Fusarium sp. DS 682, exhibited a mechanosensory response (thigmotropism) around obstacles and through pore spaces (~12 μm) in the presence of minerals.This study establishes the significance of fungal biology and nutrient translocation mechanisms in maintaining fungal growth under water and nutrient limitations in a soil-like microenvironment, using a high-throughput multi-omic analysis approach. Data package KS4A-Omics.1.0_FspDS682 (Publication: Fungal Mineral Weathering Mechanisms Revealed Through Direct Molecular Visualization) contents reported here are the first version (1.0) and contain pre- and post-processed data using high throughput data capture technologies for multi-omic analysis and integration, this data package contains raw and post-processed experimental data for X-Ray Absorption Near Edge Structure Spectroscopy (XANES/XRF), Optical Microscopy, Proteomics, Scanning Electron Microscope (SEM), Time-of-Flight Secondary Ion Mass Spectroscopy (ToF-SIMS), X-Ray Diffraction Spectroscopy (XRD) files, and X- Ray Photoelectron Spectroscopy (XPS) using EMSL capabilities. This data package DOI contains a comprehensive collection of high-throughput multi-omics data and process metadata catalog. Support files include additional data download contents “Read Me” with dataset descriptor information and data source method application ontologies (see data dictionary section). Reported data download content is structured for compliance with reported guidelines provided by community standard initiatives and publisher stakeholder policies supporting FAIR data principles.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

O'Hare Airport roadway traffic prediction via data fusion and Gaussian process regression

This study proposes an approach of leveraging information gathered from multiple traffic data sources at different resolutions to obtain approximate inference on the traffic distribution of Chicago's O'Hare Airport area. Specifically, it proposes the ingestion of traffic datasets at different resolutions to build spatiotemporal models for predicting the distribution of traffic volume on the road network. Due to its good adaptability and flexibility for spatiotemporal data, the Gaussian process (GP) regression was employed to provide short-term forecasts using data collected by loop detectors (sensors) and supplemented by telematics data. The GP regression is used to make predictions of the distribution of the proportion of sensor data traffic volume represented by the telematics data for each location of the sensors. Consequently, the fitted GP model can be used to determine the approximate traffic distribution for a testing location outside of the training points. Policymakers in the transportation sector can find the results of this work helpful for making informed decisions relating to current and future transportation conditions in the area.

42 ENGINEERING↗