Results for “multimodal dataset” — Page 2

DOE OSTI2024

Towards Unlocking Insights from Logbooks Using AI

Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL, and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, enable macro-analysis for root cause analysis, and facilitate problem-solving automation.

43 PARTICLE ACCELERATORS↗

DOE OSTI2026

SpaceNet 9—Cross-Sensor Alignment of Optical and SAR Imagery

Precise registration of high-resolution synthetic aperture radar (SAR) and optical imagery is necessary for realizing the full potential and benefits of multimodal image analysis. However, two significant challenges presently exist. First, there is a lack of annotated datasets and benchmarks available for high-resolution SAR–optical image registration. Second, an assessment of efficient and reliable image registration methods that can precisely align these modalities is lacking. Here, we present a holistic description of the SpaceNet 9 Challenge and its results. We present a description of the dataset and baseline algorithm along with the results of the challenge, including a description of the winning algorithms. We release the SpaceNet 9 dataset along with open-sourcing the winning algorithms and baseline. The objective of SpaceNet 9 was to compute a dense displacement map that indicates the shift needed to align pixels in an optical image to the pixels in a SAR image. The challenge launched in April 2025 and was active for approximately two months. The top five solutions reduced image alignment error from approximately 34 m to under 13 m for public and private test data, with the best results obtaining a registration error of only 8.5 and 6.7 m on the public testing and private testing dataset, respectively. Usage of pretrained image matching models, robust outlier rejection with RANSAC, and estimating local displacement were common among the top solutions. The results of this challenge provide insight into high-resolution SAR–optical image registration and offer opportunities for future benchmarking in this domain. The baseline algorithm, winning solutions, and datasets are available at https://spacenet.ai/sn9-challenge/.

benchmark datasets↗

DOE OSTI2024

Artificial intelligence to unlock real-world evidence in clinical oncology: A primer on recent advances

Purpose: Real world evidence is crucial to understanding the diffusion of new oncologic therapies, monitoring cancer outcomes, and detecting unexpected toxicities. In practice, real world evidence is challenging to collect rapidly and comprehensively, often requiring expensive and time-consuming manual case-finding and annotation of clinical text. In this Review, we summarise recent developments in the use of artificial intelligence to collect and analyze real world evidence in oncology. Methods: We performed a narrative review of the major current trends and recent literature in artificial intelligence applications in oncology. Results: Artificial intelligence (AI) approaches are increasingly used to efficiently phenotype patients and tumors at large scale. These tools also may provide novel biological insights and improve risk prediction through multimodal integration of radiographic, pathological, and genomic datasets. Custom language processing pipelines and large language models hold great promise for clinical prediction and phenotyping. Conclusions: Despite rapid advances, continued progress in computation, generalizability, interpretability, and reliability as well as prospective validation are needed to integrate AI approaches into routine clinical care and real-time monitoring of novel therapies.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2026

AI-Based Analytics and Energy Modeling Framework for Characterizing Urban Energy Systems

Developing location-specific district energy models is essential for understanding energy patterns and supporting efficient management and planning decisions. However, accurately characterizing these models remains challenging due to gaps in building characteristics and labor-intensive traditional modeling workflows. To address these challenges, we develop an AI-based framework that integrates top-down and bottom-up building energy data to automate urban energy model characterization. The framework trains multimodal deep learning models using heterogeneous ResStockTM datasets to infer missing building characteristics from varying levels of known information and generate simulation-ready inputs for district-scale energy modeling. It also employs a conditioning-based injection approach to generate ”what-if” scenarios, enabling users to explore retrofit, efficiency, and technology-upgrade pathways. Integrated within URBANoptTM, a bottom-up district energy modeling platform for simulating co-located buildings, the framework infers detailed building-level inputs required for bottom-up simulations. Both localized and generalized AI models are developed to learn relationships across categorical, numerical, and time-series data, enabling reconstruction of missing attributes and generation of targeted upgrade scenarios. We demonstrate this methodology on a residential neighborhood in Baltimore, MD, assessing internal consistency against ResStock reference data and URBANopt simulation, and comparing selected attributes against real-world building characteristics. Results show strong overall predictive accuracy in data completion and scenario generation, with localized and generalized models offering complementary trade-offs between precision and scalability. Overall, our automated framework streamlines energy modeling and provides a reliable framework for urban building energy characterization.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2023

Are Phosphatidic Acids Ubiquitous in Mammalian Tissues or Overemphasized in Mass Spectrometry Imaging Applications?

Abstract Mass spectrometry imaging (MSI) is an invaluable tool for the spatial visualization of molecules in vivo. However, the question of whether observed annotations are endogenous or artificial (i. e., from in‐source fragmentation) is critical and has been largely unexplored in multimodal MSI. In matrix‐assisted laser desorption/ionization (MALDI)‐MSI datasets from researchers worldwide, PAs were found to represent up to 18 % of annotations in rat brain. Rat brain was additionally imaged here using nanospray desorption electrospray ionization (nano‐DESI), a softer ionization strategy. No PAs observed with MALDI were present in the nano‐DESI dataset. Further investigation strongly indicated lipid fragmentation to PAs for MALDI‐MSI, but not with nano‐DESI‐MSI. We finally extend this observation to the MALDI‐MSI analyses of human tissues, showing that PA annotations comprised up to 16 % of annotations. Therefore, this study shows that MSI annotations should be carefully interrogated, as in‐source fragmentation or modification of lipids may contribute substantially to false annotations and incorrect biological interpretations.

Vandergrift, Gregory W.↗

DOE OSTI2026

2020 Can Do Colorado E-Bike Mini Pilot Program Study

### The Colorado Energy Office conducted a mini pilot program study as part of the Can Do Colorado initiative, providing e-bikes to 13 low-income participants. The program aimed to encourage energy-efficient transportation during the COVID-19 pandemic as transit services were reduced and people were concerned about exposure. The insights garnered from this small-scale pilot study informed the design of a full-scale, 2-year pilot in locations across Colorado. For more information about the mini pilot program, see NLR's [Preliminary Results Report](https://www.nlr.gov/docs/fy21osti/79657.pdf). Micromobility options such as e-bikes offer a solution for improving energy efficiency for short-distance trips, especially in urban areas. Pedal-assist e-bikes use an electric motor and battery to help power the bike. The motor amplifies the power behind each pedal stroke, augmenting the energy you put into the bike. #### Data Collection Agency The Colorado Energy Office conducted the survey. #### Survey Methodology Participants in the program received a Momentum LaFree E+ e-bike (Class 1) and accessories at no cost and manually submitted travel data and feedback for 3 months using the CanBikeCo App. The smartphone app, developed in partnership with NLR, used a customized version of the [NLR OpenPATH platform](https://www.nlr.gov/transportation/openpath.html). #### Survey Records and Data Survey records include a total of 13 participants. This dataset contains 3 months of end-to-end, multimodal travel data manually submitted via smartphone app by 13 low-income essential workers in the greater Denver area. The data includes distance, mode (e.g., e-bike, car, transit), trip purpose, and demographic information.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2023

Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images

Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies. A primary challenge in modeling drug response prediction (DRP) with PDXs and neural networks (NNs) is the limited number of drug response samples. We investigate multimodal neural network (MM-Net) and data augmentation for DRP in PDXs. The MM-Net learns to predict response using drug descriptors, gene expressions (GE), and histology whole-slide images (WSIs). We explore whether combining WSIs with GE improves predictions as compared with models that use GE alone. We propose two data augmentation methods which allow us training multimodal and unimodal NNs without changing architectures with a single larger dataset: 1) combine single-drug and drug-pair treatments by homogenizing drug representations, and 2) augment drug-pairs which doubles the sample size of all drug-pair samples. Unimodal NNs which use GE are compared to assess the contribution of data augmentation. The NN that uses the original and the augmented drug-pair treatments as well as single-drug treatments outperforms NNs that ignore either the augmented drug-pairs or the single-drug treatments. In assessing the multimodal learning based on the MCC metric, MM-Net outperforms all the baselines. Our results show that data augmentation and integration of histology images with GE can improve prediction performance of drug response in PDXs.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2024

Unsupervised multimodal fusion of in-process sensor data for advanced manufacturing process monitoring

Effective monitoring of manufacturing processes is crucial for maintaining product quality and operational efficiency. Modern manufacturing environments often generate vast amounts of complementary multimodal data, including visual imagery from various perspectives and resolutions, hyperspectral data, and machine health monitoring information such as actuator positions, accelerometer readings, and temperature measurements. However, fusing and interpreting this complex, high-dimensional data presents significant challenges, particularly when labeled datasets are unavailable or impractical to obtain. This paper presents a novel approach to multimodal sensor data fusion in manufacturing processes, inspired by the Contrastive Language-Image Pre-training (CLIP) model. We leverage contrastive learning techniques to correlate different data modalities without the need for labeled data, overcoming limitations of traditional supervised machine learning methods in manufacturing contexts. Our proposed method demonstrates the ability to handle and learn encoders for five distinct modalities: visual imagery, audio signals, laser position (x and y coordinates), and laser power measurements. By compressing these high-dimensional datasets into low-dimensional representational spaces, our approach facilitates downstream tasks such as process control, anomaly detection, and quality assurance. The unsupervised nature of our method makes it broadly applicable across various manufacturing domains, where large volumes of unlabeled sensor data are common. We evaluate the effectiveness of our approach through a series of experiments, demonstrating its potential to enhance process monitoring capabilities in advanced manufacturing systems. This research contributes to the field of smart manufacturing by providing a flexible, scalable framework for multimodal data fusion that can adapt to diverse manufacturing environments and sensor configurations. The proposed method paves the way for more robust, data-driven decision-making in complex manufacturing processes.

Contrastive Learning↗

DOE OSTI2024

A Case Study of Multimodal, Multi-institutional Data Management for the Combinatorial Materials Science Community

Although the convergence of high-performance computing, automation, and machine learning has significantly altered the materials design timeline, transformative advances in functional materials and acceleration of their design will require addressing the deficiencies that currently exist in materials informatics, particularly a lack of standardized experimental data management. The challenges associated with experimental data management are especially true for combinatorial materials science, where advancements in automation of experimental workflows have produced datasets that are often too large and too complex for human reasoning. The data management challenge is further compounded by the multimodal and multi-institutional nature of these datasets, as they tend to be distributed across multiple institutions and can vary substantially in format, size, and content. Furthermore, modern materials engineering requires the tuning of not only composition but also of phase and microstructure to elucidate processing–structure–property–performance relationships. To adequately map a materials design space from such datasets, an ideal materials data infrastructure would contain data and metadata describing (i) synthesis and processing conditions, (ii) characterization results, and (iii) property and performance measurements. In this work, we present a case study for the low-barrier development of such a dashboard that enables standardized organization, analysis, and visualization of a large data lake consisting of combinatorial datasets of synthesis and processing conditions, X-ray diffraction patterns, and materials property measurements generated at several different institutions. While this dashboard was developed specifically for data-driven thermoelectric materials discovery, we envision the adaptation of this prototype to other materials applications, and, more ambitiously, future integration into an all-encompassing materials data management infrastructure.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Utah FORGE: Well 16B(78)-32 DTS, RFS DSS Strain, and Absolute Strain Circulation Test Fiber Optic Data - August 2024

This dataset contains processed fiber optic measurements collected during the extended cross-well circulation test at the Utah FORGE site in August 2024. The data was acquired from the 16B(78)-32 well using distributed fiber optic sensing (DFOS) technology, including distributed temperature sensing (DTS) on multimode fiber and distributed strain sensing (DSS) measurements on single-mode fiber. The dataset includes Brillouin absolute strain, Rayleigh frequency shift (RFS) DSS strain change, RFS DSS strain change rate, and temperature (DTS) data, all stored in HDF5 format. Time coordinates are provided in UTC, and depth measurements are given in measured depth relative to the rotary kelly bushing (MD RKB) in feet. The spatial sampling on the RFS DSS strain data and the Brillouin Absolute Total Strain data is 20 cm and the spatial sampling on the DTS data is 1m. The dataset is accompanied by a report from Neubrex, which provides further documentation.

15 GEOTHERMAL ENERGY↗

DOE OSTI2026

FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation

Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipelines ability to ingest several modalities of information. We present (1) a curated dataset of 93 questions designed to evaluate a pipeline's ability to ingest textual data, tables, images, multimodal data, and cross-document multimodal data; (2) a phrase-level recall metric for correctness; (3) a nearest-neighbor embedding classifier in an attempt to classify pipeline hallucinations; (4) a comparative evaluation of 2 pipelines built with open-source retrieval mechanisms and 4 closed-source foundational models; and (5) a third-party human evaluation of the alignment of our correctness and hallucination metrics. We find that closed-source pipelines significantly outperform open-source pipelines in both the correctness and halucination metrics, with a wider performance gap in questions relying on multimodal and cross-document information. We also find after a human evaluation of our correctness and hallucination metric compared with our questions and pipeline responses, average agreement was 4.62 for correctness 4.53 for hallucination detection on a 1-5 Likert scale with 5 being strongly agree with our determination.

Hildebrand, Samuel [ORNL] (ORCID:0009000465963104)↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study

# 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction Study The 2023 University of Puerto Rico at Mayagüez National Institute for Congestion Reduction (NCIR) Study provided insight into the travel patterns and associated energy consumption of participants. Study results helped researchers identify opportunities for the development of policies that could incentivize the use of alternative modes of travel such as transit and micromobility. Such travel modes reduce congestion by reducing the miles traveled by privately owned vehicles in urban and rural areas. The [National Institute for Congestion Reduction](https://nicr.usf.edu/) provides multimodal congestion reduction strategies through real-world deployments that leverage advances in technology, big data science, and innovative transportation options to optimize the efficiency and reliability of the transportation system for all users. ## Data Collection Agency The University of Puerto Rico at Mayagüez conducted the study. ## Survey Methodology The study was conducted in Spanish. Data collection was enabled via the open-source [NREL OpenPATH platform](https://www.nrel.gov/transportation/openpath). The resulting dataset consists of partially automated travel diaries—combining sensed and surveyed data reflecting patterns of multimodal, end-to-end, individual human mobility—as well as demographic and socioeconomic information from the 17 participants. ## Survey Records, Data, and Documentation Study records include 17 participants. The total number of trips was 458 and total non-air-miles traveled was approximately 1,469.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗