Search NASA⌕ Search

SEARCH · Search NASA

Results for “multi-modal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Do we have globally representative data to understand soil processes?

Understanding and modeling soils and soil organic matter (SOM) are central to a variety of human needs, from food production to ecosystem management. Soil data have been collected for over a century, but the global spatial and process representativeness of soil data remains unclear. We assessed the representativeness of currently available soil data that could be used to understand a variety of SOM processes. We used 16 open-source soil databases and data from over 281,000 unique locations globally, categorizing the databases into three main data types necessary to understand SOM processes: soil carbon stocks and fluxes, mechanistic drivers of these stocks and fluxes, and soil carbon gain or loss potential. We found that stock and driver data have extensive global coverage. However, data on soil carbon gain or loss potential, particularly data describing change in soils over time such as time series data, are severely limited in their global coverage. We conclude that while significant strides have been made in measuring soil carbon stocks and fluxes, and their drivers, we are limited in global data related to changes in soils over time. Our recommendations for soil data generators are to ensure precise metadata reporting and prioritizing sampling in underrepresented areas like tropical, arctic, mountainous, wetland and arid regions. We also encourage designing revisit schemes that explicitly support change detection and reporting multi-modal datasets that can aid in model development. Targeted measurement of low coverage soil data types and regions is necessary for a range of applications including current and future biogeochemical predictions, and their management and policy implications.

carbon fluxes↗
DOE OSTI2026

NREL OpenPATH: An Open-Source, Extensible Platform for Instrumenting Travel Behavior Data

NREL OpenPATH is an open-source, extensible platform that allows communities to instrument their own travel behavior data. The platform consists of a smartphone app, server and analysis pipeline, and enables collection of opt-in, multi-modal, end-to-end travel diaries. It makes the aggregate statistics available via a public dashboard, and allows deployers to download and visualize trip and trajectory data through the admin dashboard. It also allows for customization of the initial demographic survey and the trip-level qualitative information collected. Our goal is to provide an easy-to-use tool that can democratize travel behavior data collection by empower communities of all sizes to recruit participants and obtain a holistic picture of their travel patterns. The platform has been used by close to 40 partners, to collect data from thousands of participants. Upon signing a simple MOU, it is currently available for free to universities, non-profits and public agencies in the United States.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Revealing the evolution of order in materials microstructures using multi-modal computer vision

The development of high-performance materials for microelectronics, energy storage, and extreme environments depends on our ability to describe and direct property-defining microstructural order. Our present understanding is typically derived from laborious manual analysis of imaging and spectroscopy data, which is difficult to scale, challenging to reproduce, and lacks the ability to reveal latent associations needed for mechanistic models. Here, we demonstrate a multi-modal machine learning (ML) approach to describe order from electron microscopy analysis of the complex oxide La 1−x Sr x FeO 3 . We construct a hybrid pipeline based on fully and semi-supervised classification, allowing us to evaluate both the characteristics of each data modality and the value each modality adds to the ensemble. We observe distinct differences in the performance of uni- and multi-modal models, from which we draw general lessons in describing crystal order using computer vision.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Object detection with deep learning for rare event search in the GADGET II TPC

In the pursuit of identifying rare two-particle events within the GADGET II Time Projection Chamber (TPC), this paper presents a comprehensive approach for leveraging Convolutional Neural Networks (CNNs) and various data processing methods. To address the inherent complexities of 3D TPC track reconstructions, the data is expressed in 2D projections and 1D quantities. This approach capitalizes on the diverse data modalities of the TPC, allowing for the efficient representation of the distinct features of the 3D events, with no loss in topology uniqueness. Additionally, it leverages the computational efficiency of 2D CNNs and benefits from the extensive availability of pre-trained models. Given the scarcity of real training data for the rare events of interest, simulated events are used to train the models to detect real events. To account for potential distribution shifts when predominantly depending on simulations, significant perturbations are embedded within the simulations. This produces a broad parameter space that works to account for potential physics parameter and detector response variations and uncertainties. These parameter-varied simulations are used to train sensitive 2D CNN object detectors. When combined with 1D histogram peak detection algorithms, this multi-modal detection framework is highly adept at identifying rare, two-particle events in data taken during experiment 21072 at the Facility for Rare Isotope Beams (FRIB), demonstrating a 100% recall for events of interest. Here, we present the methods and outcomes of our investigation and discuss the potential future applications of these techniques.

Convolutional neural network↗
DOE OSTI2026

A call to standardize metrics for monitoring baleen whales near marine construction activities

Effective monitoring is necessary to protect marine mammal species during the construction of offshore infrastructure. The tools for detecting or monitoring marine mammals span traditional (e.g., visual observers, optical cameras), to newer (e.g., passive acoustic monitoring, infrared cameras, tags), and emerging (e.g., satellite imagery, environmental DNA, dimethyl sulfide concentration) technologies. Some are better suited for use during offshore development; however, peer-reviewed literature does not typically evaluate and report on the performance of these various technologies. We define a minimum set of metrics related to efficacy (i.e., confusion matrix, precision and recall, probability of missed mitigation), detection range (i.e., maximum and reliable detection range, spatial resolution), and data delivery (i.e., detection latency, system reliability, temporal resolution) that we recommend are needed to assess the utility of monitoring technologies for this purpose. Following a literature review of relevant studies, we highlight which publications reported these metrics and used multiple technologies to compare relative performance. We also emphasize the benefits of multi-modal approaches and recommend performance assessments through modeling or large-scale collaborative field testing. These metrics will standardize data collection, reporting, and analysis; promote consistent and comparable results; and foster collaboration among developers, regulatory agencies, and scientists. This may lead to the co-development of technology that achieves multiple goals, has greater application, and can answer research questions while collecting data to fulfill permitting requirements. These metrics may also inform decisions on what systems regulatory agencies might consider using and reduce monitoring costs, which is critical to support the marine sector's rapid growth alongside marine mammal conservation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

A Representation Fusion Framework for Decoupling Diagnostic Information in Multimodal Learning

Modern medicine increasingly relies on multimodal data, ranging from clinical notes to imaging and genomics, to guide diagnosis and treatment. However, integrating these heterogeneous data sources in a principled and interpretable manner remains a major challenge. We present MODES (Multi-mOdal Disentangled Embedding Space), a representation fusion framework that explicitly separates shared and modality-specific factors of variation, offering a structured latent space for multimodal information that improves both prediction and interpretability. By leveraging pre-trained unimodal foundation models, MODES mitigates the dependency on extensive paired datasets, crucial in data-scarce clinical settings. We introduce a masking strategy that optimizes representation dimensionality by eliminating low-information dimensions, to achieve compact, information-rich representations. Our framework demonstrates superior performance in predicting diagnoses and phenotypes compared to unimodal and conventional fusion models. MODES also enables robust diagnostic inference in missing data scenarios, offering an opportunity toward interpretable and efficient multimodal diagnostics in personalized healthcare.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

Integration and Demonstration of Monitoring, Modeling, and Prediction of DV-1 Amendment Performance at the Bench Scale: DV-1 Amendment Demonstration

During fiscal years 2024 and 2025, the U.S. Department of Energy’s Hanford Field Office commissioned Pacific Northwest National Laboratory to conduct applied research aimed at reducing the cost, time, and uncertainty associated with in situ treatment of vadose zone contaminants at the Hanford Site. This report outlines the integration of three key research efforts into a meso-scale demonstration designed to advance field-scale solutions that aim to (1) optimize the delivery of chemical amendments to contaminated soils, (2) reduce uncertainty in amendment delivery performance assessment using advanced monitoring techniques, and (3) provide real-time insights into when and where amendment-induced precipitation reactions occur in the subsurface. To achieve these objectives, the tank-scale (~ 1 cubic meter) Geophysical Imaging of Flow and Transport (GIFT) system was developed. GIFT enables experimental testing of amendment delivery while incorporating automated multi-modal monitoring approaches, including pressure measurements, direct fluid sampling, and remote time-lapse geophysical imaging. The data generated from these monitoring techniques will serve as inputs for a generative artificial-intelligence-driven digital twin – a numerical simulation model designed to honor observed data while quantifying uncertainty in simulation accuracy. Using this simulator, researchers will refine an amendment injection strategy to maximize delivery efficiency within a low-permeability soil zone. Monitoring data will be interpreted through simulated outputs to enhance understanding of the injection process. The efficacy of this integrated approach will be evaluated through direct sampling at the conclusion of the experiment.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2026

Data & Code from Phoenix CPPP Phase 2 Analysis

This data and code package supports the analysis presented in “Beyond Surface Cooling: Comprehensive Field Assessment of Reflective Pavement Thermal Performance in Phoenix, Arizona” and provides fully reproducible workflows for evaluating the thermal performance of cool pavement treatments in a hot urban environment. The dataset integrates multi-modal field measurements collected across residential and nonresidential settings, including mobile air temperature traverses, stationary air temperature monitoring, residential mean radiant temperature (MRT) measurements, subsurface temperature profiles, and controlled testbed observations. The data package contains raw and processed datasets in comma-separated value (CSV) format, accompanying metadata files describing site characteristics and measurement protocols, and R scripts (.R files) used for data cleaning, time synchronization, spatial and temporal matching, quality control filtering, statistical comparison, and figure generation. All analyses were conducted using R (version ≥ 4.2.0) with commonly available packages (e.g., tidyverse, lubridate, data.table, ggplot2). No proprietary software is required to reproduce results. Field campaigns were designed to quantify the effects of high-reflectance pavement coatings on surface temperature, near-surface air temperature, subsurface heat propagation, and radiative heat exposure. Temporal alignment procedures include standardized timestamp conversion and nearest-neighbor matching of high-frequency sensor measurements to stop-based metadata within defined tolerance windows to ensure comparability across instruments. The workflows generate summary statistics, treatment–control contrasts, depth-dependent thermal gradients, and time-series visualizations used in the associated publication. By integrating mobile, stationary, radiative, and subsurface measurements within a unified and transparent processing framework, this package enables comprehensive evaluation of cool pavement performance across multiple thermal exposure pathways and supports reuse in future urban heat mitigation and climate resilience studies.

AIR TEMPERATURE↗
DOE OSTI2025

Unveiling the Hidden Evolution of Crystal Defects and Disorder in Energy Materials

Control of point defects and disorder in functional thin films and 2D materials is critical to realizing their full potential in applications ranging from energy storage to advanced electronics. However, these phenomena are often poorly understood, difficult to characterize, and challenging to direct with precision. This presentation explores emerging multi-modal computer vision to decipher and predict order in materials across multiple length scales in the electron microscope, from the atomic to the nanoscale. By fusing data from diverse sources, these powerful models provide unprecedented insights into materials' lifecycles, enabling the control of defects and their associated properties at a fundamental level. This capability promises to transform materials design and accelerate the development of next-generation technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Energy-resolved neutron imaging and diffraction including grain orientation mapping using event camera technology

Time-of-flight neutron diffraction and energy-resolved imaging each provide unique perspectives into material properties. Neutron diffraction is useful for assessing microstructural parameters such as phase composition, texture, and dislocation densities, though it typically provides averaged data over the sampled volume. Energy-resolved imaging, on the other hand, offers both spatial and spectral information by detecting Bragg edges and neutron absorption resonances, which enables detailed mapping of microstructure and isotopic composition. When combined, these techniques have the potential to enrich our understanding of material behavior across different scales, enhancing our understanding of complex materials. Traditionally, these modalities are conducted on separate instruments, which is time-consuming and poses challenges for data integration. Here, we report the integration of the LumaCam, an event-mode energy-resolved neutron imaging camera with the HIPPO time-of-flight diffractometer at LANSCE. This integration enables simultaneous diffraction and imaging across the full spectrum, with analysis optimized for diffraction and Bragg-edge imaging in the thermal range (0.45–10 Å) and resonance imaging in the epithermal range (0.5–3000 eV), facilitating comprehensive multi-modal analysis. We demonstrate its capabilities through case studies, including spatial mapping of grain orientations in a steel sample and accurate thickness estimations for irregular samples including a depleted uranium cylinder and a natural silver-containing mineral specimen. The combined setup enhances real-time sample alignment and provides comprehensive data for crystal structure, texture, and isotopic composition analysis. This approach opens new possibilities for advanced applications in nuclear engineering, archaeology, and materials science.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Edge AI-Enhanced Traffic Monitoring and Anomaly Detection Using Multimodal Large Language Models

This paper addresses the challenge of traffic monitoring and incident detection in remote areas, utilizing multimodal large language models (LLMs) deployed on edge AI devices. The key novelty of the LLM is to convert real-time video streams into descriptive texts, enabling low-bandwidth transmissions and reliable detection of anomalies and incidents in environments of intermittent connectivity. The model is developed based on fine-tuning open-source LLMs and extending it with multi-modal capabilities to analyze video frames. Our work also involves deploying this model on edge devices such as Nvidia IGX Orin and is planned to be tested in realistic environments in future work. The methodology includes data set curation, iterative model fine-tuning and compression, and hardware-based optimization. This approach aims to enhance traffic safety and response speed in remote areas, marking a significant advancement in the application of AI for traffic monitoring and safety management.

Peruski, Ryan [University of Tennessee, Knoxville ↗
DOE OSTI2025

An Advanced Machine Learning and Artificial Intelligence System for Demonstrating Radiation Regulatory Compliance in DOE Accelerator Facilities

In this Phase II proposal, Applied Research LLC (ARLLC), Thomas Jefferson National Accelerator Facility (Jefferson Lab), and Old Dominion University (ODU) propose the combination of domain knowledge (beam characteristics, fixed structural shielding, earthen burden (the soil and foliage added to the dome of the experimental halls as additional shielding), etc.), machine learning (ML) and/or artificial intelligence (AI) to correlate a variety of multi-modal onsite signals and the radiation fields seen in accessible areas of the accelerator site and the site boundary. The ML/AI will consider the complex influence of environmental parameters affecting the radon contribution of the measurements, focusing on actual data obtained from Jefferson Lab. In Phase I, the coded beam and location data were fed into a deep learning model to predict doses at several designated locations in Jefferson Lab’s facility. Moreover, a dense radiation map was generated using only a sparse collection of the samples in a facility. In Phase II, we will develop a software prototype containing a radiation prediction algorithm, dense radiation map algorithms, and background noise prediction algorithms, with actual data used to evaluate the prototype. This work will provide a framework for evaluation of radiation measurement results around the site based on learned responses. In addition, the proposed approach allows more granular mapping of radiation levels. Better understanding and communication of these levels is related to the overall approach in keeping doses to personnel ALARA.

43 PARTICLE ACCELERATORS↗
DOE OSTI2026

Vulcan-Forge: Architecture and Design of a Multi-Modal Forensic Analysis Plugin for CALDERA

Forge and VULCAN together describe an open-architecture cybersecurity analysis ecosystem that unifies forensic artifact processing, detection engineering, and vulnerability intelligence within integrated platforms. Forge operates as a plugin for MITRE CALDERA, ingesting diverse evidence formats—including EVTX, PCAP/PCAPNG, CSV, JSON, YAML, XML, binaries, and archives—to construct a unified artifact graph enriched with severity scoring, TLP classification, and audit trails. It provides subsystems for artifact parsing, streaming structured-data visualization, NetworkMiner-based packet inspection, PE/.NET binary analysis, and LLM-assisted triage and rule generation, with outputs validated against CCCS-YARA and pySigma schemas. VULCAN complements this by serving as a cybersecurity analyst platform that integrates a Neo4j knowledge graph, Qdrant vector retrieval, SSVC-based triage, and a local LLM to deliver CVE intelligence and forensic analysis through a multi-source ingest pipeline drawing from NVD, CISA KEV, EPSS, MITRE ATT&CK, and CAPEC. Together, they bridge structured threat intelligence with automated forensic analysis and detection workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Enhancing Automotive Intrusion Detection Through Multi-Modal Fusion: A CAN FD-LiDAR Approach

As vehicles become smarter and more autonomous, they increasingly depend on advanced sensors and communication technologies to operate securely. However, such growing dependence on technology—whether it’s CAN (Controller Area Network) for internal communication or LiDAR (Light Detection and Ranging) for sensing the world around them—also expands the attack surface for the types of cyber attacks. Traditional intrusion detection systems (IDS) typically monitor these systems in isolation, limiting their ability to detect sophisticated, crosssystem attacks. To address this, we propose a multi-modal fusion approach that combines real-world CAN FD signals (from the HCRL dataset) with LiDAR features (from the nuScenes dataset) to enhance attack detection. Our method employs a twostage ensemble approach. Calibrated XGBoost and LightGBM models initially process CAN FD (Fuzzing Data) and LiDAR data independently, detecting timing anomalies and space abnormalities. They are subsequently logarithmically combined with a logistic regression meta-model along with 17 engineered features capturing cross-modal behavior, prediction conflicts, and nonlinear interactions. This approach achieves an AUC of 0.87 and an F1-score of 0.82, surpassing single-modality baselines and early fusion methods, at merely 2 ms inference latency. Compared with deep learning competitors, it is 3 times more efficient, providing a lightweight, interpretable, and real time solution to automotive cybersecurity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Towards Secure Autonomous Vehicles: An Integrated Edge and Multi-Modal Machine Learning Framework for Intrusion Detection

Autonomous vehicles (AVs) are vulnerable to cyberattacks targeting both internal communication networks and external perception sensors. While edge-based intrusion de- tection for Controller Area Network (CAN) buses offers real-time protection, it cannot detect cross-modal threats. Conversely, multi-modal fusion approaches improve coverage but often lack efficiency for in-vehicle deployment. This thesis integrates two complemen- tary solutions: (1) a lightweight, edge-deployable machine learning framework for CAN bus intrusion detection, and (2) a late-fusion system combining CAN FD and LiDAR data. Together, they form a hierarchical defense capable of handling single-modality and coordi- nated attacks. Simulations show that CAN-only models reach 93% accuracy on simulated DoS, spoofing, replay, and fuzzy attacks, while the fusion system achieves 0.87 AUC and 0.82 F1-score at 2 ms latency. This unified framework establishes a scalable, explainable, and field-ready strategy for AV cybersecurity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

LAF-Net: A Deep Residual and Cross-Attention Framework for Day-Ahead Load Forecasting: Preprint

Accurate day-ahead load forecasting is essential for reliable power system operations and market efficiency. System operators such as the Midcontinent Independent System Operator (MISO) rely on forecasts from multiple vendors, yet combining them effectively remains a persistent challenge due to vendor-specific biases. This paper presents a novel LSTM-Attention Fusion Network with Error Representation (LAF-Net) that enhances day-ahead hourly load forecasting through deep residual learning and multi-modal cross-attention. The proposed model builds a historical error memory from past vendor performance and dynamically queries it with future hour context to generate adaptive, hour-specific trust weights for each vendor. A bounded residual correction further refines forecasts by mitigating systematic and temporally localized errors. Tested on real MISO LBA data with multi-vendor forecasts, LAF-Net consistently outperforms the best vendor baseline across all 38 LBAs, achieving more than a 40% reduction in system-level mean absolute error (MAE) during peak load hours relative to the best vendor baseline.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Multi-modality deep learning for pulse prediction in homogeneous nonlinear systems via parametric conversion

In this Letter, we introduce FusionNet, a multi-modality deep learning framework designed to predict and analyze output pulses in high-power rare-earth-doped laser systems driving parametric conversion in homogeneous guided nonlinear media. FusionNet integrates temporal, spectral, and physical experimental conditions to model ultrafast nonlinear phenomena, including parametric nonlinear frequency conversion, self-phase modulation, and cross-phase modulation in homogeneous guided systems such as gas-filled hollow-core fibers. These systems bridge physical models with experimental data, advancing our understanding of light-guiding principles and nonlinear interactions while expediting the design and optimization of on-demand high-power, high-brightness systems. Our results demonstrate a 73% reduction in prediction error and an 83% improvement in computational efficiency compared to conventional neural networks. This work establishes a new paradigm for accelerating parametric simulations and optimizing experimental designs in high-power laser systems, with further implications for high-precision spectroscopy, quantum information science, and distributed entangled interconnects.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

Throughput Estimation of Data Transport Networks From Digital Twin Measurements

Digital twins of networked infrastructures, known as Virtual Infrastructure Twins (VITs), are increasingly used for software development, pre-deployment testing, and design space exploration. While VITs avoid the costs and potential disruptions associated with experiments on operational networks, their throughput measurements are typically not sufficiently accurate for performance profiling of wide-area networks that they emulate. Here, machine learning (ML) methods are developed to transform these inaccurate VIT network throughput measurements to closely match in peak and overall profile of those from a physical testbed or production network. First, a micro kernel network reflecting a physical network is utilized to collect one-time measurements on a host to support this ML transformation. Then, a generic multi-modal ML method is developed to learn a map that transforms measurements from subsequent VITs on the same host to match past, current and follow-on testbed and cloud networks. ML generalization equations are derived to establish its correctness and probabilistically guarantee its generalization accuracy. Experimental results are presented for a variety of VIT hosts with target testbed and cloud networks; they include a case study of a four-site science ecosystem wherein inaccurate convex VIT measurement profiles are transformed into accurate concave profiles of target networks.

97 MATHEMATICS AND COMPUTING↗