Search NASA⌕ Search

SEARCH · Search NASA

Results for “multi-modal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Energy-resolved neutron imaging and diffraction including grain orientation mapping using event camera technology

Time-of-flight neutron diffraction and energy-resolved imaging each provide unique perspectives into material properties. Neutron diffraction is useful for assessing microstructural parameters such as phase composition, texture, and dislocation densities, though it typically provides averaged data over the sampled volume. Energy-resolved imaging, on the other hand, offers both spatial and spectral information by detecting Bragg edges and neutron absorption resonances, which enables detailed mapping of microstructure and isotopic composition. When combined, these techniques have the potential to enrich our understanding of material behavior across different scales, enhancing our understanding of complex materials. Traditionally, these modalities are conducted on separate instruments, which is time-consuming and poses challenges for data integration. Here, we report the integration of the LumaCam, an event-mode energy-resolved neutron imaging camera with the HIPPO time-of-flight diffractometer at LANSCE. This integration enables simultaneous diffraction and imaging across the full spectrum, with analysis optimized for diffraction and Bragg-edge imaging in the thermal range (0.45–10 Å) and resonance imaging in the epithermal range (0.5–3000 eV), facilitating comprehensive multi-modal analysis. We demonstrate its capabilities through case studies, including spatial mapping of grain orientations in a steel sample and accurate thickness estimations for irregular samples including a depleted uranium cylinder and a natural silver-containing mineral specimen. The combined setup enhances real-time sample alignment and provides comprehensive data for crystal structure, texture, and isotopic composition analysis. This approach opens new possibilities for advanced applications in nuclear engineering, archaeology, and materials science.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Edge AI-Enhanced Traffic Monitoring and Anomaly Detection Using Multimodal Large Language Models

This paper addresses the challenge of traffic monitoring and incident detection in remote areas, utilizing multimodal large language models (LLMs) deployed on edge AI devices. The key novelty of the LLM is to convert real-time video streams into descriptive texts, enabling low-bandwidth transmissions and reliable detection of anomalies and incidents in environments of intermittent connectivity. The model is developed based on fine-tuning open-source LLMs and extending it with multi-modal capabilities to analyze video frames. Our work also involves deploying this model on edge devices such as Nvidia IGX Orin and is planned to be tested in realistic environments in future work. The methodology includes data set curation, iterative model fine-tuning and compression, and hardware-based optimization. This approach aims to enhance traffic safety and response speed in remote areas, marking a significant advancement in the application of AI for traffic monitoring and safety management.

Peruski, Ryan [University of Tennessee, Knoxville ↗
DOE OSTI2025

An Advanced Machine Learning and Artificial Intelligence System for Demonstrating Radiation Regulatory Compliance in DOE Accelerator Facilities

In this Phase II proposal, Applied Research LLC (ARLLC), Thomas Jefferson National Accelerator Facility (Jefferson Lab), and Old Dominion University (ODU) propose the combination of domain knowledge (beam characteristics, fixed structural shielding, earthen burden (the soil and foliage added to the dome of the experimental halls as additional shielding), etc.), machine learning (ML) and/or artificial intelligence (AI) to correlate a variety of multi-modal onsite signals and the radiation fields seen in accessible areas of the accelerator site and the site boundary. The ML/AI will consider the complex influence of environmental parameters affecting the radon contribution of the measurements, focusing on actual data obtained from Jefferson Lab. In Phase I, the coded beam and location data were fed into a deep learning model to predict doses at several designated locations in Jefferson Lab’s facility. Moreover, a dense radiation map was generated using only a sparse collection of the samples in a facility. In Phase II, we will develop a software prototype containing a radiation prediction algorithm, dense radiation map algorithms, and background noise prediction algorithms, with actual data used to evaluate the prototype. This work will provide a framework for evaluation of radiation measurement results around the site based on learned responses. In addition, the proposed approach allows more granular mapping of radiation levels. Better understanding and communication of these levels is related to the overall approach in keeping doses to personnel ALARA.

43 PARTICLE ACCELERATORS↗
NASA NTRS1976

Diurnal measurements with prototype CMOS Omega receivers

Diurnal signals from eight omega channels have been monitored at 10.2 KHz for selected station pairs. All eight Omega stations have been received at least 50 percent of the time over a 24 hour period during the month of October 1976. The data presented confirm the expected performance of the CMOS omega sensor processor in being able to digsignals out of a noisy environment. Of particular interest are possibilities for use of antipodal reception phenomena and a need for some ways of correcting for multi-modal propagation effects.

Burhans, R. W.↗
DOE OSTI2026

Vulcan-Forge: Architecture and Design of a Multi-Modal Forensic Analysis Plugin for CALDERA

Forge and VULCAN together describe an open-architecture cybersecurity analysis ecosystem that unifies forensic artifact processing, detection engineering, and vulnerability intelligence within integrated platforms. Forge operates as a plugin for MITRE CALDERA, ingesting diverse evidence formats—including EVTX, PCAP/PCAPNG, CSV, JSON, YAML, XML, binaries, and archives—to construct a unified artifact graph enriched with severity scoring, TLP classification, and audit trails. It provides subsystems for artifact parsing, streaming structured-data visualization, NetworkMiner-based packet inspection, PE/.NET binary analysis, and LLM-assisted triage and rule generation, with outputs validated against CCCS-YARA and pySigma schemas. VULCAN complements this by serving as a cybersecurity analyst platform that integrates a Neo4j knowledge graph, Qdrant vector retrieval, SSVC-based triage, and a local LLM to deliver CVE intelligence and forensic analysis through a multi-source ingest pipeline drawing from NVD, CISA KEV, EPSS, MITRE ATT&CK, and CAPEC. Together, they bridge structured threat intelligence with automated forensic analysis and detection workflows.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1995

Multi-modal virtual environment research at Armstrong Laboratory

One mission of the Paul M. Fitts Human Engineering Division of Armstrong Laboratory is to improve the user interface for complex systems through user-centered exploratory development and research activities. In support of this goal, many current projects attempt to advance and exploit user-interface concepts made possible by virtual reality (VR) technologies. Virtual environments may be used as a general purpose interface medium, an alternative display/control method, a data visualization and analysis tool, or a graphically based performance assessment tool. An overview is given of research projects within the division on prototype interface hardware/software development, integrated interface concept development, interface design and evaluation tool development, and user and mission performance evaluation tool development.

Eggleston, Robert G.↗
DOE OSTI2025

Enhancing Automotive Intrusion Detection Through Multi-Modal Fusion: A CAN FD-LiDAR Approach

As vehicles become smarter and more autonomous, they increasingly depend on advanced sensors and communication technologies to operate securely. However, such growing dependence on technology—whether it’s CAN (Controller Area Network) for internal communication or LiDAR (Light Detection and Ranging) for sensing the world around them—also expands the attack surface for the types of cyber attacks. Traditional intrusion detection systems (IDS) typically monitor these systems in isolation, limiting their ability to detect sophisticated, crosssystem attacks. To address this, we propose a multi-modal fusion approach that combines real-world CAN FD signals (from the HCRL dataset) with LiDAR features (from the nuScenes dataset) to enhance attack detection. Our method employs a twostage ensemble approach. Calibrated XGBoost and LightGBM models initially process CAN FD (Fuzzing Data) and LiDAR data independently, detecting timing anomalies and space abnormalities. They are subsequently logarithmically combined with a logistic regression meta-model along with 17 engineered features capturing cross-modal behavior, prediction conflicts, and nonlinear interactions. This approach achieves an AUC of 0.87 and an F1-score of 0.82, surpassing single-modality baselines and early fusion methods, at merely 2 ms inference latency. Compared with deep learning competitors, it is 3 times more efficient, providing a lightweight, interpretable, and real time solution to automotive cybersecurity.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2003

Enhanced Multi-Modal Access to Planetary Exploration

Tomorrow's Interplanetary Network (IPN) will evolve from JPL's Deep-Space Network (DSN) and provide key capabilities to future investigators, such as simplified acquisition of higher-quality science at remote sites and enriched access to these sites. These capabilities could also be used to foster public interest, e.g., by making it possible for students to explore these environments personally, eventually perhaps interacting with a virtual world whose models could be populated by data obtained continuously from the IPN. Our paper looks at JPL's approach to making this evolution happen, starting from improved communications. Evolving space protocols (e.g., today's CCSDS proximity and file-transfer protocols) will provide the underpinning of such communications in the next decades, just as today's rich web was enabled by progress in Internet Protocols starting from the early 1970's (ARPAnet research). A key architectural thrust of this effort is to deploy persistent infrastructure incrementally, using a layered service model, where later higher-layer capabilities (such as adaptive science planning) are enabled by earlier lower-layer services (such as automated routing of object-based messages). In practice, there is also a mind shift needed from an engineering culture raised on point-to-point single-function communications (command uplink, telemetry downlink), to one in which assets are only indirectly accessed, via well-defined interfaces. We are aiming to foster a 'community of access' both among space assets and the humans who control them. This enables appropriate (perhaps eventually optimized) sharing of services and resources to the greater benefit of all participants. We envision such usage to be as automated in the future as using a cell phone is today - with all the steps in creating the real-time link being automated.

interplanetary network (IPN)↗
DOE OSTI2026

Towards Secure Autonomous Vehicles: An Integrated Edge and Multi-Modal Machine Learning Framework for Intrusion Detection

Autonomous vehicles (AVs) are vulnerable to cyberattacks targeting both internal communication networks and external perception sensors. While edge-based intrusion de- tection for Controller Area Network (CAN) buses offers real-time protection, it cannot detect cross-modal threats. Conversely, multi-modal fusion approaches improve coverage but often lack efficiency for in-vehicle deployment. This thesis integrates two complemen- tary solutions: (1) a lightweight, edge-deployable machine learning framework for CAN bus intrusion detection, and (2) a late-fusion system combining CAN FD and LiDAR data. Together, they form a hierarchical defense capable of handling single-modality and coordi- nated attacks. Simulations show that CAN-only models reach 93% accuracy on simulated DoS, spoofing, replay, and fuzzy attacks, while the fusion system achieves 0.87 AUC and 0.82 F1-score at 2 ms latency. This unified framework establishes a scalable, explainable, and field-ready strategy for AV cybersecurity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

LAF-Net: A Deep Residual and Cross-Attention Framework for Day-Ahead Load Forecasting: Preprint

Accurate day-ahead load forecasting is essential for reliable power system operations and market efficiency. System operators such as the Midcontinent Independent System Operator (MISO) rely on forecasts from multiple vendors, yet combining them effectively remains a persistent challenge due to vendor-specific biases. This paper presents a novel LSTM-Attention Fusion Network with Error Representation (LAF-Net) that enhances day-ahead hourly load forecasting through deep residual learning and multi-modal cross-attention. The proposed model builds a historical error memory from past vendor performance and dynamically queries it with future hour context to generate adaptive, hour-specific trust weights for each vendor. A bounded residual correction further refines forecasts by mitigating systematic and temporally localized errors. Tested on real MISO LBA data with multi-vendor forecasts, LAF-Net consistently outperforms the best vendor baseline across all 38 LBAs, achieving more than a 40% reduction in system-level mean absolute error (MAE) during peak load hours relative to the best vendor baseline.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Multi-modality deep learning for pulse prediction in homogeneous nonlinear systems via parametric conversion

In this Letter, we introduce FusionNet, a multi-modality deep learning framework designed to predict and analyze output pulses in high-power rare-earth-doped laser systems driving parametric conversion in homogeneous guided nonlinear media. FusionNet integrates temporal, spectral, and physical experimental conditions to model ultrafast nonlinear phenomena, including parametric nonlinear frequency conversion, self-phase modulation, and cross-phase modulation in homogeneous guided systems such as gas-filled hollow-core fibers. These systems bridge physical models with experimental data, advancing our understanding of light-guiding principles and nonlinear interactions while expediting the design and optimization of on-demand high-power, high-brightness systems. Our results demonstrate a 73% reduction in prediction error and an 83% improvement in computational efficiency compared to conventional neural networks. This work establishes a new paradigm for accelerating parametric simulations and optimizing experimental designs in high-power laser systems, with further implications for high-precision spectroscopy, quantum information science, and distributed entangled interconnects.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

Throughput Estimation of Data Transport Networks From Digital Twin Measurements

Digital twins of networked infrastructures, known as Virtual Infrastructure Twins (VITs), are increasingly used for software development, pre-deployment testing, and design space exploration. While VITs avoid the costs and potential disruptions associated with experiments on operational networks, their throughput measurements are typically not sufficiently accurate for performance profiling of wide-area networks that they emulate. Here, machine learning (ML) methods are developed to transform these inaccurate VIT network throughput measurements to closely match in peak and overall profile of those from a physical testbed or production network. First, a micro kernel network reflecting a physical network is utilized to collect one-time measurements on a host to support this ML transformation. Then, a generic multi-modal ML method is developed to learn a map that transforms measurements from subsequent VITs on the same host to match past, current and follow-on testbed and cloud networks. ML generalization equations are derived to establish its correctness and probabilistically guarantee its generalization accuracy. Experimental results are presented for a variety of VIT hosts with target testbed and cloud networks; they include a case study of a four-site science ecosystem wherein inaccurate convex VIT measurement profiles are transformed into accurate concave profiles of target networks.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2005

Inductive Learning Approaches for Improving Pilot Awareness of Aircraft Faults

Neural network flight controllers are able to accommodate a variety of aircraft control surface faults without detectable degradation of aircraft handling qualities. Under some faults, however, the effective flight envelope is reduced; this can lead to unexpected behavior if a pilot performs an action that exceeds the remaining control authority of the damaged aircraft. The goal of our work is to increase the pilot s situational awareness by informing him of the type of damage and resulting reduction in flight envelope. Our methodology integrates two inductive learning systems with novel visualization techniques. One learning system, the Inductive Monitoring System (IMS), learns to detect when a simulation includes faulty controls, while two others, Inductive Classification System (INCLASS) and multiple binary decision tree system (utilizing C4.5), determine the type of fault. In off-line training using only non-failure data, IMS constructs a characterization of nominal flight control performance based on control signals issued by the neural net flight controller. This characterization can be used to determine the degree of control augmentation required in the pitch, roll, and yaw command channels to counteract control surface failures. This derived information is typically sufficient to distinguish between the various control surface failures and is used to train both INCLASS and C4.5. Using data from failed control surface flight simulations, INCLASS and C4.5 independently discover and amplify features in IMS results that can be used to differentiate each distinct control surface failure situation. In real-time flight simulations, distinguishing features learned during training are used to classify control surface failures. Knowledge about the type of failure can be used by an additional automated system to alter its approach for planning tactical and strategic maneuvers. The knowledge can also be used directly to increase the pilot s situational awareness and inform manual maneuver decisions. Our multi-modal display of this information provides speech output to issue control surface failure warnings to a lesser-used communication channel and provides graphical displays with pilot-selectable !eve!s of details to issues additional information about the failure. We also describe a potential presentation for flight envelope reduction that can be viewed separately or integrated with an existing attitude indicator instrument. Preliminary results suggest that the inductive approach is capable of detecting that a control surface has failed and determining the type of fault. Furthermore, preliminary evaluations suggest that the interface discloses a concise summary of this information to the pilot.

Spikovska, Lilly↗
NASA NTRS2003

Performance Evaluation Modeling of Network Sensors

Substantial benefits are promised by operating many spatially separated sensors collectively. Such systems are envisioned to consist of sensor nodes that are connected by a communications network. A simulation tool is being developed to evaluate the performance of networked sensor systems, incorporating such metrics as target detection probabilities, false alarms rates, and classification confusion probabilities. The tool will be used to determine configuration impacts associated with such aspects as spatial laydown, and mixture of different types of sensors (acoustic, seismic, imaging, magnetic, RF, etc.), and fusion architecture. The QualNet discrete-event simulation environment serves as the underlying basis for model development and execution. This platform is recognized for its capabilities in efficiently simulating networking among mobile entities that communicate via wireless media. We are extending QualNet's communications modeling constructs to capture the sensing aspects of multi-target sensing (analogous to multiple access communications), unimodal multi-sensing (broadcast), and multi-modal sensing (multiple channels and correlated transmissions). Methods are also being developed for modeling the sensor signal sources (transmitters), signal propagation through the media, and sensors (receivers) that are consistent with the discrete event paradigm needed for performance determination of sensor network systems. This work is supported under the Microsensors Technical Area of the Army Research Laboratory (ARL) Advanced Sensors Collaborative Technology Alliance.

probability of detection↗
DOE OSTI2025

SANE: strategic autonomous non-smooth exploration for multiple optima discovery in multi-modal and non-differentiable black-box functions

Both computational and experimental material discovery bring forth the challenge of exploring multidimensional and multimodal parameter spaces, such as phase diagrams of Hamiltonians with multiple interactions, composition spaces of combinatorial libraries, material structure image spaces, and molecular embedding spaces. Often these systems are black-boxes and time-consuming to evaluate, which resulted in strong interest towards active learning methods such as Bayesian optimization (BO). However, these systems are often noisy which make the black box function severely multi-modal and non-differentiable, where a vanilla BO can get overly focused near a single or faux optimum, deviating from the broader goal of scientific discovery. To address these limitations, here we developed Strategic Autonomous Non-Smooth Exploration (SANE) to facilitate an intelligent Bayesian optimized navigation with a proposed cost-driven probabilistic acquisition function to find multiple global and local optimal regions, avoiding the tendency to becoming trapped in a single optimum. To distinguish between a true and false optimal region due to noisy experimental measurements, a human (domain) knowledge driven dynamic surrogate gate is integrated with SANE. We implemented the gate-SANE into pre-acquired piezoresponse spectroscopy data of a ferroelectric combinatorial library with high noise levels in specific regions, and piezoresponse force microscopy (PFM) hyperspectral data. SANE demonstrated better performance than classical BO to facilitate the exploration of multiple optimal regions and thereby prioritized learning with higher coverage of scientific values in autonomous experiments. Our work showcases the potential application of this method to real-world experiments, where such combined strategic and human intervening approaches can be critical to unlocking new discoveries in autonomous research.

Biswas, Arpan [University of Tennessee, Knoxville,↗
NASA NTRS2003

Development of a Dynamically Configurable, Object-Oriented Framework for Distributed, Multi-modal Computational Aerospace Systems Simulation: Second Year Progress Report

Mesh generation has long been recognized as a bottleneck in the CFD process. While much research on automating the volume mesh generation process have been relatively successful,these methods rely on appropriate initial surface triangulation to work properly. Surface discretization has been one of the least automated steps in computational simulation due to its dependence on implicitly defined CAD surfaces and curves. Differences in CAD peometry engines manifest themselves in discrepancies in their interpretation of the same entities. This lack of "good" geometry causes significant problems for mesh generators, requiring users to "repair" the CAD geometry before mesh generation. The problem is exacerbated when CAD geometry is translated to other forms (e.g., IGES )which do not include important topological and construction information in addition to entity geometry. One technique to avoid these problems is to access the CAD geometry directly from the mesh generating software, rather than through files. By accessing the geometry model (not a discretized version) in its native environment, t h s a proach avoids translation to a format which can deplete the model of topological information. Our approach to enable models developed in the Denali software environment to directly access CAD geometry and functions is through an Application Programming Interface (API) known as CAPRI. CAPRI provides a layer of indirection through which CAD-specific data may be accessed by an application program using CAD-system neutral C and FORTRAN language function calls. CAPRI supports a general set of CAD operations such as truth testing, geometry construction and entity queries.

Afjeh, Abdollah A.↗
DOE OSTI2026

Quantum-Inspired Bayesian Sampling for Uncertainty Quantification and Machine Learning (Final Technical Report)

With increasing simulation and measurement data, machine learning and artificial intelligence have been widely used in computational decision-making of complex engineering systems. The resulting tools, such as uncertainty quantification solvers, reinforcement learning, and physics-informed machine learning, have achieved great success in critical DOE tasks such as material discovery and design, energy system modeling and control, and numerical weather and climate prediction. A core topic in scientific machine learning and artificial intelligence is Bayesian inference: given an observed data set, people want to estimate the posterior distribution of a (possibly large) number of hidden parameters. Due to the flexibility and weak assumptions, Bayesian sampling has been the mainstream Bayesian inference solvers despite the rapid progress of approximate Bayesian inference. Classical Bayesian sampling methods such as Markov-chain Monte Carlo suffer from a low-acceptance rate due to the random walk nature, therefore state-of-the-art techniques use Hamiltonian Monte Carlo and its variants to efficiently draw posterior samples in a high dimension. The key idea of Hamiltonian Monte Carlo and its variants is to simulate the Hamiltonian dynamics of a classical particle with a fixed mass, and their performance significantly degrades when the posterior distribution is highly spiky or has multiple modes. Leveraging the idea of quantum physics, this project has investigated new theory, algorithms and applications of Bayesian inference (especially Bayesian sampling). The main results include: (1) novel quantum-inspired Bayesian sampling methods that can lead to better accuracy for challenging multi-modal or spiky distributions, (2) more scalable machine learning framework leveraging tensor-compressed Bayesian inference, and (3) Bayesian and sampling approaches for verifying the robustness of continuous and binary neural networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Scalable Multi-Facility Workflows for Artificial Intelligence Applications in Climate Research

Earth observation satellites and earth system models are sources of vast, multi-modal datasets that are invaluable for advancing climate and environmental research. However, their scale and complexity pose significant challenges for processing and analysis. In this paper we discuss our experiences in developing and using a scientific research application using an automated multi-facility workflow that orchestrates data collection, preprocessing, artificial intelligence (AI) inferencing, and data movement across diverse computational resources, leveraging the Advanced Computing Ecosystem Testbed at the Oak Ridge Leadership Computing Facility (OLCF). We demonstrate that our workflow can be seamlessly integrated and orchestrated across research facilities managed by different federal agencies, thus allowing users to extract new scientific insights from climate datasets. The experimental results indicate that the multi-facility workflow significantly reduces processing time, enhances scalability, and maintains high efficiency across varying workloads. Notably, our workflow processes 12,000 high-resolution satellite images in just 44 seconds using 80 workers distributed across 10 nodes on the OLCF systems. Such high throughput is essential for dynamic tokenization and sharding of petascale satellite data for distributed AI model training and inferencing at scale across thousands of GPUs.

Kurihana, Takuya [ORNL] (ORCID:0000000156698565)↗