Search NASA⌕ Search

SEARCH · Search NASA

Results for “Multi-modal Sensing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

An organic electrochemical transistor for multi-modal sensing, memory and processing

By integrating sensing, memory and processing functionalities, biological nervous systems are energy and area efficient. Emulating such capabilities in artificial systems is, however, challenging and is limited by the device heterogeneity of sensing and processing cores. Here we report an organic electrochemical transistor capable of sensing, memory and processing. The device has a vertical traverse architecture and a crystalline–amorphous channel that can be selectively doped by ions to enable two reconfigurable modes: a volatile receptor and a non-volatile synapse. As a volatile receptor, the device is capable of multi-modal sensing and is responsive to stimuli such as ions and light. As a non-volatile synapse, it is capable of 10-bit analogue states, low switching stochasticity and good state retention. We also show that the homogeneous integration of the devices could provide functions such as conditioned reflexes and could be used for real-time cardiac disease diagnoses via reservoir computing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Integrating multi-modal remote sensing, deep learning, and attention mechanisms for yield prediction in plant breeding experiments

In both plant breeding and crop management, interpretability plays a crucial role in instilling trust in AI-driven approaches and enabling the provision of actionable insights. The primary objective of this research is to explore and evaluate the potential contributions of deep learning network architectures that employ stacked LSTM for end-of-season maize grain yield prediction. A secondary aim is to expand the capabilities of these networks by adapting them to better accommodate and leverage the multi-modality properties of remote sensing data. In this study, a multi-modal deep learning architecture that assimilates inputs from heterogeneous data streams, including high-resolution hyperspectral imagery, LiDAR point clouds, and environmental data, is proposed to forecast maize crop yields. The architecture includes attention mechanisms that assign varying levels of importance to different modalities and temporal features that, reflect the dynamics of plant growth and environmental interactions. The interpretability of the attention weights is investigated in multi-modal networks that seek to both improve predictions and attribute crop yield outcomes to genetic and environmental variables. This approach also contributes to increased interpretability of the model's predictions. The temporal attention weight distributions highlighted relevant factors and critical growth stages that contribute to the predictions. The results of this study affirm that the attention weights are consistent with recognized biological growth stages, thereby substantiating the network's capability to learn biologically interpretable features. Accuracies of the model's predictions of yield ranged from 0.82-0.93 R 2 ref in this genetics-focused study, further highlighting the potential of attention-based models. Further, this research facilitates understanding of how multi-modality remote sensing aligns with the physiological stages of maize. The proposed architecture shows promise in improving predictions and offering interpretable insights into the factors affecting maize crop yields, while demonstrating the impact of data collection by different modalities through the growing season. By identifying relevant factors and critical growth stages, the model's attention weights provide valuable information that can be used in both plant breeding and crop management. The consistency of attention weights with biological growth stages reinforces the potential of deep learning networks in agricultural applications, particularly in leveraging remote sensing data for yield prediction. To the best of our knowledge, this is the first study that investigates the use of hyperspectral and LiDAR UAV time series data for explaining/interpreting plant growth stages within deep learning networks and forecasting plot-level maize grain yield using late fusion modalities with attention mechanisms.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Progress of Tripwire: Multi-Modal Distributed Sensing for Repository Verification

This project is developing and demonstrating a multi-modal sensor system, TRIPWIRE, for containment verification in inaccessible radiological and nuclear waste repositories. The TRIPWIRE system will continuously monitor ionizing radiation and electromagnetic fields in the vicinity of emplaced nuclear materials buried in a repository, reporting on disturbances with a real-time alarm control station. The system will use long-length scintillating fiber bundles (SFBs) to perform area radiation monitoring; these will be coupled to kilometer-scale multimodal optical communication fibers – all light sensors and electronic components used with this system will be located above ground. Electromagnetic fields, and changes in local dielectric conditions caused by intrusion and soil movement, will be monitored using commercial grade, ported "leaky" coaxial cables (PCCs), with control electronics also located above ground. This paper documents the results of the simulations and modeling efforts that have helped identify the optimal placement of the SFBs. Additionally, the paper includes the current progress on the multi-modal SFB-PCC system, including the fabrication of long-length (>3-m) fibers and their coupling to optical fibers.

98 - NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL↗
DOE OSTI2022

Progress of Tripwire: Multi-modal Distributed Sensing for Repository Verification

This project is developing and demonstrating a multi-modal sensor system, TRIPWIRE, for containment verification in inaccessible radiological and nuclear waste repositories. The TRIPWIRE system will continuously monitor ionizing radiation and electromagnetic fields in the vicinity of emplaced nuclear materials buried in a repository, reporting on disturbances with a real-time alarm control station. The system will use long-length scintillating fiber bundles (SFBs) to perform area radiation monitoring; these will be coupled to kilometer-scale multimodal optical communication fibers – all light sensors and electronic components used with this system will be located above ground. Electromagnetic fields, and changes in local dielectric conditions caused by intrusion and soil movement, will be monitored using commercial grade, ported "leaky" coaxial cables (PCCs), with control electronics also located above ground. This paper documents the results of the simulations and modeling efforts that have helped identify the optimal placement of the SFBs. Additionally, the paper includes the current progress on the multi-modal SFB-PCC system, including the fabrication of long-length (>3-m) fibers and their coupling to optical fibers.

98 - NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL↗
DOE OSTI2026

Curation and Dissemination of Complex Multi-Modal Datasets for Radiation Detection, Localization, and Tracking

The PANDAWN sensor network in Chicago, IL, is a state-of-the-art testbed for networked, multi-modal sensing. It integrates AI/data science methods into its operation, from data acquisition to automated data labeling and curation workflows. The curation and dissemination of diverse multi-modal datasets will enable the development of new radiological/nuclear (R/N) detection, localization, and tracking algorithms and methods relevant across the nonproliferation mission space. This article first introduces the PANDAWN sensor network and the features that make it stand out from previous multi-modal data acquisition efforts. We then review the various data streams acquired on the PANDAWN nodes and present the implementation of an automated data curation pipeline that includes the labeling of radiation and contextual data streams. Here, we finally provide a short overview of different studies that leveraged the curated datasets.

Data curation↗
DOE OSTI2023

Face mask integrated with flexible and wearable manganite oxide respiration sensor

Face masks are key personal protective equipment for reducing exposure to viruses and other environmental hazards such as air pollution. Integrating flexible and wearable sensors into face masks can provide valuable insights into personal and public health. The advantages that a breath-monitoring face mask requires, including multi-functional sensing ability and continuous, long-term dynamic breathing process monitoring, have been underdeveloped to date. Here, we design an effective human breath monitoring face mask based on a flexible La 0.7 Sr 0.3 MnO 3 (LSMO)/Mica respiration sensor. The sensor’s capabilities and systematic measurements are investigated under two application scenes, namely clinical monitoring mode and daily monitoring mode, to monitor, recognise, and analyse different human breath status, i.e., cough, normal breath, and deep breath. This sensing system exhibits super-stability and multi-modal capabilities in continuous and long-time monitoring of the human breath. We determine that during monitoring human breath, thermal diffusion in LSMO is responsible for the change of resistance in flexible LSMO/Mica sensor. Both simulated and experimental results demonstrate good discernibility of the flexible LSMO/Mica sensor operating at different breath status. Our work opens a route for the design of novel flexible and wearable electronic devices.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Potential-induced wetting and dewetting in pH-responsive block copolymer membranes for mass transport control

Wetting and dewetting behavior in channel-confined hydrophobic volumes is used in biological membranes to effect selective ion/molecular transport. Artificial biomimetic hydrophobic nanopores have been devised utilizing wetting and dewetting, however, tunable mass transport control utilizing multiple transport modes is required for applications such as controllable release/transport, water separation/purification and energy conversion. Here, we investigate the potential-induced wetting and dewetting behavior in a pH-responsive membrane composed of a polystyrene-b-poly(4-vinylpyridine) (PS-b-P4VP) block copolymer (BCP) when fabricated as a hierarchically-organized sandwich structure on a nanopore electrode array (NEA), i.e. BCP@NEA. At pH < pK a (P4VP) (pK a ~ 4.8), the BCP acts as an anion-exchange membrane due to the hydrophilic, protonated P4VP cylindrical nanodomains, but at pH > pK a (P4VP), the P4VP domains exhibit charge-neutral, hydrophobic and collapsed structures, blocking mass transport via the hydrophobic membrane. However, when originally prepared in a dewetted condition, mass transport in the BCP membrane may be switched on if sufficiently negative potentials are applied to the BCP@NEA architecture. When the hydrophobic BCP membrane is introduced on top of 2-electrode-embedded nanopore arrays, electrolyte solution in the nanopores is introduced, then isolated, by exploiting the potential-induced wetting and dewetting transitions in the BCP membrane. The potential-induced wetting/dewetting transition and the effect on cyclic voltammetry in the BCP@NEA structures is characterized as a function of the potential, pH and ionic strength. Additionally chronoamperometry and redox cycling experiments are used to further characterize the potential response. The multi-modal mass transport system proposed in this work will be useful for ultrasensitive sensing and single-molecule studies, which require long-time monitoring to explore reaction dynamics as well as molecular heterogeneity in nanoconfined volumes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

TRIPWIRE – Deployment and Field-Testing Report

TRIPWIRE is a multi-modal system for containment verification in inaccessible radiological and nuclear waste repositories. The sensing modes are focused on radiation, electromagnetic and vibration. The radiation detection component has been developed over the last three years at Idaho National Laboratory (INL). This development has focused on the fabrication, characterization, and testing of long-length (greater than 10-m) plastic scintillating fibers. This fiscal year, the focus of TRIPWIRE has been on field testing and demonstration of the radiation sensing component in a relevant environment. After reviewing the static storage facilities at INL, the Radioactive Scrap Waste Facility (RSWF) south storage area outside the Materials and Fuels Complex (MFC) was selected due to its periodic changes in configuration and inventory. This report summarizes the research outcomes for the work performed in fiscal year (FY) 2023.

98 - NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL↗
DOE OSTI2021

Iterative self-organizing SCEne-LEvel sampling (ISOSCELES) for large-scale building extraction

Convolutional neural networks (CNN) provide state-of-the-art performance in many computer vision tasks, including those related to remote-sensing image analysis. Successfully training a CNN to generalize well to unseen data, however, requires training on samples that represent the full distribution of variation of both the target classes and their surrounding contexts. With remote sensing data, acquiring a sufficiently representative training set is a challenge due to both the inherent multi-modal variability of satellite or aerial imagery and the general high cost of labeling data. To address this challenge, we have developed ISOSCELES, an Iterative Self-Organizing SCEne LEvel Sampling method for hierarchical sampling of large image sets. Using affinity propagation, ISOSCELES automates the selection of highly representative training images. Compared to random sampling or using available reference data, the distribution of the training is principally data driven, reducing the chance of oversampling uninformative areas or undersampling informative ones. In comparison to manual sample selection by an analyst, ISOSCELES exploits descriptive features, spectral and/or textural, and eliminates human bias in sample selection. Using a hierarchical sampling approach, ISOSCELES can obtain a training set that reflects both between-scene variability, such as in viewing angle and time of day, and within-scene variability at the level of individual training samples. We verify the method by demonstrating its superiority to stratified random sampling in the challenging task of adapting a pre-trained model to a new image and spatial domain for country-scale building extraction. Using a pair of hand-labeled training sets comprising 1,987 sample image chips, a total of 496,000,000 individually labeled pixels, we show, across three distinct model architectures, an increase in accuracy, as measured by F1-score, of 2.2–4.2%.

42 ENGINEERING↗
DOE OSTI2025

Enhancing Automotive Intrusion Detection Through Multi-Modal Fusion: A CAN FD-LiDAR Approach

As vehicles become smarter and more autonomous, they increasingly depend on advanced sensors and communication technologies to operate securely. However, such growing dependence on technology—whether it’s CAN (Controller Area Network) for internal communication or LiDAR (Light Detection and Ranging) for sensing the world around them—also expands the attack surface for the types of cyber attacks. Traditional intrusion detection systems (IDS) typically monitor these systems in isolation, limiting their ability to detect sophisticated, crosssystem attacks. To address this, we propose a multi-modal fusion approach that combines real-world CAN FD signals (from the HCRL dataset) with LiDAR features (from the nuScenes dataset) to enhance attack detection. Our method employs a twostage ensemble approach. Calibrated XGBoost and LightGBM models initially process CAN FD (Fuzzing Data) and LiDAR data independently, detecting timing anomalies and space abnormalities. They are subsequently logarithmically combined with a logistic regression meta-model along with 17 engineered features capturing cross-modal behavior, prediction conflicts, and nonlinear interactions. This approach achieves an AUC of 0.87 and an F1-score of 0.82, surpassing single-modality baselines and early fusion methods, at merely 2 ms inference latency. Compared with deep learning competitors, it is 3 times more efficient, providing a lightweight, interpretable, and real time solution to automotive cybersecurity.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Methods for rapid identification of anomalous layers in laser powder bed fusion

In situ process monitoring is a key requirement for increased industry acceptance of powder bed Additive Manufacturing. As sensing technologies increase in maturity, attention must also be given to effective data exploration techniques. These data are often high-resolution and multi-modal, with each build consisting of thousands of layers. Here, in this paper, the authors propose two methods enabling users to rapidly identify layers of interest within a build. Both methods leverage results from deep learning based segmentations of in situ powder bed images. The first method is an unsupervised “reverse layer search” algorithm while the second method uses supervised machine learning.

36 MATERIALS SCIENCE↗
DOE OSTI2025

A case study in contrastive learning information combination: Application to technical forensics of additive manufacturing filament source identification

Combination of information from disparate data sources into a single decision is a core challenge in many fields, including the field of technical forensics. Technical forensics (TF) utilizes technical characterization of questioned samples to determine properties of that sample; these properties are then used to infer information of forensic interest, such as provenance, age, or attribution. TF is utilized in traditional forensic applications, such as the attribution of material fragments from an explosive, and in nuclear forensic applications, such as the attribution of actinides which have been interdicted out of regulatory control. The challenge of combining information from disparate sources, described alternately by many terms including “Data Fusion” and “Data Integration”, is exacerbated in the technical forensics domain due to at least two factors: the challenge of interpreting each information source singularly, and the relatively small data set sizes available. Extensive literature exists attempting to combine technical forensics information sources, both in manual and automated processes. These attempts are often bespoke to the specific information sources (such as the bi-, tri-, or quad-isotope chart (Moody, Grant, and Hutcheon 2005)), with some emerging examples of simple early- and late- fusion (, respectively). Simultaneous to the information combination efforts described in the previous paragraph, the field of natural language processing attempted (and largely succeeded) in combining information from multiple non-technical information sources. The ecosystem of “multi-modal” language models, which can take text and images as input, and generate text and images as output, became large and diverse by 2025 (Khan et al. 2025). In a generalized sense, many of these methods are trained by learning neural networks which can convert raw text or images into a vector of numbers describing the text or image, hereafter called “embeddings” and the neural networks performing the conversion are called “embedders”. By using a separate embedder for text and images, finding coincident text and images (such as images with their captions), and optimizing the parameters of the embedders such that the embeddings for the text and the image are similar, the field has found a bridge between text and images (Girdhar et al. 2023). It is the contention of the authors of this report that this insight is not limited to text and images but instead can be extended to any modality which can be found coincidently. The subject of the rest of this report is the application of this method to example multi-modal technical forensic data. Some details about the data used in this report are not appropriate for this report, and are included in a companion report (PNNL-38669).

36 MATERIALS SCIENCE↗
DOE OSTI2022

Microcam: A Low Power and Privacy Preserving Multi-modal Platform for Occupancy Detection (Final Report)

Heating, ventilation, and air conditioning (HVAC) consumes a significant portion of the energy used in buildings. Much of this is wasted energy, used when buildings are either not occupied at all, or occupied well under their maximum design conditions. This project has focused on residential occupancy detection to autonomously control HVAC systems and save energy. Limitations of existing occupancy sensors include one or more of the following: (i) they employ sensors or algorithms that are not able to detect stationary occupants; (ii) they cannot classify the source of the motion (such as a pet); (iii) depending on the camera resolution and employed algorithms, they do not allow for embedded or onboard computation, and require external or cloud-based processing; (iv) many algorithms developed for camera-based systems are sensitive to lighting changes, and thus prone to missed detections or false alarms; (v) Most existing systems depend on adjustment of settings for different scenarios, complicating self-commissioning; (vi) they cannot provide high enough accuracy; (vii) they are costly; (viii) they are not battery-powered, thus limiting ease of use and installation. In this project, Syracuse University and its partner SRI have developed a low-cost, high accuracy, standalone residential occupancy sensing platform, referred to as the MicroCam, to address all of the aforementioned challenges. MicroCam can operate on typical alkaline batteries without relying on the “cloud” or external computing resources, and consists of low-power, Artificial Intelligence (AI)-based, IoT platforms. Each platform has multi-modal sensors and can process motion, audio and video data, and send binary occupancy result to a lead platform. All sensor data is processed locally on platforms, and the only transmitted data is the binary occupancy state. In addition, preliminary work has been done on images wherein occupants are not discernable. Thus, MicroCam is a standalone solution preserving privacy of the occupants.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗