Search NASA⌕ Search

SEARCH · Search NASA

Results for “multimodal data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Advanced tracking and data relay experiments study: Multimode transponder experiment equipment

Plans and implementation concepts were developed for a series of experiments utilizing a Multimode Transponder mounted in an aircraft working either through a spacecraft or directly with a ground station which would simulate a TDRSS user working through the TDRSS. The purpose of the experiments is to determine the best modulation and encoding techniques for combating RFI in discreet bands. The experiments also determine the feasibility and accuracy of range and range rate measurements with the various modulation and encoding techniques. An analysis of the Multimode Transponder and ground support equipment is presented, and the additional equipment required to perform the experiments described above is determined.

Cnossen, R. S.↗

Generation of a suite of 3D computer-generated breast phantoms from a limited set of human subject data

Purpose: The authors previously reported on a three-dimensional computer-generated breast phantom, based on empirical human image data, including a realistic finite-element based compression model that was capable of simulating multimodality imaging data. The computerized breast phantoms are a hybrid of two phantom generation techniques, combining empirical breast CT (bCT) data with flexible computer graphics techniques. However, to date, these phantoms have been based on single human subjects. In this paper, the authors report on a new method to generate multiple phantoms, simulating additional subjects from the limited set of original dedicated breast CT data. The authors developed an image morphing technique to construct new phantoms by gradually transitioning between two human subject datasets, with the potential to generate hundreds of additional pseudoindependent phantoms from the limited bCT cases. The authors conducted a preliminary subjective assessment with a limited number of observers (n= 4) to illustrate how realistic the simulated images generated with the pseudoindependent phantoms appeared. Methods: Several mesh-based geometric transformations were developed to generate distorted breast datasets from the original human subject data. Segmented bCT data from two different human subjects were used as the 'base' and 'target' for morphing. Several combinations of transformations were applied to morph between the 'base' and 'target' datasets such as changing the breast shape, rotating the glandular data, and changing the distribution of the glandular tissue. Following the morphing, regions of skin and fat were assigned to the morphed dataset in order to appropriately assign mechanical properties during the compression simulation. The resulting morphed breast was compressed using a finite element algorithm and simulated mammograms were generated using techniques described previously. Sixty-two simulated mammograms, generated from morphing three human subject datasets, were used in a preliminary observer evaluation where four board certified breast radiologists with varying amounts of experience ranked the level of realism (from 1 ='fake' to 10 ='real') of the simulated images. Results: The morphing technique was able to successfully generate new and unique morphed datasets from the original human subject data. The radiologists evaluated the realism of simulated mammograms generated from the morphed and unmorphed human subject datasets and scored the realism with an average ranking of 5.87 {+-} 1.99, confirming that overall the phantom image datasets appeared more 'real' than 'fake.' Moreover, there was not a significant difference (p > 0.1) between the realism of the unmorphed datasets (6.0 {+-} 1.95) compared to the morphed datasets (5.86 {+-} 1.99). Three of the four observers had overall average rankings of 6.89 {+-} 0.89, 6.9 {+-} 1.24, 6.76 {+-} 1.22, whereas the fourth observer ranked them noticeably lower at 2.94 {+-} 0.7. Conclusions: This work presents a technique that can be used to generate a suite of realistic computerized breast phantoms from a limited number of human subjects. This suite of flexible breast phantoms can be used for multimodality imaging research to provide a known truth while concurrently producing realistic simulated imaging data.

60 APPLIED LIFE SCIENCES↗

Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences

Fueled by breakthrough technology developments, the biological, biomedical, and behavioral sciences are now collecting more data than ever before. There is a critical need for time- and cost-efficient strategies to analyze and interpret these data to advance human health. The recent rise of machine learning as a powerful technique to integrate multimodality, multifidelity data, and reveal correlations between intertwined phenomena presents a special opportunityin this regard. However, classical machine learning techniques often ignore the fundamental laws of physics and result in ill-posed problems or non-physical solutions. Multiscale modeling is a successful strategy to integrate multiscale, multiphysics data and uncover mechanisms that explain the emergence of function. However, multiscale modeling alone often fails to efficiently combine large data sets from different sources and different levels of resolution. Here we demonstrate that machine learning and multiscale modeling can naturally complement each other to create robust predictive models that integrate the underlying physics to manage ill-posed problems and explore massive design spaces. We critically review the current literature, highlight applications and opportunities, address open questions, and discuss potential challenges and limitations in four overarching topical areas: ordinary differential equations, partial differential equations, data-driven approaches, and theory-driven approaches. Towards these goals, we leverage expertise in applied mathematics, computer science, computational biology, biophysics, biomechanics, engineering mechanics, experimentation, and medicine. Our multidisciplinary perspective suggests that integrating machine learning and multiscale modeling can provide new insights into disease mechanisms, help identify new targets and treatment strategies, and inform decision making for the benefit of human health.

97 MATHEMATICS AND COMPUTING↗

2020 Can Do Colorado E-Bike Mini Pilot Program Study

### The Colorado Energy Office conducted a mini pilot program study as part of the Can Do Colorado initiative, providing e-bikes to 13 low-income participants. The program aimed to encourage energy-efficient transportation during the COVID-19 pandemic as transit services were reduced and people were concerned about exposure. The insights garnered from this small-scale pilot study informed the design of a full-scale, 2-year pilot in locations across Colorado. For more information about the mini pilot program, see NLR's [Preliminary Results Report](https://www.nlr.gov/docs/fy21osti/79657.pdf). Micromobility options such as e-bikes offer a solution for improving energy efficiency for short-distance trips, especially in urban areas. Pedal-assist e-bikes use an electric motor and battery to help power the bike. The motor amplifies the power behind each pedal stroke, augmenting the energy you put into the bike. #### Data Collection Agency The Colorado Energy Office conducted the survey. #### Survey Methodology Participants in the program received a Momentum LaFree E+ e-bike (Class 1) and accessories at no cost and manually submitted travel data and feedback for 3 months using the CanBikeCo App. The smartphone app, developed in partnership with NLR, used a customized version of the [NLR OpenPATH platform](https://www.nlr.gov/transportation/openpath.html). #### Survey Records and Data Survey records include a total of 13 participants. This dataset contains 3 months of end-to-end, multimodal travel data manually submitted via smartphone app by 13 low-income essential workers in the greater Denver area. The data includes distance, mode (e.g., e-bike, car, transit), trip purpose, and demographic information.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Descriptor: Infrastructure Perception and Control: Multi-Sensor Object Tracking Dataset (IPC-MSOT)

Traffic intersections are crucial and challenging nodes in transportation networks where multiple lanes of vehicles and pedestrians converge. Traffic accidents often occur at traffic intersections, including a large proportion of traffic fatalities and about one-half of all traffic injuries in the United States. Object detection data were collected in 2024 across three intersections in Colorado Springs, CO, USA, over the course of multiple days and various times to induce a heterogeneous mix of traffic conditions and behaviors. The purpose of the data collection exercises was to learn various attributes about infrastructure sensors and to build a repository of high-resolution, object-level data that can be used for research and development (e.g., to develop multisensor data fusion algorithms). The Infrastructure Perception and Control:Multi-Sensor Object tracking (IPC-MSOT) dataset was collected as part of the U.S. Department of Transportation's Strengthening Mobility and Revolutionizing Transportation (SMART) project, where the city of Colorado Springs, Colorado, and the National Renewable Energy Laboratory collaborated to collect object-level trajectory data from road users using multiple types of infrastructure sensors deployed at different intersections. This dataset allows for testing of late-stage sensor fusion algorithms and their ability to ingest multimodal sensor data, and it can be utilized by traffic engineers to design and evaluate trajectory-based signal control strategies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Doppler Global Velocimetry at NASA Glenn Research Center: System Discussion and Results

A ruggedized Doppler Global Velocimetry system has been built and tested at NASA Glenn Research Center. One component of planar velocity measurements of subsonic and supersonic flows from an under-expanded free jet are reported, which agree well with predicted values. An error analysis evaluates geometric and spectral error terms, and characterizes speckle noise in isotropic data. A multimode, fused fiber optic bundle is demonstrated to couple up to 650 mJ/pulse of laser light without burning or fiber ablation, and without evidence of Stimulated Brillouin Scattering or other spectral-broadening problems. Comparisons are made between spinning wheel data using illumination by freespace beam propagation and fiber optic beam delivery. The fiber bundle illumination is found to provide more spatially even and stable illumination than is typically available from pulsed Nd:YAG laser beams. The fiber bundle beam delivery is also a step toward making remote measurements and automatic real-time plume sectioning feasible in wind tunnel environments.

Lant, Christian T.↗

Audiovisual Fusion: Challenges and New Approaches

In this study, we review recent results on audiovisual (AV) fusion. We also discuss some of the challenges and report on approaches to address them. One important issue in AV fusion is how the modalities interact and influence each other. This review will address this question in the context of AV speech processing, and especially speech recognition, where one of the issues is that the modalities both interact but also sometimes appear to desynchronize from each other. An additional issue that sometimes arises is that one of the modalities may be missing at test time, although it is available at training time; for example, it may be possible to collect AV training data while only having access to audio at test time. We will review approaches to address this issue from the area of multiview learning, where the goal is to learn a model or representation for each of the modalities separately while taking advantage of the rich multimodal training data. In addition to multiview learning, we also discuss the recent application of deep learning (DL) toward AV fusion. We finally draw conclusions and offer our assessment of the future in the area of AV fusion.

47 OTHER INSTRUMENTATION↗

Solving multiphysics-based inverse problems with learned surrogates and constraints

Abstract Solving multiphysics-based inverse problems for geological carbon storage monitoring can be challenging when multimodal time-lapse data are expensive to collect and costly to simulate numerically. We overcome these challenges by combining computationally cheap learned surrogates with learned constraints. Not only does this combination lead to vastly improved inversions for the important fluid-flow property, permeability, it also provides a natural platform for inverting multimodal data including well measurements and active-source time-lapse seismic data. By adding a learned constraint, we arrive at a computationally feasible inversion approach that remains accurate. This is accomplished by including a trained deep neural network, known as a normalizing flow, which forces the model iterates to remain in-distribution, thereby safeguarding the accuracy of trained Fourier neural operators that act as surrogates for the computationally expensive multiphase flow simulations involving partial differential equation solves. By means of carefully selected experiments, centered around the problem of geological carbon storage, we demonstrate the efficacy of the proposed constrained optimization method on two different data modalities, namely time-lapse well and time-lapse seismic data. While permeability inversions from both these two modalities have their pluses and minuses, their joint inversion benefits from either, yielding valuable superior permeability inversions and CO 2 plume predictions near, and far away, from the monitoring wells.

Yin, Ziyi (ORCID:0000000250248771)↗

Pixel-Registered Multimodal Synchrotron XRF and FTIR Microscopies Reveal Salinity Stress Response Mechanisms in Pistachio

Background: Salinity is a major abiotic stress that negatively affects nearly all plant species at all stages of growth. Drought and poor-quality irrigation cause high soil salinity and salt accumulation via evaporation, reducing crop productivity. Despite its critical importance, the spatial localization of salt ions and associated biochemical changes within plants experiencing high salinity remains largely unknown. In this study, we developed a multimodal imaging pipeline to understand the impact of salinity on the pistachio rootstock UCB-1 (Pistacia atlantica x Pistacia integerrima). We directly link biochemical fingerprints in stem tissue architecture with salt ion localization to provide insights into the strategies pistachio uses to tolerate salinity. Results: We observed that Pistacia spp. exposed to high salt conditions accumulated Ca, Si, Cl, Al and Mg as hotspots within the pith, compared to the control (of which only Ca and Al co-locate). In contrast, there was a decrease in K between the control and salinity treatment. Hotspots of amide I and II were present in the cortex and pith of the salinity treated sample. Additionally, the salinity treatment resulted in an increased abundance of pectin and carbohydrates within the pith compared to the control, and the abundance of esters/carboxylic acid was greater in the salinity treatment. Conclusions: We determined that Cl and K, S and P, and biochemical components polysaccharide and pectin, esters and carboxylic acid, amide I and cellulose are the strongest drivers of salinity- treatment induced variability. In the cortex and phloem/xylem, a negative K-Ca correlation decreases in the salinity treatment. Several hotspots of elements and amide I (proteins) appear under salinity treatment, particularly in the cortex, suggesting an increase in the production of stress-related proteins (in response to high Cl) and/or structural proteins (i.e. Ca). Together, these results indicate that pistachio responds to salinity through ion compartmentalization coupled with a targeted biochemical adjustment, rather than a broadscale tissue-wide response. Overall, these novel, spatially resolved pixel-registered multimodal imaging data provide an enabling platform to understand the mechanisms of salinity tolerance in Pistacia spp and can be broadly applied to studying stress-related phenotype response in various plant tissues.

FTIR spectromicroscopy↗

Multimodal Atomic Force Microscopy for the Characterization of Metallic Particulates

This study investigates the utility of multimodal, or functional, atomic force microscopy (AFM) for the characterization of individual particles and particle ensembles. In single-particle analyses, AFM imaging modes provided orthogonal insights into mechanical and magnetic properties that are not accessible through conventional electron microscopy. Although the experiments were inherently delicate and time-intensive, with an observed sample loss rate of approximately 30%, these techniques enabled the qualitative differentiation of grain structures and magnetic domains, underscoring the challenges of experimental robustness. For particle ensembles, AFM enabled the extraction of reliable 2D and 3D particle size distributions. However, efforts to chemically differentiate particles based on mechanical property contrasts were limited by scaling effects and the qualitative nature of the data. Overall, multimodal AFM offers valuable complementary information, but its application requires careful consideration of methodological limitations, particularly for sample integrity, calibration, and scalability of mechanical and magnetic measurements. This report is organized to first address the application of AFM to single-particle analysis using various imaging modalities, followed by its potential role in ensemble-level particle screening and differentiation.

36 MATERIALS SCIENCE↗

Improving streamflow predictions across CONUS by integrating advanced machine learning models and diverse data

Accurate streamflow prediction is crucial to understand climate impacts on water resources and develop effective adaption strategies. A global long short-term memory (LSTM) model, using data from multiple basins, can enhance streamflow prediction, yet acquiring detailed basin attributes remains a challenge. To overcome this, we introduce the Geo-vision transformer (ViT)-LSTM model, a novel approach that enriches LSTM predictions by integrating basin attributes derived from remote sensing with a ViT architecture. Applied to 531 basins across the Contiguous United States, our method demonstrated superior prediction accuracy in both temporal and spatiotemporal extrapolation scenarios. Geo-ViT-LSTM marks a significant advancement in land surface modeling, providing a more comprehensive and effective tool for better understanding the environment responses to climate change.

Tayal, Kshitij↗

Peak2Patch: High-Fidelity Functional Group Identification through Attention-Based Fusion of Infrared and Mass Spectra

Identifying molecular structure based on spectroscopic readings is a key task in a variety of chemical and biological applications. Common spectroscopy techniques, such as Infrared (IR) Spectroscopy and Mass Spectrometry (MS), provide detailed information on the structure of molecular compounds but nonetheless require expert-level knowledge to decode. Machine learning has emerged as a potential solution for automating structure prediction from chemical spectra; however, current approaches generally focus on single sensor modalities, neglecting to leverage the complementary information contained within differing spectra. In this paper, we introduce Peak2Patch, a novel approach to fusion-enhanced prediction of functional groups from IR and mass spectra. First, we perform a detailed comparison of backbone networks for encoding both sparse mass spectra and dense IR spectra and demonstrate the superior performance of transformer neural networks over current state-of-the-art convolutional neural networks. Second, we evaluate three broad categories of fusion: early (raw feature), middle (deep feature), and late (decision) fusion, demonstrating the potential of a deep feature fusion-based approach. Lastly, we present Peak2Patch, our attention-based fusion scheme, which leverages cross-attention to mix features between encoded tokens of the two modalities. We validate our approach on a publicly available multimodal spectroscopic data set of 790k simulated molecules, demonstrating a large improvement in functional group prediction over both the previous state-of-the-art and our own strong single-modal baselines.

Jacobson, Philip [Sandia National Laboratories (SN↗

Deep learning with plasma plume image sequences for anomaly detection and prediction of growth kinetics during pulsed laser deposition

Abstract Materials synthesis platforms that are designed for autonomous experimentation are capable of collecting multimodal diagnostic data that can be utilized for feedback to optimize material properties. Pulsed laser deposition (PLD) is emerging as a viable autonomous synthesis tool, and so the need arises to develop machine learning (ML) techniques that are capable of extracting information from in situ diagnostics. Here, we demonstrate that intensified-CCD image sequences of the plasma plume generated during PLD can be used for anomaly detection and the prediction of thin film growth kinetics. We develop multi-output (2 + 1)D convolutional neural network regression models that extract deep features from plume dynamics that not only correlate with the measured chamber pressure and incident laser energy, but more importantly, predict parameters of an auto-catalytic film growth model derived from in situ laser reflectivity experiments. Our results demonstrate how ML with in situ plume diagnostics data in PLD can be utilized to maintain deposition conditions in an optimal regime. Further, the predictive capabilities of plume dynamics on the kinetics of film growth or other film properties prior to deposition provides a means for rapid pre-screening of growth conditions for the non-expert, which promises to accelerate materials optimization with PLD.

36 MATERIALS SCIENCE↗

Roadmap for transforming heterogeneous catalysis with artificial intelligence

Artificial intelligence (AI) is poised to transform heterogeneous catalysis, opening avenues for catalytic materials discovery. By uncovering intricate patterns in high-dimensional data, AI has been reshaping our pursuit of sustainable catalytic processes across the energy, environmental and chemical sectors. This promise, however, hinges on overcoming fundamental barriers, including limitations in data availability and quality, challenges in the generalizability and interpretability of data-augmented decisions, and the persistent gap between in silico predictions and experiments. Furthermore, we outline a forward-looking roadmap for deeply integrating AI into heterogeneous catalysis with an AI-ready data ecosystem, multimodal foundation models, and ultimately autonomous laboratories to accelerate the development of next-generation catalytic technologies via AI-empowered human–machine collaboration.

Computational methods↗

IBEX: An open infrastructure software platform to facilitate collaborative work in radiomics

Purpose: Radiomics, which is the high-throughput extraction and analysis of quantitative image features, has been shown to have considerable potential to quantify the tumor phenotype. However, at present, a lack of software infrastructure has impeded the development of radiomics and its applications. Therefore, the authors developed the imaging biomarker explorer (IBEX), an open infrastructure software platform that flexibly supports common radiomics workflow tasks such as multimodality image data import and review, development of feature extraction algorithms, model validation, and consistent data sharing among multiple institutions. Methods: The IBEX software package was developed using the MATLAB and C/C++ programming languages. The software architecture deploys the modern model-view-controller, unit testing, and function handle programming concepts to isolate each quantitative imaging analysis task, to validate if their relevant data and algorithms are fit for use, and to plug in new modules. On one hand, IBEX is self-contained and ready to use: it has implemented common data importers, common image filters, and common feature extraction algorithms. On the other hand, IBEX provides an integrated development environment on top of MATLAB and C/C++, so users are not limited to its built-in functions. In the IBEX developer studio, users can plug in, debug, and test new algorithms, extending IBEX’s functionality. IBEX also supports quality assurance for data and feature algorithms: image data, regions of interest, and feature algorithm-related data can be reviewed, validated, and/or modified. More importantly, two key elements in collaborative workflows, the consistency of data sharing and the reproducibility of calculation result, are embedded in the IBEX workflow: image data, feature algorithms, and model validation including newly developed ones from different users can be easily and consistently shared so that results can be more easily reproduced between institutions. Results: Researchers with a variety of technical skill levels, including radiation oncologists, physicists, and computer scientists, have found the IBEX software to be intuitive, powerful, and easy to use. IBEX can be run at any computer with the windows operating system and 1GB RAM. The authors fully validated the implementation of all importers, preprocessing algorithms, and feature extraction algorithms. Windows version 1.0 beta of stand-alone IBEX and IBEX’s source code can be downloaded. Conclusions: The authors successfully implemented IBEX, an open infrastructure software platform that streamlines common radiomics workflow tasks. Its transparency, flexibility, and portability can greatly accelerate the pace of radiomics research and pave the way toward successful clinical translation.

60 APPLIED LIFE SCIENCES↗

SU-C-BRA-06: Automatic Brain Tumor Segmentation for Stereotactic Radiosurgery Applications

Purpose: Stereotactic radiosurgery (SRS), which delivers a potent dose of highly conformal radiation to the target in a single fraction, requires accurate tumor delineation for treatment planning. We present an automatic segmentation strategy, that synergizes intensity histogram thresholding, super-voxel clustering, and level-set based contour evolving methods to efficiently and accurately delineate SRS brain tumors on contrast-enhance T1-weighted (T1c) Magnetic Resonance Images (MRI). Methods: The developed auto-segmentation strategy consists of three major steps. Firstly, tumor sites are localized through 2D slice intensity histogram scanning. Then, super voxels are obtained through clustering the corresponding voxels in 3D with reference to the similarity metrics composited from spatial distance and intensity difference. The combination of the above two could generate the initial contour surface. Finally, a localized region active contour model is utilized to evolve the surface to achieve the accurate delineation of the tumors. The developed method was evaluated on numerical phantom data, synthetic BRATS (Multimodal Brain Tumor Image Segmentation challenge) data, and clinical patients’ data. The auto-segmentation results were quantitatively evaluated by comparing to ground truths with both volume and surface similarity metrics. Results: DICE coefficient (DC) was performed as a quantitative metric to evaluate the auto-segmentation in the numerical phantom with 8 tumors. DCs are 0.999±0.001 without noise, 0.969±0.065 with Rician noise and 0.976±0.038 with Gaussian noise. DC, NMI (Normalized Mutual Information), SSIM (Structural Similarity) and Hausdorff distance (HD) were calculated as the metrics for the BRATS and patients’ data. Assessment of BRATS data across 25 tumor segmentation yield DC 0.886±0.078, NMI 0.817±0.108, SSIM 0.997±0.002, and HD 6.483±4.079mm. Evaluation on 8 patients with total 14 tumor sites yield DC 0.872±0.070, NMI 0.824±0.078, SSIM 0.999±0.001, and HD 5.926±6.141mm. Conclusion: The developed automatic segmentation strategy, which yields accurate brain tumor delineation in evaluation cases, is promising for its application in SRS treatment planning.

60 APPLIED LIFE SCIENCES↗

Study of the surface heterogeneity of topographic and electric nature by atomic force microscopy by the example of triglycine sulfate

Triglycine sulfate crystals with an ideal (010) cleavage plane are used as model objects to reveal problems in interpreting atomic force microscopy (AFM) images of surfaces with nonuniform charge distribution. Specific microrelief features of two types are found: lenslike formations with different contrast and rounded protrusions/valleys of different size but fixed height. An analysis of their evolution with a change in temperature and under an electric field and mechanical impacts has made it possible to separate relief elements from the crystal domain structure. The interpretation proposed is confirmed by the multimode AFM data. The specific features of the images of dynamic domains and aged domains (which cannot undergo polarization reversal) are studied. The domain-wall width found in the AFM measurements depends on the technique used and the specificity of probe-surface interaction; it varies from 9 to 2000 nm. The most reliable data on the domain-wall width in triglycine sulfate crystals are provided by piezoelectric force microscopy, according to which the wall width does not exceed 30 nm.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗