Search NASA⌕ Search

SEARCH · Search NASA

Results for “Multimodal integration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Integrating multimodal data through interpretable heterogeneous ensembles

Motivation: Integrating multimodal data represents an effective approach to predicting biomedical characteristics, such as protein functions and disease outcomes. However, existing data integration approaches do not sufficiently address the heterogeneous semantics of multimodal data. In particular, early and intermediate approaches that rely on a uniform integrated representation reinforce the consensus among the modalities but may lose exclusive local information. The alternative late integration approach that can address this challenge has not been systematically studied for biomedical problems. Results: We propose Ensemble Integration (EI) as a novel systematic implementation of the late integration approach. EI infers local predictive models from the individual data modalities using appropriate algorithms and uses heterogeneous ensemble algorithms to integrate these local models into a global predictive model. We also propose a novel interpretation method for EI models. We tested EI on the problems of predicting protein function from multimodal STRING data and mortality due to coronavirus disease 2019 (COVID-19) from multimodal data in electronic health records. We found that EI accomplished its goal of producing significantly more accurate predictions than each individual modality. It also performed better than several established early integration methods for each of these problems. The interpretation of a representative EI model for COVID-19 mortality prediction identified several disease-relevant features, such as laboratory test (blood urea nitrogen and calcium) and vital sign measurements (minimum oxygen saturation) and demographics (age). These results demonstrated the effectiveness of the EI framework for biomedical data integration and predictive modeling.

59 BASIC BIOLOGICAL SCIENCES↗

Pan-cancer evaluation of clinical value of mitotic network activity index (MNAI) and its predictive value for immunotherapy

Increased mitotic activity is associated with the genesis and aggressiveness of many cancers. To assess the clinical value of mitotic activity as prognostic biomarker, we performed a pan-cancer study on the mitotic network activity index (MNAI) constructed based on 54-gene mitotic apparatus network. Our pan-cancer assessment on TCGA (33 tumor types, 10,061 patients) and validation on other publicly available cohorts (23 tumor types, 9,209 patients) confirmed the significant association of MNAI with overall survival, progression-free survival, and other prognostic endpoints in multiple cancer types, including lower-grade gliomas (LGG), breast invasive carcinoma (BRCA), as well as many others. We also showed significant association between MNAI and genetic instability, which provides a biological explanation of its prognostic impact at pan-cancer landscape. Our association analysis revealed that patients with high MNAI benefitted more from anti-PD-1 and Anti-CTLA-4 treatment. In addition, we demonstrated that multimodal integration of MNAI and the AI-empowered Cellular Morphometric Subtypes (CMS) significantly improved the predictive power of prognosis compared to using MNAI and CMS alone. Our results suggest that MNAI can be used as a potential prognostic biomarker for different tumor types toward different clinical endpoints, and multimodal integration of MNAI and CMS exceeds individual biomarker for precision prognosis.

60 APPLIED LIFE SCIENCES↗

A multimodal and integrated approach to interrogate human kidney biopsies with rigor and reproducibility: guidelines from the Kidney Precision Medicine Project

Comprehensive and spatially mapped molecular atlases of organs at a cellular level are a critical resource to gain insights into pathogenic mechanisms and personalized therapies for diseases. The Kidney Precision Medicine Project (KPMP) is an endeavor to generate three-dimensional (3-D) molecular atlases of healthy and diseased kidney biopsies by using multiple state-of-the-art omics and imaging technologies across several institutions. Obtaining rigorous and reproducible results from disparate methods and at different sites to interrogate biomolecules at a single-cell level or in 3-D space is a significant challenge that can be a futile exercise if not well controlled. Here we describe a “follow the tissue” pipeline for generating a reliable and authentic single-cell/region 3-D molecular atlas of human adult kidney. Our approach emphasizes quality assurance, quality control, validation, and harmonization across different omics and imaging technologies from sample procurement, processing, storage, shipping to data generation, analysis, and sharing. We established benchmarks for quality control, rigor, reproducibility, and feasibility across multiple technologies through a pilot experiment using common source tissue that was processed and analyzed at different institutions and different technologies. A peer review system was established to critically review quality control measures and the reproducibility of data generated by each technology before their being approved to interrogate clinical biopsy specimens. The process established economizes the use of valuable biopsy tissue for multiomics and imaging analysis with stringent quality control to ensure rigor and reproducibility of results and serves as a model for precision medicine projects across laboratories, institutions and consortia.

59 BASIC BIOLOGICAL SCIENCES↗

Revealing Local Structures through Machine-Learning-Fused Multimodal Spectroscopy

Atomistic structures of materials offer valuable insights into their functionality. Determining these structures remains a fundamental challenge in materials science, especially for systems with defects. While both experimental and computational methods exist, each has limitations in resolving nanoscale structures. Core-level spectroscopies, such as X-ray absorption (XAS) or electron energy-loss spectroscopies (EELS), have been used to determine the local bonding environment and structure of materials. Recently, machine learning (ML) methods have been applied to extract structural and bonding information from XAS/EELS data. However, frameworks relying solely on a single data stream, defined as characterization data derived from a single element using one technique, are often insufficient because multiple local environments can yield similar spectral features, making it challenging to differentiate between competing structural hypotheses. Here, in this work, we address this challenge by integrating multimodal ab initio simulations, experimental data acquisition, and ML techniques for structure characterization. Our goal is to determine local structures and properties using EELS and XAS data from multiple elements and edges. To showcase our approach, we use various lithium nickel manganese cobalt (NMC) oxide compounds which are used for lithium ion batteries, including those with oxygen vacancies and antisite defects, as the sample material system. We successfully inferred local element content, ranging from lithium to transition metals, with quantitative agreement with experimental data. Beyond local element inference, we find that ML model based on multimodal spectroscopic data is able to determine whether local defects such as oxygen vacancy and antisites are present, a task which is impossible for single mode spectra or other experimental techniques. Furthermore, our framework is able to provide physical interpretability, bridging spectroscopy with the local atomic and electronic structures.

battery↗

Mondo: integrating disease terminology across communities

Precision medicine aims to enhance diagnosis, treatment, and prognosis by integrating multimodal data at the point of care. However, challenges arise due to the vast number of diseases, differing methods of classification, and conflicting terminological coding systems and practices used to represent molecular definitions of disease. This lack of interoperability artificially constrains the potential for diagnosis, clinical decision support, care outcome analysis, as well as data linkage across research domains to support the development or repurposing of therapeutics. There is a clear and pressing need for a unified system for managing disease entities⁠—including identifiers, synonyms, and definitions. To address these issues, we created the Mondo disease ontology—a community-driven, open-source, unified disease classification system that harmonizes diverse terminologies into a consistent, computable framework. Mondo integrates key medical and biomedical terminologies, including Online Mendelian Inheritance in Man (OMIM), Orphanet, Medical Subject Headings (MeSH), National Cancer Institute Thesaurus (NCIt), and more, to provide a comprehensive and accurate representation of disease concepts with fully provenanced and attributed links back to the sources. Mondo can be used as the handle for curation of gene–disease associations utilized in diagnostic applications, research applications such as computational phenotyping, and in clinical coding systems in clinical decision support by pointing the clinician to the numerous knowledge resources linked to the Mondo identifier. Mondo's community-centric approach, stewarded by the Monarch Initiative's expertise in ontologies, ensures that the ontology remains adaptable to the evolving needs of biomedical research and clinical communities, as well as the knowledge providers.

biomedical informatics↗

Artificial intelligence to unlock real-world evidence in clinical oncology: A primer on recent advances

Purpose: Real world evidence is crucial to understanding the diffusion of new oncologic therapies, monitoring cancer outcomes, and detecting unexpected toxicities. In practice, real world evidence is challenging to collect rapidly and comprehensively, often requiring expensive and time-consuming manual case-finding and annotation of clinical text. In this Review, we summarise recent developments in the use of artificial intelligence to collect and analyze real world evidence in oncology. Methods: We performed a narrative review of the major current trends and recent literature in artificial intelligence applications in oncology. Results: Artificial intelligence (AI) approaches are increasingly used to efficiently phenotype patients and tumors at large scale. These tools also may provide novel biological insights and improve risk prediction through multimodal integration of radiographic, pathological, and genomic datasets. Custom language processing pipelines and large language models hold great promise for clinical prediction and phenotyping. Conclusions: Despite rapid advances, continued progress in computation, generalizability, interpretability, and reliability as well as prospective validation are needed to integrate AI approaches into routine clinical care and real-time monitoring of novel therapies.

60 APPLIED LIFE SCIENCES↗

Scalable algorithms for physics-informed neural and graph networks

Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for which conventional methods may fail. Unlike commercial machine learning where training of deep neural networks requires big data, in PIML big data are not available. Instead, we can train such networks from additional information obtained by employing the physical laws and evaluating them at random points in the space–time domain. Such PIML integrates multimodality and multifidelity data with mathematical models, and implements them using neural networks or graph networks. Here, we review some of the prevailing trends in embedding physics into machine learning, using physics-informed neural networks (PINNs) based primarily on feed-forward neural networks and automatic differentiation. For more complex systems or systems of systems and unstructured data, graph neural networks (GNNs) present some distinct advantages, and here we review how physics-informed learning can be accomplished with GNNs based on graph exterior calculus to construct differential operators; we refer to these architectures as physics-informed graph networks (PIGNs). We present representative examples for both forward and inverse problems and discuss what advances are needed to scale up PINNs, PIGNs and more broadly GNNs for large-scale engineering problems.

42 ENGINEERING↗

Enhanced prediction of terrestrial feedbacks to the coastal carbon cycle:using machine learning to improve sub-grid biogeochemical processes

Focal Areas: The paper aims to improve predictability of the coastal carbon cycle through improved model representation and quantification of the terrestrial feedbacks to aquatic ecosystems. The paper covers two focus areas, 1. Improved predictive modeling of terrestrial and aquatic biogeochemistry through ML-enabled surrogate models, and 2. Employing machine learning to integrate multimodal data sets collected across terrestrial and aquatic ecosystems at high spatial and temporal resolution.

54 ENVIRONMENTAL SCIENCES↗

New computer system for aircraft noise prediction

The purpose of the Aircraft Noise Prediction Office (ANOPO) at Langley Research Center is to provide a focal point for NASA's aircraft noise prediction activities and an appropriate interface with other agencies and industry. An interim prediction system is now in operation, which includes a program for source modeling leading to noise prediction for a single event, two complementary multiple event prediction programs which predict NEF contours on the basis of interpolation of noise, thrust, and altitude data, and the FAA data base for the commercial fleet. An integrated multimode aircraft noise prediction program has been designed, and a mechanism for continuously providing current prediction technology to be incorporated in the program has been implemented.

Raney, J. P.↗

Supersonic cruise vehicle research/business jet

A comparison study of a GE-21 variable propulsion system with a Multimode Integrated Propulsion System (MMIPS) was conducted while installed in small M = 2.7 supersonic cruise vehicles with military and business jet possibilities. The 1984 state of the art vehicles were sized to the same transatlantic range, takeoff distance, and sideline noise. The results indicate the MMIPS would result in a heavier vehicle with better subsonic cruise performance. The MMIPS arrangement with one fan engine and two satellite turbojet engines would not be appropriate for a small supersonic business jet because of design integration penalties and lack of redundancy.

Kelly, R. J.↗

A US History of Airbreathing/Rocket Combined-Cycle (RBCC) Propulsion for Powering Future Aerospace Transports, with a Look Ahead to the Year 2020

A technohistorical and forward-planning overview of U.S. developments in combined airbreathing/rocket propulsion for advanced aerospace vehicle applications is presented. Such system approaches fall into one of two categories: (1) Combination propulsion systems (separate, non-interacting engines installed), and (2) Combined-Cycle systems. The latter, and main subject, comprises a large family of closely integrated engine types, made up of both airbreathing and rocket derived subsystem hardware. A single vehicle-integrated, multimode engine results, one capable of operating efficiently over a very wide speed and altitude range, atmospherically and in space. While numerous combination propulsion systems have reached operational flight service, combined-cycle propulsion development, initiated ca. 1960, remains at the subscale ground-test engine level of development. However, going beyond combination systems, combined-cycle propulsion potentially offers a compelling set of new and unique capabilities. These capabilities are seen as enabling ones for the evolution of Spaceliner class aerospace transportation systems. The following combined-cycle hypersonic engine developments are reviewed: (1) RENE (rocket engine nozzle ejector), (2) Cryojet and LACE, (3) Ejector Ramjet and its derivatives, (4) the seminal NASA NAS7-377 study, (5) Air Force/Marquardt Hypersonic Ramjet, (6) Air Force/Lockheed-Marquardt Incremental Scramjet flight-test project, (7) NASA/Garrett Hypersonic Research Engine (HRE), (8) National Aero-Space Plane (NASP), (9) all past projects; and such current and planned efforts as (10) the NASA ASTP-ART RBCC project, (11) joint CIAM/NASA DNSCRAM flight test,(12) Hyper-X, (13) Trailblazer,( 14) W-Vehicle and (15) Spaceliner 100. Forward planning programmatic incentives, and the estimated timing for an operational Spaceliner powered by combined-cycle engines are discussed.

Escher, William J. D.↗

CAMFeND: Credibility-Aware Multimodal Fake News Detection with Rotational Attention

In the evolving digital landscape, fake news is a significant challenge, influencing public perception and decision-making. Traditional detection approaches focus on single-modal data or simple multimodal fusion, often overlooking deeper interactions and news credibility. We propose a novel model addressing these limitations by introducing rotational attention and news domain information as a feature. Unlike static attention mechanisms, our rotational attention dynamically shifts query, key, and value roles across text and image inputs, enabling richer cross-modal interaction. Incorporating news domain information further enhances the model’s reliability by associating news posts with top domains extracted from Google search results, reducing false detections. This approach assesses both the content and the broader web context in which the news is discussed. Our model outperforms existing state-of-the-art methods by providing deeper, layered multimodal integration and domain information analysis, resulting in a more robust and adaptive fake news detection system.

Gupta, Nidhi↗

A Dynamic Hierarchical Attention Framework for Multimodal Malware Detection

The increasing use of Android in the worldwide mobile ecosystem has come along with a significant increase in advanced malware, highlighting the critical necessity for efficient, scalable, and adaptable detection systems. Despite recent advancements in machine learning improving malware detection, the majority of current solutions are limited to one, two, or three data modalities, hence neglecting the comprehensive behavioral spectrum of contemporary multi-vector threats. This thesis presents the first comprehensive multimodal framework for Android malware detection, which combines textual, time-series (temporal), graph-based (structural), and visual information using an innovative hierarchical attention mechanism and Dynamic Fusion Controller (DFC). Our methodology consistently classifies and processes modalities as either sequential or structural, facilitating content-adaptive weighting and resilient cross-modal representation learning. We advance the implementation of cutting-edge time series techniques, such as MiniRocket, for malware detection, hence creating new opportunities for temporal analysis in cybersecurity. Comprehensive experimental assessment shows that our framework performs exceptionally well, with 99.46% classification accuracy and 97.15% detection accuracy, significantly outperforming existing approaches through effective multimodal integration and hierarchical attention mechanisms.

Nazmin, Tamanna↗

miss-SNF: a multimodal patient similarity network integration approach to handle completely missing data sources

Abstract Motivation Precision medicine leverages patient-specific multimodal data to improve prevention, diagnosis, prognosis, and treatment of diseases. Advancing precision medicine requires the non-trivial integration of complex, heterogeneous, and potentially high-dimensional data sources, such as multi-omics and clinical data. In the literature, several approaches have been proposed to manage missing data, but are usually limited to the recovery of subsets of features for a subset of patients. A largely overlooked problem is the integration of multiple sources of data when one or more of them are completely missing for a subset of patients, a relatively common condition in clinical practice. Results We propose miss-Similarity Network Fusion (miss-SNF), a novel general-purpose data integration approach designed to manage completely missing data in the context of patient similarity networks. miss-SNF integrates incomplete unimodal patient similarity networks by leveraging a non-linear message-passing strategy borrowed from the SNF algorithm. miss-SNF is able to recover missing patient similarities and is “task agnostic”, in the sense that can integrate partial data for both unsupervised and supervised prediction tasks. Experimental analyses on nine cancer datasets from The Cancer Genome Atlas (TCGA) demonstrate that miss-SNF achieves state-of-the-art results in recovering similarities and in identifying patients subgroups enriched in clinically relevant variables and having differential survival. Moreover, amputation experiments show that miss-SNF supervised prediction of cancer clinical outcomes and Alzheimer’s disease diagnosis with completely missing data achieves results comparable to those obtained when all the data are available. Availability and implementation miss-SNF code, implemented in R, is available at https://github.com/AnacletoLAB/missSNF.

Biochemistry & Molecular Biology↗

A 20-dB quasi-integrated horn antenna

A multimode quasi-integrated dipole-fed horn antenna is presented with a performance comparable to that of waveguide-fed corrugated horn antennas. The antenna has been designed using fullwave analysis and has been fabricated and tested at 91 GHz. The horn has a gain of 20 dB with very symmetric patterns, a Gaussian coupling efficiency of 97 percent, and a cross-polarization level of -22.7 dB. The antenna provides a significant improvement in integrated antenna designs and is suitable for millimeter-wave communication and radar systems and as a Gaussian-beam launcher in quasi-optical receiver systems.

Eleftheriades, George V.↗

Dara: Automated Multiple-Hypothesis Phase Identification and Refinement from Powder X-ray Diffraction

Powder X-ray diffraction (XRD) is a foundational technique for characterizing crystalline materials. However, the reliable interpretation of XRD patterns, particularly in multiphase systems, remains a manual and expertise-demanding task. As a characterization method that only provides structural information, multiple reference phases can often be fit to a single pattern, leading to potential misinterpretation when alternative solutions are overlooked. To ease humans’ efforts and address the challenge, we introduce Dara (data-driven automated Rietveld analysis), a framework designed to automate the robust identification and refinement of multiple phases from powder XRD data. Dara performs an exhaustive tree search over all plausible phase combinations within a given chemical space and validates each hypothesis using the BGMN Rietveld refinement routine. Key features include structural database filtering, automatic clustering of isostructural phases during tree expansion, and peak-matching-based scoring to identify promising phases for refinement. When ambiguity exists, Dara generates multiple hypothesis which can then be decided between by human experts or with further characterization tools. By enhancing the reliability and accuracy of phase identification, Dara enables scalable analysis of realistic complex XRD patterns and provides a foundation for integration into multimodal characterization workflows, moving toward fully self-driving materials discovery.

Biological databases↗

Structure and Sulfur: Tuning the Viscoelastic and Surface Properties of Natural Keratin Fibers

Natural keratin fibers, such as wool, possess a complex hierarchical structure that governs their mechanical properties and surface energy. However, the extent to which these characteristics are influenced by combined contributions of structural variations (e.g., fiber diameter, intermediate filament (IF) packing) and chemical composition (e.g., disulfide bond density) remains poorly understood. In this study, we investigate wool fibers from five sheep breeds (Merino, Polwarth, Cheviot, Eider, and Devon) to elucidate how these factors influence viscoelasticity and surface interactions. Using a multimodal approach integrating interfacial and bulk characterization methods, including inverse gas chromatography (IGC), atomic force microscopy-infrared spectroscopy (AFM-IR), X-ray photoelectron spectroscopy (XPS), uniaxial tensile testing, and synchrotron small-angle X-ray scattering (SAXS), we show that the nanometer-thick 18-methyleicosanoic acid (18-MEA) layer is consistently present across all wool types and plays a key role in governing hydrophobicity and surface heterogeneity. A controlled isothermal treatment at 200 °C, designed to cleave disulfide bonds, results in a nearly 40% reduction in specific surface area across all fiber types, accompanied by a significant decrease in tensile strength and 80% reduction in elongation at break for Merino and Devon wool, but limited influence on the mechanical properties of Eider fibers. Furthermore, rate-dependent tensile testing within the elastic regime reveals distinct viscoelastic responses among the fiber types, suggesting that the sulfur-rich protein matrix surrounding IFs and its structure contribute actively to stress partitioning. Altogether, when combined with conclusions from SAXS measurements of IF spacing, our work offers compelling insights into the role of the keratin-associated protein (KAP) matrix in shaping wool fiber mechanics. Differences in mechanical behavior among wool types, despite similar IF spacing or sulfur content, highlight the importance of matrix composition and cross-linking density, suggesting that the molecular architecture of the KAP network may be a dominant factor in determining fiber performance.

X-ray scattering↗

Prediction of carbon nanostructure mechanical properties and the role of defects using machine learning

Graphene-based nanostructures hold immense potential as strong and lightweight materials, however, their mechanical properties such as modulus and strength are difficult to fully exploit due to challenges in atomic-scale engineering. This study presents a database of over 2,000 pristine and defective nanoscale CNT bundles and other graphitic assemblies, inspired by microscopy, with associated stress–strain curves from reactive molecular dynamics (MD) simulations using the reactive INTERFACE force field (IFF-R). These 3D structures, containing up to 80,000 atoms, enable detailed analyses of structure-stiffness-failure relationships. By leveraging the database and physics- and chemistry-informed machine learning (ML), accurate predictions of elastic moduli and tensile strength are demonstrated at speeds 1,000 to 10,000 times faster than efficient MD simulations. Hierarchical Graph Neural Networks with Spatial Information (HS-GNNs) are introduced, which integrate chemistry knowledge. HS-GNNs as well as extreme gradient boosted trees (XGBoost) achieve forecasts of mechanical properties of arbitrary carbon nanostructures with only 3 to 6% mean relative error. The reliability equals experimental accuracy and is up to 20 times higher than other ML methods. Predictions maintain 8 to 18% accuracy for large CNT bundles, CNT junctions, and carbon fiber cross-sections outside the training distribution. The physics- and chemistry-informed HS-GNN works remarkably well for data outside the training range while XGBoost works well with limited training data inside the training range. The carbon nanostructure database is designed for integration with multimodal experimental and simulation data, scalable beyond 100 nm size, and extendable to chemically similar compounds and broader property ranges. The ML approaches have potential for applications in structural materials, nanoelectronics, and carbon-based catalysts.

Winetrout, Jordan J.↗