Search NASA⌕ Search

SEARCH · Search NASA

Results for “embedding model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models

Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs. To better understand LLM scaling, we study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in smaller models. We observe a geometric phenomenon which we term embedding condensation, where token embeddings collapse into a narrow cone-like subspace in some language models. Through systematic analyses across multiple Transformer families, we show that small models such as GPT2 and Qwen3-0.6B exhibit severe condensation, whereas larger models such as GPT2-x1 and Qwen3-32B are more resistant to this phenomenon. Additional observations show that embedding condensation is not reliably mitigated by knowledge distillation from larger models. To fight against it, we formulate a dispersion loss that explicitly encourages embedding dispersion during training. Experiments demonstrate that it mitigates condensation, recovers dispersion patterns seen in larger models, and yields performance gains across 10 benchmarks. We believe this work offers a principled path toward improving smaller Transformers without additional parameters.

Xiao, Xi [ORNL] (ORCID:0009000009316982)↗
NASA NTRSDate not supplied

ES2Vec: Earth Science Metadata Suggestions and Analogical Reasoning

As the volume of text-based Earth science research grows, it is increasingly possible to discover latent relationships in the literature. However, traditional methodologies are restricted by limited computational capabilities and intractable problem spaces. Advancements in natural language processing (NLP) have allowed us to use an extensive Earth science corpus to create a domain-specific word vector model, Es2Vec, which we have used to surface latent relationships between Earth science concepts and generate improved keyword tags. Earth science metadata keyword assignment is a challenging problem. Dataset curators select appropriate keywords from the Global Change Master Directory (GCMD) set of keywords. The keywords an are integral part of the search and discovery of these datasets. Hence, the selection of keywords is crucial to increasing the discoverability of datasets. Utilizing machine learning techniques, we provide users with automated keyword suggestions to complement manual selection. We trained a machine learning model that leverages the semantic embedding ability of Word2Vec models to process abstracts and suggest relevant keywords. A user interface tool we built to assist data curators in the assignment of such keywords is also described.

word vectors↗
NASA NTRS2024

Accurate Assessment of Land-Atmosphere Coupling in Climate Models Requires High Frequency Data Output

Land-atmosphere (L-A) interactions are important for understanding convective processes, climate feedbacks, the development and perpetuation of droughts, heatwaves, pluvials, and other land-centred climate anomalies. Local L-A coupling (LoCo) metrics capture relevant L-A processes, highlighting the impact of soil and vegetation states on surface flux partitioning, and the impact of surface fluxes on boundary layer (BL) growth, development, and entrainment of air above the BL. A primary goal of the Climate Process Team on Coupling Land and Atmospheric Subgrid Parameterizations (CLASP) is parameterizing and characterizing the impact of subgrid heterogeneity in global and regional earth system models (ESMs) to improve the connection between land and atmospheric states and processes. A critical step in achieving that aim is the incorporation of L-A metrics, especially LoCo metrics, into climate model diagnostic process streams. However, because land-atmosphere interactions span time scales of minutes (e.g., turbulent fluxes), hours (e.g., BL growth and decay), days (e.g., soil moisture memory), and seasons (e.g., variability of behavioural regimes between soil moisture and latent heat flux), with multiple processes of interest happening in different geographic regions at different times of year, there is not a single metric that captures all the modes, means, and methods of interaction between the land and the atmosphere. And while monthly means of most of the LoCo-relevant variables are routinely saved from ESM simulations, data storage constraints typically preclude routine archival of the hourly data that would enable the calculation of all LoCo metrics. Here we outline a reasonable data request that would allow for adequate characterization of sub-daily coupling processes between the land and the atmosphere, preserving enough sub-daily output to describe, analyse, and better understand L-A coupling in modern climate models. A secondary request involves embedding calculations within the models to determine mean properties in and above the BL to further improve characterization of model behaviour. Higher-frequency model output will (i) allow for more direct comparison with observational field campaigns on process-relevant time scales, (ii) enable demonstration of inter-model spread in L-A coupling processes, and (iii) aid in targeted identification of sources of deficiencies and opportunities for improvement of the models.

Kirsten L. Findell↗
NASA NTRS1991

A model for the evaporation of clusters of drops embedded in jet vortices. I - Steady injection of identical clusters

A model is developed for describing the interaction of vortex-drop clusters in a flowing gaseous jet, convecting downstream from an injection location. Results are presented for a stationary case representing the situation when identical clusters are continuously injected and the injection rate is constant. The results indicate that, in a rich mixture high-drop-number density regime, the mass evaporated from the drops controls the velocity of the cluster-in-vortex as it propagates downstream.

Bellan, J.↗
NASA NTRS2020

Integrated Modeling and Development of Component-Based Embedded Software in Scala

Programming of embedded systems is challenging due to the low-level design patterns normally applied in the implementation of such systems. Furthermore, programming languages normally considered suitable for this level of programming, such as C and C++, are themselves low-level compared to more modern programming languages. We report on an effort exploring modeling and programming of embedded systems in modern high-level programming languages combining object-oriented and functional programming. We present an integration of four separate internal DSLs (libraries), considered useful for embedded program- ming, in the Scala programming language, for programming and testing component-based systems. These include a DSL for defining components and their connections, and a DSL for programming the individual components as hierarchical state machines. Two additional DSLs support testing, and include a DSL for writing temporal logic flavored test oracles for monitoring program executions, and a DSL for rule-based test input generation. The paper discusses the gap between Scala as used here and the needs for embedded systems programming.

Bocchino, Robert↗
NASA NTRS1997

Micromechanical Modeling of Woven Metal Matrix Composites

This report presents the results of an extensive micromechanical modeling effort for woven metal matrix composites. The model is employed to predict the mechanical response of 8-harness (8H) satin weave carbon/copper (C/Cu) composites. Experimental mechanical results for this novel high thermal conductivity material were recently reported by Bednarcyk et al. along with preliminary model results. The micromechanics model developed herein is based on an embedded approach. A micromechanics model for the local (micro-scale) behavior of the woven composite, the original method of cells (Aboudi), is embedded in a global (macro-scale) micromechanics model (the three-dimensional generalized method of cells (GMC-3D) (Aboudi). This approach allows representation of true repeating unit cells for woven metal matrix composites via GMC-3D, and representation of local effects, such as matrix plasticity, yarn porosity, and imperfect fiber-matrix bonding. In addition, the equations of GMC-3D were reformulated to significantly reduce the number of unknown quantities that characterize the deformation fields at the microlevel in order to make possible the analysis of actual microstructures of woven composites. The resulting micromechanical model (WCGMC) provides an intermediate level of geometric representation, versatility, and computational efficiency with respect to previous analytical and numerical models for woven composites, but surpasses all previous modeling work by allowing the mechanical response of a woven metal matrix composite, with an elastoplastic matrix, to be examined for the first time. WCGMC is employed to examine the effects of composite microstructure, porosity, residual stresses, and imperfect fiber-matrix bonding on the predicted mechanical response of 8H satin C/Cu. The previously reported experimental results are summarized, and the model predictions are compared to monotonic and cyclic tensile and shear test data. By considering appropriate levels of porosity, residual stresses, and imperfect fiber-matrix debonding, reasonably good qualitative and quantitative correlation is achieved between model and experiment.

Bednarcyk, Brett A.↗
NASA NTRS1990

Embedded CLIPS for SDI BM/C3 simulation and analysis

Nichols Research Corporation is developing the BM/C3 Requirements Analysis Tool (BRAT) for the U.S. Army Strategic Defense Command. BRAT uses embedded CLIPS/Ada to model the decision making processes used by the human commander of a defense system. Embedding CLlPS/Ada in BRAT allows the user to explore the role of the human in Command and Control (C2) and the use of expert systems for automated C2. BRAT models assert facts about the current state of the system, the simulated scenario, and threat information into CLIPS/Ada. A user-defined rule set describes the decision criteria for the commander. We have extended CLIPS/Ada with user-defined functions that allow the firing of a rule to invoke a system action such as weapons release or a change in strategy. The use of embedded CLIPS/Ada will provide a powerful modeling tool for our customer at minimal cost.

Gossage, Brett↗
NASA NTRS1988

Thermospheric dynamics during November 21-22, 1981 - Dynamics Explorer measurements and thermospheric general circulation model predictions

Time-dependent aurora and magnetospheric convection parameterizations have been derived from solar wind and aurora particle data for November 21-22, 1981, and are used to drive the auroral and magnetospheric convection models that are embedded in the National Center for Atmospheric Research thermospheric general circulation model (TGCM). Neutral wind speeds and transition boundaries between the midlatitude solar-driven circulation and the high-latitude magnetospheric convection-driven circulation are examined on an orbit-by-orbit basis. The results show that TGCM-calculated winds and reversal boundary locations are in generally good agreement with Dynamics Explorer 2 measurements for the orbits studied. This suggests that, at least for this particular period of relatively moderate geomagnetic activity, the TGCM parameterizations on the eveningside of the auroral oval and polar cap are adequate.

Roble, R. G.↗
NASA NTRS1993

Demonstration of the Dynamic Flowgraph Methodology using the Titan 2 Space Launch Vehicle Digital Flight Control System

Dynamic Flowgraph Methodology (DFM) is a new approach developed to integrate the modeling and analysis of the hardware and software components of an embedded system. The objective is to complement the traditional approaches which generally follow the philosophy of separating out the hardware and software portions of the assurance analysis. In this paper, the DFM approach is demonstrated using the Titan 2 Space Launch Vehicle Digital Flight Control System. The hardware and software portions of this embedded system are modeled in an integrated framework. In addition, the time dependent behavior and the switching logic can be captured by this DFM model. In the modeling process, it is found that constructing decision tables for software subroutines is very time consuming. A possible solution is suggested. This approach makes use of a well-known numerical method, the Newton-Raphson method, to solve the equations implemented in the subroutines in reverse. Convergence can be achieved in a few steps.

Yau, M.↗
NASA NTRS1999

Real-time Adaptive Control Using Neural Generalized Predictive Control

The objective of this paper is to demonstrate the feasibility of a Nonlinear Generalized Predictive Control algorithm by showing real-time adaptive control on a plant with relatively fast time-constants. Generalized Predictive Control has classically been used in process control where linear control laws were formulated for plants with relatively slow time-constants. The plant of interest for this paper is a magnetic levitation device that is nonlinear and open-loop unstable. In this application, the reference model of the plant is a neural network that has an embedded nominal linear model in the network weights. The control based on the linear model provides initial stability at the beginning of network training. In using a neural network the control laws are nonlinear and online adaptation of the model is possible to capture unmodeled or time-varying dynamics. Newton-Raphson is the minimization algorithm. Newton-Raphson requires the calculation of the Hessian, but even with this computational expense the low iteration rate make this a viable algorithm for real-time control.

Haley, Pam↗
NASA NTRS2012

Metroplex Optimization Model Expansion and Analysis: The Airline Fleet, Route, and Schedule Optimization Model (AFRS-OM)

This report describes the Airline Fleet, Route, and Schedule Optimization Model (AFRS-OM) that is designed to provide insights into airline decision-making with regards to markets served, schedule of flights on these markets, the type of aircraft assigned to each scheduled flight, load factors, airfares, and airline profits. The main inputs to the model are hedged fuel prices, airport capacity limits, and candidate markets. Embedded in the model are aircraft performance and associated cost factors, and willingness-to-pay (i.e. demand vs. airfare curves). Case studies demonstrate the application of the model for analysis of the effects of increased capacity and changes in operating costs (e.g. fuel prices). Although there are differences between airports (due to differences in the magnitude of travel demand and sensitivity to airfare), the system is more sensitive to changes in fuel prices than capacity. Further, the benefits of modernization in the form of increased capacity could be undermined by increases in hedged fuel prices

Sherry, Lance↗
NASA NTRS1989

Observations and numerical simulations of precipitation development is seeded clouds over the Sierra Nevada

The evolution of precipitation in seeded wintertime orographically induced convective and stratiform clouds with embedded convection were studied using in situ observations and particle growth and trajectory models. The particle growth model of Heymsfield embedded in a kinematic flow field representative of the Sierra barrier was used to study the ice particle growth by diffusion, accretion and subsequent fall trajectories. The particles observed by the aircraft were classified into habits. The growth of observed particles were compared with the model predicted evolution. Using the aggregation model of Heymsfield, the observation of formation of aggregates is less than 10 minutes was verified. The key findings of this study are presented.

Prasad, N.↗
DOE OSTI2025

Knowledge Oriented Graph Unified Transformer (KOGUT) v0.1

KOGUT — Knowledge Oriented Graph Unified Transformer KOGUT implements the Relational Graph Transformer (RelGT) architecture for knowledge graph link prediction in biological domains, with a primary focus on microbial growth media prediction. While the original RelGT (arXiv:2505.10960) targets relational tables, time series, and multi-table databases, KOGUT adapts this architecture for heterogeneous biological knowledge graphs, providing first-in-class AI predictive models for microbial cultivation. Key Adaptations Beyond Original RelGT: - Knowledge Graph Focus: Applied to biological KGs with semantic node types (taxa, chemicals, media, phenotypes, environments) versus generic relational database tables, trained on the KG-Microbe knowledge graph (1.3M entities, 2.9M edges, 24 relation types). - Multimodal Node Encoding: Integrates node labels, categories, descriptions, and synonyms from KG metadata through learned embedding layers—adapting relational column features to graph node attributes with textual semantics. - Extended K-Hop Subgraph Strategy: Optimized neighborhood sampling (3-hop default, configurable up to 200 nodes) tuned for sparse biological networks, building on the original local-global attention framework with biological relation preservation. - Biolink Predicate Preservation: Type-specific transformations for 24 biological edge semantics (occurs_in, consumes, produces, has_phenotype, subclass_of) beyond standard relational foreign keys, enabling multi-relation link prediction. - Inductive Learning Support: Enables zero-shot predictions for novel taxa through feature-based embeddings (temperature, oxygen requirements, gram stain, cell shape), extending the original transductive relational benchmark scope to uncultured microorganisms. CheapSOTA Performance Optimizations (This Distribution): - VQ-EMA Centroid Attention: Vector quantization with exponential moving average for improved global context modeling (+5-10% MRR improvement). - HDF5 Precomputed Data Loading: One-time preprocessing of k-hop subgraphs to eliminate redundant graph traversals (2-5× training speedup). - Distributed Data Parallel Training: Multi-GPU support for scaling to larger knowledge graphs (tested on 4× NVIDIA A100 GPUs at NERSC Perlmutter). - Mixed Precision Training: Automatic mixed precision (AMP) for memory efficiency and faster training. Advantages Over Standard Knowledge Graph Embedding Models: Combines RelGT's proven multi-element tokenization (features, type, hop, structure) with graph-native biological representations, enabling interpretable link prediction across heterogeneous entities that standard embedding models (TransE, RotatE, ComplEx) and table-based transformers cannot directly model. Achieves near-perfect performance on microbial growth media prediction (MRR: 0.9966, Precision@1: 0.9932, Hit@10: 1.0000) while maintaining explainability through attention-based reasoning over biological pathways. Training Data: - KG-Microbe merged knowledge graph: 1,379,337 nodes, 2,960,472 edges - 24 biological relation types including taxonomic hierarchies, metabolic interactions, phenotype associations, and environmental relationships - Primary prediction task: Growth media suitability for microbial taxa (biolink:occurs_in, 50K edges) - Multi-relation capability: Predicts links for any of the 24 relation types, including chemical consumption/production, phenotype associations, and taxonomic classification Citation: Original RelGT Architecture: Dwivedi et al., "Relational Graph Transformer", arXiv:2505.10960, 2025 KOGUT Implementation: Knowledge Oriented Graph Unified Transformer for Microbial Growth Media Prediction Developed at Lawrence Berkeley National Laboratory (LBNL) Trained on NERSC Perlmutter supercomputer

Joachimiak, Marcin [Lawrence Berkeley National Lab↗
DOE OSTI2026

Active learning of a crystal plasticity flow rule from discrete dislocation dynamics simulations

Continuum-scale material deformation models, such as crystal plasticity (CP), can significantly enhance their predictive accuracy by incorporating input from lower-scale (i.e. mesoscale) models. The procedure to generate and extract the relevant information is however typically complex and ad hoc, involving decision and intervention by domain experts, leading to long development times. In this study, we develop a principled approach for calibration of continuum-scale models using lower scale information by representing a CP flow rule as a Gaussian process model. This representation allows for efficient parameter space exploration, guided by the uncertainty embedded in the model through a process known as Bayesian optimization (BO). We demonstrate a semi-autonomous BO loop which instantiates discrete dislocation dynamics simulations whose initial conditions are automatically chosen to optimize the uncertainty of a model CP flow rule. Our self-guided computational pipeline efficiently generated a dataset and corresponding model whose error, uncertainty, and physical feature sensitivities were validated with comparison to an independent dataset four times larger, demonstrating a valuable and efficient active learning implementation readily transferable to similar material systems.

36 MATERIALS SCIENCE↗
NASA NTRS1991

PDEMOD: Software for control/structures optimization

Because of the possibility of adverse interaction between the control system and the structural dynamics of large, flexible spacecraft, great care must be taken to ensure stability and system performance. Because of the high cost of insertion of mass into low earth orbit, it is prudent to optimize the roles of structure and control systems simultaneously. Because of the difficulty and the computational burden in modeling and analyzing the control structure system dynamics, the total problem is often split and treated iteratively. It would aid design if the control structure system dynamics could be represented in a single system of equations. With the use of the software PDEMOD (Partial Differential Equation Model), it is now possible to optimize structure and control systems simultaneously. The distributed parameter modeling approach enables embedding the control system dynamics into the same equations for the structural dynamics model. By doing this, the current difficulties involved in model order reduction are avoided. The NASA Mini-MAST truss is used an an example for studying integrated control structure design.

Taylor, Lawrence W., Jr.↗
DOE OSTI2026

Modeling CO 2 flow through faulted/fractured reservoirs using tEDFM in corner-point grids

The interest in underground CO 2 storage has increased significantly over the last decade because of the rising concern about global warming due to the growing levels of greenhouse gases in the atmosphere. Considering that CO 2 accounts for 80% of these greenhouse gases, carbon capture, utilization, and storage (CCUS) is regarded as one of the most direct approaches to achieving the net zero carbon target. Although CO 2 storage in deep saline aquifers and depleted gas reservoirs has been studied extensively, most studies use commercial simulators that model faults/fractures by simply modifying the transmissibility in the direction perpendicular to the fault surfaces. Here, this work shows that this simplistic approach ignores the accelerated flow in the directions parallel to the fault plane, leading to significantly higher leakage along the fault surface. To accurately model the flow of CO 2 in faulted reservoirs, we present the first transient embedded discrete fracture model for corner-point grids (tEDFM-CPG). By comparing the results of the tEDFM-CPG to high-resolution reference solutions, we show that this approach is accurate and efficient at predicting CO 2 flow in faulted/fractured reservoirs. Finally, this work presents the use of mixed reality (MR) to efficiently observe CO 2 gas migration in the interior of these corner-point grid systems.

25 ENERGY STORAGE↗
DOE OSTI2024

OPER: Optimality-Guided Embedding Table Parallelization for Large-scale Recommendation Model

With the sharp increasing volume of user data, Deep Learning Recommendation Model (DLRM) becomes an indispensable infrastructure in large technology companies. However, large-scale DLRM on the multi-GPU platform is still inefficient due to unbalanced workload partitioning and intensive inter-GPU communication. To this end, we propose OPER, an OPtimality guided Embedding table placement for large-scale Recommendation model training and inference. OPER explores the potential of mitigating remote memory access latency in DLRM through fine-grained embedding table placement. Specifically, OPER proposes a theoretical modeling that builds up the relationship between EMT placement and the embedding communication latency in both training and inference. OPER proves the NP hardness of finding the optimal embedding table placement and proposes a heuristic algorithm that yields near optimal placement. OPER implements a SHMEM-based embedding table training system and a unified embedding index mapping to support fine-grained embedding table sharding and placement. Comprehensive experiments reveal that OPER achieves on average 3.4× and 5.1× speedup on training and inference respectively over state-of-the-art DLRM frameworks.

Wang, Zheng↗
NASA NTRS1994

Computer-aided operations engineering with integrated models of systems and operations

CONFIG 3 is a prototype software tool that supports integrated conceptual design evaluation from early in the product life cycle, by supporting isolated or integrated modeling, simulation, and analysis of the function, structure, behavior, failures and operation of system designs. Integration and reuse of models is supported in an object-oriented environment providing capabilities for graph analysis and discrete event simulation. Integration is supported among diverse modeling approaches (component view, configuration or flow path view, and procedure view) and diverse simulation and analysis approaches. Support is provided for integrated engineering in diverse design domains, including mechanical and electro-mechanical systems, distributed computer systems, and chemical processing and transport systems. CONFIG supports abstracted qualitative and symbolic modeling, for early conceptual design. System models are component structure models with operating modes, with embedded time-related behavior models. CONFIG supports failure modeling and modeling of state or configuration changes that result in dynamic changes in dependencies among components. Operations and procedure models are activity structure models that interact with system models. CONFIG is designed to support evaluation of system operability, diagnosability and fault tolerance, and analysis of the development of system effects of problems over time, including faults, failures, and procedural or environmental difficulties.

Malin, Jane T.↗