SEARCH · Search NASA

Results for “word embeddings”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2021

An Introduction to Word Embeddings and Language Models

Language models have advanced at a phenomenal pace over the past decade. This document provides a short introduction to terminology, word embeddings (aka low-dimensional representations), and popular large-scale language models (LMs). Word embeddings are used to represent words as numerical vectors and are context-independent, meaning a word can only have a single representation (e.g., club can only be club sandwich, not golf club ). Language models can determine the probability of a given sequence of words occurring in a sentence and can provide context to distinguish between words and phrases that sound similar. LMs are context-dependent (e.g., club can be club sandwich or golf club ) and largely fall in two main classes – autoregressive and autoencoding models. Autoregressive models are pretrained on the classic language modeling task: guess the next token having read all the previous ones. Those models can be fine-tuned and achieve great results on many tasks, the most natural application is text generation. A typical example of such models is GPT, but others include GPT-2, GPT-3, CTLR, TRANSFORMER-XL, REFORMER, XLNET. Autoencoding models are pretrained by corrupting the input tokens in some way and trying to reconstruct the original sentence. They can be fine-tuned and achieve great results on many tasks such as text generation, but their most natural application is sentence classification or token classification. A typical example of such models is BERT, but others include ROBERTA, ALBERT, XML, XML-ROBERTA, FLAUBERT AND LONGFORMER.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Understanding Event Trajectories Across Massive Temporal Datasets with Word Embeddings and Visualization

In collaboration with researchers from Virginia Tech, Savannah River National Laboratory has continued development of a natural language processing pipeline to identify and extract events of interest from massive open data sources in the domain of worldwide state-sponsored civil nuclear energy. The foundation of the pipeline is built on compass aligned temporal word embedding models, whereby contextual shifts are automatically identified by comparing keyword embedding vectors across successive time windows. Within the approach, a contextual shift indicates the occurrence of a potential event of interest. However, in such a broad topical domain that captures events at a global scale, across various life cycle stages, and across numerous different technology types, a user that is monitoring events may have broad interests in capturing many different event types with varying degrees of signal. As such, the quantity of information that may be returned from an automated event extraction pipeline can be substantial, requiring manual effort to sift through the information to identify any relevant bits of information. Therefore, a more streamlined workflow that aids in directing a user toward specific information at different points in time is necessary. The workflow presented here has been developed with this concept in mind, built on top of the initial prototype event extraction pipeline, whereby a user can analyze temporal text-based data sources at multiple different contextual levels to isolate key points in time and key subdomains captured within a data corpus. Using multiple corpuses that consist of approximately 7 million Tweets and 7 million news articles, the team has extended compass aligned temporal word embedding models to establish an interconnected and hierarchical structure that relates known key words of interest to documents, local topics (i.e., within a time window), and global topics across the corpuses. All of this information is packaged into a visual analytics system that is linked to the information extraction pipeline and enables a user to identify contextual information that describes the evolution of a high dimensional embedding space across time to isolate changes of interest and explore associated events. This report demonstrates the use of these analytics and a means to fuse information across multiple datasets.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Machine learning with persistent homology and chemical word embeddings improves prediction accuracy and interpretability in metal-organic frameworks

Machine learning has emerged as a powerful approach in materials discovery. Its major challenge is selecting features that create interpretable representations of materials, useful across multiple prediction tasks. We introduce an end-to-end machine learning model that automatically generates descriptors that capture a complex representation of a material’s structure and chemistry. This approach builds on computational topology techniques (namely, persistent homology) and word embeddings from natural language processing. It automatically encapsulates geometric and chemical information directly from the material system. We demonstrate our approach on multiple nanoporous metal–organic framework datasets by predicting methane and carbon dioxide adsorption across different conditions. Our results show considerable improvement in both accuracy and transferability across targets compared to models constructed from the commonly-used, manually-curated features, consistently achieving an average 25–30% decrease in root-mean-squared-deviation and an average increase of 40–50% in R 2 scores. A key advantage of our approach is interpretability: Our model identifies the pores that correlate best to adsorption at different pressures, which contributes to understanding atomic-level structure–property relationships for materials design.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Explaining word embeddings with perfect fidelity: a case study in predicting research impact

The best-performing approaches for scholarly document quality prediction are based on embedding models. In addition to their performance when used in classifiers, embedding models can also provide predictions even for words that were not contained in the labelled training data for the classification model, which is important in the context of the ever-evolving research terminology. Although model-agnostic explanation methods, such as Local interpretable model-agnostic explanations, can be applied to explain machine learning classifiers trained on embedding models, these produce results with questionable correspondence to the model. We introduce a new feature importance method, Self-Model Entities Rated (SMER), for logistic regression-based classification models trained on word embeddings. We show that SMER has theoretically perfect fidelity with the explained model, as the average of logits of SMER scores for individual words (SMER explanation) exactly corresponds to the logit of the prediction of the explained model. Quantitative and qualitative evaluation is performed through five diverse experiments conducted on 50,000 research articles (papers) from the CORD-19 corpus. In conclusion, through an AOPC curve analysis, we experimentally demonstrate that SMER produces better explanations than LIME, SHAP and global tree surrogates.

Coarse-grained models↗

DOE OSTI2021

Supervised learning with word embeddings derived from PubMed captures latent knowledge about protein kinases and cancer

Abstract Inhibiting protein kinases (PKs) that cause cancers has been an important topic in cancer therapy for years. So far, almost 8% of >530 PKs have been targeted by FDA-approved medications, and around 150 protein kinase inhibitors (PKIs) have been tested in clinical trials. We present an approach based on natural language processing and machine learning to investigate the relations between PKs and cancers, predicting PKs whose inhibition would be efficacious to treat a certain cancer. Our approach represents PKs and cancers as semantically meaningful 100-dimensional vectors based on word and concept neighborhoods in PubMed abstracts. We use information about phase I-IV trials in ClinicalTrials.gov to construct a training set for random forest classification. Our results with historical data show that associations between PKs and specific cancers can be predicted years in advance with good accuracy. Our tool can be used to predict the relevance of inhibiting PKs for specific cancers and to support the design of well-focused clinical trials to discover novel PKIs for cancer therapy.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2022

Teaching AI when to care about gender

Natural Language Processing (NLP) is a branch of Artificial Intelligence (AI) concerned with solving language tasks by modeling large amounts of textual data. Some NLP techniques use word embeddings which are semantic models where machine learning (ML) is used to learn to cluster semantically related words by learning about word co-occurrences in the original training text. Unfortunately, these models tend to reflect or even exaggerate biases that are present in the training corpus. Here we describe the Word Embedding Navigator (WEN), which is a tool for exploring word embedding models. We examine a specific potential use case for this tool: interactive discovery and neutralization of gender bias in word embedding models, and compare this human-in-the-loop approach to reducing bias in word embeddings with a debiasing post-processing technique.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Multimodal Data Representation with Deep Learning for Extracting Cancer Characteristics from Clinical Text

This paper presents a multimodal data representation to improve the performance of deep learning models for extracting cancer key characteristics from unstructured text in pathology reports. Specifically, in addition to using the text as the input to deep learning models, we use concept unique identifiers (CUIs) as another source of information to the models. We analyze the performance of different text and CUI data representations, including word embeddings and bag of embeddings (BOE), with a convolutional neural network (CNN) and a fully connected multilayer perceptron neural network (MLP-NN). The high level document embeddings from text and CUI inputs are combined by concatenating them and then applying a classifier. The model is used for extracting cancer subsite and histology from pathology reports. These two classification tasks have a large number of labels, i.e. 317 for subsite and 556 for histology, with extreme class imbalance. We compare the performance of the developed DL models across the two tasks based on micro- and macro-F1 scores. The evaluation shows that a multi-channel DL model that utilizes text represented by word embeddings and CUIs represented by BOE outperforms other DL models. Also, this approach significantly improves the model performance on low prevalence classes.

Alawad, Mohammed↗

DOE OSTI2026

Evolving language of pediatric anxiety in electronic health records

Objectives This study aimed to identify and quantify semantic drift (ie, the change in semantic meaning over time) within expert-defined anxiety-related (AR) terminology and compare it to common electronic health record (EHR) vocabulary across longitudinal pediatric clinical notes. Materials and Methods A corpus of pediatric clinical notes from 2009 to 2022 was analyzed using computational methods. Semantic drift for each term was quantified using cosine similarity between annual temporal word embeddings. Contextual meaning was examined through changes in nearest neighbors across years. The Laws of Semantic Change were applied to assess the influence of word frequency and polysemy. Vocabulary terms were categorized as AR or common EHR. Results 98% of AR terminology maintained a cosine similarity between 0.00 and 0.50, indicating moderate semantic stability, whereas 90% of common EHR terms remained between 0.00 and 0.25, showing greater contextual stability overall. Frequent terms exhibited minimal change (Frequency Coefficient = 0.04), whereas highly polysemous or abbreviated terms showed less stability (Polysemy Coefficient = 0.630). AR terminology drifted more slowly than general EHR vocabulary (Type Coefficient = −0.179), further supported by significant year–type interactions (Coef = −0.09 to −0.523). Discussion Although anxiety-related terminology demonstrates slower semantic drift than general EHR vocabulary, subtle contextual shifts still occur that may affect downstream interpretability and retrieval in automated systems. Conclusion Continuous linguistic monitoring and adaptive modeling are essential to maintain semantic fidelity and ensure the long-term reliability of clinical decision support systems as healthcare documentation evolves.

Pediatric anxiety disorders↗

DOE OSTI2022

Hierarchical Conceptual Model Event and Activity Domains for Forecasting State-Sponsored Civil Nuclear Power Activities

In FY20, the Savannah River National Laboratory and the Sanghani Center for Artificial Intelligence and Data Analytics (Virginia Tech) entered a collaboration funded by the Department of Energy’s Office of Defense Nuclear Nonproliferation Research and Development to develop a machine learning based modeling pipeline to extract proliferation events of interest from open data sources. The prototype modeling pipeline that was developed relies on the use of time dependent word embedding models to identify contextual shifts in key words and phrases that act as indicators of events of interest. The FY20- 21 efforts were focused on a narrow topical domain of forecasting “fissile core fabrication” at the Savannah River Site prior to its official announcement in 2018. In FY22, the research team was funded to continue development of the modeling pipeline by applying it to the problem of forecasting new, and/or significant changes to existing, civil nuclear power reactors around the world. In this effort, the development will focus on proving applicability to a broader topical domain and in data environments that may contain more sparse information, relative to the United States. In this report, the worldwide landscape of civil nuclear reactors is outlined, the timeline of interest is defined, and a hierarchical conceptual model is established to identify the activity domains of interest and the event domains of interest. This preliminary effort will guide the curation of a glossary of key terms for data acquisition, as well as the downstream modeling efforts.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2020

A Computational Theory for the Emergence of Grammatical Categories in Cortical Dynamics

A general agreement in psycholinguistics claims that syntax and meaning are unified precisely and very quickly during online sentence processing. Although several theories have advanced arguments regarding the neurocomputational bases of this phenomenon, we argue that these theories could potentially benefit by including neurophysiological data concerning cortical dynamics constraints in brain tissue. In addition, some theories promote the integration of complex optimization methods in neural tissue. In this paper we attempt to fill these gaps introducing a computational model inspired in the dynamics of cortical tissue. In our modeling approach, proximal afferent dendrites produce stochastic cellular activations, while distal dendritic branches–on the other hand–contribute independently to somatic depolarization by means of dendritic spikes, and finally, prediction failures produce massive firing events preventing formation of sparse distributed representations. The model presented in this paper combines semantic and coarse-grained syntactic constraints for each word in a sentence context until grammatically related word function discrimination emerges spontaneously by the sole correlation of lexical information from different sources without applying complex optimization methods. By means of support vector machine techniques, we show that the sparse activation features returned by our approach are well suited—bootstrapping from the features returned by Word Embedding mechanisms—to accomplish grammatical function classification of individual words in a sentence. In this way we develop a biologically guided computational explanation for linguistically relevant unification processes in cortex which connects psycholinguistics to neurobiological accounts of language. We also claim that the computational hypotheses established in this research could foster future work on biologically-inspired learning algorithms for natural language processing applications.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

REGULATORY REQUIREMENTS, GUIDANCE, AND REVIEW OF EMBEDDED DIGITAL DEVICES IN SAFETY SYSTEMS

Interest in embedded digital devices (EDDs) for monitoring and providing control of components is increasing in the nuclear power market as more and more users both within and outside the nuclear sector are employing EDDs. EDDs in other industries or countries may be known by the related terms as smart devices, devices of limited functionality, or intelligent devices. EDDs can sometimes perform a safety role when embedded in components such as temperature transmitters, pressure transmitters, voltage regulators, gas analyzers, boiler controllers, relays, and radiation monitors. The introduction of EDDs can affect safety by creating new hazards, vulnerabilities, failure modes, triggering mechanisms, and other potential safety concerns at both the component and system levels. The U.S. Nuclear Regulatory Commission’s (NRC’s) existing regulations and guidance cover the use of EDDs in safety-related and important to safety systems in nuclear power plants (NPPs). In addition to potential future challenges, the nature of EDDs being implemented in similar or identical manners between different plants and industries opens up the potential of leveraging the reviews of other regulators or utilization of industry certifications.Key words: embedded digital device, firmware

Halverson, D. S.↗

DOE OSTI2021

OPERATING EXPERIENCE AND ITS RELATIONSHIP TO EMBEDDED DIGITAL DEVICES IN SAFETY SYSTEMS

Interest in embedded digital devices (EDDs) for monitoring and providing control of components is increasing and the added functionality digital brings such as diagnostics, displays, control functions is welcomed in both the nuclear and other industries. As a result, pure analog-based components continue to disappear from the industrial I&C marketplace. Equipment with EDDs, also known by related terms such as smart devices, intelligent devices, and digital devices of limited functionality, can include sensors, breakers, priority logic modules, time delay relays, pumps, valve actuators, motor control centers, and uninterruptible power supplies. Not only is equipment consisting of analog and older digital technology being replaced with commercial grade products containing EDDs, but EDDs can also be retrofitted to work with existing components. EDDs may be installed physically within a component, such as in a relay. There are also external devices that some would not consider an EDD, but that operate in almost all technical ways identically to one internal to a component such as in a valve controller. In fact, the digital portions may be located hundreds of feet from the actual component.Due to the potential of being used in an identical manner and without alteration from their application in other industries, operating experience (OpE) for EDDs may be much more applicable than in the case of other more system level devices. Clear separation and independence of functionality provided with EDDs may also allow greater applicability of existing OpE from industry for changes that can be shown not to affect its safety function. However, OpE also demonstrates issues with different conditions between other industries and nuclear power plants, such as issues with undeclared changes or undeclared digital devices and the new failure modes they may introduce, issues with the reporting of OpE in industry, and a lack of transparency into OpE for EDDs.Understanding and addressing these issues will be key in the future use of EDDs in nuclear power plants, but failure rate data is difficult to obtain for EDDs, and lessons learned from the use of EDDs are difficult to collect and identify.Key words: embedded digital device, smart device, operating experience, failure modes

Muhlheim, Michael↗

DOE OSTI2025

An unsupervised machine learning based approach to identify efficient spin-orbit torque materials

Materials with large spin–orbit torque (SOT) hold considerable significance for many spintronic applications because of their potential for energy-efficient magnetization switching. Unfortunately, most of the existing materials exhibit an SOT efficiency factor that is much less than unity, requiring a large current for magnetization switching. The search for new materials that can exhibit an SOT efficiency much greater than unity is a topic of active research, and only a few such materials have been identified using conventional approaches. In this paper, we present a machine learning-based approach using a word embedding model that can identify new results by deciphering non-trivial correlations among various items in a specialized scientific text corpus. We show that such a model can be used to identify materials likely to exhibit high SOT and rank them according to their expected SOT strengths. The model captured the essential spintronics knowledge embedded in scientific abstracts within various materials science, physics, and engineering journals and identified 97 new materials to exhibit high SOT. Among them, 16 candidate materials are expected to exhibit an SOT efficiency greater than unity, and one of them has recently been confirmed with experiments with quantitative agreement with the model prediction.

Sayed, Shehrin↗

DOE OSTI2021

Advances in scientific literature mining for interpreting materials characterization

Abstract Using synchrotron light sources, such as the National Synchrotron Light Source II at Brookhaven National Laboratory, scientists in fields as diverse as physics, biology, and materials science, identify the atomic structure, chemical composition, or other important properties of varied specimens. x-ray spectroscopy from light sources is particularly valuable for materials research with vast information available about reference spectra in the scientific literature. However, as the technique is applicable to many science domains, searching for information about select x-ray spectroscopy spectra is impeded by the sheer number of publications. Moreover, useful information about the context of an experiment or figures presented in papers can be buried among the details, which takes time to assess. This work presents a scientific literature mining system that supports data acquisition, information extraction, and user interaction for referencing x-ray spectra identification and spectral interpretation. The goal is to provide efficient access to useful spectral data to researchers who may spend only a few days at a synchrotron light source. With this system, users browse a classification tree for papers arranged according to x-ray spectroscopic methods, chemical elements, and x-ray absorption spectroscopy edges. Relevant figures are extracted with sentences from the paper that explain them, known as ‘figure explanatory text.’ Notably, this system focuses on semantic aspects (logical analysis) to find figure explanatory text using deep contextualized word embeddings techniques and contains an interface to obtain labeled data from domain experts that is used to evaluate and improve the model.

Park, Gilchan (ORCID:0000000201536646)↗

DOE OSTI2023

Natural Language Processing for Text Based Event Extraction: Identifying Events of Interest Related to Worldwide State-Sponsored Civil Nuclear Power

Beginning in FY20, SRNL was funded by the National Nuclear Security Administration’s Office of Defense Nuclear Non-Proliferation Research and Development to develop a prototype natural language processing/natural language understating machine learning-based modeling and analysis pipeline to extract and forecast events of interest from massive open data sources. The working hypothesis within the approach is that contextual shifts in key words and phrases act as indicators of events of interest over time. Therefore, by identifying points in time where contextual shifts occur, events of interest can be extracted along with explicit and implicit connections of entities and activities. The development of the preliminary prototype pipeline proved successful, meriting further testing of the pipeline on more broad topical domains and in a worldwide data environment. Therefore, SRNL, in collaboration with the Sanghani Center for Artificial Intelligence and Data Analytics at Virginia Tech, have continued development with a test case of identifying events of interest related to worldwide state-sponsored civil nuclear power in open data sources. In the first year of this follow-on effort, the team has curated domain-specific data corpuses using an automated scheme and applied the modeling and analysis pipeline. This robust, focused, and efficient approach consists of an ensemble of analyses applied to time dependent word embedding models that are trained on the data corpuses. In this report, the team has demonstrated the capability of the existing pipeline (as development has continued in parallel) by exploring several specific case-studies centered around Rosatom’s international activities regarding the planning, construction, operation, and/or shutdown of nuclear reactors. A basic timeline events has been generated by manually cataloging known “milestone” events that have occurred at reactors in Turkey, Finland, Hungary, and Egypt and compared with the output of the modeling pipeline. In this approach, the team has characterized the lead time using the prototype pipeline, as well as the ability to capture relevant information, which proved 100% successful. A deep dive example of the Akkuyu reactor (Turkey) is presented that shows the breadth of information that can be captured using the approach. In this case study, events were extracted pertaining to the planning/construction of Akkuyu including protests from the population, information campaigns in response to the protests, forged regulatory documents and lawsuits, budgetary/shareholder information, geopolitical tensions, and the various construction milestones. This has demonstrated the pipeline’s utility as a research aid or real-time event extraction tool, where summary-level information and detailed text extractions from millions of articles or Tweets across long time periods can be generated with significantly less effort than current techniques.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2023

SULI Oral Presentation

Furthering our understanding of the prevalence and severity of issues that customers face when charging their electric vehicles (EVs) is crucial in order to improve the charging experience across the United States. This project utilizes web-scraping, machine leaning (ML), and natural language processing (NLP) techniques to analyze and categorize user-generated reviews. Selenium was used to build a data collection tool that can scrape vast amounts of user review data from the PlugShare website. Sentiment analysis was employed on this dataset in order to filter out negative reviews for further analysis. NLP techniques such as tokenization and word embedding were then used to convert user-written comments into a numerical format that a ML model can interpret. Multiple ML approaches are currently being explored in order to identify and categorize the charging issues being talked about in each review. Ultimately, the results from the ML model will be visualized and explained in a report on customer pain points to be delivered to the ChargeX Consortium, therefore revealing specific areas for improvement in the customer charging experience.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2023

SULI Oral Presentation

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2022

Unstructured clinical notes within the 24 hours since admission predict short, mid & long-term mortality in adult ICU patients

Mortality prediction for intensive care unit (ICU) patients is crucial for improving outcomes and efficient utilization of resources. Accessibility of electronic health records (EHR) has enabled data-driven predictive modeling using machine learning. However, very few studies rely solely on unstructured clinical notes from the EHR for mortality prediction. In this work, we propose a framework to predict short, mid, and long-term mortality in adult ICU patients using unstructured clinical notes from the MIMIC III database, natural language processing (NLP), and machine learning (ML) models. Depending on the statistical description of the patients’ length of stay, we define the short-term as 48-hour and 4-day period, the mid-term as 7-day and 10-day period, and the long-term as 15-day and 30-day period after admission. We found that by only using clinical notes within the 24 hours of admission, our framework can achieve a high area under the receiver operating characteristics (AU-ROC) score for short, mid and long-term mortality prediction tasks. The test AU-ROC scores are 0.87, 0.83, 0.83, 0.82, 0.82, and 0.82 for 48-hour, 4-day, 7-day, 10-day, 15-day, and 30-day period mortality prediction, respectively. We also provide a comparative study among three types of feature extraction techniques from NLP: frequency-based technique, fixed embedding-based technique, and dynamic embedding-based technique. Lastly, we provide an interpretation of the NLP-based predictive models using feature-importance scores.

60 APPLIED LIFE SCIENCES↗