Search NASA⌕ Search

SEARCH · Search NASA

Results for “language models”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Predictive Indicators of the Performance of Large Language Models

In several mission contexts, it is desirable to estimate the performance of large language models (LLMs) on tasks that we cannot run directly. In light of published “scaling laws” our hypothesis is that some tasks should be consistently more challenging than others based on characteristics of the task. The goal of this project was to begin quantifying how much information about LLM performance can be gained from the features of a model and a task. Two of our statistical models struggled to converge. Pass/fail test results may provide limited information for inference beyond model quality and task difficulty, but we see no evidence at this time for significant feature interaction effect sizes, arguing for simple models. Future work extending the models to capitalize on perplexity of ground truth answers is suggested. This project also introduces “Depth of Knowledge Variant Testing” as a strategy for more finely assessing language models on open domain question and answer tasks. We developed sets of questions that ask a language model to produce similar information while demonstrating increasing depth of knowledge, and also relabeled existing Q&A test questions with their depth of knowledge. Our results suggest further consideration of Bloom’s taxonomy and further refinement of prompts to properly elicit information at varying depths. In the course of this work, we set up a basic infrastructure for standardizing tasks and testing many language models on these tasks. In addition to testing the predictive quality of model features and performance across test suites, with this project we have introduced two new task features to contextualize each test question: the Dewey Classification main category of information covered, and the Bloom’s taxonomy level that corresponds to the depth of knowledge probed by the question. Splits across these and other features produced over five hundred task subtypes with distinct feature vectors, which we tested on half a dozen models.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Genomic Language model for Annotation of Repetitive Elements (GLARE) v1.0

GLARE (Genomic Language model for Annotation of Repetitive Elements) is a tool that classifies transposable elements (TEs)—the mobile, repetitive DNA sequences that make up large fractions of eukaryotic genomes. GLARE fine-tunes the NTv3-650M genomic language model on a harmonized collection of curated TE sequences from the PanTEon and Repbase reference databases, assigning each input sequence to one of 11 orders and 32 superfamilies in a Wicker-compatible taxonomy. Features. From nucleotide FASTA input, GLARE outputs per-sequence predictions, class summaries, composition figures, and an annotated FASTA. It provides calibrated confidence scores with optional abstention and runs on CPU or GPU. Uses. GLARE serves as a classification component in genome-annotation pipelines, downstream of TE discovery, supporting genome annotation and comparative and evolutionary genomics. Advantages. GLARE is the first repeat-element classifier to leverage a pretrained genomic language model. Combined with multi-database training, this approach outperformed all nine classifiers in the PanTEon benchmark, generalized better to unseen taxonomic clades, and remained robust to sequence orientation—a common failure mode of existing tools.

Bruna, Tomas [Lawrence Berkeley National Laborator↗
DOE OSTI2026

SmileyLlama: modifying large language models for directed chemical space exploration

Here we show that large language models (LLMs) can be transformed via supervised fine-tuning of engineered prompts into SmileyLlama for exploring the chemical space of drug molecules. We benchmark SmileyLlama against pretrained LLMs and chemical language models trained from scratch for generating valid and novel drug-like molecules, and use direct preference optimization to both improve SmileyLlama’s adherence to a prompt and as part of the iMiner reinforcement learning framework to predict molecules with optimized three-dimensional conformations and high binding affinity to drug targets. By training an LLM to speak directly as a chemical language model, while retaining most of its natural language capabilities, we show that SmileyLlama can reliably generate molecules with user-specified properties rather than acting only as a chatbot with knowledge of chemistry or as a virtual assistant. While SmileyLlama is geared toward drug discovery, the supervised fine-tuning/direct preference optimization/LLM framework can be extended to other chemical, biological and materials applications.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRSDate not supplied

Defining A Modelling Language to Support Functional Hazard Assessment

Functional Hazard Assessment (FHA) is a key early-stage engineering process that supports the incorporation of safety in design by identifying the high-level functional hazards the system may encounter. While many FHA-like methodologies have been proposed in the design engineering literature, many of these methodologies have had difficulty becoming accepted industry practice. Industry standards, on the other hand, either provide too little recommendation on how to represent the function of the system to perform FHA, or rely on existing design artefacts which insufficiently support the goals of the process. This paper presents some of the problems with current modeling languages (both proposed and used) for FHA which limit the scope, expressiveness, flexibility, and precision of the analysis. It then outlines desirable principles an FHA-supporting analysis language should embody, and introduces the Functional Reasoning Design Language (FRDL), a formal modeling language for describing the functional elements of a system and their interactions, which aims to satisfy these principles. To demonstrate the use of this language, the modeling and hazard analysis of a disaster response drone is presented. While this case study is limited in scope, it highlights how FRDL can represent system function while reducing the ambiguity present in typical FHA-supporting functional modeling languages

Hazard Assessment↗
NASA NTRS2006

Adaptive Modeling Language and Its Derivatives

Adaptive Modeling Language (AML) is the underlying language of an object-oriented, multidisciplinary, knowledge-based engineering framework. AML offers an advanced modeling paradigm with an open architecture, enabling the automation of the entire product development cycle, integrating product configuration, design, analysis, visualization, production planning, inspection, and cost estimation.

Chemaly, Adel↗
NASA NTRSDate not supplied

BERT-E: An Earth Science Specific Language Model for Domain-Specific Downstream Tasks

Language models are fast approaching human-like understanding of natural language. They have been shown to perform equally, if not better than humans in a myriad of language tasks such as next sentence prediction, question answering, entity extraction etc. Part of the success of the models are owed to the fact that they have been trained on varied natural language text over the internet. By virtue of this, the models do not contain the semantic information present in Earth science literature. Hence, there is a lot of room for improvement when using these models for earth science specific tasks. In this work, we showcase our approach on developing Earth science specific language models. Furthermore, we justify the need for such a model by using the embeddings generated by the model to perform a domain specific downstream task that performs better than a generic model.

Prasanna Koirala↗
DOE OSTI2024

Comparative Study of Large Language Model Architectures on Frontier

Large language models (LLMs) have garnered significant attention in both the AI community and beyond. Among these, the Generative Pre-trained Transformer (GPT) has emerged as the dominant architecture, spawning numerous variants. However, these variants have undergone pre-training under diverse conditions, including variations in input data, data preprocessing, and training methodologies, resulting in a lack of controlled comparative studies. Here we meticulously examine two prominent open-sourced GPT architectures, GPT-NeoX and LLaMA, leveraging the computational power of Frontier, the world’s first Exascale supercomputer. Employing the same materials science text corpus and a comprehensive end-to-end pipeline, we conduct a comparative analysis of their training and downstream performance. Our efforts culminate in achieving state-of-the-art performance on a challenging materials science benchmark. Furthermore, we investigate the computation and energy efficiency, and propose a computationally efficient method for architecture design. To our knowledge, these pre-trained models represent the largest available for materials science. Our findings provide practical guidance for building LLMs on HPC platforms.

Yin, Junqi↗
DOE OSTI2025

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Chitty-Venkata, Krishna Teja↗
DOE OSTI2025

Demystifying the Resilience of Large Language Models: An End-to-End Perspective

Deep neural networks are known to be resilient to random bit-wise faults in their parameters. However, this resilience has primarily been established through evaluations of classification models. The extent to which this claim holds for large-language models remains underexplored. In this work, we conduct an extensive measurement study on the impact of random bitwise faults in commercial-scale language models. We perform an in-depth analysis of the resulting generation outputs. We first expose that these language models are not truly resilient to random bit-flips. While aggregate metrics such as accuracy may suggest resilience, an in-depth inspection of the generated outputs shows significant degradation in text quality. Our analysis also shows that tasks requiring more complex reasoning suffer more from performance and quality degradation. Moreover, we extend our analysis to models with augmented reasoning capabilities, such as Chain-of-Thought or Mixture of Experts architectures, and characterize their failure scenarios under random bit-flips.

Sun, Yu↗
NASA NTRSDate not supplied

Harnessing Large Language Models for Scientific Endeavors

The rapid proliferation of Large Language Models (LLMs) such as GPT, Bard, and Llama has revolutionized various sectors, including the scientific community. These models, with their potential to automate and augment tasks, are increasingly being recognized as both a valuable asset and a potential challenge in the realm of scientific research and data management. However, the current LLMs, primarily trained on general corpora, exhibit a limited understanding of scientific concepts and terminologies due to the lack of scientific corpus in their training data. Recognizing this gap, several groups are now advocating for the development of LLMs specifically tailored for scientific applications. A notable initiative in this direction is the Large Language Model effort initiated by NASA's CSDO. This endeavor aims to align LLM efforts across NASA’s Science Mission Directorate, develop a science-specific corpus and validation test set for model training, and create an encoder-only model for various downstream tasks. Moreover, the initiative also plans to develop a decoder-only model to explore the potential benefits and risks associated with a generative LLM for science. Lastly, the project aims to create a science evaluation suite, encompassing various categories of downstream scientific tasks, to serve as a benchmark for assessing the value of any LLM for future use. This presentation will provide an overview and current status of this ongoing initiative, highlighting its potential to reshape the use of LLMs in the scientific domain.

Rahul Ramachandran↗
NASA NTRSDate not supplied

Language Model For Earth Science: Exploring Potential Downstream Applications As Well As Current Challenges

The use of deep learning techniques to build transformer language models such as SciBERT and GPT3 have transformed the natural language technology (NLT) landscape. These new NLTs are being used in speech to text and vice versa, auto-mated text classification, sentiment analysis, topic modeling, text summarization, and cognitive assistants. While Earth science has no shortage of unstructured data such as journal and conference papers, little efforts have focused on harnessing NLTs for knowledge extraction and supporting the scientific process. This paper surveys the use of language models in different science. BERT-E, a new Earth science-specific language model, is presented. BERT-E is generated using a transfer learning solution. A language model that has already been trained for general Science (SciBERT) is fine-tuned using abstracts and full text extracted from various Earth science-related articles. A downstream keywords classification application is used for evaluation, and the use of BERT-E shows improved performance. The need to develop a robust set of benchmarks in evaluating the language model such as BERT-E is discussed. Finally, example applications are presented to inspire additional ideas for applications using domain-specific language models.

R Ramachandran↗
DOE OSTI2024

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

A Large Language Model for Determining Partial Tripping of Distributed Energy Resources

Knowing the status of individual distributed energy resources, i.e., being tripped or not, after a contingency can inform the development of an aggregated DER model. Here, this letter presents a large language model application to determine the partial tripping of distributed energy resources depending on the types, locations, and duration of faults in the transmission network. The large language model, or more specifically BERT-based approach can streamline the fault information into tokenized input, which not only reduces the complexity of the machine learning model but also demonstrates a robust performance with only limited data sets.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

Prompt Phrase Ordering Using Large Language Models in HPC: Evaluating Prompt Sensitivity

Large language models (LLMs) have demonstrated effective performance in domain-specific tasks, often requiring a well-designed prompt to guide their responses. However, optimizing the right prompt is challenging due to prompt sensitivity—the phenomenon where small changes in the prompt can lead to significant variations in performance. In this study, we evaluate prompt performance by examining all permutations of independent phrases to investigate prompt sensitivity and robustness. We used two datasets: the GSM8k dataset, which assesses mathematical reasoning, and a custom template prompt for summarizing database metadata. Our goal was to evaluate the performance across all permutations of a sequence of prompt phrases. The study was conducted using the llama3-instruct- 7B model hosted on Ollama, with computations parallelized in a high-performance computing environment. By comparing the average index of phrases in the best and worst-performing prompts, we found that the order of independent phrases within a prompt significantly impacts LLM performance. Additionally, we used Hamming distance to assess changes between phrase orderings, concluding that prompt modifications can dramatically affect scores, often by almost random chance. These findings support existing research on prompt sensitivity. We discuss the challenges of prompt optimization, noting that altering phrases in a successful prompt does not always result in another successful prompt.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Dataset for "Large Language Models as molecular design engines"

This dataset contains data and results associated with the paper "Large Language Models as molecular design engines" The paper investigates the use of large language models, specifically Claude 3 Opus, for generating and analyzing chemical structures based on various prompts from A-H (as mentioned in the manuscript), and guided design related to electron-withdrawing groups (EWG), electron-donating groups (EDG).

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

Leveraging Large Language Models for Understanding Fundamental Principles of Catalysis

Heterogeneous catalysis presents a distinct challenge for artificial intelligence (AI). Data sets are often small and inconsistently reported, catalyst representations are not standardized, and extracting fundamental knowledge requires integrating performance data, spectroscopic characterizations, and mechanistic models across multiple scales. Language offers a unifying representation across these modalities, making catalysis well suited for leveraging large language models (LLMs). By standardizing how catalytic data is represented, LLMs make dispersed experimental results more accessible to downstream statistical modeling. In this perspective, we focus our discussion around three opportunities where LLMs can significantly contribute to catalysis: (1) text to properties; (2) text to structure; and (3) text to mechanistic models. The discussion is followed by a perspective section on LLM-readiness of data, aligning LLM outputs with scientific correctness, and bridging lab-scale discovery to industrial deployment. Across each area, the most productive applications couple dispersed chemical knowledge with physics-grounded validation to produce verifiable hypotheses and actionable representations.

Catalysts↗
NASA NTRSDate not supplied

Defining A Modelling Language to Support Functional Hazard Assessment

Functional Hazard Assessment (FHA) is a key early-stage engineering process that supports the incorporation of safety in design by identifying the high-level functional hazards the system may encounter. While many FHA-like methodologies have been proposed in the design engineering literature, many of these methodologies have had difficulty becoming accepted industry practice. Industry standards, on the other hand, either provide little recommendation on how to represent the function of the system to perform FHA, or rely on readily-available models with little justification in design theory. This paper presents some of the problems with current modelling languages used for FHA which limit the scope, expressiveness, flexibility, and precision of the analysis, as well as desirable principles an FHA-supporting analysis language should embody. It further introduces the Functional Reasoning Design Language (FRDL), a formal modelling language for describing the functional behaviors of a system and their interactions which satisfies these principles. To demonstrate the use of this language, the modelling and hazard analysis of a disaster response drone is presented.

safety analysis↗
DOE OSTI2026

Ten questions concerning Large Language Models (LLMs) for building applications

Large Language Models (LLMs) are emerging as powerful AI tools capable of transforming how building information is collected, processed, analyzed, and applied across diverse research areas. Their capabilities can help building operators, facility managers and other stakeholders such as designers, architects and engineers by providing actionable insights for decision-making across planning, construction, operations, and maintenance of buildings and facilities. This paper explores ten key questions concerning the role of LLMs in shaping sustainable, intelligent, and human-centric buildings. From fundamental definitions to advanced applications, we examine how LLMs facilitate decision-making across the life cycle of buildings and energy systems. LLMs can enhance life cycle assessments (LCA), building energy simulations, and real-time data integration, empowering more efficient and adaptive human-AI environments. They can also contribute to streamlining regulatory compliance, improving post-occupancy evaluations, and fostering more inclusive and participatory design processes. Additionally, this paper addresses the ethical challenges posed by LLMs, such as bias, data privacy, and environmental impacts, and explores their potentials in advancing intelligent digital twins (DT) for ongoing building operations and maintenance. Built upon our applied research using LLMs and the review of tools, datasets, and research gaps, we provide a forward-looking perspective on how LLMs can drive innovation, collaboration, and productivity in the built environment while supporting ethical and effective implementation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗