Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large language models”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

A Framework for Evaluating the Implementation Cost of Attacks on Large Language Models

Large Language Models (LLMs) have been increasingly proposed as a method to enhance productivity in tasks that involve language and code. However, these models are large, complex, and their capabilities are not easily understood and controlled, meaning that their adoption opens many possibilities for new cyberattacks and misuse. Numerous attacks on LLMs have been reported and summarized in literature reviews, but we found existing reviews lacking in understanding the implementation cost of the attacks - i.e., how much effort would an attacker need in terms of coding, expertise, and resources to adopt attacks presented in the literature. Therefore, we divide existing attacks on LLMs into a taxonomy, and define a cost evaluation framework to determine the cost of the attack. An attack’s cost can be 1) estimated from reading the publication about the attack or 2) determined by implementing the attack from that publication. We provide an example evaluation of a couple jailbreaking frameworks based on experiments, and then apply the more lightweight cost estimate to a representative selection of attacks across the taxonomy we define. We discuss the relative difficulty of the attacks and also highlight defenses that have attempted to mitigate these attacks and assess their effectiveness.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Optimizing Distributed Training on Frontier for Large Language Models

Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Loss scaling studies have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of 38.38%, 36.14%, and 31.96%, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved 100% weak scaling efficiency on 1024 and 3072 Mi250X GPUs, respectively. We also achieved strong scaling efficiencies of 89% and 87% for these two models. We trained these models only tens of iterations instead of training till completion.

Yin, Junqi↗
DOE OSTI2024

Towards the holistic design of alloys with large language models

Large language models are very effective at solving general tasks, but can also be useful in materials design and extracting and using information from the scientific literature and unstructured corpora. For instance, in the domain of alloy design and manufacturing, they can expedite the materials design process and enable the inclusion of holistic criteria.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Advancing Building Energy Modeling with Large Language Models: Exploration and Case Studies

The rapid progression in artificial intelligence has facilitated the emergence of large language models like ChatGPT, offering potential applications extending into specialized engineering modeling, especially physics-based building energy modeling. This paper investigates the innovative integration of large language models with building energy modeling software, focusing specifically on the fusion of ChatGPT with EnergyPlus. A literature review is first conducted to reveal a growing trend of incorporating large language models in engineering modeling, albeit limited research on their application in building energy modeling. We underscore the potential of large language models in addressing building energy modeling challenges and outline potential applications including simulation input generation, simulation output analysis and visualization, conducting error analysis, co-simulation, simulation knowledge extraction and training, and simulation optimization. Three case studies reveal the transformative potential of large language models in automating and optimizing building energy modeling tasks, underscoring the pivotal role of artificial intelligence in advancing sustainable building practices and energy efficiency. The case studies demonstrate that selecting the right large language model techniques is essential to enhance performance and reduce engineering efforts. The findings advocate a multidisciplinary approach in future artificial intelligence research, with implications extending beyond building energy modeling to other specialized engineering modeling.

building energy modeling↗
DOE OSTI2025

ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training

Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, the training of these models is computationally intensive and susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs. In this paper, we investigate the impact of faults on LLM training, focusing on INF, NaN, and near-INF values in the computation results with systematic fault injection experiments. We observe the propagation patterns of these errors, which can trigger non-trainable states in the model and disrupt training, forcing the procedure to load from checkpoints. To mitigate the impact of these faults, we propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs. ATTNChecker is designed based on fault propagation patterns of LLM and incorporates performance optimization to adapt to both system reliability and model vulnerability while providing lightweight protection for fast LLM training. Evaluations on four LLMs show that ATTNChecker on average incurs on average 7% overhead on training while detecting and correcting all extreme errors. Compared with the state-of-the-art checkpoint/restore approach, ATTNChecker reduces recovery overhead by up to 49×.

Liang, Yuhang [University of Alabama - Birmingham]↗
DOE OSTI2026

Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models

Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs. To better understand LLM scaling, we study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in smaller models. We observe a geometric phenomenon which we term embedding condensation, where token embeddings collapse into a narrow cone-like subspace in some language models. Through systematic analyses across multiple Transformer families, we show that small models such as GPT2 and Qwen3-0.6B exhibit severe condensation, whereas larger models such as GPT2-x1 and Qwen3-32B are more resistant to this phenomenon. Additional observations show that embedding condensation is not reliably mitigated by knowledge distillation from larger models. To fight against it, we formulate a dispersion loss that explicitly encourages embedding dispersion during training. Experiments demonstrate that it mitigates condensation, recovers dispersion patterns seen in larger models, and yields performance gains across 10 benchmarks. We believe this work offers a principled path toward improving smaller Transformers without additional parameters.

Xiao, Xi [ORNL] (ORCID:0009000009316982)↗
DOE OSTI2025

Large language models for batteries

Large Language Models (LLMs) are advanced artificial intelligence systems capable of solving diverse tasks using language, reasoning, and external tools. Despite their growing deployment in academia and industry, their potential remains underexplored in battery research. This review presents a comprehensive overview of existing and emerging applications of LLMs in batterie field, addressing two critical questions: What can LLMs offer to support battery-related tasks, and how to develop more effective models for this purpose. We begin by outlining the principles of LLMs and criteria for selecting appropriate models and tools for battery research and development. We then explore their roles in text-mining, data interpretation, and the development of intelligent battery systems. In parallel, we discuss technical challenges, such as data standardizing and sharing, model evaluation, and tool integration. Lastly, we propose future research directions with short-, medium-, and long-term goals and highlight more broad perspectives for connecting experts and cross-disciplinary collaborations.

SoC↗
DOE OSTI2024

Sentiment analysis of the United States public support of nuclear power on social media using large language models

This study utilized large language models (LLMs) to analyze public sentiment in the United States (US) regarding nuclear power on social media, focusing on X/Twitter, considering climate change challenges and advancements in nuclear power technology. Approximately, 1.26 million nuclear tweets from 2008–2023 were examined to fine-tune LLMs for sentiment classification. We found the crucial role of accurate data labeling for model performance, with potential implications for a 15% improvement, achieved through high-confidence labels. LLMs demonstrated better performance compared to traditional machine learning classifiers, with reduced susceptibility to overfitting and up to 96% classification accuracy. LLMs are used to segment the US public tweets into policy and energy-related categories, revealing that 68% are politically themed. Policy tweets tended to convey negative sentiment, often reflecting opposing political perspectives and focusing on nuclear deals and international relations. Energy-related tweets covered diverse topics with predominantly neutral to positive sentiment, indicating broad support for nuclear power in 48 out of 50 US states. The US public positive sentiments toward nuclear power stemmed from its high power density, reliability regardless of weather conditions, environmental benefits, application versatility, and recent innovations and advancements in both fission and fusion technologies. Negative sentiments primarily focused on waste management, high capital costs, and safety concerns. The neutral campaign highlighted global nuclear facts and advancements, with varying tones leaning towards positivity or negativity. An interesting neutral theme was the advocacy for the combined use of renewable and nuclear energy to attain net-zero goals.

Energy & Fuels↗
DOE OSTI2024

Comparative Study of Large Language Model Architectures on Frontier

Large language models (LLMs) have garnered significant attention in both the AI community and beyond. Among these, the Generative Pre-trained Transformer (GPT) has emerged as the dominant architecture, spawning numerous variants. However, these variants have undergone pre-training under diverse conditions, including variations in input data, data preprocessing, and training methodologies, resulting in a lack of controlled comparative studies. Here we meticulously examine two prominent open-sourced GPT architectures, GPT-NeoX and LLaMA, leveraging the computational power of Frontier, the world’s first Exascale supercomputer. Employing the same materials science text corpus and a comprehensive end-to-end pipeline, we conduct a comparative analysis of their training and downstream performance. Our efforts culminate in achieving state-of-the-art performance on a challenging materials science benchmark. Furthermore, we investigate the computation and energy efficiency, and propose a computationally efficient method for architecture design. To our knowledge, these pre-trained models represent the largest available for materials science. Our findings provide practical guidance for building LLMs on HPC platforms.

Yin, Junqi↗
DOE OSTI2025

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose significant challenges, requiring efficient hardware acceleration. Benchmarking the performance of LLMs across diverse hardware platforms is crucial to understanding their scalability and throughput characteristics. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks. We provide an interactive dashboard to help identify configurations for optimal performance for a given hardware platform.

Chitty-Venkata, Krishna Teja↗
DOE OSTI2025

Automatic building energy model development and debugging using large language models agentic workflow

Building energy modeling (BEM) is a complex process that demands significant time and expertise, limiting its broader application in building design and operations. While Large Language Models (LLMs) agentic workflow have facilitated complex engineering processes, their application in BEM has not been specifically explored. This paper investigates the feasibility of automating BEM using LLM agentic workflow. Here, we developed a generic LLM-planning-based workflow that takes a building description as input and generates an error-free EnergyPlus building energy model. Our robust workflow includes four core agents: 1) Building Description Pre-Processing, 2) IDF Object Information Extraction, 3) Single IDF Object Generator Suite, and 4) IDF Debugging Agent. These agents divide the complex tasks into manageable sub-steps, enabling LLMs to generate accurate and reliable results at each stage. The case study demonstrates the successful translation of a building description into an error-free EnergyPlus model for the iUnit modular building at the National Renewable Energy Laboratory. The effectiveness of our workflow surpasses: 1) naive prompt engineering, 2) other LLM-based workflows, and 3) manual modeling, in terms of accuracy, reliability, and time efficiency. The paper concludes with a discussion on the interplay between foundational models and LLM agent planning design, advocating for the use of fine-tuned, specialized models to advance this field.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or aerial wildfire operations reports to better understand the risks present. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement. These applications can benefit from the use of state-of-the-art natural language processing techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of adaptation of NLP tools to the aviation domain by fine-tuning transformer based models using aviation data sets. In 2018, a novel language model based on neural units (also called transformers) was created and became known as “Bidirectional Encoder Representations from Transformers” or BERT. This architecture combined with large amounts of English training data and innovative semi-supervised training tasks set the standard for what would later emerge as Large Language Models. The performance of these models was further improved by hyperparameter tuning and refinement of the semi-supervised training task and resulted in “Robustly Optimized BERT Pre-training Approach through hyperparameter tuning” or RoBERTa models. These pre-trained Large Language Models proved to be useful for a wide variety of natural language processing tasks such as text classification and question answering through a process called fine-tuning. The transformer architecture with pre-trained weights served as the basis with the last few layers replaced with layers fine-tuned to perform a new task e.g., a layer that provides a label for the entire input text. This process of fine-tuning can also be used to adapt the models to new domains; e.g., BioBERT started with the pre-trained BERT model and was completed by additional fine-tuning and training on biomedical documents. Transformer-based architectures can also be used to create rich representations of text called embeddings which can serve as the input to other machine learning models. This allows simpler algorithms such as logistic regression to use context-rich representations of the text while still remaining quick to train and evaluate. In the world of aviation, there is a growing demand for natural language processing and understanding but the domain presents unique challenges. Due to the technical content (and specialized language) of most aviation documents, fine-tuning pre-trained Large Language Models to specific tasks has not met the benchmark on natural language processing tasks set by simpler models trained from scratch on the data. To address this deficiency, this paper evaluates the improvements from fine-tuning a Large Language Model on a large set of aviation documents using the original semi-supervised training tasks before performing specific natural language tasks. In fine-tuning, a domain-specific dataset is used on the original training task but with the pre-trained Large Language Model instead of starting from a random initialization. This approach allows the model to be adapted to the specific domain language without discarding the information gained from training on general English data. This paper utilized two major dataset types to train and assess the RoBERTa fine-tuning performance. The first are 7,057 Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the semi-supervised fine-tuning. The second type is the 494 document classification labels to be used for evaluation. This down-stream evaluation aims to show the performance of the fine-tuned model, better understand how much data is needed for an effective fine-tuning, and how fine-tuning can be adapted for different applications in-the domain. After semi-supervised training, evaluation begins by encoding the documents for classification using the fine-tuned RoBERTa model. Then a logistic regression classifier is trained to label the document type and compared against our ground truth labels. This currently leads to a 82.8% accuracy on 10-fold cross validation showing improvement over baseline RoBERTa which achieved 81.0%. We plan to measure the improvements on additional tasks and it is expected that these improvements will lead to more robust models that can tackle the natural language processing challenges present in aviation datasets.

ATM↗
DOE OSTI2026

Ten questions concerning Large Language Models (LLMs) for building applications

Large Language Models (LLMs) are emerging as powerful AI tools capable of transforming how building information is collected, processed, analyzed, and applied across diverse research areas. Their capabilities can help building operators, facility managers and other stakeholders such as designers, architects and engineers by providing actionable insights for decision-making across planning, construction, operations, and maintenance of buildings and facilities. This paper explores ten key questions concerning the role of LLMs in shaping sustainable, intelligent, and human-centric buildings. From fundamental definitions to advanced applications, we examine how LLMs facilitate decision-making across the life cycle of buildings and energy systems. LLMs can enhance life cycle assessments (LCA), building energy simulations, and real-time data integration, empowering more efficient and adaptive human-AI environments. They can also contribute to streamlining regulatory compliance, improving post-occupancy evaluations, and fostering more inclusive and participatory design processes. Additionally, this paper addresses the ethical challenges posed by LLMs, such as bias, data privacy, and environmental impacts, and explores their potentials in advancing intelligent digital twins (DT) for ongoing building operations and maintenance. Built upon our applied research using LLMs and the review of tools, datasets, and research gaps, we provide a forward-looking perspective on how LLMs can drive innovation, collaboration, and productivity in the built environment while supporting ethical and effective implementation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Prompt Phrase Ordering Using Large Language Models in HPC: Evaluating Prompt Sensitivity

Large language models (LLMs) have demonstrated effective performance in domain-specific tasks, often requiring a well-designed prompt to guide their responses. However, optimizing the right prompt is challenging due to prompt sensitivity—the phenomenon where small changes in the prompt can lead to significant variations in performance. In this study, we evaluate prompt performance by examining all permutations of independent phrases to investigate prompt sensitivity and robustness. We used two datasets: the GSM8k dataset, which assesses mathematical reasoning, and a custom template prompt for summarizing database metadata. Our goal was to evaluate the performance across all permutations of a sequence of prompt phrases. The study was conducted using the llama3-instruct- 7B model hosted on Ollama, with computations parallelized in a high-performance computing environment. By comparing the average index of phrases in the best and worst-performing prompts, we found that the order of independent phrases within a prompt significantly impacts LLM performance. Additionally, we used Hamming distance to assess changes between phrase orderings, concluding that prompt modifications can dramatically affect scores, often by almost random chance. These findings support existing research on prompt sensitivity. We discuss the challenges of prompt optimization, noting that altering phrases in a successful prompt does not always result in another successful prompt.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Evaluation of pre-training large language models on leadership-class supercomputers

Large language models (LLMs) have arisen rapidly to the center stage of artificial intelligence as the foundation models applicable to many downstream learning tasks. However, how to effectively build, train, and serve such models for many high-stake and first-principle-based scientific use cases are both of great interests and of great challenges. Moreover, pre-training LLMs with billions or even trillions of parameters can be prohibitively expensive not just for academic institutions, but also for well-funded industrial and government labs. Furthermore, the energy cost and the environmental impact of developing LLMs must be kept in mind. Here, in this work, we conduct a first-of-its-kind performance analysis to understand the time and energy cost of pre-training LLMs on the Department of Energy (DOE)’s leadership-class supercomputers. Employing state-of-the-art distributed training techniques, we evaluate the computational performance of various parallelization approaches at scale for a range of model sizes, and establish a projection model for the cost of full training. Our findings provide baseline results, best practices, and heuristics for pre-training such large models that should be valuable to HPC community at large. We also offer insights and optimization strategies for using the first exascale computing system, Frontier, to train models of the size of GPT-3 and beyond.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Harnessing Large Language Models for Scientific Endeavors

The rapid proliferation of Large Language Models (LLMs) such as GPT, Bard, and Llama has revolutionized various sectors, including the scientific community. These models, with their potential to automate and augment tasks, are increasingly being recognized as both a valuable asset and a potential challenge in the realm of scientific research and data management. However, the current LLMs, primarily trained on general corpora, exhibit a limited understanding of scientific concepts and terminologies due to the lack of scientific corpus in their training data. Recognizing this gap, several groups are now advocating for the development of LLMs specifically tailored for scientific applications. A notable initiative in this direction is the Large Language Model effort initiated by NASA's CSDO. This endeavor aims to align LLM efforts across NASA’s Science Mission Directorate, develop a science-specific corpus and validation test set for model training, and create an encoder-only model for various downstream tasks. Moreover, the initiative also plans to develop a decoder-only model to explore the potential benefits and risks associated with a generative LLM for science. Lastly, the project aims to create a science evaluation suite, encompassing various categories of downstream scientific tasks, to serve as a benchmark for assessing the value of any LLM for future use. This presentation will provide an overview and current status of this ongoing initiative, highlighting its potential to reshape the use of LLMs in the scientific domain.

Rahul Ramachandran↗
DOE OSTI2026

A Benchmarking Framework for Evaluating Large Language Model Capabilities in Nuclear Reactor Safety Applications

Large language models (LLMs) are increasingly capable of answering technical questions, synthesizing domain knowledge, and supporting engineering workflows. For nuclear science and engineering, these capabilities require careful, domain-specific evaluation before they can be credibly incorporated into safety-related activities, regulatory review, or technical decision support. This paper presents preliminary results from benchmarking framework for evaluating LLM capabilities in nuclear contexts. The framework is organized into three evaluation categories: nuclear fundamentals, general dual-use knowledge, and plant specific knowledge. These categories are intended to distinguish general nuclear engineering competence from broader technical reasoning and more context-dependent nuclear knowledge. Initial evaluations focus on nuclear fundamentals using questions representative of the knowledge expected of a nuclear professional engineer. Results indicate that contemporary frontier models perform at a high level and substantially exceed the performance of older model generations, with some models approaching saturation of the current benchmark. These findings suggest both the rapid improvement of LLM capabilities in specialized technical domains and the need for more discriminating evaluation methods. The paper presents the benchmark structure, preliminary model-comparison results, and ongoing work. This work supports development of verifiable, responsible, and safety-conscious methods for assessing AI systems in nuclear engineering applications.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗
DOE OSTI2024

Improving Reliability of Large Language Models for Nuclear Power Plant Diagnostics [Poster]

Large Language Models (LLMs) struggle out of the box when answering factually about detailed questions, especially in domains that are sparsely represented in their training data. This causes hallucinations and reduces reliability making it difficult for them to be used in practice. This work shows that using RAG techniques can improve factual accuracy and reliability, allowing for the application of LLMs in specialized areas, even when those areas that aren’t extensively covered in their initial training.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗