Assessment of chemistry knowledge in large language models that generate code
In this work, we investigate the question: do code-generating large language models know chemistry? Our results indicate, mostly yes.
White, Andrew D.↗
SEARCH · Search NASA
Results for “Language model”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Exploring Large Language Model Agents in Cybersecurity: A Literature Review with Experiments
The accelerated development and integration of large language model (LLM) agents have led researchers and developers to explore their effectiveness in cybersecurity, specifically with penetration testing (pentesting). Recent research efforts have attempted to use LLM agents to automate the process of pentesting because of the cost and time requirements that are required to perform a manual review. However, not all of the tools perform as expected. This paper reviews some of the newest and most popular autonomous pentesting frameworks, highlighting the capabilities and limitations of each one with the goal of providing the components needed to successfully and effectively build an autonomous pentesting agent in the future.
97 MATHEMATICS AND COMPUTING↗
Cross-domain digital twin architecture for predictive maintenance via machine learning and Large Language Models
This research introduces a comprehensive framework for creating and deploying a digital twin platform for continuous monitoring and predictive maintenance within industrial settings. Through utilizing advanced technologies, including Unreal Engine 5, Unity 3D, the Message Queue Telemetry Transport protocol, Random Forest machine learning algorithms, and Large Language Models (LLMs), we establish a platform that digitally reproduces physical equipment and translates digital controls into real-world actions. This facilitates preventive maintenance approaches and improves operational effectiveness. The digital twin platform gathers sensor data from operational equipment, analyzes it using machine learning, and delivers practical insights to prevent potential malfunctions and enhance equipment performance. Furthermore, the incorporation of a web portal enables efficient monitoring and access to historical data, educational materials, and equipment status information. Preliminary findings indicate that digital twins can transform industrial equipment management and maintenance methodologies.
97 MATHEMATICS AND COMPUTING↗
Business Case Analysis for Artificial Intelligence-Large Language Model Technology Integration
AI-assisted processes are expected to enhance operational efficiency and improve decision-making, supporting the long-term economic viability of nuclear power plants. However, detailed business analyses of AI-generated cost savings are rarely performed. Given the recent industry interest in Large Language Model (LLM), the U.S. Department of Energy (DOE) Light Water Reactor Sustainability (LWRS) Program has conducted a comprehensive business case analysis of LLM Artificial Intelligence (AI) implementation in nuclear plant engineering workflows. The research employed three complementary business case approaches to evaluate impact of an LLM, using three representative engineering processes as use-cases: Boric Acid Corrosion (BAC) Evaluations, Maintenance Rule Evaluations, and 10 CFR 50.59 Screenings. Through detailed workload analyses and structured interviews, the study quantified significant efficiency improvements ranging from 11% to 59% across these processes. The research further considers how these efficiency gains could translate into tangible reliability improvements through enhanced engineering capacity. Analysis of historical plant trip data indicates that enabling engineers to focus on proactive reliability activities could provide substantial financial benefits through avoided outages, potentially generating greater value than the direct efficiency improvements alone. By documenting successful applications, implementation challenges, and strategic opportunities, this research provides nuclear utilities with a practical framework for evaluating the value of AI technology to support long-term operations through advanced digital technologies.
97 MATHEMATICS AND COMPUTING↗
ChatGPT and Other Large Language Models for Cybersecurity of Smart Grid Applications
Cybersecurity breaches targeting electrical substations constitute a significant threat to the integrity of the power grid, necessitating comprehensive defense and mitigation strategies. Any anomaly in information and communication technology (ICT) should be detected for secure communications between devices in digital substations. This paper proposes large language models (LLMs), e.g., ChatGPT, for the cybersecurity of IEC 61850-based communications. Multi-cast messages such as generic object oriented system events (GOOSE) and sampled values (SV) are used for case studies. The proposed LLM-based cybersecurity framework includes, for the first time, data pre-processing of communication systems and human-in-the-loop (HITL) training (considering the cybersecurity guidelines recommended by humans). The results show a comparative analysis of detected anomaly data carried out based on the performance evaluation metrics for different LLMs. A hardware-in-the-loop (HIL) testbed is used to generate and extract a dataset of IEC 61850 communications.
ChatGPT↗
14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon
We report the findings of a hackathon focused on exploring the diverse applications of large language models in molecular and materials science.
36 MATERIALS SCIENCE↗
Reductive Analysis with Compiler-Guided Large Language Models for Input-Centric Code Optimizations
Input-centric program optimization aims to optimize code by considering the relations between program inputs and program behaviors. Despite its promise, a long-standing barrier for its adoption is the difficulty of automatically identifying critical features of complex inputs. This paper introduces a novel technique, reductive analysis through compiler-guided Large Language Models (LLMs), to solve the problem through a synergy between compilers and LLMs. It uses a reductive approach to overcome the scalability and other limitations of LLMs in program code analysis. The solution, for the first time, automates the identification of critical input features without heavy instrumentation or profiling, cutting the time needed for input identification by 44× (or 450× for local LLMs), reduced from 9.6 hours to 13 minutes (with remote LLMs) or 77 seconds (with local LLMs) on average, making input characterization possible to be integrated into the workflow of program compilations. Optimizations on those identified input features show similar or even better results than those identified by previous profiling-based methods, leading to optimizations that yield 92.6% accuracy in selecting the appropriate adaptive OpenMP parallelization decisions, and 20-30% performance improvement of serverless computing while reducing resource usage by 50-60%.
Input-Centric Optimization↗
Edge AI-Enhanced Traffic Monitoring and Anomaly Detection Using Multimodal Large Language Models
This paper addresses the challenge of traffic monitoring and incident detection in remote areas, utilizing multimodal large language models (LLMs) deployed on edge AI devices. The key novelty of the LLM is to convert real-time video streams into descriptive texts, enabling low-bandwidth transmissions and reliable detection of anomalies and incidents in environments of intermittent connectivity. The model is developed based on fine-tuning open-source LLMs and extending it with multi-modal capabilities to analyze video frames. Our work also involves deploying this model on edge devices such as Nvidia IGX Orin and is planned to be tested in realistic environments in future work. The methodology includes data set curation, iterative model fine-tuning and compression, and hardware-based optimization. This approach aims to enhance traffic safety and response speed in remote areas, marking a significant advancement in the application of AI for traffic monitoring and safety management.
Peruski, Ryan [University of Tennessee, Knoxville ↗
Mitigative Strategies for Recovering From Large Language Model Trust Violations
In this study, we investigated strategies to address trust issues arising from errors in large language models (LLMs). The study examined the impact of confidence scores, system capability explanations, and user feedback on trust restoration post-error. 68 participants viewed the responses of an LLM to 20 general trivia questions, with an error introduced on the third trial. Each participant was presented with one mitigation strategy. Participants rated their overall trust in the model and the reliability of the answer. Results showed an immediate drop in trust after the error; however, there were no differences across the three strategies in trust recovery. All conditions had a logarithmic trend in trust recovery following error. Differences in overall trust were predicted by perceived reliability of the answer, suggesting that participants were evaluating results critically and using that to inform their trust in the model. Qualitative data supported this finding; participants expressed lasting distrust despite the LLM’s later accuracy. Results showcase the need to prioritize accuracy in LLM deployment, because early errors may irrevocably damage user trust calibration and later adoption.
97 MATHEMATICS AND COMPUTING↗
Derivation of physical equations for high-speed laser welding using large language models
It is challenging to formulate complex physical phenomena that occur in a manufacturing process, particularly when the available data are limited, rendering conventional data-driven approaches ineffective. This study aims to predict humping onset in high-speed laser welding by introducing a novel framework, namely text-to-equations generative pre-trained transformer (T2EGPT). This method leverages the capabilities of large language models (LLMs), in combination with sparse experimental data and enriched literature data, to derive an interpretable and generalizable equation for predicting humping initiation. By capturing key correlations among physical parameters, T2EGPT generates a compact and dimensionless expression that accurately predicts hump formation. The equation reveals that humping arises from the interplay between inertia-driven backward melt flow and capillary-driven surface stabilization, where inertial forces drive molten metal backward and capillary forces resist surface deformation. Furthermore, compared to traditional data-driven models, T2EGPT demonstrates enhanced predictive accuracy and cross-material transferability. More broadly, this study highlights the potential of LLMs to integrate textual information with data-driven discovery, enabling the extraction of physical laws in data-scarce scientific domains.
36 MATERIALS SCIENCE↗
Generative large language models for predictive maintenance planning
Maintenance planning and the generation of necessary components for tasks can prove time-consuming and complex. Automating the creation of recurring or similar tasks by leveraging previous planning packages and data, while uncovering insights to automate planning package generation, presents an opportunity to conserve valuable time and resources. This work aims to harness the textual and probabilistic capabilities of large language models (LLMs) to automate the generation of planning packages. Utilizing diverse data sources ranging from raw data to handwritten text, both singular and collaborative LLMs are trained and tested. Results demonstrate their capability to generate essential planning package components, effectively replicating the statistical patterns in the data. This demonstrates the use of these tools inside a digital asset for automated planning. This work outlines a methodology for constructing datasets, a training suite, and evaluation methods for LLM-based textual and conversational planning tools utilized in an asset digital twin. Results indicate that the fine-tuned models generate estimated planning information within the statistical ranges observed in real maintenance data. The models achieve high accuracy (>90%) in document question-answering and instruction generation tasks. Furthermore, the conversational retrieval-augmented generation (RAG) assistant system achieves 100% document retrieval accuracy, while conversational information capture exceeds 98% across the majority of work-package assistant modules.
97 MATHEMATICS AND COMPUTING↗
Hybrid-LLM-GNN: integrating large language models and graph neural networks for enhanced materials property prediction
This study combines Graph Neural Networks (GNNs) and Large Language Models (LLMs) to improve material property predictions. By leveraging both embeddings, this hybrid approach achieves up to a 25% improvement over GNN-only model in accuracy.
Li, Youjia [Department of Electrical and Computer ↗
ESAC (EQ-SANS Assisting Chatbot): Application of large language models and retrieval-augmented generation for enhanced user experience at EQ-SANS
Neutron scattering experiments have played vital roles in exploring materials properties in the past decades. While user interfaces have been improved over time, neutron scattering experiments still require specific knowledge or training by an expert due to the complexity of such advanced instrumentation and the limited number of experiments each person may perform each year. This paper introduces an innovative chatbot application that leverages Large Language Models(LLM) and Retrieval-Augmented Generation (RAG) technologies to significantly enhance the user experience at the EQ-SANS, a small-angle neutron scattering instrument at the Spallation Neutron Source of Oak Ridge National Laboratory. Through a user-centric design approach, the EQ-SANS Assisting Chatbot (ESAC) serves as an interactive reference for users, thereby facilitating the use of the instrument by visiting scientists. By bridging the gap between the users of EQ-SANS and the control systems required to perform their experiments, the ESAC sets a new standard for interactive learning and support for the scientific community using large-scale scientific facilities.
97 MATHEMATICS AND COMPUTING↗
Data Driven Approach to Public Opinion Mining on Autonomous Vehicles: Sentiment Analysis of Social Media Comments Using Large Language Models
In the realm of online identity, social media has emerged as a rich and dynamic source of user-generated content, making it an invaluable resource for understanding public sentiment on a wide range of topics. Individuals often share their raw emotions and candid opinions on these platforms without fear of judgment or backlash. In this study, we conduct a sentiment analysis on user comments collected from various online platforms, with a specific focus on discussions surrounding autonomous vehicles. Leveraging the capabilities of large language models (LLMs), we classify each comment into one of five sentiment categories: Very Negative, Negative, Neutral, Positive, and Very Positive. Our approach demonstrates the effectiveness of LLMs in capturing nuanced contextual sentiment, offering a scalable and state-of-the-art alternative to traditional manual annotation methods. The results reveal key trends and insights into public perception, enabling a deeper understanding of how autonomous vehicle technologies are received by the online community. Our findings underscore the dynamic nature of public sentiment, which is shaped not only by advances in autonomous vehicle technology but also by contextual events such as regulatory developments, political adjustment and safety incidents.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Leveraging large language models to address data scarcity in machine learning for graphene synthesis
Machine learning in experimental materials science faces significant challenges due to the scarcity of data, which are costly and time-consuming to generate, particularly when relying on in-house experiments. Literature data mining offers a potential solution but introduces issues like mixed data quality, inconsistent formats, and non-uniform reporting of synthesis parameters, resulting in partially missing and heterogeneous features across the dataset. Here, we propose data imputation and feature engineering methods that employ pre-trained large language models (LLMs) to enhance machine learning performance on scarce, heterogeneous datasets, demonstrated on graphene CVD synthesis data and the ML-HydPARK hydrogen storage dataset. GPT models perform data imputation via tailored prompting and semantic normalization of inconsistently reported features through embeddings, for example, to harmonize the complex nomenclature of CVD substrates. Beyond yielding more diverse and richer feature representations than traditional methods such as K-nearest neighbors (KNN) and Multivariate Imputation by Chained Equations (MICE), LLM-based data imputation is evaluated against dataset characteristics and prompting strategies. We vary the level of autonomy granted to the LLM, from generic prompting that leverages pre-trained knowledge for autonomous data generation to data-informed prompting that constrains outputs using target-specific information, and demonstrate which level of autonomy yields superior imputation performance across datasets and feature types. The proposed data engineering methods markedly improve downstream performance; for example, in graphene layer number classification using a support vector machine (SVM), binary accuracy increases from 39% to 65% and ternary accuracy from 52% to 72%. Fine-tuning experiments on both datasets show that combining our proposed LLM-based data imputation and feature encoding methods with numerical machine learning predictors outperforms standalone fine-tuned LLM predictors in data-scarce settings. The proposed strategies emphasize data enhancement techniques rather than refining learning architectures or regularizing loss functions, offering a broadly applicable framework for improving machine learning performance on scarce, inhomogeneous datasets.
Chemical vapor deposition↗
Can Large Language Models Understand Intermediate Representations?
Intermediate Representations (IRs) are essential in compiler design and program analysis, yet their comprehension by Large Language Models (LLMs) remains underexplored. This paper presents a pioneering empirical study to investigate the capabilities of LLMs, including GPT-4, GPT-3, Gemma 2, LLaMA 3.1, and Code Llama, in understanding IRs. We analyze their performance across four tasks: Control Flow Graph (CFG) reconstruction, decompilation, code summarization, and execution reasoning. Our results indicate that while LLMs demonstrate competence in parsing IR syntax and recognizing high-level structures, they struggle with control flow reasoning, execution semantics, and loop handling. Specifically, they often misinterpret branching instructions, omit critical IR operations, and rely on heuristic-based reasoning, leading to errors in CFG reconstruction, IR decompilation, and execution reasoning. The study underscores the necessity for IR-specific enhancements in LLMs, recommending fine-tuning on structured IR datasets and integration of explicit control flow models to augment their comprehension and handling of IR-related tasks.
Jiang, Hailong↗
RLMolLM: Reinforcement Learning-Enhanced Language Model Framework for Inverse Molecular Design
Inverse molecular design faces significant challenges due to vast chemical space and complex property requirements. While language models show promise for molecular generation, they struggle with validity, multi-property optimization, and structural constraints. This work presents RLMolLM, a reinforcement learning framework combining Proximal Policy Optimization (PPO) with genetic algorithms to address these limitations. Our approach optimizes multiple user-specified properties including quantitative estimates of drug-likeness (QED), synthetic accessibility (SA), and ADMET (absorption, distribution, metabolism, excretion, and toxicity) endpoints without requiring complete model retraining, while maintaining capability for scaffold-constrained generation where specific substructures must be preserved. We outperform state-of-the-art methods for molecular optimization, achieving best QED scores across GDB13, Moses, and Zinc datasets with up to 31% improvement over previous methods while maintaining excellent validity, uniqueness, and novelty metrics. For simultaneous multi-property optimization, our framework achieves substantial improvements in ADMET properties including 4.5-fold reduction in hERG toxicity and enhanced Caco-2 permeability compared to Moses dataset. Under structural constraints, the framework significantly improves molecular validity while preserving scaffolds and effectively optimizing properties. In conclusion, this versatile solution advances pharmaceutical and materials molecular design through effective integration of reinforcement learning and genetic algorithms with multi-property optimization and scaffold preservation.
Genetic algorithms↗
Towards Next-Generation Urban Decision Support Systems through AI-Powered Construction of Scientific Ontology Using Large Language Models—A Case in Optimizing Intermodal Freight Transportation
The incorporation of Artificial Intelligence (AI) models into various optimization systems is on the rise. However, addressing complex urban and environmental management challenges often demands deep expertise in domain science and informatics. This expertise is essential for deriving data and simulation-driven insights that support informed decision-making. In this context, we investigate the potential of leveraging the pre-trained Large Language Models (LLMs) to create knowledge representations for supporting operations research. By adopting ChatGPT-4 API as the reasoning core, we outline an applied workflow that encompasses natural language processing, Methontology-based prompt tuning, and Generative Pre-trained Transformer (GPT), to automate the construction of scenario-based ontologies using existing research articles and technical manuals of urban datasets and simulations. From these ontologies, knowledge graphs can be derived using widely adopted formats and protocols, guiding various tasks towards data-informed decision support. The performance of our methodology is evaluated through a comparative analysis that contrasts our AI-generated ontology with the widely recognized pizza ontology, commonly used in tutorials for popular ontology software. We conclude with a real-world case study on optimizing the complex system of multi-modal freight transportation. Our approach advances urban decision support systems by enhancing data and metadata modeling, improving data integration and simulation coupling, and guiding the development of decision support strategies and essential software components.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗