Dodge, Haley Diane | Search NASA

DOE OSTI2024

Predictive Indicators of the Performance of Large Language Models

In several mission contexts, it is desirable to estimate the performance of large language models (LLMs) on tasks that we cannot run directly. In light of published “scaling laws” our hypothesis is that some tasks should be consistently more challenging than others based on characteristics of the task. The goal of this project was to begin quantifying how much information about LLM performance can be gained from the features of a model and a task. Two of our statistical models struggled to converge. Pass/fail test results may provide limited information for inference beyond model quality and task difficulty, but we see no evidence at this time for significant feature interaction effect sizes, arguing for simple models. Future work extending the models to capitalize on perplexity of ground truth answers is suggested. This project also introduces “Depth of Knowledge Variant Testing” as a strategy for more finely assessing language models on open domain question and answer tasks. We developed sets of questions that ask a language model to produce similar information while demonstrating increasing depth of knowledge, and also relabeled existing Q&A test questions with their depth of knowledge. Our results suggest further consideration of Bloom’s taxonomy and further refinement of prompts to properly elicit information at varying depths. In the course of this work, we set up a basic infrastructure for standardizing tasks and testing many language models on these tasks. In addition to testing the predictive quality of model features and performance across test suites, with this project we have introduced two new task features to contextualize each test question: the Dewey Classification main category of information covered, and the Bloom’s taxonomy level that corresponds to the depth of knowledge probed by the question. Splits across these and other features produced over five hundred task subtypes with distinct feature vectors, which we tested on half a dozen models.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Stingray Case Study

Improvised explosive devices (IEDs) have injured and killed numerous soldiers and civilians as a consequence of military operations in the Middle East. With gaps in existing technologies, the US military required devices for quickly addressing deadly IEDs without harming military personnel or inflicting severe damage to the environment. In response to this need, Sandia developed Stingray, a clear, plastic handheld device to quickly and safely disable threatening IEDs. Stingray is designed to be used in two configurations: a coherent water blade for cutting operations and as a water slug for general device disruption. Prior to disabling an IED, Explosive Ordnance Disposal (EOD) technicians will x-ray the target using tools such as Sandia's X-ray Toolkit (XTK) to determine which function operators should use to dismantle the IED. For example, if the operator knows exactly which wires to cut, they can use the precision water blade. If the operator wants to create a general disruption, they can use the water slug function.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

DOE OSTI2020

GEMINI Case Study

Sandia's GEMINI-Scout Mine Rescue Robot is an unmanned ground vehicle designed to enter potentially hazardous environments to explore, assess, and evaluate dangerous situations first responders may face when conducting a rescue mission. GEMINI is approximately four feet long and two feet tall, which enables the robot to maneuver through small locations on rough terrains caused by earthquakes, fires, or radiological incidents. GEMINI uses track propulsion to climb stairs, travel through gravel and sand pits, pivot in place, and traverse 45-degree climbs with few problems. Furthermore, the vehicle's dual tracked-chassis design allows it to operate in hostile, dark, muddy, high-temperature, and explosive debris-strewn environments, while maintaining efficient ground mobility. The mobility and modularity of the vehicle allow for easy integration of sensors to conduct gas and temperature sensing and offers pan/tilt, zoom color, and thermal camera video streaming capabilities. The vehicle is also able to carry a payload of about 50 pounds of batteries and can handle an additional 200 pounds of payload, whether for additional diagnostics, supplies, or clothing for those trapped in an effected area. GEMINI is remotely operated through a wireless connection and an onboard computer running a customized embedded control application, which directly communicates to all onboard components except for the audio and video systems. When line of sight is not possible, operators use a shockresistant fiber optic cable to ensure continuous functionality of the vehicle. This allows for direct local control of the vehicle, which streams collected data back to the operator for enhanced situational awareness. In addition, the vehicle incorporates safety features such as explosion proof housing to ensure safe electronic operations in hazardous gas or flooded environments; a four-channel video link and two-way audio to ensure located survivors can communicate with operators; and an MSHA-approved multi-gas sensor to monitor air quality.

42 ENGINEERING↗

DOE OSTI2020

MRDIS Case Study

The Mobile Radiation Detection and Identification System (MRDIS) is a large mobile scanner that inspects containers in transit from cargo ships for radiological materials. The MRDIS platform operates as a two-part system with one MRDIS using a plastic Polyvinyl Toleune (synthetic polymer) for primary detection and another MRDIS that uses spectroscopic detectors for secondary isotopic identification. MRDIS can operate either independently or as part of a team, depending on the needs of the port. MRDIS is controlled by a human operator, who searches the computer monitor for any traces of radiological materials when the containers pass through the center of the system. Each MRDIS can also feed data into a central system or collect data on its own for additional material analysis. The system integrates radiation detection, radioisotope identification, an optical character recognition system, occupancy/speed sensors, wireless communications, and data processing capabilities to discern what specific radiological materials are of particular interest. In addition, Sandia engineers created a detailed set of requirements for subsequent models, allowing for faster implementation of additional detection systems.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2020

Xyce Case Study

With the elimination of underground nuclear testing and declining defense budgets, science-based stockpile stewardship requires increased reliance on high performance modeling and simulation of weapon systems. Today's weapon systems are comprised of various electrical components and systems. As a result, there is a need for tools that will allow the use of massively parallel modeling and simulation techniques on high performance computers in existing and future weapons' electrical systems models. The Xyce Parallel Electronic Simulator is a SPICE (Simulation Program with Integrated Circuit Emphasis)- compatible circuit simulator designed to run on large-scale parallel computing platforms, though it can also execute efficiently on a variety of architectures including single processor workstations. As a mature platform for large-scale parallel circuit simulation, Xyce supports standard capabilities available in commercial simulators, in addition to various devices and models specific to Sandia's needs. Specifically, Xyce aids in the design and verification of electrical and electronic circuits and systems prior to weapons' manufacturing and deployment.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

DOE OSTI2020

Xyce Case Study

With the elimination of underground nuclear testing and declining defense budgets, science-based stockpile stewardship requires increased reliance on high performance modeling and simulation of weapon systems. Today's weapon systems are comprised of various electrical components and systems. As a result, there is a need for tools that will allow the use of massively parallel modeling and simulation techniques on high performance computers in existing and future weapons' electrical systems models. The Xyce Parallel Electronic Simulator is a SPICE (Simulation Program with Integrated Circuit Emphasis)- compatible circuit simulator designed to run on large-scale parallel computing platforms, though it can also execute efficiently on a variety of architectures including single processor workstations. As a mature platform for large-scale parallel circuit simulation, Xyce supports standard capabilities available in commercial simulators, in addition to various devices and models specific to Sandia's needs. Specifically, Xyce aids in the design and verification of electrical and electronic circuits and systems prior to weapons' manufacturing and deployment.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗