Search NASA⌕ Search

SEARCH · Search NASA

Results for “metric categorization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2015

Probability of Loss of Crew Achievability Studies for NASA's Exploration Systems Development

Over the last few years, NASA has been evaluating various vehicle designs for multiple proposed design reference missions (DRM) beyond low Earth orbit in support of its Exploration Systems Development (ESD) programs. This paper addresses several of the proposed missions and the analysis techniques used to assess the key risk metric, probability of loss of crew (LOC). Probability of LOC is a metric used to assess the safety risk as well as a design requirement. These assessments or studies were categorized as LOC achievability studies to help inform NASA management as to what "ball park" estimates of probability of LOC could be achieved for each DRM and were eventually used to establish the corresponding LOC requirements. Given that details of the vehicles and mission are not well known at this time, the ground rules, assumptions, and consistency across the programs become the important basis of the assessments as well as for the decision makers to understand.

Boyer, Roger L.↗
DOE OSTI2026

Public Reference Data for Megawatt-Scale Hydrogen Electrolysis - NLR Historical Solar PV

The U.S. Department of Energy and National Laboratory of the Rockies (NLR) demonstrate hydrogen electrolysis from variable sources, hydrogen compression and storage, and hydrogen fuel cell power production using megawatt-scale equipment at NLR’s Flatirons Campus as part of the Advanced Research on Integrated Energy Systems (ARIES) research platform. This dataset represents part of that effort and is intended for academic, national laboratory, industrial, and other stakeholders to plan, design, and validate models of megawatt-scale hydrogen technologies and diverse energy infrastructure nationwide. These data provide a baseline for how existing hydrogen electrolysis technologies perform when coupled with various energy technologies. Future datasets will demonstrate how existing hydrogen fuel cell technologies can provide controllable, dispatchable, and variable power output for artificial intelligence data centers and other variable loads. This dataset entry describes the behavior of a 1.25-MW proton exchange membrane MC250 electrolyzer system, manufactured by Nel Hydrogen , [1] when fed historical data generated by the 430-kW, fixed-axis solar photovoltaic (PV) array located at NLR’s Flatirons Campus. (While the electrolyzer balance of plant supports up to 2.5 MW of electrolysis, NLR only has a single 1.25-MW electrolysis stack.) Solar PV power output data for the 2020 calendar year were categorized on a daily basis by total energy generation and standard deviation. Each day was then ranked by these metrics, and the 25th, 50th, and 100th percentiles were selected. The 75th percentile day did not exhibit sufficient variability to make for a valuable experiment. A similar process was used for the related historical wind dataset . [2] The historical days in 2020 that represented these percentiles are Dec. 19, March 29, and May 4, respectively. The entire solar day’s power profile was then fed through the MC250 electrolyzer. Due to its length, the 100th percentile day experiment was split into two parts, and the final 3 hours of the solar day were not captured. These final 3 hours contained no spikes or dips of interest and simply represented a slow decay of input solar power. Also, a single timestamp (13:13:47 on Jan. 14, 2026) was lost in the hydrogen system supervisory control and data acquisition. Finally, during the 25th percentile experiment (solar day Dec. 19, 2020) data recording was lost from 11:00:13 to 11:14:45. The roughly 15 minutes of the solar profile were rerun at the end of the experiment and spliced into this time slot during post-processing. The electrolysis system controls hydrogen production by varying direct current applied to the stack, from a maximum of 3,000 A to a minimum safe operation of 300 A, or 10%. Because the current–voltage characteristic changes as the stack ages and efficiency degrades, the actual minimum safe operating power changes over time. The historical solar profiles were translated from power (kilowatts) to current (amperes) using a curve fit with calibration data and sent to the electrolyzer power supply at 1-Hz frequency. For more details on the statistical analysis process, see the slide deck “Public Reference Data for Megawatt-Scale Hydrogen Electrolysis: NLR Historical Solar PV Analysis and Profile Generation” accessible with this data entry. These datasets report relevant hydrogen balance-of-plant and system data, all captured at 1 Hz, including hydrogen mass production measured with an Emerson Coriolis flow meter. Each .zip file represents a single solar PV electrolysis experiment and is formatted as: {technology}_{percentile}_{scaling factor} For instance, “solarPV-430kW_25_2x.zip” reports the experiment using the 25th percentile solar data from the historical 2020 solar PV dataset, scaled to 200%. Scaling factors were applied to the generated solar PV power output files to more closely match the 1.25-MW capacity of the electrolyzer. Each .zip folder contains the following files: A .csv file containing raw data. An .xlsx file explaining all the fields in the raw data. A .png plot showing the time series of hydrogen production, electrolysis power consumption, and solar power input. A PDF file detailing the historical solar data statistical analysis used to generate the solar profile. An experiment labeled “characterization_200.zip” demonstrates the MC250 electrolyzer steady-state response with 30-minute load steps for a total duration of 5 hours. Finally, a .csv file is provided with all experiments combined into one dataset labeled "combined_solarPV_experiments.csv". [1] nelhydrogen.com/product/mc-series-electrolyser . [2] data.nlr.gov/submissions/316 .

08 HYDROGEN↗
DOE OSTI2024

A network approach for multiscale catchment classification using traits

Abstract. The classification of river catchments into groups with similar biophysical characteristics is useful to understand and predict their hydrological behavior. The increasing availability of remote sensing and other large-scale geospatial datasets has enabled the use of advanced data-driven approaches to classify catchments using traits such as topography, geology, climate, land cover, land use, and human influence. Unsupervised clustering algorithms based on the Euclidean distance are commonly used for trait-based classification but are not suitable for highly dimensional data. In this study we present a new network-based method for multi-scale catchment classification, which can be applied to large datasets and used to determine the traits associated with different catchment groups. In this framework, two networks are analyzed in parallel: the first being where the nodes are traits and the second being where the nodes are catchments. In both cases, edges represent pairwise similarity, and a network cluster detection algorithm is used for the classification. The trait network is used to investigate redundancy in the trait data and to condense this information into a small number of interpretable categories. The catchments network is used to classify the catchments into clusters and to identify representative catchments for the different groups using the degree centrality metric. We apply this method to classify 9067 river catchments across the contiguous United States at both regional and continental scales using 274 non-categorical traits. At the continental scale, we identify 25 interpretable trait categories and 34 catchment clusters of sizes greater than 50. We find that catchments with similar trait categories are typically located in the same region, with different spatial patterns emerging among clusters dominated by natural and anthropogenic traits. We also find that the catchment clusters exhibit distinct hydrological behavior based on an analysis of streamflow indices. This network approach provides several advantages over traditional means of classification, including better separation of clusters, the use of alternate similarity metrics that are more suitable for highly dimensional data, and reducing redundancy in the trait information. The paired catchment–trait networks enable analysis of hydrological behavior using the dominant trait categories for each catchment cluster. The approach can be used at multiple spatial scales since the network topologies adjust automatically to reflect the trait patterns at the scale of investigation. Finally, the representative catchments identified as hub nodes in the network can be used to guide transferable observational and modeling strategies. The method is broadly applicable beyond hydrology for classification of other complex systems that utilize different types of trait datasets.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2014

Complexity and Pilot Workload Metrics for the Evaluation of Adaptive Flight Controls on a Full Scale Piloted Aircraft

Flight research has shown the effectiveness of adaptive flight controls for improving aircraft safety and performance in the presence of uncertainties. The National Aeronautics and Space Administration's (NASA)'s Integrated Resilient Aircraft Control (IRAC) project designed and conducted a series of flight experiments to study the impact of variations in adaptive controller design complexity on performance and handling qualities. A novel complexity metric was devised to compare the degrees of simplicity achieved in three variations of a model reference adaptive controller (MRAC) for NASA's F-18 (McDonnell Douglas, now The Boeing Company, Chicago, Illinois) Full-Scale Advanced Systems Testbed (Gen-2A) aircraft. The complexity measures of these controllers are also compared to that of an earlier MRAC design for NASA's Intelligent Flight Control System (IFCS) project and flown on a highly modified F-15 aircraft (McDonnell Douglas, now The Boeing Company, Chicago, Illinois). Pilot comments during the IRAC research flights pointed to the importance of workload on handling qualities ratings for failure and damage scenarios. Modifications to existing pilot aggressiveness and duty cycle metrics are presented and applied to the IRAC controllers. Finally, while adaptive controllers may alleviate the effects of failures or damage on an aircraft's handling qualities, they also have the potential to introduce annoying changes to the flight dynamics or to the operation of aircraft systems. A nuisance rating scale is presented for the categorization of nuisance side-effects of adaptive controllers.

pilot workload metrics↗
NASA NTRS2022

MLtool: Universal Supervised Machine Learning Tool to Model Tabulated Data

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine learning↗
NASA NTRS2022

MLtool Python Code

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine Learning↗
DOE OSTI2025

Interplay of Topography, Fire History, and Climate on Interior Alaska Boreal Forest Vegetation Dynamics in the 21st Century: A Landsat Time-Series Analysis

This study investigates vegetation dynamics in boreal forests of Interior Alaska, focusing on topography, fire history, and climate influences. The study area includes Bonanza Creek Experimental Forest (BCEF) and surrounding region, categorized by topography (upland, floodplain, lowland) and fire history. Using Mann–Kendall trend and Theil–Sen slope analyses on Landsat-derived spectral metrics: Normalized Difference Vegetation Index (NDVI) and Normalized Burn Ratio (NBR), we observed a shift from browning to greening trends, particularly in historically burned areas. The photosynthetic activity in burned upland converged with unburned areas ~30 years post-fire, coincident with a shift towards deciduous dominance during post-fire succession. Normalized Difference Moisture Index (NDMI) trends revealed a significant increase in vegetation moisture content across all topographies. We introduce Effective Seasonal Precipitation Index (ESPI), which combines prior-year annual precipitation with current-year spring snow depth. Its positive correlation with NDMI highlights its potential for monitoring vegetation moisture dynamics at the landscape scale. Furthermore, by correlating dendrochronology-based climate indices, we found strong correlation between NDMI and normalized Supplemental Precipitation Index (nSPI), across topographies. Overall, this research provides critical insights into how climate and fire influence interior boreal vegetation, highlighting the effects of increased precipitation, and topography on shaping differential vegetation responses across the landscape.

Google Earth Engine↗
NASA NTRS1985

FH/MFSK performance in multitone jamming

The performance of frequency-hopped (FH) M-ary frequency-shift keyed (MFSK) signals in partial-band noise was analyzed in the open literature. The previous research is extended to the usually more effective class of multitone jamming. Some objectives researched are: (1) To categorize several different multitone jamming strategies; (2) To analyze the performance of FH/MSFK signaling, both uncoded with diversity, assuming a noncoherent energy detection metric with linear combining and perfect jamming state side information, in the presence of worst case interference for each of these multitone categories; and (3) To compare the effectiveness of the various multitone jamming techniques, and contrast the results with the partial band noise jamming case.

Levitt, B. K.↗
NASA NTRS1985

FH/MFSK performance in multitone jamming

The performance of frequency-hopped (FH) M-ary frequency-shift keyed (MFSK) signals in partial-band noise as analyzed in the open literature. The previous research is extended to the usually more effective class of multitone jamming. Some objectives researched are: (1) To categorize several different multitone jamming strategies; (2) To analyze the performance of FH/MSFK signaling, both uncoded with diversity, assuming a noncoherent energy detection metric with linear combining and perfect jamming state side information, in the presence of worst case interference for each of these multitone categories; and (3) To compare the effectiveness of the various multitone jamming techniques, and contrast the results with the partial band noise jamming case.

Levitt, B. K.↗
NASA NTRS2003

The relationship between lifestyle regularity and subjective sleep quality

In previous work we have developed a diary instrument-the Social Rhythm Metric (SRM), which allows the assessment of lifestyle regularity-and a questionnaire instrument--the Pittsburgh Sleep Quality Index (PSQI), which allows the assessment of subjective sleep quality. The aim of the present study was to explore the relationship between lifestyle regularity and subjective sleep quality. Lifestyle regularity was assessed by both standard (SRM-17) and shortened (SRM-5) metrics; subjective sleep quality was assessed by the PSQI. We hypothesized that high lifestyle regularity would be conducive to better sleep. Both instruments were given to a sample of 100 healthy subjects who were studied as part of a variety of different experiments spanning a 9-yr time frame. Ages ranged from 19 to 49 yr (mean age: 31.2 yr, s.d.: 7.8 yr); there were 48 women and 52 men. SRM scores were derived from a two-week diary. The hypothesis was confirmed. There was a significant (rho = -0.4, p < 0.001) correlation between SRM (both metrics) and PSQI, indicating that subjects with higher levels of lifestyle regularity reported fewer sleep problems. This relationship was also supported by a categorical analysis, where the proportion of "poor sleepers" was doubled in the "irregular types" group as compared with the "non-irregular types" group. Thus, there appears to be an association between lifestyle regularity and good sleep, though the direction of causality remains to be tested.

NASA Program Biomedical Research and Countermeasur↗
DOE OSTI2021

Compensatory reserve detects subclinical shock with more expeditious prediction for need of life‐saving interventions compared to systolic blood pressure and blood lactate

Abstract Introduction We conducted a prospective observational study on 205 trauma patients at a level I trauma facility to test the hypothesis that a compensatory reserve measurement (CRM) would identify higher risk for progression to shock and/or need a life‐saving interventions (LSIs) earlier than systolic blood pressure (SBP) and blood lactate (LAC). Methods A composite outcome metric included blood transfusion, procedural LSI, and mortality. Discrete measures assessed as abnormal (ab) were SBP <90 mmHg, CRM <60%, and LAC >2.0. A graded categorization of shock was defined as: no shock (normal [n] SBP [n‐SBP], n‐CRM, n‐LAC); sub‐clinical shock (ab‐CRM, n‐SBP, n‐LAC); occult shock (n‐SBP, ab‐CRM, ab‐LAC); or overt shock (ab‐SBP, ab‐CRM, ab‐LAC). Results Three patients displayed overt shock, 53 displayed sub‐clinical shock, and 149 displayed no shock. After incorporating lactate into the analysis, 86 patients demonstrated no shock, 25 were classified as sub‐clinical shock, 91 were classified as occult shock, and 3 were characterized as overt shock. Each shock subcategory revealed a graded increase requiring LSI and transfusion. Initial CRM was associated with progression to shock (odds ratio = 0.97; p < .001) at an earlier time than SBP or LAC. Conclusions Initial CRM uncovers a clinically relevant subset of patients who are not detected by SBP and LAC. Our results suggest CRM could be used to more expeditiously identify injured patients likely to deteriorate to shock, with requirements for blood transfusion or procedural LSI.

Convertino, Victor A.↗
NASA NTRS2019

Quantitative Metrics from 20 Years of Terra Data Usage

NASA's Terra flagship satellite carries five Earth-observing instruments that have collected data for almost 20 years. NASA's Earth Science Data and Information System (ESDIS) Project makes these data, along with derived products, available to worldwide data users. Since the launch of Terra on December 18, 1999, more than 10,000 data products have been archived and distributed by NASA-funded Distributed Active Archive Centers (DAACs) that are part of NASA's Earth Observing System Data and Information System (EOSDIS). At the end of the 2018 Fiscal Year, about 1,000 Terra data products constituted almost 22% of the entire EOSDIS data archive volume (6 PB out of approximately 27.5 PB), and 6 PB of Terra data were distributed to over half-a-million public users worldwide.By categorizing the Terra data products and their distribution, we can get a quantitative assessment of Terra data usage. NASA's ESDIS Project has collected archive, distribution, and user information from EOSDIS data users since February 2000. These metrics are available through the ESDIS Metrics System (EMS). EMS information is stored in a relational database from which quantitative metrics of Terra data use can be retrieved and analyzed.The purposes of this study are to: 1) perform a comprehensive investigation of the 20-year trend in the archive and distribution of Terra data products; 2) identify and characterize data product usage over the last 20 years; and 3) identify and characterize the global user community for these data. In addition to revealing how Terra data use has evolved over time, the results of this study provide insights on identifying the various user communities for different kinds of Earth science data products. Also, because of the enormous quantity of data handled by EOSDIS DAACs, the study provides guidance of the requirements for future data systems that will be needed to effectively and efficiently handle the ever-increasing amounts of Earth science data produced by future (and ongoing) Earth science missions.

Wanchoo, Lalit↗
DOE OSTI2023

Computer Vision: EO Imaging & Video Tasks, Metrics, and Datasets [Slides]

Classification is an assignment of an image to a certain category. Localization is identified by the box surrounding the object in the image. Object detection detects instances of semantic objects of a certain class. Semantic segmentation categorizes all the pixels of an image into classes of objects. Instance segmentation identifies each instance of each object in the image. Panoptic (i.e., showing or seeing everything at once) segmentation is a combination of semantic and instance segmentation and classifies all the pixels in the image.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Use of Historical EM Practices and Policies for Management of Cleanup Program and Address GAO Concerns - 20222

As Department of Energy (DOE) Environmental Management (EM) program policies and management practices are revised over time, it might be useful to periodically reflect on the DOE EM program's past to identify practices that have worked and those that might be appropriate for possible reuse in the future. An added impetus to examine alternative approaches is the recent Government Accountability Office (GAO) report entitled 'Nuclear Waste Cleanup - DOE Could Improve Program and Project Management by Better Classifying Work and Following Leading Practices.' The February 2019 GAO report states that EM categorizes most of its work in a way that does not adequately involve independent experts and DOE senior leadership. It also noted that EM has not followed leading practices for program and project management that could help keep the cleanup efforts on schedule and control costs. The GAO report recommended that EM, working with DOE Office of Project Management (PM): - Establish requirements and then assess EM's ongoing operations activities to determine if some activities should be reclassified as capital asset projects based on these newly established requirements. - Review and revise EM's 2017 cleanup policy to include program management leading practices related to scope, cost, schedule performance, and independent reviews. - Update the cleanup policy to require that earned value management (EVM) systems be maintained and used in a way that follows EVM best practices. - Develop a policy to ensure that work is categorized as level of effort (LOE) only in appropriate, specified circumstances, such as when work is not measurable or when measurement is impractical. - Integrate EVM data into EM's performance metrics for operations activities. To assist with improving program performance and implementing audit recommendations, this paper reviewed past successful cleanups such as Weldon Springs, Rocky Flats, Fernald, Ashtabula, Battelle Columbus (abs, and other historical resources and people with experience during the early years of EM program. As such, there are several past practices which may well be suited for reuse today. Some of these practices include: - In 1990's, as the DOE EM cleanup program, known as Environmental Restoration came into effect, the known cleanup scope at each geographic location or site was organized into four levels ranging from overall site-wide cleanup effort to the individual 'release sites', providing high granularity to the program. - The cleanup program was organized by geography (Northwest, Southwest, Eastern Area Programs, etc.). These 'sub-program' managers provided a degree of Headquarters oversight and management allowing upper management (EM-1, 2, and 3) to focus on major decisions and issues. - Budget levels at selected cleanup sites were funded above 'minimum-safety' levels to accelerate actual cleanup work. - Past practice utilized EVM techniques and uniform EM oversight over all discrete project-like activities, providing for a coordinated cleanup effort. - EM managed LOE activities using benchmarking and best in class management practice techniques to help manage costs. - EM managed and reviewed entire site cleanup baselines, not only individual projects, focusing on a successful overall outcome. - Sites had a 'Road map' or 'Management Action Plan' that provided the overall strategy or plan for that site's cleanup - this helped with understanding and coordination of efforts. - EM had a robust lessons-learned program and database to capture and transmit worthy ideas. - EM had well-structured cleanup baseline change control policies to control cost and schedule changes, at the contractor and higher Headquarters levels. Finally, some of the GAO observations may be valid, however it is not clear as to their extent. For example, for some operation types of work, the use of the Level of Effort (LOE) earned value method to track progress may be appropriate. Also, to ensure independence of thought, and to obtain useful insight on past practices, it may be suitable to have former EM employees to assist with understanding past practices or to verify that activities have been properly identified. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2022

Review of Controllers for Low-Power Free-Piston Stirling Convertors

This work provides a complete review of controller strategies for free-piston Stirling convertors (FPSCs) and design recommendations for future controller development efforts. First, basic operating principles for the control of FPSCs and design variables for the FPSC controllers are described. Next, past controller approaches are reviewed and categorized based on the design variables. State-of-the-art controllers are also summarized by their key features, which are then hypothetically extended to flight designs to compare their critical metrics such as the estimated size, weight, and power with cost (SWaP-C) and reliability. Finally, design recommendations are suggested to improve the state-of-the-art controllers for future flight development projects.

Free-Piston Stirling Convertor Controller↗
DOE OSTI2020

Generation and evaluation of synthetic patient data

Background: Machine learning (ML) has made a significant impact in medicine and cancer research; however, its impact in these areas has been undeniably slower and more limited than in other application domains. A major reason for this has been the lack of availability of patient data to the broader ML research community, in large part due to patient privacy protection concerns. High-quality, realistic, synthetic datasets can be leveraged to accelerate methodological developments in medicine. By and large, medical data is high dimensional and often categorical. These characteristics pose multiple modeling challenges. Methods: In this paper, we evaluate three classes of synthetic data generation approaches; probabilistic models, classification-based imputation models, and generative adversarial neural networks. Metrics for evaluating the quality of the generated synthetic datasets are presented and discussed. Results: While the results and discussions are broadly applicable to medical data, for demonstration purposes we generate synthetic datasets for cancer based on the publicly available cancer registry data from the Surveillance Epidemiology and End Results (SEER) program. Specifically, our cohort consists of breast, respiratory, and non-solid cancer cases diagnosed between 2010 and 2015, which includes over 360,000 individual cases. Conclusions: We discuss the trade-offs of the different methods and metrics, providing guidance on considerations for the generation and usage of medical synthetic data.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Modern chemical graph theory

Abstract Graph theory has a long history in chemistry. Yet as the breadth and variety of chemical data is rapidly changing, so too do graph encoding methods and analyses that yield qualitative and quantitative insights. Using illustrative cases within a basic mathematical framework, we showcase modern chemical graph theory's utility in Chemists' analysis and model development toolkit. The encoding of both experimental and simulation data is discussed at various levels of granularity of information. This is followed by a discussion of the two major classes of graph theoretical analyses: identifying connectivity patterns and partitioning methods. Measures, metrics, descriptors, and topological indices are then introduced with an emphasis upon enhancing interpretability and incorporation into physical models. Challenging data cases are described that include strategies for studying time dependence. Throughout, we incorporate recent advancements in computer science and applied mathematics that are propelling chemical graph theory into new domains of chemical study. This article is categorized under: Molecular and Statistical Mechanics > Molecular Dynamics and Monte‐Carlo Methods Structure and Mechanism > Computational Materials Science Structure and Mechanism > Molecular Structures

Leite, Leonardo S. G.↗
DOE OSTI2025

Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models

Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models (LLMs) have been incorporated into the training dataset of the models. This gives a false sense of performance that can overestimate how these models will function on truly unseen data. This problem becomes worse with commercial LLMs with larger and non-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box (text access only) and grey box (access to model loss/probabilities required) techniques and apply them to both commercial and non-commercial models. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness (or ineffectiveness) of these contamination detection techniques. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination (broadly) in fine-tuned models when both a baseline and fine-tuned model is present. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar (e.g., drawn from the same set of code projects).

97 MATHEMATICS AND COMPUTING↗