Search NASA⌕ Search

SEARCH · Search NASA

Results for “Python computer language”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Integrating ORNL’s HPC and Neutron Facilities with a Performance-Portable CPU/GPU Ecosystem

We explore the development of a performance-portable CPU/GPU ecosystem to integrate two of the US Department of Energy’s (DOE’s) largest scientific instruments, the Oak Ridge Leadership Computing facility and the Spallation Neutron Source (SNS), both of which are housed at Oak Ridge National Laboratory. We select a relevant data reduction workflow use-case to obtain the differential scattering cross-section from data collected by SNS’s CORELLI and TOPAZ instruments. We compare the current CPU-only production implementation using the Garnet Python multiprocess package based on the Mantid C++ framework against our proposed CPU/GPU implementation that uses the LLVM-based, just-in-time Julia scientific language and the JACC.jl performance-portable package. Two proxy apps were developed: (i) an app for extracting relevant Mantid kernels (MDNorm) in C++ and (ii) the Julia MiniVATES.jl miniapp. We present performance results for NVIDIA A100 and AMD MI100 GPUs and AMD EPYC 7513 and 7662 CPUs. The results provide insights for future generations of data reduction software that can embrace performance portability for an integrated research infrastructure across DOE’s experimental and computational facilities.

Hahn, Steven↗
NASA NTRS2007

Solving Equations of Multibody Dynamics

Darts++ is a computer program for solving the equations of motion of a multibody system or of a multibody model of a dynamic system. It is intended especially for use in dynamical simulations performed in designing and analyzing, and developing software for the control of, complex mechanical systems. Darts++ is based on the Spatial-Operator- Algebra formulation for multibody dynamics. This software reads a description of a multibody system from a model data file, then constructs and implements an efficient algorithm that solves the dynamical equations of the system. The efficiency and, hence, the computational speed is sufficient to make Darts++ suitable for use in realtime closed-loop simulations. Darts++ features an object-oriented software architecture that enables reconfiguration of system topology at run time; in contrast, in related prior software, system topology is fixed during initialization. Darts++ provides an interface to scripting languages, including Tcl and Python, that enable the user to configure and interact with simulation objects at run time.

Jain, Abhinandan↗
NASA NTRS2026

Hydrology Copilot: A Cloud-Native Ai System for Hydrological Data Analysis

The emergence of AI-driven Earth observation systems promises to broaden access to petabyte-scale geospatial data beyond domain specialists. However, translating this vision into operational scientific infrastructure requires addressing fundamental challenges in data virtualization, code transparency, and domain-specific reasoning. We present Hydrology Copilot, a cloud-native AI framework for natural-language-driven analysis of Earth observation data. To demonstrate operational capabilities at scale, we implement the system using NASA's North American Land Data Assimilation System version 3 (NLDAS-3), which provides surface meteorological forcing and land-surface model output across North and Central America at 1-km resolution, from which drought diagnostics are derived. The system integrates five core contributions: (1) scalable data virtualization using Kerchunk-based cloud optimized access, achieving a 1.5 to 4.6 times improvement in I/O latency across benchmark queries spanning regional single-day extractions (4.6 times speedup) to continental monthly aggregations (1.5 times speedup); (2) transparent code generation through Microsoft Azure AI Foundry agents that expose executable Python workflows for scientific verification; (3) persistent conversational memory enabling multi-turn analytical discourse across sessions; (4) intelligent query validation that enforces dataset boundaries and resolves ambiguous requests before execution; and (5) a multi-agent architecture coordinating query parsing, code generation, and visualization. We evaluate the system through drought-monitoring workflows, demonstrating reliable code generation, accurate results validated against reference computations and the operational U.S. Drought Monitor, and efficient operation across increasingly complex tasks. By bridging natural-language interfaces with rigorous hydrological analysis, Hydrology Copilot advances beyond proof-of-concept demonstrations to provide a deployable framework for operational Earth science applications.

Data virtualization↗
DOE OSTI2025

Accelerating Floating-Point Computations with Intel AMX

Intel AMX is a built-in component of recent Intel CPU architectures, first supported by the Intel Sapphire Rapids in 2023, that enables efficient dense matrix multiplications using mixed precision with low-precision data types. The popularity of mixed-precision algorithms has grown recently, primarily due to their use on GPUs to enhance the efficiency of HPC applications, particularly for the training of large language models. The availability of mixed precision on CPUs represents a cost-effective solution for applications where high speed is not critical. This report shows how to use the Intel AMX accelerator through examples in C++ and Python. The examples will focus on mixed-precision floating-point operations obtained by the use of bfloat16 (or BF16) to accelerate code in single precision. We employ a bottom-up methodology, starting from specific register instructions (TMUL operation) to higher-level applications in libraries such as Intel MKL, PyTorch, and TensorFlow, ensuring a comprehensive understanding of the accelerator's potential. Additionally, we provide insights into the expected performance gains when leveraging the accelerator on the Kestrel HPC machine at the National Renewable Energy Laboratory.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2006

Simulating Responses of Gravitational-Wave Instrumentation

Synthetic LISA is a computer program for simulating the responses of the instrumentation of the NASA/ESA Laser Interferometer Space Antenna (LISA) mission, the purpose of which is to detect and study gravitational waves. Synthetic LISA generates synthetic time series of the LISA fundamental noises, as filtered through all the time-delay-interferometry (TDI) observables. (TDI is a method of canceling phase noise in temporally varying unequal-arm interferometers.) Synthetic LISA provides a streamlined module to compute the TDI responses to gravitational waves, according to a full model of TDI (including the motion of the LISA array and the temporal and directional dependence of the arm lengths). Synthetic LISA is written in the C++ programming language as a modular package that accommodates the addition of code for specific gravitational wave sources or for new noise models. In addition, time series for waves and noises can be easily loaded from disk storage or electronic memory. The package includes a Python-language interface for easy, interactive steering and scripting. Through Python, Synthetic LISA can read and write data files in Flexible Image Transport System (FITS), which is a commonly used astronomical data format.

Armstrong, John↗
NASA NTRS2012

HURON (HUman and Robotic Optimization Network) Multi-Agent Temporal Activity Planner/Scheduler

HURON solves the problem of how to optimize a plan and schedule for assigning multiple agents to a temporal sequence of actions (e.g., science tasks). Developed as a generic planning and scheduling tool, HURON has been used to optimize space mission surface operations. The tool has also been used to analyze lunar architectures for a variety of surface operational scenarios in order to maximize return on investment and productivity. These scenarios include numerous science activities performed by a diverse set of agents: humans, teleoperated rovers, and autonomous rovers. Once given a set of agents, activities, resources, resource constraints, temporal constraints, and de pendencies, HURON computes an optimal schedule that meets a specified goal (e.g., maximum productivity or minimum time), subject to the constraints. HURON performs planning and scheduling optimization as a graph search in state-space with forward progression. Each node in the graph contains a state instance. Starting with the initial node, a graph is automatically constructed with new successive nodes of each new state to explore. The optimization uses a set of pre-conditions and post-conditions to create the children states. The Python language was adopted to not only enable more agile development, but to also allow the domain experts to easily define their optimization models. A graphical user interface was also developed to facilitate real-time search information feedback and interaction by the operator in the search optimization process. The HURON package has many potential uses in the fields of Operations Research and Management Science where this technology applies to many commercial domains requiring optimization to reduce costs. For example, optimizing a fleet of transportation truck routes, aircraft flight scheduling, and other route-planning scenarios involving multiple agent task optimization would all benefit by using HURON.

Hua, Hook↗
DOE OSTI2025

Benchmark Tracking System for Performance Monitoring

Benchmarking is essential for high-performance software development, particularly for monitoring performance across code iterations. This project focused on enhancing the benchmarking process for Lamellar, an asynchronous runtime for High-Performance Computing (HPC) systems developed at Pacific Northwest National Laboratory. Prior to this work, benchmark results were difficult to track and compare across code versions, presenting significant challenges in identifying performance regressions and long-term trends. The primary objective was to establish a systematic, reproducible approach for measuring performance and detecting regressions following code commits. Our methodology involved three key components: standardizing benchmark outputs, implementing data versioning, and developing analysis tools. We standardized the benchmark output format to JSON Line records containing specific fields (execution time, hardware specifications, and environmental variables). To address data management challenges, we evaluated several options and eventually chose a git repository dedicated to benchmark data. We developed a suite of Python tools that processed benchmark results, enriched them with metadata, and facilitated search in the repository. The resulting system enables more efficient filtering and comparison of performance metrics across commit histories, hardware configurations, and benchmark variants through a unified query interface. Our implementation reduces computational overhead by first checking for existing results through configuration matching before initiating new benchmark runs, thereby conserving resources. The system has been validated by Lamellar developers. It organizes results by benchmark type and build configurations for efficient retrieval. Future developments include a planned Large Language Model interface for predicting benchmark performance, incorporating the criterion package for statistical analysis, which will enable automated detection of statistically significant performance changes, and integration with continuous integration pipelines. Despite these enhancements being reserved for future work, this project has successfully provided the Lamellar development team with a framework for maintaining consistent performance standards and identifying optimization opportunities across workloads and hardware environments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

MontePy: a Python library for reading, editing, and writing MCNP input files.

The Monte Carlo N-Particle (MCNP) radiation transport code is a highly capable and accurate code with a long legacy. MCNP uses the Monte Carlo simulation process to simulate the path of particles (e.g., neutrons, photons, charged particles, etc.), and their interaction with materials. It is widely used in nuclear engineering, high-energy physics, and other fields. Its origins in the mid-twentieth century predate many modern software conventions. MCNP users provide an input file to MCNP, which it then uses to create an internal representation of the simulation problem. These input files originally had to be stored as punchcard decks, and the user manual still uses the terminology of cards and decks, despite moving beyond punchcards. MCNP predates nearly all modern human readable markup or data serialization languages, such as the extensible Markup Language (XML), the Standard Generalized Markup Language (SGML), YAML (YAML Ain’t Markup Language), and Javascript Object Notation (JSON). Due to this, MCNP uses an entirely custom defined syntax language for its input, making off-the-shelf libraries for XML, YAML, and JSON impossible to use for scripting various operations on MCNP input files (Kulesza et al., 2022).

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Data From: "Warming and snow loss increase reliance on old groundwater in a Colorado River headwater"

This repository contains the data and code associated with the paper titled "Warming and snow loss increase reliance on old groundwater in a Colorado River headwater," published in Nature Geoscience, 2026. This study seeks to answer how various ages of groundwater interact with mountainous streamflow in mountainous headwaters such as the East River. It includes various model-data processing scripts, primarily for ParFlow-CLM analysis of simulated water years 2015-2021, and two numerical warming experiments (+2.5 and +4.0 degrees C), including run scripts, forcing scripts, and post-processing, as well as comparison to observation datasets, detailed below. This data requires the use of R (.r, .rmd), Python (.py), Jupyter Notebook or Jupyter Lab (.ipynb), ParFLOW-CLM, EcoSLIM. Further information on the use of all file formats mentioned below (e.g. .tff. .nc) are provided within the associated scripts and directory where the files are located. Contents & Usage ASO/: ​​Contains the bash and python scripts used to convert airborne snow observatory (ASO) data (ASO, 2023) in various data formats (georeferenced tiff file, NetCDF, UTM, and to latitude/longitude) then regrided to the ParFlow equivalent grid. Output data are in regrid_regll_data.zip and subsequently visualized and analyzed in plot_and_compare.py for Supplementary Figures A14 and A15. The wksht_ASO_comparison.xlsx spreadsheet is used to calculate the data for Supplementary Figure A16. EcoSLIM/: Contains the scripts and input files to run the EcoSLIM particle tracking simulations (/run_scripts) and the post-processing python script (/plot_scripts/eco_agedist_plots.ipynb). Jasechko et al./: Contains the jupyter notebook (Extract_Elevation.ipynb) to determine the outlet elevations of the 260 watersheds used in Jasechko et al. (2016), and the corresponding table, Table_S1_Watersheds_alt.csv. Used to create Supplementary Information Figure A2. PLM_Wells/: Contains the QA/QC-ed groundwater level time series of the PLM-1 and PLM-6 Monitoring Wells from Faybishenko et al. (2023), reformatted to water years used for Supplementary Figures A19 and and A20. ParFlow/: Contains the input files and run scripts to run ParFlow-CLM (/run_scripts), the python and tool command language (Tcl) scripts to create and distribute the ParFlow forcing simulation files (/forcing), and various scripts and intermediary files to analyze the model outputs (/post_process). SQUIRE/: Contains the processing scripts and intermediary files for the Surface QUantitatIve pRecipitation Estimation (SQUIRE) data (Grover, 2023) used to generate Supplementary Figure A18. USGS_Streamflow/: Contains the raw and gap-filled United States Geological Survey streamflow data (U.S. Geological Survey, 2026) used at the Almont station (site number 09112500). Gap-filling is performed in the R script with data from the Taylor station (site number 09110000). (/USGS_09112500_EAST_RIVER_AT_ALMONT_GAP_FILLED/code_almont_streamflow_gap_fill.Rmd). discharge/: Contains the gap-filled discharge data at the Watershed Function SFA East River pumphouse site (Newcomer et al., 2022) used to generate Supplementary Figure A13 and to compute hourly Nash-Sutcliffe model efficiency coefficients (NSE) in Table A4. snotel_and_flux_tower/: Contains the snow telemetry data (U.S. Department of Agriculture, 2024) from the Butte (site ID 380) and Schofield (site ID 737) stations, reformatted by water year, accessed with the snotelr R package. Used to create Supplementary Figure A17. Also contains the flux tower observational data (FluxTower_Pumphouse_ESS-DIVE.ET_only.h.txt) from Ryken et al. (2022) and sap flux transpiration data (MaxB_Transpiration_5Sites.daily_sums.h.txt) from Ryken (2021), used to create Supplementary Figures A22 and A23, respectively. Raw EcoSLIM model outputs are in excess of 24TB, and are stored on National Energy Research Scientific Computing Center (NERSC) and publicly available via the external link provided in the paper.

atmospheric warming↗
DOE OSTI2026

SetGo: Metadata Readiness for Scientific AI Datasets

Scientific datasets intended for AI use require both computational readiness for model training and metadata readiness for discovery, sharing, and reuse. The Readiness Engine for Data Integration (REDI) addresses computational readiness, but no corresponding tool evaluates whether a dataset’s metadata are sufficiently complete, governed, and standards-compliant for publication and agent-based consumption. Existing FAIR assessors operate only on published repository records, and no single system covers FAIR compliance, licensing, provenance, governance, reproducibility, and catalog readiness together. We present SetGo, an open-source Python toolkit that assesses and repairs metadata readiness across these six dimensions before a dataset is published or archived. Applied to four scientific corpora, SetGo surfaces deficiencies that general-purpose tools do not detect: ERA5 climate metadata scores 4% on ACDD 1.3 compliance; materials datasets fail OPTIMADE species-definition requirements; and PDB-derived proteomics data carries licensing terms incompatible with standard SPDX identifiers. Guided enrichment raises overall FAIR scores from 52–57% to 81–91%, and a single setgo publish command pushes to Hugging Face Hub, CKAN, or OpenMetadata with ML Commons Croissant 1.0 metadata sidecars. To support interactive and automated workflows, SetGo integrates with coding agents powered by large language models (LLMs) through a /setgo skill that enables natural-language execution of the full assess–enrich–publish loop, with user involvement limited to supplying missing metadata values.

Wilkinson, Sean [ORNL] (ORCID:0000000214437479)↗
DOE OSTI2026

TRACE Input Modernization

This work presents a Tom’s Obvious Minimal Language (TOML)-based representation of input for the US Nuclear Regulatory Commission’s TRAC/RELAP Advanced Computational Engine (TRACE) thermal hydraulics code. Implemented using the Workbench Analysis Sequence Processor (WASP), the approach maps traditional TRACE input structures to a hierarchical format composed of named parameters, typed values, and native data collections. The resulting representation preserves TRACE’s existing modeling capabilities while providing a modern, structured interface for model development and management. WASP further extends TOML through a file import directive that supports modular model composition and reusable input organization. In addition, WASP provides extended array data entry convenience with various data repeat and interpolation capabilities. Examples of the new TOML syntax are provided for major TRACE input categories, including hydraulic components, heat structures, control systems, and trip logic. The TOML representation establishes a foundation for improved validation, tooling, automation, and model maintainability while remaining compatible with existing TRACE workflows. To facilitate migration to the TOML-based input format, the TRACE executable now supports conversion of native TRACE input into an intermediate JSON representation. A Python utility subsequently transforms the JSON data into an equivalent TOML model. Lastly, the TRACE executable now supports execution using TOML-formatted input.

Lefebvre, Robert A. [Oak Ridge National Laboratory↗
DOE OSTI2026

BASIN-3D Data Integration for Selected ARM Data Field Campaign Report

The purpose of this data services request was to demonstrate integration of the Atmospheric Radiation Measurement (ARM) User Facility’s “met” datastreams with time series data from other earth science data sources using the BASIN-3D data synthesis software tool. BASIN-3D is an open-source Python library that enables researchers to integrate data across configured public and private data sources. It provides a common query language for researchers to request measurement locations and time series data based on specified locations, variables, time period, statistics, aggregation, and data quality. BASIN-3D acquires the data that match the query from each configured data source and translates the results into harmonized vocabularies, thus reducing researchers' data-wrangling effort. In addition, because the queries are executed on demand, researchers can easily regenerate their synthesized data sets as new data and/or data updates become available, eliminating one-off data products. BASIN-3D can output data using a variety of different data structures for end-user applications including Python pandas data frames and hdf5 output formats.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗
DOE OSTI2025

RANGE: A robust adaptive nature-inspired global explorer of potential energy surfaces

With the growing demand for realistic representations of chemical structures and the advent of exascale computing, the intelligent sampling of potential energy surfaces and efficient identification of global minima have become more essential but also more feasible. Building on prior studies demonstrating the efficiency of the Artificial Bee Colony (ABC) swarm intelligence algorithm, we report a hybrid metaheuristic framework that integrates the adaptive exploration capabilities of ABC coupled with the exploitation strengths of genetic algorithms (GA) in a scalable, Python-based implementation. The resulting tool, RANGE (Robust Adaptive Nature-inspired Global Explorer), provides seamless interfaces to multiple potential energy evaluators, either directly or via widely used Python libraries, and is designed for high-performance computing environments. We describe the implementation details of RANGE and evaluate its performance, relative to ABC- or GA-alone based algorithms, on a variety of chemical systems, including molecular clusters and heterogeneous surfaces. In conclusion, our results demonstrate RANGE’s efficiency, robustness, and broad applicability in addressing challenging global optimization problems in computational chemistry and materials science.

Algorithms and data structure↗
DOE OSTI2026

PVDeg: Enhancing Usability and AI-Driven Multi-Mechanism Degradation Modeling

PVDeg version 0.7.0, released in December 2025, introduced major enhancements to improve usability and performance. This update reorganized tutorials and tool notebooks to create a more intuitive experience, enabling users to easily follow and adapt workflows for their specific analyses. In addition to structural improvements, both the notebooks and core logic underwent significant optimization for efficiency, robustness, and style. These refinements were supported by new testing frameworks built on nbval and pytest, adherence to PEP8 standards, and extensive code refactoring, which collectively simplify onboarding for new developers. Looking ahead, version 0.8.0 will deliver advanced AI-driven capabilities. The primary focus is to further develop and automate the degradation workflow, designed to analyze PV module degradation across diverse locations and system configurations. By integrating large language models (LLMs) to scan literature and compile a comprehensive database of materials and degradation rates, this feature will enable modeling of multiple materials and mechanisms within a single, streamlined workflow. Users will be able to evaluate degradation impacts on different system architectures under varying environmental conditions, facilitating informed decisions on bill-of-materials optimization for specific deployment scenarios. These advancements position PVDeg as a powerful, user-friendly tool for accelerating PV reliability research and system design.

14 SOLAR ENERGY↗
DOE OSTI2026

pyRMG: A framework for high-throughput, large-cell DFT calculations on supercomputers

Exascale computing delivers the raw power to simulate ever larger and more chemically realistic systems, but realizing this potential requires codes that can efficiently use thousands of processors. Our real-space multigrid (RMG) density functional theory (DFT) code’s grid-decomposition approach scales nearly linearly with the number of graphics processing units (GPUs), even for simulations exceeding thousands of atoms. This scalability makes RMG a compelling tool for high-throughput DFT studies of materials that would otherwise be bottlenecked in other codes (for example, by global fast Fourier transforms in plane-wave DFT). However, the limited workflow infrastructure for RMG has thus far constrained its adoption to a small user community. In this work, we present pyRMG, a Python package designed to streamline the setup and execution of RMG DFT calculations. Built on the pymatgen and ASE (Atomic Simulation Environment) computational materials science Python packages, pyRMG automates input generation and convergence checking, and it integrates with modern job schedulers (e.g., Flux) on leadership-class platforms such as Frontier and Perlmutter. Here, we demonstrate pyRMG for a high-throughput study of strain effects in 2D 2L-Bi 2 Se 3 /2L-NbSe 2 heterostructures, which offers chemical insights into this system and shows that RMG-based workflows can converge with limited user intervention.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models

Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models (LLMs) have been incorporated into the training dataset of the models. This gives a false sense of performance that can overestimate how these models will function on truly unseen data. This problem becomes worse with commercial LLMs with larger and non-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box (text access only) and grey box (access to model loss/probabilities required) techniques and apply them to both commercial and non-commercial models. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness (or ineffectiveness) of these contamination detection techniques. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination (broadly) in fine-tuned models when both a baseline and fine-tuned model is present. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar (e.g., drawn from the same set of code projects).

97 MATHEMATICS AND COMPUTING↗