Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78

GLM Proxy Data Generation: Methods for Stroke/Pulse Level Inter-Comparison of Ground-Based Lightning Reference Networks

In order to produce useful proxy data for the GOES-R Geostationary Lightning Mapper (GLM) in regions not covered by VLF lightning mapping systems, we intend to employ data produced by ground-based (regional or global) VLF/LF lightning detection networks. Before using these data in GLM Risk Reduction tasks, it is necessary to have a quantitative understanding of the performance of these networks, in terms of CG flash/stroke DE, cloud flash/pulse DE, location accuracy, and CLD/CG classification error. This information is being obtained through inter-comparison with LMAs and well-quantified VLF/LF lightning networks. One of our approaches is to compare "bulk" counting statistics on the spatial scale of convective cells, in order to both quantify relative performance and observe variations in cell-based temporal trends provided by each network. In addition, we are using microsecond-level stroke/pulse time correlation to facilitate detailed inter-comparisons at a more-fundamental level. The current development status of our ground-based inter-comparison and evaluation tools will be presented, and performance metrics will be discussed through a comparison of Vaisala s Global Lightning Dataset (GLD360) with the NLDN at locations within and outside the U.S.

Cummins, Kenneth L.↗

Characterizing the Vertical Profile of Aerosol Particle Extinction and Linear Depolarization over Southeast Asia and the Maritime Continent: The 2007-2009 View from CALIOP

Vertical profiles of 0.532 μm aerosol particle extinction coefficient and linear volume depolarization ratio are described for Southeast Asia and the Maritime Continent. Quality-screened and cloud-cleared Version 3.01 Level 2 NASA Cloud Aerosol Lidar with Orthogonal Polarization (CALIOP) 5-km Aerosol Profile datasets are analyzed from 2007 to 2009. Numerical simulations from the U.S. Naval Aerosol Analysis and Predictive System (NAAPS), featuring two-dimensional variational assimilation of NASA Moderate Resolution Imaging Spectroradiometer and Multi-angle Imaging Spectro- Radiometer quality-assured datasets, combined with regional ground-based lidar measurements, are considered for assessing CALIOP retrieval performance, identifying bias, and evaluating regional representativeness. CALIOP retrievals of aerosol particle extinction coefficient and aerosol optical depth (AOD) are high over land and low over open waters relative to NAAPS (0.412/0.312 over land for all data points inclusive, 0.310/0.235 when the per bin average is used and each is treated as single data points; 0.102/0.151 and 0.086/0.124, respectively, over ocean). Regional means, however, are very similar (0.180/0.193 for all data points and 0.155/0.159 when averaged per normalized bin), as the two factors offset one another. The land/ocean offset is investigated, and discrepancies attributed to interpretation of particle composition and a-priori assignment of the extinction-to-backscatter ratio ("lidar ratio") necessary for retrieving the extinction coefficient from CALIOP signals. Over land, NAAPS indicates more dust present than CALIOP algorithms are identifying, indicating a likely assignment of a higher lidar ratio representative of more absorptive particles. NAAPS resolvesmore smoke overwater than identified with CALIOP, indicating likely usage of a lidar ratio characteristic of less absorptive particles to be applied that biases low AOD there. Over open waters except within the Bay of Bengal, aerosol particle scattering is largely capped below 1.5 km MSL, though ground-based lidar measurements at Singapore differ slightly from this finding. Significant aerosol particle presence over land is similarly capped near 3.0 km MSL over most regions. Particle presence at low levels regionally, except over India, is dominated by relatively non-depolarizing particles. Industrial haze, sea salt droplets and fresh smoke are thus most likely present.

linear volume depolarization ratio↗

Meteorite Dust and Health - A Novel Approach for Determining Bulk Compositions for Toxicological Assessments of Precious Materials

With the resurgence of human curiosity to explore planetary bodies beyond our own, comes the possibility of health risks associated with the materials covering the surface of these planetary bodies. In order to mitigate these health risks and prepare ourselves for the eventuality of sending humans to other planetary bodies, toxicological evaluations of extraterrestrial materials is imperative (Harrington et al. 2017). Given our close proximity, as well as our increased datasets from various missions (e.g., Apollo, Mars Exploration Rovers, Dawn, etc…), the three most likely candidates for initial human surface exploration are the Moon, Mars, and asteroid 4Vesta. Seven samples, including lunar mare basalt NWA 4734, lunar regolith breccia NWA 7611, martian basalt Tissint, martian regolith breccia NWA 7034, a vestian basalt Berthoud, a vestian regolith breccia NWA 2060, and a terrestrial mid-ocean ridge basalt, were examined for bulk chemistry, mineralogy, geochemical reactivity, and inflammatory potential. In this study, we have taken alliquots from these samples, both the fresh samples and those that underwent iron leaching (Tissint, NWA 7034, NWA 4734, MORB), and performed low pressure, high temperature melting experiments to determine the bulk composition of the materials that were previously examined.

Vander Kaaden, K. E.↗

An Evaluation of Teleconnections Over the United States in an Ensemble of AMIP Simulations with the MERRA-2 Configuration of the GEOS Atmospheric Model

The atmospheric general circulation model that is used in NASA's Modern Era Retrospective Analysis for Research and Applications Version 2 (MERRA-2) is evaluated with respect to the relationship between large-scale teleconnection patterns and daily temperature and precipitation over the United States (US) using a ten-member ensemble of simulations, referred to as M2AMIP. A focus is placed on four teleconnection patterns that are known to influence weather and climate in the US: El Nino Southern Oscillation, the Pacific Decadal Oscillation, the North Atlantic Oscillation, and the Pacific-North American Pattern. The monthly and seasonal indices associated with the patterns are correlated with daily temperature and precipitation statistics including: (i) monthly mean 2 m temperature and precipitation, (ii) the frequency of extreme temperature events at the 90th, 95th, and 99th percentiles, and (iii) the frequency and intensity of extreme precipitation events classified at the 90th, 95th, and 99th percentiles.Correlations obtained with M2AMIP data and thus the strength of teleconnections in the free-running model are evaluated through comparison against corresponding correlations computed from observations and from MERRA-2. Overall, the strongest teleconnections in all datasets occur during the winter and coincide with the largest agreement between the observations, MERRA-2, and M2AMIP. When M2AMIP does capture the correlation seen in observations, there is a tendency for the spatial extent to be exaggerated. The weakest agreement between the data sources, for all teleconnection patterns, is in the correlation with extreme precipitation; however there are discrepancies between the datasets in the number of days with at least 1 mm of precipitation: M2AMIP has too few days with precipitation in the Northwest and the Northern Great Plains and too many days in the Northeast. In JJA, M2AMIP has too few days with precipitation in the western two-thirds of the country and too many days with precipitation along the east coast.

JJA↗

Analyzing Natural Language Context in Human-Machine Teaming using Supervised Machine Learning

Building a foundation for trustworthiness and trust verification in multi-asset teaming is the research challenge of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR). The Design Reference Mission (DRM) for ATTRACTOR is a search and rescue mission objective governed by a multi-member team consisting of human and machine operators. A crucial component to the effort is the communication between humans and autonomous agents throughout both planning and execution stages of the mission. Intuitive communication methods and modalities are posited as critical enablers for certifying trust and trustworthiness. This paper reports on the data collection and analysis conducted in support of the Human Informed Natural-language GANs Evaluation (HINGE)project to attain explainable and trusted communication between human-machine assets. Two identically curated image description datasets were acquired for HINGE, both consisting of two unique input modalities (typed vs. verbal) and retrieved in two distinct contexts (general vs. specific). The gathered datasets were assessed and compared using Parts-of-Speech (POS)features, sentence similarity metrics, and linguistic analysis. Then, the datasets were modeled and tested separately and in combination with one another using machine learning algorithms. The comparison and testing results reveal a superior dataset, by which a preferred context and input is understood, for generating image representations of missing persons using a Generative Adversarial Network (GAN).

Bryan A Barrows↗

TPSAS-NF1676L-23246-DND

The Arctic Radiation-IceBridge Sea and Ice Experiment (ARISE) was conducted from Fairbanks, Alaska during the September transition in the Beaufort-Chukchi Seas. Fifteen missions were flown with a NASA C-130 to characterize clouds, sea-ice, and associated radiative properties over a wide range of conditions. The advanced instrumentation payload included broadband and spectral radiometers, in-situ cloud probes, a laser altimeter and digital camera system. The flight profiles were tightly coordinated with multiple intersecting satellite overpasses allowing for a thorough evaluation of the Clouds and the Earth's Radiant Energy System (CERES) satellite climate data products. Thus, a rich dataset was acquired to help better understand cloud radiative effects as sea-ice conditions are changing due to arctic warming. The mission also acquired measurements to supplement the ongoing Operation Icebridge campaign. An overview of the mission operations, preliminary results, and a description of the datasets now available to the scientific community, will be presented.

William L Smith, Jr↗

Linear Mixed-Effects Models for Human-in-the-Loop Tracking Experiment Data

Linear mixed-effects models provide several benefits over more traditional statistical inference tests that are particularly useful for most human-in-the-loop tracking experiment data. However, surprisingly, mixed models are virtually not used for the analysis of tracking experiment data. This paper uses of linear mixed-effects models to analyze combined tracking data from two previous human-in-the-loop roll tracking experiments that compared control behavior metrics collected in both a research aircraft and a motion-base simulator. In the experiments, pilots' behavior under 10 different motion configurations with varying motion filter gains and break frequencies was evaluated and compared to that in the real aircraft. The linear mixed-effects model analysis on the combined dataset confirmed the main statistical outcomes of the individual experiments. The main benefits of mixed models for this type of data were demonstrated by successfully combining data from two experiments that used different experimental conditions and of which one had an additional apparatus and the other a missing participant. Finally, the mixed-model analysis was able to explicitly test for scientifically relevant statistical differences in the dependent measures between the aircraft and simulator, as well as both experiments.

manual control↗

Passive Microwave Brightness Temperature Assimilation to Improve Snow Mass Estimation across Complex Terrain in Pakistan, Afghanistan, and Tajikistan

An ensemble Kalman filter is used to assimilate Advanced Microwave Scanning Radiometer-2 (AMSR2) observations of passive microwave (PMW) brightness temperatures (spectral differences, ΔT b ) into land surface model estimates of snow mass over northwestern high mountain Asia (HMA). Trained support vector machines serve as the observation operator and map the geophysical modeled variables into ΔT b space within the assimilation framework. Evaluation of the assimilation routine is carried out through comparison of assimilated snow mass estimates with an in situ dataset. The assimilation framework helps improve the land surface model estimates through PMW ΔT b assimilation, particularly in terms of decreasing the domain-wide bias. The assimilation framework proved more effective during the (dry) snow accumulation season and decreased the bias and root-mean-square error (RMSE) in snow mass estimates at 76% and 58% of the comparative pixels, respectively. During the snow ablation season, the PMW brightness temperature signal contained less information related to snow mass due to the presence of other concurrent geophysical features that effectively serve as noise during the snow mass update. The utilization of PMW ΔT b for accurate snow mass estimation in complex terrain such as HMA is dependent on a multitude of factors for optimal results; however, it does add utility to the land surface model if the relevant pitfalls are taken into consideration prior to the state variable update.

Jawairia Ahmad↗

The Coupling Between Tropical Meteorology, Aerosol Lifecycle, Convection, and Radiation, During the Clouds, Aerosol and Monsoon Processes Philippines Experiment (CAMP2Ex)

The NASA Cloud, Aerosol, and Monsoon Processes Philippines Experiment (CAMP2Ex) employed the NASA P-3, Stratton Park Engineering Company (SPEC) Learjet 35, and a host of satellites and surface sensors to characterize the coupling of aerosol processes, cloud physics, and atmospheric radiation within the Maritime Continent’s complex southwest monsoonal environment. Conducted in the late summer of 2019 from Luzon Philippines in conjunction with the Office of Naval Research Propagation of Intraseasonal Tropical OscillatioNs (PISTON) experiment with its R/VSally Ride stationed in the North Western Tropical Pacific, CAMP2Ex documented diverse biomass burning, industrial and natural aerosol populations and their interactions with small to congestus convection. The 2019 season exhibited El Nino and associated drought, high biomass burning emissions, and an early monsoon transition allowing for observation of pristine to massively polluted environments as they advected through intricate diurnal mesoscale and radiative environments into the monsoonal trough. CAMP2Ex’s preliminary results indicate 1) increasing aerosol loadings tend to invigorate congestus convection in height and increase liquid water paths; 2) lidar, polarimetry, and geostationary Advanced Himawari Imager remote sensing sensors have skill in quantifying diverse aerosol and cloud properties and their interaction; and 3) high resolution remote sensing technologies are able to greatly improve our ability to evaluate the radiation budget in complex cloud systems. Through the development of innovative informatics technologies, CAMP2Ex provides a benchmark dataset of an environment of extremes for the study of aerosol, cloud and radiation processes as well as a crucible for the design of future observing systems.

CAMP2Ex↗

Ambient Synchrophasor Measurement Based System Inertia Estimation

This paper develops an algorithm to estimate the system inertia value based on ambient synchrophasor measurement. Informative features are extracted from ambient synchrophasor measurements for machine-learning-based inertia estimation. Besides ambient synchrophasor measurements of FNET/GridEye, other available data relevant to inertia (such as weather and system load data) are also used to improve the inertia estimation accuracy. Then a machine learning algorithm to estimate system inertia is developed. A test dataset including ambient synchrophasor data from FNET/GridEye measurements and the WECC system inertia data from NERC is used to evaluate the performance of the developed inertia estimation method. The average and maximum estimation errors of the developed inertia estimation method is lower than 5% and 10%, respectively. This accuracy is higher than reported accuracy values in existing literature.

CUI, YI↗

Techno-Economic Viability of Flexible Dispatch of Unconventional Geothermal Systems

Flexible geothermal operations could boost project returns through the allocation of improved power purchase agreements and/or exploitation of power price arbitrage opportunities. In this study, we investigated the techno-economic feasibility of variable flow rate control and time-of-day pricing in closed-loop geothermal systems. We considered U-shaped multilateral system configurations and modeled a variety of technical system parameters. These designs were simulated using a slender-body theory (SBT) model for transient heat transfer and fluid flow. This subsurface model was integrated into the flexible geothermal economic model (FGEM) tool to evaluate the overall flexible geothermal system techno-economics. Future hourly ambient temperature conditions were based on the Sup3rCC dataset. Published datasets were used for future hourly wholesale electricity prices. We analyzed four operating strategies: 1) baseload operation, 2) seasonal dispatch (high flow rate during summer and nominal flow rate during the rest of the year), 3) net generation maximization by varying flow rate to maximize net power output, and 4) revenue maximization by varying flow rate to maximize revenue. We ran all four scenarios for a multiloop configuration with 12 lateral passes, 7-km vertical depth and 87-km total drilling length. Furthermore, we assumed a 60 degrees C/km geothermal gradient and ambient temperature and wholesale electricity prices for New Mexico as a typical state location. The nominal flow rate was set to 80 kg/s. When considering drilling costs of $1,000/m and a discount rate of 7%, the generation maximization scenario resulted in the lowest levelized cost of electricity (LCOE) of ~$150/MWh. When considering project return on investment (ROI), defined as lifetime net income divided by upfront capital costs, all flexible operation scenarios performed better than the base case scenario. The highest ROI of 80% was obtained with the revenue maximization scenario. With drilling costs of $200/m and a discount rate of 5%, the generation maximization scenario resulted in LCOE of $49/MWh.

flexible geothermal↗

Improving Subsurface Stress Characterization for Carbon Dioxide Storage Projects by Incorporating Machine Learning Techniques

The overall objective of this project is to develop a framework for reliable characterization and prediction of the state of stress in the overburden and underburden (including the basement) in CO 2 storage reservoirs using machine learning and integrated geomechanics and geophysical methods. Specifically, we propose to develop workflow encompassing of technologies and/or methods to predict stress and pressure changes due to CO 2 injection in an active tertiary recovery site and their impacts on subtle fault activation, fractures and occurrence of microseismic events and compare responses to field observations. In this project, we anticipate using dataset from the Farnsworth field Unit (FWU) which is operated by Purdure Petroleum. A novel elastic-waveform VSP inversion technique will be used to estimate high-resolution spatial and temporal changes of elastic moduli in CO 2 storage reservoirs, which will be combined with velocity-stress relationship derived from laboratory tests to obtain subsurface pressure and stress. Clustered microseismic data will be jointly inverted for improved focal mechanisms. Least-squares reverse-time migration of microseismic waveform data will be performed to directly image fracture/fault zones. Additionally, a deep neural network machine learning technique with convolutional and recurrent layers will be used for learning the spectro-temporal structures in microseismic waveforms. The results of this geotechnical data analysis will be integrated to develop a high-resolution 3D mechanical earth model extending from the overburden sealing formations to the underburden including the basement. Mechanical properties will be derived through integration of mechanical logs, tests, available results from chemo-mechanical laboratory tests, and elastic inversion of seismic data using a combination of Bayesian and stochastic methods as well as machine learning technique. Failure features (faults/fractures) will be represented and/or modeled based on seismic and core data analysis. A transient hydrodynamic-geomechanical model will be developed through coupling with the calibrated FWU reservoir simulation model. The full physics coupled model will be used to train a reduced order proxy model using machine learning algorithm for estimating stress which will then be used with appropriate constitutive relationships and forward seismological models to simulate pressure changes and induced microseismicity. An advanced optimization framework will be developed to perform a history match to minimize error between field observations and simulated. The history matched proxy model will be verified against the full-physics equivalent. The field observations that will be used in the coupled model calibration process include pressure/stress inverted from VSP, moment magnitude from microseismic analysis, real time downhole pressure measurements, production and injection data. Parameter sensitivity and uncertainty analysis will be performed to characterize the impact of model parameter uncertainty on stress estimates. The proposed project will have significant impact on future field implementation of the proposed technology. Because the project field site is an ongoing CO 2 EOR development, the value of the new technology will be demonstrated in an operational context and evaluated as a viable risk mitigation strategy. Cost/benefit will be evaluated together with the various commercial incentives for CO 2 sequestration available to oil and gas operators. The extensive available dataset and ongoing data acquisition under the SWP Phase III work plan provides flexibility for investigation of multiple approaches and reduces technical risk.

58 GEOSCIENCES↗

Basin-Scale Structural Features Database

The Basin-Scale Structural Features database provides spatial datasets of faults, fractures, folds, and earthquakes compiled from public, authoritative sources (e.g., U.S. Geological Survey and State Geological Surveys) and aggregated into derivative forms to support subsurface assessments. Recognizing that characterizing basin-scale structural features requires interpreting data that are often ambiguous or lack key information, the source data were evaluated using a knowledge-data framework and geospatial fuzzy logic method (Justman et al., 2020) to represent both measured (observed) and predicted (inferred or potential) structural features as derivative datasets. This workflow employs conceptual models for known structural features and predicted structural features, incorporating geospatial data to estimate potential, even with limited data. The aim is to aid and support an understanding of basin-scale features and identify potential gaps in data and knowledge. As of 4/30/2025, the database includes resources for nine sedimentary basins: Appalachian, Denver, U.S. Gulf Coast, Illinois, Michigan, Permian, Sacramento, San Joquin and Williston. The database is organized by basin and then data category: 1) Faults, fractures, folds, 2) Earthquakes, 3) Topographic, 4) Structural contours and isopachs, 5) Geophysical, and 6) Structural feature density assessment maps.

basin scale↗

Assessment of the National Solar Radiation Database (NSRDB 1998-2016)

Applying traceable and standardized uncertainty characterization for solar resource data provides confidence in the dataset for use by financiers, developers and site operators of solar energy conversion systems, and ultimately reduces deployment cost. Performance guarantees of solar energy conversion systems are based on the available solar resource from measurement stations or modeled dataset such as the National Solar Radiation Database (NSRDB). In this study we implemented a comprehensive uncertainty determination approach [1]. The study also analyzed how the NSRDB (19982016) - Version 3 compares with the previous NSRDB (19982015) - Version 2. The study also attempted to estimate the uncertainty differences derived by comparing theNSRDB data to the seven measurement stations from the National Oceanic and Atmospheric Administration's Surface Radiation Budget Network (SURFRAD) and University of Oregon Solar Radiation Monitoring Laboratory (SRML). The evaluation was conducted for hourly values, daily totals, monthly mean daily totals, and annual mean monthly mean daily totals and demonstrate the qualityof the new datasets currently available from the NSRDB.

14 SOLAR ENERGY↗

Synoptic NO3 in Slate River Watershed, Colorado (2022)

The synoptic nitrate (NO3) dataset in the Slate River Watershed, Colorado consists of NO3 data collected at 19 locations three times during the summer of 2022. Stream samples were collected in early summer (early July), mid summer (late August), and late summer (late September). The samples include mainstem, tributary, and point source input water samples. These data were collected to evaluate spatiotemporal variability in stream NO3 during the summer, and evaluate anthropogenic controls on stream NO3 dynamics. This data package contains: (1) a csv of all NO3 samples and (2) a csv of locations for each sampling site. The dataset additionally includes a file-level metadata (flmd.csv) file that lists each file contained in the dataset with associated metadata; and a data dictionary (dd.csv) file that contains column/row headers used throughout the files along with a definition, units, and data type. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

EARTH SCIENCE > BIOSPHERE > ECOSYSTEMS > FRESHWATE↗

Predicting Overall Survival After Stereotactic Ablative Radiation Therapy in Early-Stage Lung Cancer: Development and External Validation of the Amsterdam Prognostic Model

Purpose: A prognostic model for 5-year overall survival (OS), consisting of recursive partitioning analysis (RPA) and a nomogram, was developed for patients with early-stage non-small cell lung cancer (ES-NSCLC) treated with stereotactic ablative radiation therapy (SABR). Methods and Materials: A primary dataset of 703 ES-NSCLC SABR patients was randomly divided into a training (67%) and an internal validation (33%) dataset. In the former group, 21 unique parameters consisting of patient, treatment, and tumor factors were entered into an RPA model to predict OS. Univariate and multivariate models were constructed for RPA-selected factors to evaluate their relationship with OS. A nomogram for OS was constructed based on factors significant in multivariate modeling and validated with calibration plots. Both the RPA and the nomogram were externally validated in independent surgical (n=193) and SABR (n=543) datasets. Results: RPA identified 2 distinct risk classes based on tumor diameter, age, World Health Organization performance status (PS) and Charlson comorbidity index. This RPA had moderate discrimination in SABR datasets (c-index range: 0.52-0.60) but was of limited value in the surgical validation cohort. The nomogram predicting OS included smoking history in addition to RPA-identified factors. In contrast to RPA, validation of the nomogram performed well in internal validation (r{sup 2}=0.97) and external SABR (r{sup 2}=0.79) and surgical cohorts (r{sup 2}=0.91). Conclusions: The Amsterdam prognostic model is the first externally validated prognostication tool for OS in ES-NSCLC treated with SABR available to individualize patient decision making. The nomogram retained strong performance across surgical and SABR external validation datasets. RPA performance was poor in surgical patients, suggesting that 2 different distinct patient populations are being treated with these 2 effective modalities.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

An Evaluation of Clouds and Radiation in a Large-Scale Atmospheric Model Using a Cloud Vertical Structure Classification

We revisit the concept of the cloud vertical structure (CVS) classes we have previously employed to classify the planet's cloudiness (Oreopoulos et al., 2017). The CVS classification reflects simple combinations of simultaneous cloud occurrence in the three standard layers traditionally used to separate low, middle, and high clouds and was applied to a dataset derived from active lidar and cloud radar observations. This classification is now introduced in an atmospheric global climate model, specifically a version of NASA's GEOS-5, in order to evaluate the realism of its cloudiness and of the radiative effects associated with the various CVS classes. Such classes can be defined in GEOS-5 thanks to a sub column cloud generator paired with the model's radiative transfer algorithm, and their associated radiative effects can be evaluated against observations. We find that the model produces 50% more clear skies than observations in relative terms and produces isolated high clouds that are slightly less frequent than in observations, but optically thicker, yielding excessive planetary and surface cooling. Low clouds are also brighter than in observations, but underestimates of the frequency of occurrence (by ~20% in relative terms) help restore radiative agreement with observations. Overall the model better reproduces the longwave radiative effects of the various CVS classes because cloud vertical location is substantially constrained in the CVS framework.

Lee, Dongmin↗

AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)

Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.

97 MATHEMATICS AND COMPUTING↗