Search NASA⌕ Search

SEARCH · Search NASA

Results for “false data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15

Change Detection in Constellations of Buried Objects Extracted From Ground-Penetrating Radar Data

Detection of deliberately buried objects in ground-penetrating radar (GPR) data acquired along a path is a clutter-limited problem. Detection-false alarm rate performance can be improved by replacing the detection statistic with a change statistic that incorporates information from previous path traversals. A constellation matching approach is developed for buried-object change detection in GPR data. Network topologies of buried objects detected in GPR data from previous path traversals are maintained in a constellation database. Localized groups of buried objects newly detected on the latest path traversal are matched to the constellation. Buried objects from the latest path traversal whose locations or strengths cannot be reconciled with the constellation are identified as changes. The system has one component that generates constellation databases offline and another component suitable for change detection in real time. It can tolerate paths with significant translational misalignments. The system uses the following: 1) a customized translational relaxation algorithm for point pattern matching that incorporates detection strength and a probabilistic uncertainty model for buried-object location into the objective function and 2) a change statistic that accounts for the magnitude of change relative to predicted detection strength. A constellation database can typically be generated offline from a single path traversal roughly two orders of magnitude faster than the time typically required for a vehicle to travel the extent of the path. Database sizes are typically four to five orders of magnitude smaller than the data sets of GPR signal scans or focused 3-D GPR images that they were generated from. On bumpy dirt roads buried exclusively with nonmetallic objects at various depths, detection-false alarm rate performance is shown to be significantly better for our change statistics than for our detection statistics.

47 OTHER INSTRUMENTATION↗

GOTTCHA, Version2

One major challenge in the field of shotgun metagenomics is the accurate identification of the organisms present within the community, based on classification of short sequence reads. Though microbial community profiling methods have emerged to attempt to rapidly classify the millions of reads output from contemporary sequencers, the combination of incomplete databases, similarity among otherwise divergent genomes, and the large volumes of sequencing data required for metagenome sequencing has led to unacceptably high false discovery rates (FDR). Here we present the application of a novel, gene-independent and signature-based metagenomic taxonomic profiling tool with significantly smaller FDR, which is also capable of classifying never-before seen genomes into the appropriate parent taxa.The algorithm is based upon three primary computational phases: (I) genomic decomposition into bit vectors, (II) bit vector intersections to identify shared regions, and (III) bit vector subtractions to remove shared regions and reveal unique, signature regions. In the Decomposition phase, genomic data is first masked to highlight only the valid (non-ambiguous) regions and then decomposed into overlapping 24-mers. The k-mers are sorted along with their start positions, de-replicated, and then prefixed, to minimize data duplication. The prefixes are indexed and an identical data structure is created for the start positions to mimic that of the k-mer data structure. During the Intersection phase -- which is the most computationally intensive phase -- as an all-vs-all comparison is made, the number of comparisons is first reduced by four methods: (a) Prefix restriction, (b) Overlap detection, (c) Overlap restriction, and (d) Result recording. In Prefix restriction, only k-mers of the same prefix are compared. Within that group, potential overlap of k-mer suffixes that would result in a non-empty set intersection are screened for. If such an overlap exists, the region which intersects is first reduced by performing a binary search of the boundary suffixes of the smaller set into the larger set, which defines the limits of the zipper-based intersection process. Rather than recording the actual k-mers of the intersection, another data structure of identical "shape" is created which consists of only bit vectors so that only a 1 or 0 will be stored in the location of the k-mer suffix that was found in the intersection. This reduces the amount of data generated and stored considerably. During the Subtraction phase, relevant intersection bitmasks are first unionized together to form a single bitmask which is then applied over the original genome to reveal only those regions of the genome that are unique. These regions are then exported to disk in FASTA format and used in the application of determining the constituents of an unknown metagenomic community.

Freitas, Tracey↗

GOTTCHA, Version 1

One major challenge in the field of shotgun metagenomics is the accurate identification of the organisms present within the community, based on classification of short sequence reads. Though microbial community profiling methods have emerged to attempt to rapidly classify the millions of reads output from contemporary sequencers, the combination of incomplete databases, similarity among otherwise divergent genomes, and the large volumes of sequencing data required for metagenome sequencing has led to unacceptably high false discovery rates (FDR). Here we present the application of a novel, gene-independent and signature-based metagenomic taxonomic profiling tool with significantly smaller FDR, which is also capable of classifying never-before seen genomes into the appropriate parent taxa.The algorithm is based upon three primary computational phases: (I) genomic decomposition into bit vectors, (II) bit vector intersections to identify shared regions, and (III) bit vector subtractions to remove shared regions and reveal unique, signature regions. In the Decomposition phase, genomic data is first masked to highlight only the valid (non-ambiguous) regions and then decomposed into overlapping 24-mers. The k-mers are sorted along with their start positions, de-replicated, and then prefixed, to minimize data duplication. The prefixes are indexed and an identical data structure is created for the start positions to mimic that of the k-mer data structure. During the Intersection phase -- which is the most computationally intensive phase -- as an all-vs-all comparison is made, the number of comparisons is first reduced by four methods: (a) Prefix restriction, (b) Overlap detection, (c) Overlap restriction, and (d) Result recording. In Prefix restriction, only k-mers of the same prefix are compared. Within that group, potential overlap of k-mer suffixes that would result in a non-empty set intersection are screened for. If such an overlap exists, the region which intersects is first reduced by performing a binary search of the boundary suffixes of the smaller set into the larger set, which defines the limits of the zipper-based intersection process. Rather than recording the actual k-mers of the intersection, another data structure of identical "shape" is created which consists of only bit vectors so that only a 1 or 0 will be stored in the location of the k-mer suffix that was found in the intersection. This reduces the amount of data generated and stored considerably. During the Subtraction phase, relevant intersection bitmasks are first unionized together to form a single bitmask which is then applied over the original genome to reveal only those regions of the genome that are unique. These regions are then exported to disk in FASTA format and used in the application of determining the constituents of an unknown metagenomic community.

Freitas, Tracey↗

A Novel Method to Train Classification Models for Structure Detection in In Situ Spacecraft Data

We present a method for creating spacecraft-like data which can be used to train Machine Learning (ML) models to detect and classify structures in in situ spacecraft data. First, we use the Grad-Shafranov equation to numerically solve for several magnetohydrostatic equilibria which are variations on a known analytic equilibrium. These equilibria are then used as the initial conditions for Particle-In-Cell simulations in which the structures of interest are observed and labeled. We then take one-dimensional slices through the simulations to replicate what a spacecraft collecting data from the simulation would observe. This sliced data then can be used as training data for the initial training of ML models intended for use on spacecraft data. We demonstrate the method applied to the problem of detecting small-scale plasmoids in the magnetotail, which is important for understanding complex magnetotail reconnection dynamics. The simple 1D classifier we train is able to detect more than 70% of the plasmoid points in the data set but also produces a large number of false positives. Our further work on this example problem is detailed, and further potential uses of the method are discussed.

79 ASTRONOMY AND ASTROPHYSICS↗

ON THE SOLAR CHROMOSPHERE OBSERVED AT THE LIMB WITH HINODE

Broadband images in the Ca II H line, from the Broadband Filter Imager (BFI) instrument on the Hinode spacecraft, show emission from spicules emerging from and visible right down to the observed limb. Surprisingly, little absorption of spicule light is seen along their lengths. We present formal solutions to the transfer equation for given (ad hoc) source functions, including a stratified chromosphere from which spicules emanate. The model parameters are broadly compatible with earlier studies of spicules. The visibility of Ca II spicules down to the limb in Hinode data seems to require that spicule emission be Doppler shifted relative to the stratified atmosphere, either by supersonic turbulent or organized spicular motion. The non-spicule component of the chromosphere is almost invisible in the broadband BFI data, but we predict that it will be clearly visible in high spectral resolution data. Broadband Ca II H limb images give the false impression that the chromosphere is dominated by spicules. Our analysis serves as a reminder that the absence of a signature can be as significant as its presence.

79 ASTRONOMY AND ASTROPHYSICS↗

MARVELS-1: A FACE-ON DOUBLE-LINED BINARY STAR MASQUERADING AS A RESONANT PLANETARY SYSTEM AND CONSIDERATION OF RARE FALSE POSITIVES IN RADIAL VELOCITY PLANET SEARCHES

We have analyzed new and previously published radial velocity (RV) observations of MARVELS-1, known to have an ostensibly substellar companion in a {approx}6 day orbit. We find significant ({approx}100 m s{sup -1}) residuals to the best-fit model for the companion, and these residuals are naievely consistent with an interior giant planet with a P = 1.965 days in a nearly perfect 3:1 period commensurability (|P{sub b} /P{sub c} - 3| < 10{sup -4}). We have performed several tests for the reality of such a companion, including a dynamical analysis, a search for photometric variability, and a hunt for contaminating stellar spectra. We find many reasons to be critical of a planetary interpretation, including the fact that most of the three-body dynamical solutions are unstable. We find no evidence for transits, and no evidence of stellar photometric variability. We have discovered two apparent companions to MARVELS-1 with adaptive optics imaging at Keck; both are M dwarfs, one is likely bound, and the other is likely a foreground object. We explore false-alarm scenarios inspired by various curiosities in the data. Ultimately, a line profile and bisector analysis lead us to conclude that the {approx}100 m s{sup -1} residuals are an artifact of spectral contamination from a stellar companion contributing {approx}15%-30% of the optical light in the system. We conclude that origin of this contamination is the previously detected RV companion to MARVELS-1, which is not, as previously reported, a brown dwarf, but in fact a G dwarf in a face-on orbit.

79 ASTRONOMY AND ASTROPHYSICS↗

Discovering faint and high apparent motion rate near-Earth asteroids using a deep learning program

ABSTRACT Although many near-Earth objects have been found by ground-based telescopes, some fast-moving ones, especially those near detection limits, have been missed by observatories. We developed a convolutional neural network for detecting faint fast-moving near-Earth objects. It was trained with artificial streaks generated from simulations and was able to find these asteroid streaks with an accuracy of 98.7 per cent and a false positive rate of 0.02 per cent on simulated data. This program was used to search image data from the Zwicky Transient Facility (ZTF) in four nights in 2019, and it identified six previously undiscovered asteroids. The visual magnitudes of our detections range from ∼19.0 to 20.3 and motion rates range from ∼6.8 to 24 deg d−1, which is very faint compared to other ZTF detections moving at similar motion rates. Our asteroids are also ∼1–51 m diameter in size and ∼5–60 lunar distances away at close approach, assuming their albedo values follow the albedo distribution function of known asteroids. The use of a purely simulated data set to train our model enables the program to gain sensitivity in detecting faint and fast-moving objects while still being able to recover nearly all discoveries made by previously designed neural networks which used real detections to train neural networks. Our approach can be adopted by any observatory for detecting fast-moving asteroid streaks.

Wang, Franklin (ORCID:0000000344887355)↗

Patterns in the Use of Axillary Operations for Patients with Node-Positive Breast Cancer After Neoadjuvant Chemotherapy: A National Cancer Database (NCDB) Analysis

Background: The American College of Surgeons Oncology Group (ACOSOG) Z1071 and Sentinel Neoadjuvant (SENTINA) trials of sentinel node biopsy for node-positive breast cancer treated with neoadjuvant chemotherapy (NAC) demonstrated false-negative rates that varied on the basis of surgical technique. This study evaluated trends in axillary operations before and after publication of these trials. Methods: This study analyzed patients from National Cancer Database (NCDB) with clinical T0 through T4, N1 and N2, M0 breast cancer who received NAC from 1 January 2012 to 31 December 2015 and sentinel lymph node biopsy (SNB) or axillary lymph node dissection (ALND). The patients were divided into the following groups: SNB, ALND, and (SNB + ALND). Results: Of the 32,036 evaluable patients identified in this study. 5565 had SNB, 19,930 had ALND, and 6541 had SNB + ALND. Compared with the ALND group, the SNB group was younger, had more invasive ductal cancers, and had lower clinical T- and N-stage disease (p < 0.001). The patients in the SNB group had a higher rate of estrogen receptor-positive and triple-negative breast cancers, but a lower rate of human epidermal growth factor receptor 2 (HER2)-positive cancer (p < 0.001). The nodal pathologic complete response (PCR) rate, defined as no residual invasive cancer, was 66.5% in the SNB group and 33.1% in the ALND group. Since 2013, the rate of ALND has decreased from 88.7 to 77.1% in both community and academic institutions (p < 0.001). Conclusion: Since publication of the ACOSOG Z1071 and SENTINA trials, the national rates of ALND in node positive breast cancer treated with NAC have decreased despite reported false-negative SNB rates and lack of prospective outcome data regarding the oncologic safety of ALND omission.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Enhancing Docking Accuracy with PECAN2, a 3D Atomic Neural Network Trained without Co-Complex Crystal Structures

Decades of drug development research have explored a vast chemical space for highly active compounds. The exponential growth of virtual libraries enables easy access to billions of synthesizable molecules. Computational modeling, particularly molecular docking, utilizes physics-based calculations to prioritize molecules for synthesis and testing. Nevertheless, the molecular docking process often yields docking poses with favorable scores that prove to be inaccurate with experimental testing. To address these issues, several approaches using machine learning (ML) have been proposed to filter incorrect poses based on the crystal structures. However, most of the methods are limited by the availability of structure data. Here, we propose a new pose classification approach, PECAN2 (Pose Classification with 3D Atomic Network 2), without the need for crystal structures, based on a 3D atomic neural network with Point Cloud Network (PCN). The new approach uses the correlation between docking scores and experimental data to assign labels, instead of relying on the crystal structures. We validate the proposed classifier on multiple datasets including human mu, delta, and kappa opioid receptors and SARS-CoV-2 Mpro. Our results demonstrate that leveraging the correlation between docking scores and experimental data alone enhances molecular docking performance by filtering out false positives and false negatives.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DeepZipper. II. Searching for Lensed Supernovae in Dark Energy Survey Data with Deep Learning

Gravitationally lensed supernovae (LSNe) are important probes of cosmic expansion, but they remain rare and difficult to find. Current cosmic surveys likely contain 5–10 LSNe in total while next-generation experiments are expected to contain several hundred to a few thousand of these systems. We search for these systems in observed Dark Energy Survey (DES) five year SN fields—10 3 sq. deg. regions of sky imaged in the griz bands approximately every six nights over five years. To perform the search, we utilize the DeepZipper approach: a multi-branch deep learning architecture trained on image-level simulations of LSNe that simultaneously learns spatial and temporal relationships from time series of images. We find that our method obtains an LSN recall of 61.13% and a false-positive rate of 0.02% on the DES SN field data. DeepZipper selected 2245 candidates from a magnitude-limited (m i < 22.5) catalog of 3,459,186 systems. We employ human visual inspection to review systems selected by the network and find three candidate LSNe in the DES SN fields.

79 ASTRONOMY AND ASTROPHYSICS↗

False alarm recognition in hyperspectral gas plume identification

According to one embodiment, a method for analyzing hyperspectral data includes collecting first hyperspectral data of a scene using a hyperspectral imager during a no-gas period and analyzing the first hyperspectral data using one or more gas plume detection logics. The gas plume detection logic is executed using a low detection threshold, and detects each occurrence of an observed hyperspectral signature. The method also includes generating a histogram for all occurrences of each observed hyperspectral signature which is detected using the gas plume detection logic, and determining a probability of false alarm (PFA) for all occurrences of each observed hyperspectral signature based on the histogram. Possibly at some other time, the method includes collecting second hyperspectral data, and analyzing the second hyperspectral data using the one or more gas plume detection logics and the PFA to determine if any gas is present. Other systems and methods are also included.

97 MATHEMATICS AND COMPUTING↗

Monitoring and flaw detection during wire-based directed energy deposition using in-situ acoustic sensing and wavelet graph signal analysis

The goal of this work is to detect flaw formation in the wire-based directed energy deposition (W-DED) process using in-situ sensor data. The W-DED studied in this work is analogous to metal inert gas electric arc welding. The adoption of W-DED in industry is limited because the process is susceptible to stochastic and environmental disturbances that cause instabilities in the electric arc, eventually leading to flaw formation, such as porosity and suboptimal geometric integrity. Moreover, due to the large size of W-DED parts, it is difficult to detect flaws post-process using non-destructive techniques, such as X-ray computed tomography. Accordingly, the objective of this work is to detect flaw formation in W-DED parts using data acquired from an acoustic (sound) sensor installed near the electric arc. To realize this objective, we develop and apply a novel wavelet integrated graph theory approach. The approach extracts a single feature called graph Laplacian Fiedler number from the noise-contaminated acoustic sensor data, which is subsequently tracked in a statistical control chart. Using this approach, the onset of various types of flaws are detected with a false alarm rate less-than 2%. This work demonstrates the potential of using advanced data analytics for in-situ monitoring of W-DED.

42 ENGINEERING↗

Checking Questionable Entry of Personally Identifiable Information Encrypted by One-Way Hash Transformation

Background: As one of the several effective solutions for personal privacy protection, a global unique identifier (GUID) is linked with hash codes that are generated from combinations of personally identifiable information (PII) by a one-way hash algorithm. On the GUID server, no PII is permitted to be stored, and only GUID and hash codes are allowed. The quality of PII entry is critical to the GUID system. Objective: The goal of our study was to explore a method of checking questionable entry of PII in this context without using or sending any portion of PII while registering a subject. Methods: According to the principle of GUID system, all possible combination patterns of PII fields were analyzed and used to generate hash codes, which were stored on the GUID server. Based on the matching rules of the GUID system, an error-checking algorithm was developed using set theory to check PII entry errors. We selected 200,000 simulated individuals with randomly-planted errors to evaluate the proposed algorithm. These errors were placed in the required PII fields or optional PII fields. The performance of the proposed algorithm was also tested in the registering system of study subjects. Results: There are 127,700 error-planted subjects, of which 114,464 (89.64%) can still be identified as the previous one and remaining 13,236 (10.36%, 13,236/127,700) are discriminated as new subjects. As expected, 100% of nonidentified subjects had errors within the required PII fields. The possibility that a subject is identified is related to the count and the type of incorrect PII field. For all identified subjects, their errors can be found by the proposed algorithm. The scope of questionable PII fields is also associated with the count and the type of the incorrect PII field. The best situation is to precisely find the exact incorrect PII fields, and the worst situation is to shrink the questionable scope only to a set of 13 PII fields. In the application, the proposed algorithm can give a hint of questionable PII entry and perform as an effective tool. Conclusions: The GUID system has high error tolerance and may correctly identify and associate a subject even with few PII field errors. Correct data entry, especially required PII fields, is critical to avoiding false splits. In the context of one-way hash transformation, the questionable input of PII may be identified by applying set theory operators based on the hash codes. The count and the type of incorrect PII fields play an important role in identifying a subject and locating questionable PII fields.

59 BASIC BIOLOGICAL SCIENCES↗

Multivariate analysis of progressive thermal desorption coupled gas chromatography-mass spectrometry.

Thermal decomposition of poly dimethyl siloxane compounds, Sylgard{reg_sign} 184 and 186, were examined using thermal desorption coupled gas chromatography-mass spectrometry (TD/GC-MS) and multivariate analysis. This work describes a method of producing multiway data using a stepped thermal desorption. The technique involves sequentially heating a sample of the material of interest with subsequent analysis in a commercial GC/MS system. The decomposition chromatograms were analyzed using multivariate analysis tools including principal component analysis (PCA), factor rotation employing the varimax criterion, and multivariate curve resolution. The results of the analysis show seven components related to offgassing of various fractions of siloxanes that vary as a function of temperature. Thermal desorption coupled with gas chromatography-mass spectrometry (TD/GC-MS) is a powerful analytical technique for analyzing chemical mixtures. It has great potential in numerous analytic areas including materials analysis, sports medicine, in the detection of designer drugs; and biological research for metabolomics. Data analysis is complicated, far from automated and can result in high false positive or false negative rates. We have demonstrated a step-wise TD/GC-MS technique that removes more volatile compounds from a sample before extracting the less volatile compounds. This creates an additional dimension of separation before the GC column, while simultaneously generating three-way data. Sandia's proven multivariate analysis methods, when applied to these data, have several advantages over current commercial options. It also has demonstrated potential for success in finding and enabling identification of trace compounds. Several challenges remain, however, including understanding the sources of noise in the data, outlier detection, improving the data pretreatment and analysis methods, developing a software tool for ease of use by the chemist, and demonstrating our belief that this multivariate analysis will enable superior differentiation capabilities. In addition, noise and system artifacts challenge the analysis of GC-MS data collected on lower cost equipment, ubiquitous in commercial laboratories. This research has the potential to affect many areas of analytical chemistry including materials analysis, medical testing, and environmental surveillance. It could also provide a method to measure adsorption parameters for chemical interactions on various surfaces by measuring desorption as a function of temperature for mixtures. We have presented results of a novel method for examining offgas products of a common PDMS material. Our method involves utilizing a stepped TD/GC-MS data acquisition scheme that may be almost totally automated, coupled with multivariate analysis schemes. This method of data generation and analysis can be applied to a number of materials aging and thermal degradation studies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Single-shot, multi-signature remote detection of uranium by filament-induced breakdown spectroscopy

Filament-induced breakdown spectroscopy (FIBS) is an attractive remote detection approach that is potentially applicable to nuclear materials, such as uranium (U). Simultaneous detection of atomic U and uranium monoxide features is demonstrated in a single laser shot with detection probabilities of 52.2% and 39.0%, respectively, and with a 1% false-alarm probability. It is shown that a nearly 100% detection probability can be reached with 1% false alarm probability in .1 second when using an 80-Hz laser and data acquisition system. Atomic U and UO signal features are identifiable in the remote measurement with comparable signal-to-background ratios, suggesting that, despite being broader than the U I emission line, the UO band is also suitable for remote detection of uranium by FIBS.

(020.3260) Isotope shifts, (140.3440) Laser-induce↗

(U) PRAD0697 & PRAD0698: Complex Loading of CeO₂ Powder

Cerium(IV) oxide (CeO₂) powder is shock compressed using the Precision High Energy-density Liner Implosion eXperiment (PHELIX) platform. Experimental results are compared against several modeling approaches. Compaction behavior is best captured with a P-∝ model, which calculates CeO₂ powder bulk densities within 80-99% of experimental values but overpredicts densi cation at the cylindrical target's outer radius and center by up to 20%. Preliminary calculations suggest that accuracy could be increased with the inclusion of a coupled strength model. Several common computational modeling approaches for the shock compression response of granular materials and the magnetohydrodynamic (MHD) force upon the impactor/liner in pulsed power compression experiments are investigated and analyzed for their validity. The Bi-linear Ramp, P-∝ PACXP, and P-∝ Menikoff-Kober continuum compaction models are calibrated to planar impact Hugoniot data for CeO₂ powder and used to predict the powder's shock compaction response under non-planar shock wave compression. MHD calculations of the PHELIX pulsed power driver are performed using an idealized resistor-inductor-capacitor (RLC) circuit calibrated to previous experiments. All simulations are performed using the LANL code FLAG. Two validation experiments are computationally designed using the calibrated compaction and circuit models, executed using the PHELIX platform on CeO₂ targets with initial porous densities of 3.95 and 4.03 g/cm³, measured with proton radiography, and analyzed against the model predictions. The two P-∝ models more accurately describe CeO₂ powder densi cation than the Bi-linear Ramp model. However, the two P-∝ models overpredict bulk density of the shock compressed CeO₂ powder by up to 20% when the appropriate impact velocities are applied. MHD calculations for both validation experiments underpredict liner impact velocities by 4-11% when using the idealized RLC circuit model calibrated to previous experiments. Compensating underpredictions of impact velocity and overpredictions of powder densication lead to a false accuracy in pre-shot calculations compared to experimental data. To improve correlation between simulations and experiments, the following improvements are suggested: 1. A coupled strength model for CeO₂ powder that updates strength as a function of porosity and applied stress. 2. An improved MHD circuit model that more accurately captures the PHELIX machine.

36 MATERIALS SCIENCE↗

Impacts of benchmarking choices on inferred model skill of the Arctic–Boreal terrestrial carbon cycle

Abstract Land surface models require continuous validation against observations to improve and reduce simulation uncertainty. However, inferred model performance can be heavily influenced by subjective choices made in the selection and application of observational data products. A key area often misrepresented by models is the Arctic–Boreal region, which is a potential tipping point region in Earth’s climate system due to large permafrost carbon stocks that are vulnerable to release with climate warming. We use the International Land Model Benchmarking (ILAMB) framework to evaluate how the model skill of TRENDY-v9 models varies based on the choice of observational-based benchmark and how benchmarks are applied in model evaluation. This analysis uses global datasets integrated into ILAMB and new, regionally-specific observational products from the Arctic–Boreal Vulnerability Experiment. Our results cover the overall time period of 1979–2019 and show that model scores can vary substantially depending on the data product applied, with higher model scores indicating better model performance against observations. The lowest model scores occur when benchmarked against regional, compared to global, datasets. We also evaluate observed and modeled functional relationships between ecosystem respiration and air temperature and between gross primary production and precipitation. Here, we find that the magnitude and shape of the responses are strongly impacted by the choice of observational dataset and the approach used to construct the functional relationship benchmark. These results suggest that model evaluation studies could conclude a false sense of model skill if only using a single benchmark data product or if not applying regional data products when performing a regional model analysis. Collectively, our findings highlight the influence of benchmarking choices on model evaluation and point to the need for benchmarking guidelines when assessing model skill.

Poe, Jeralyn (ORCID:0000000318495278)↗

Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models

Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models (LLMs) have been incorporated into the training dataset of the models. This gives a false sense of performance that can overestimate how these models will function on truly unseen data. This problem becomes worse with commercial LLMs with larger and non-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box (text access only) and grey box (access to model loss/probabilities required) techniques and apply them to both commercial and non-commercial models. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness (or ineffectiveness) of these contamination detection techniques. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination (broadly) in fine-tuned models when both a baseline and fine-tuned model is present. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar (e.g., drawn from the same set of code projects).

97 MATHEMATICS AND COMPUTING↗