Search NASA⌕ Search

SEARCH · Search NASA

Results for “Blind comparison”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Lensing without borders – I. A blind comparison of the amplitude of galaxy–galaxy lensing between independent imaging surveys

ABSTRACT Lensing without borders is a cross-survey collaboration created to assess the consistency of galaxy–galaxy lensing signals (ΔΣ) across different data sets and to carry out end-to-end tests of systematic errors. We perform a blind comparison of the amplitude of ΔΣ using lens samples from BOSS and six independent lensing surveys. We find good agreement between empirically estimated and reported systematic errors which agree to better than 2.3σ in four lens bins and three radial ranges. For lenses with zL > 0.43 and considering statistical errors, we detect a 3–4σ correlation between lensing amplitude and survey depth. This correlation could arise from the increasing impact at higher redshift of unrecognized galaxy blends on shear calibration and imperfections in photometric redshift calibration. At zL > 0.54, amplitudes may additionally correlate with foreground stellar density. The amplitude of these trends is within survey-defined systematic error budgets that are designed to include known shear and redshift calibration uncertainty. Using a fully empirical and conservative method, we do not find evidence for large unknown systematics. Systematic errors greater than 15 per cent (25 per cent) ruled out in three lens bins at 68 per cent (95 per cent) confidence at z < 0.54. Differences with respect to predictions based on clustering are observed to be at the 20–30 per cent level. Our results therefore suggest that lensing systematics alone are unlikely to fully explain the ‘lensing is low’ effect at z < 0.54. This analysis demonstrates the power of cross-survey comparisons and provides a promising path for identifying and reducing systematics in future lensing analyses.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

A computational modeling framework for pre-clinical evaluation of cardiac mapping systems

There are a variety of difficulties in evaluating clinical cardiac mapping systems, most notably the inability to record the transmembrane potential throughout the entire heart during patient procedures which prevents the comparison to a relevant “gold standard”. Cardiac mapping systems are comprised of hardware and software elements including sophisticated mathematical algorithms, both of which continue to undergo rapid innovation. The purpose of this study is to develop a computational modeling framework to evaluate the performance of cardiac mapping systems. The framework enables rigorous evaluation of a mapping system’s ability to localize and characterize (i.e., focal or reentrant) arrhythmogenic sources in the heart. The main component of our tool is a library of computer simulations of various dynamic patterns throughout the entire heart in which the type and location of the arrhythmogenic sources are known. Our framework allows for performance evaluation for various electrode configurations, heart geometries, arrhythmias, and electrogram noise levels and involves blind comparison of mapping systems against a “silver standard” comprised of computer simulations in which the precise transmembrane potential patterns throughout the heart are known. A feasibility study was performed using simulations of patterns in the human left atria and three hypothetical virtual catheter electrode arrays. Activation times (AcT) and patterns (AcP) were computed for three virtual electrode arrays: two basket arrays with good and poor contact and one high-resolution grid with uniform spacing. The average root mean squared difference of AcTs of electrograms and those of the nearest endocardial action potential was less than 1 ms and therefore appears to be a poor performance metric. In an effort to standardize performance evaluation of mapping systems a novel performance metric is introduced based on the number of AcPs identified correctly and those considered spurious as well as misclassifications of arrhythmia type; spatial and temporal localization accuracy of correctly identified patterns was also quantified. This approach provides a rigorous quantitative analysis of cardiac mapping system performance. Proof of concept of this computational evaluation framework suggests that it could help safeguard that mapping systems perform as expected as well as provide estimates of system accuracy.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

An international benchmark for wind plant wakes from the American WAKE ExperimeNt (AWAKEN)

This article introduces the first benchmark study within the International Energy Agency Wind Task 57 framework, focusing on wind plant wakes. Leveraging data from the American WAKE ExperimeNt (AWAKEN), the benchmark aims to assess the accuracy of simulation tools in modeling wind plant wakes and their impact on the downstream flow under diverse inflow conditions. The AWAKEN field campaign, conducted in Oklahoma from 2022 to 2024, provides unprecedented observations of wind plant-atmosphere interactions, thus offering a large dataset to validate numerical models of different complexity. The benchmark will include three phases—code calibration, blind comparison, and iteration—allowing participants to refine their numerical models based on the feedback from the benchmark team. This article describes the benchmark case study selected from observations providing details on atmospheric conditions, wake evidence, and wind turbine operation. The benchmark’s structure and timeline, along with the expected publication of results, are discussed as well. This collaborative effort aims to enhance the accuracy of wind plant wake simulations, thus contributing to the improvement of wind energy production estimates.

17 WIND ENERGY↗
DOE OSTI2016

Radiation Effects Science: Advancing the Frontiers of Science and Engineering

Recent improvements in the radiation effects science capability at Sandia have achieved the following: Record gamma-ray outputs at HERMES enable qualification for Life Extension Programs (LEPs) and evaluation of electromagnetic pulse (EMP) effects on electric grid elements; Completed blind comparison for model validation of cavity system generated EMP in advanced computer simulations; For the first time, new diagnostics provide a high resolution photon energy spectrum on Z (Grand Challenge LDRD); ACRR completed record number of operations, two years in a row, to support design and qualification; Installed the nation’s largest GHz electromagnetic test chamber for design and qualification.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Feature review of photovoltaic modeling software utilizing blind performance assessment

While confidence in photovoltaic (PV) modeling software has always been essential, the rapid pace of new PV plant developments makes accuracy and credibility more critical than ever. Independent assessments, particularly through blind modeling comparisons, are therefore necessary to ensure unbiased benchmarking across PV modeling software. Previous studies have been limited by a narrow range of models compared, anonymized results, or system size. This study presents results from the first-ever onymous blind modeling comparison, evaluated using both lab- and utility-scale fixed-tilt, monofacial, south-facing systems at sub-hourly time intervals. Seven commercially used PV software tools were compared: 3E SynaptiQ, PlantPredict, PVsyst, RatedPower, SAM, SolarFarmer, and Solargis Evaluate. Predictions were submitted directly by software representatives, providing unique insights into each software’s implementation and resulting prediction behavior. Notable features, including plane-of-array (POA) transposition model, module temperature model, shading model, and performance model were analyzed and compared. Four summary tables compile these features of the software, serving as a resource to help users understand the methodological differences and select the most suitable software for their applications. The software tools show deviations from mean error in annual yield up to 2.5 % in the lab-scale system, increasing to 6.0 % for the utility-scale system. These differences arise from a combination of user decisions and the inherent behavior of the software, indicating the need for continuous and rigorous validation of modeling methods using these software tools against complex, real-world systems.

14 SOLAR ENERGY↗
DOE OSTI2023

PV Performance Modeling - Data and Resources

The Photovoltaic (PV) Performance Modeling Collaborative (PVPMC) organized a blind PV performance modeling intercomparison to allow PV modelers to blindly test their models and modeling ability against real system data. Measured weather and irradiance data were provided along with detailed descriptions of PV systems from two locations (Albuquerque, New Mexico, USA and Roskilde, Denmark). Participants were asked to simulate the plane-of-array irradiance, module temperature, and DC power output from six systems and submit their results to Sandia for processing. This dataset includes seven MS-Excel sheets with instructions, notes and all necessary data (weather, irradiance, temperature, power) used for the data analysis of the blind modeling comparison. The hourly data represent six different systems from Albuquerque, NM and Roskilde, Denmark over a period of one year. These data are useful for PV performance model validation studies.

14 SOLAR ENERGY↗
DOE OSTI2016

MO-G-201-04: Knowledge-Based Planning for Single-Isocenter Stereotactic Radiosurgery to Multiple Brain Metastases

Purpose: Single-isocenter, linac-based SRS for multiple brain metastases (multi-mets) can deliver highly conformal radiation doses and reduce overall patient treatment time compared to other therapy techniques. This study aims to quantify the dosimetric benefits of knowledge-based planning (KBP) for multi-met treatments. Methods: Using a previously-published KBP methodology (an artificial neural network (ANN) trained on single-target linac-based SRS plans), 3D dose distribution predictions for multi-met patients were obtained by treating each brain lesion as a solitary target and subsequently combining individual predictions into a single distribution using a dose-weighted geometric averaging to obtain the best results in the inter-target space. 17 previously-treated multi-met plans, with target numbers ranging from N=2–5, were used to validate the ANN predictions and subsequent KBP auto-planning routine. The fully-deliverable KBP plans were developed by converting dose distribution predictions into patient-specific optimization objectives while maintaining identical target normalizations (typically PTV V100%=D98%). Plan quality improvements were quantified by the difference between SRS quality metrics (QMs): δdQM=QM(clinical)-QM(KBP). QMs of interest were: gradient measure (GM), conformity index (CI), brain V10 and V5, brainstem D0.1cc and heterogeneity index (HI). Finally, overall plan quality was judged via blinded plan comparison by SRS-specializing physicians. Results: Two clinical plans were found to be significant outliers wherein plan quality was dramatically worse than KBP. Despite indicating KBP superiority, these were removed from the QM analysis to prevent skewing the results. In the remaining cases, clinical and KBP QMs were nearly identical with modest improvements in the KBP sample: δGM=0.12±0.56mm, δCI=−0.01±0.04, Brain δV10=0.8±2.6cc, brain δV5=6.3 ±10.7cc, brainstem δD0.1cc=0.06±1.19Gy and δHI= −0.04±0.05. Ultimately, 13/17 KBP plans were deemed superior to the manual plans in blinded physician review. Conclusion: The results demonstrate that KBP-driven automated planning in linac-based single-isocenter treatments for multiple brain metastases is indistinguishable from, or even better than, traditional manual planning. J. Hattangadi: Research Grant; Varian Medical Systems; K.L. Moore: Research Grant: Varian Medical Systems.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Blind photovoltaic modeling intercomparison: A multidimensional data analysis and lessons learned

The Photovoltaic (PV) Performance Modeling Collaborative (PVPMC) organized a blind PV performance modeling intercomparison to allow PV modelers to blindly test their models and modeling ability against real system data. Measured weather and irradiance data were provided along with detailed descriptions of PV systems from two locations (Albuquerque, New Mexico, USA, and Roskilde, Denmark). Participants were asked to simulate the plane-of-array irradiance, module temperature, and DC power output from six systems and submit their results to Sandia for processing. The results showed overall median mean bias (i.e., the average error per participant) of 0.6% in annual irradiation and –3.3% in annual energy yield. While most PV performance modeling results seem to exhibit higher precision and accuracy as compared to an earlier blind PV modeling study in 2010, human errors, modeling skills, and derates were found to still cause significant errors in the estimates.

14 SOLAR ENERGY↗
DOE OSTI2022

Uniform Recalibration of Common Spectrophotometry Standard Stars onto the CALSPEC System Using the SuperNova Integral Field Spectrograph

Abstract We calibrate spectrophotometric optical spectra of 32 stars commonly used as standard stars, referenced to 14 stars already on the Hubble Space Telescope–based CALSPEC flux system. Observations of CALSPEC and non-CALSPEC stars were obtained with the SuperNova Integral Field Spectrograph over the wavelength range 3300–9400 Å as calibration for the Nearby Supernova Factory cosmology experiment. In total, this analysis used 4289 standard-star spectra taken on photometric nights. As a modern cosmology analysis, all presubmission methodological decisions were made with the flux scale and external comparison results blinded. The large number of spectra per star allows us to treat the wavelength-by-wavelength calibration for all nights simultaneously with a Bayesian hierarchical model, thereby enabling a consistent treatment of the Type Ia supernova cosmology analysis and the calibration on which it critically relies. We determine the typical per-observation repeatability (median 14 mmag for exposures ≳5 s), the Maunakea atmospheric transmission distribution (median dispersion of 7 mmag with uncertainty 1 mmag), and the scatter internal to our CALSPEC reference stars (median of 8 mmag). We also check our standards against literature filter photometry, finding generally good agreement over the full 12 mag range. Overall, the mean of our system is calibrated to the mean of CALSPEC at the level of ∼3 mmag. With our large number of observations, careful cross-checks, and 14 reference stars, our results are the best calibration yet achieved with an integral-field spectrograph, and among the best calibrated surveys.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2016

Thermo-Hydraulic-Mechanical Modelling of the Full-Scale Emplacement (FE) Experiment at the Mont Terri Rock Laboratory - 16490

The Full-Scale Emplacement Experiment at the Mont Terri underground research laboratory is a heater test carried out in a gallery constructed in Opalinus Clay. The aim of the experiment is both to develop emplacement procedures for radioactive waste canisters and their surrounding backfill as well as understanding the thermo-hydro-mechanical effects of disposing of radioactive waste on the engineering materials and the host rock. Coupled thermo-hydro-mechanical models have been used to aid planning of the locations of sensors, to inform decisions on the schedule for turning on the heaters as well as to make predictions of the temperatures and water pressures in the bentonite and the Opalinus Clay. The FE Experiment was preceded by a number of other experiments at the Mont Terri laboratory at smaller scales, but with similar materials, which provided crucial input to the models and gave confidence in the general process understanding. Blind predictions were made prior to turning on the heaters, of the temperature and relative humidity in the bentonite surrounding the heaters. The comparison between the blind predictions and early data from the experiment show that knowledge gained from smaller scale experiments on similar materials has enabled good blind predictions to be made and gives confidence that temperature in the experiment is unlikely to rise to a level that could damage the sensors. It also demonstrates that the thermal properties of the disposal system are understood well enough to produce reliable estimates of temperature in the backfill. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2016

Blind test of density-functional-based methods on intermolecular interaction energies

In the past decade, a number of approaches have been developed to fix the failure of (semi)local density-functional theory (DFT) in describing intermolecular interactions. In this work, the performance of several such approaches with respect to highly accurate benchmarks is compared on a set of separation-dependent interaction energies for ten dimers. Since the benchmarks were unknown before the DFT-based results were collected, this comparison constitutes a blind test of these methods.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Tidal Turbine Benchmarking Project: Stage I - Steady Flow Blind Predictions: Preprint

This paper presents the first blind prediction stage of the Tidal Turbine Benchmarking Project being conducted and funded by the UK's EPSRC and Supergen ORE Hub. In this first stage, only steady flow conditions, at low and elevated turbulence (3.1%) levels, were considered. Prior to the blind prediction stage, a large laboratory scale experiment was conducted in which a highly instrumented 1.6m diameter tidal rotor was towed through a large towing tank in well-defined flow conditions with and without an upstream turbulence grid. Details of the test campaign and rotor design were released as part of this community blind prediction exercise. Participants were invited to use a range of engineering modelling approaches to simulate the performance and loads of the turbine. 26 submissions were received from 12 groups from across academia and industry using solution techniques ranging from blade resolved computational fluid dynamics through actuator line, boundary integral element methods, vortex methods to engineering Blade Element Momentum methods. The comparisons between experiments and blind predictions were extremely positive helping to provide validation and uncertainty estimates for the models, but also validating the experimental tests themselves. The exercise demonstrated that the experimental turbine data provides a robust data set against which researchers and design engineers can test their models and implementations to ensure robustness in their processes, helping to reduce uncertainty and provide increased confidence in engineering processes. Furthermore, the data set provides the basis by which modellers can evaluate and refine approaches.

benchmarking↗
DOE OSTI2024

Outcomes and Conclusions from the 2022 AM Bench Measurements, Challenge Problems, Modeling Submissions, and Conference

The Additive Manufacturing Benchmark Test Series (AM Bench) provides rigorous measurement data for validating additive manufacturing (AM) simulations for a broad range of AM technologies and material systems. AM Bench includes extensive in situ and ex situ measurements, simulation challenges for the AM modeling community, and a corresponding conference series. In 2022, the second round of AM Bench measurements, challenge problems, and conference were completed, focusing primarily upon laser powder bed fusion (LPBF) processing of metals, and both material extrusion processing and vat photopolymerization of polymers. In all, more than 100 people from 10 National Institute of Standards and Technology (NIST) divisions and 21 additional organizations were directly involved in the AM Bench 2022 measurements, data management, and conference organization. The international AM community submitted 138 sets of blind modeling simulations for comparison with the in situ and ex situ measurements, up from 46 submissions for the first round of AM Bench in 2018. Analysis of these submissions provides valuable insight into current AM modeling capabilities. The AM Bench data are permanently archived and freely accessible online. The AM Bench conference also hosted an embedded workshop on qualification and certification of AM materials and components.

36 MATERIALS SCIENCE↗