Search NASA⌕ Search

SEARCH · Search NASA

Results for “Simulation Credibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Applying NASA-STD-7009 Standard for Models and Simulations to Surrogate and Other Statistical Models

NASA-STD-7009A, Standard for Models and Simulations, contains a worthy and insightfully-crafted credibility assessment. That set of metrics is intended to accumulate and communicate information that can be critical for users and their customers in evaluating findings due to a model or simulation. Was the model designed for the present use? Is it validated in the domain it’s being applied to? And so on. The standard was originally focused on physics-based models, but it can be applied more generally. It can – and should – be applied to high-impact statistical models, including surrogate models based on larger models and sims and regression models based directly on data. A key issue with many of such models is their relative simplicity compared to most M&S despite their use in high-risk engineering decisions. This report will make a case for using this Standard for a wider class of models, and how it might actually add value in excess of its cost for not only stakeholders, but modelers themselves. A rough example showing compliance for a statistical model is included.

K L Johnson↗

Magnetohydrodynamic Simulation of a Streamer Beside a Realistic Coronal Hole

Existing models of coronal streamers establish their credibility and act as the initial state for transients. The models have produced satisfactory streamer simulations, but unsatisfactory coronal hole simulations. This is a consequence of the character of the models and the boundary conditions. The models all have higher densities in the magnetically open regions than occur in coronal holes (Noci, et al., 1993).

Suess, S. T.↗

Standard for Models and Simulations

This NASA Technical Standard establishes uniform practices in modeling and simulation to ensure essential requirements are applied to the design, development, and use of models and simulations (MS), while ensuring acceptance criteria are defined by the program project and approved by the responsible Technical Authority. It also provides an approved set of requirements, recommendations, and criteria with which MS may be developed, accepted, and used in support of NASA activities. As the MS disciplines employed and application areas involved are broad, the common aspects of MS across all NASA activities are addressed. The discipline-specific details of a given MS should be obtained from relevant recommended practices. The primary purpose is to reduce the risks associated with MS-influenced decisions by ensuring the complete communication of the credibility of MS results.

verification↗

Code-verification techniques for hypersonic reacting flows in thermochemical nonequilibrium

The study of hypersonic flows and their underlying aerothermochemical reactions is particularly important in the design and analysis of vehicles exiting and reentering Earth's atmosphere. Computational physics codes can be employed to simulate these phenomena; however, code verification of these codes is necessary to certify their credibility. To date, few approaches have been presented for verifying codes that simulate hypersonic flows, especially flows reacting in thermochemical nonequilibrium. In this work, we present our code-verification techniques for verifying the spatial accuracy and thermochemical source term in hypersonic reacting flows in thermochemical nonequilibrium. Additionally, we demonstrate the effectiveness of these techniques on the Sandia Parallel Aerodynamics and Reentry Code (SPARC).

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Comparing ICME simulations with scaled laboratory experiment

In stellar physics and astrophysics, numerical simulations and laboratory experiments are often compared to observational data to support their representation of the real world. However, there is also merit in comparing numerical simulations to properly scaled experiments, especially when the experiment and the simulation are both emulating the solar phenomena. Confirming the credibility of scaled experiments and their scaling with well-validated models is important to expand our knowledge of the associated physical phenomena. This is significant because experiments and simulations can be performed frequently, whereas observations may be limited by location, field of view, and missing data. In this work, we use the Alfvén Wave Solar atmosphere Model, a well-validated magnetohydrodynamic model, to simulate an interplanetary coronal mass ejection (ICME) and compare it to an experiment which provides a scaled analog to a physical ICME. The experiment was performed on the Big Red Ball facility and scaled using dimensionless parameters such as plasma β and magnetosonic Mach number to reproduce the main structure of an ICME. We compare the model-simulated temperature, density, and magnetic field to those from the experiment, as well as the scaling parameters used in the experiment, to those calculated from the simulation. This comparison is performed to further justify the scaling arguments made by the experiment. Additionally, the comparison would lead to the development of stronger scaling arguments for future experiments.

Bryant, K. [University of Michigan, Ann Arbor, MI ↗

Comparing Automated Posterior Estimation Techniques for Modeling Strong Lenses In Ground-based Survey Data

Current and future ground-based cosmological surveys, such as the Dark Energy Survey (DES), and the Vera Rubin Observatory Legacy Survey of Space and Time (LSST), are predicted to discover thousands to tens of thousands of strong gravitational lenses. The large number of strong lenses discoverable in future surveys will make strong lensing a highly competitive and complementary cosmic probe. However, conventional lens modeling techniques are unable to scale up to the sheer number of lenses that will be discovered through upcoming surveys. Therefore, the use of automated lens analysis techniques is necessary. We demonstrate that machine learning methods can be used to automate the inference of informative model posteriors of strong lensing systems in ground-based surveys with credible uncertainty estimation. We present two Simulation-Based Inference (SBI) approaches for lens parameter estimation of galaxy-galaxy lenses. We demonstrate applications of Neural Posteriors Estima tors (NPEs) and Bayesian Neural Network (BNNs) to automate the inference of a 12-parameter lensing system for DES-like ground-based imaging data. We apply a suite of diagnostics (e.g., posterior coverage and SBC) to validate the performance of our methods. We find that NPEs outperform the BNN, producing posterior distributions that are for the most part both more accurate and more precise; in particular, several source-light model parameters are systematically biased in the BNN implementation.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Impedance-Based Stability Analysis in Grid Interconnection Impact Study Owing to the Increased Adoption of Converter-Interfaced Generators

This study investigates the emerging harmonic stability concerns to be addressed by grid planners in generation interconnection studies, owing to the increased adoption of renewable energy resources connected to the grid via power electronic converters. The wideband and high-frequency electromagnetic transient (EMT) characteristics of these converter-interfaced generators (CIGs) and their interaction with the grid impedance are not accurately captured in the typical dynamic studies conducted by grid planners. This paper thus identifies the desired components to be studied and subsequently develops a practical process for integrating a new CIG into a grid with the existing CIGs. The steps of this process are as follows: the impedance equation of a CIG using its control dynamics and an interface filter to the grid, for example, an LCL filter (inductor-capacitor-inductor type), is developed; an equivalent impedance model including the existing CIGs nearby and the grid observed from the point of common coupling are derived; the system stability for credible operating scenarios is assessed. Detailed EMT simulations validate the accuracy of the impedance models and stability assessment for various connection scenarios. Here, by complementing the conventional EMT simulation studies, the proposed analytical approach enables grid planners to identify critical design parameters for seamlessly integrating a new CIG and ensuring the reliability of the grid.

24 POWER TRANSMISSION AND DISTRIBUTION↗

A ``Tail" of Two Cities: Fatality-based Modeling of COVID-19 Evolution in New York City and Cook County, IL

I describe $SIR$ modeling of the COVID-19 pandemic in two U.S. urban environments, New York City (NYC) and Cook County, IL, from onset through the month of June, 2020. Since testing was not widespread early in the pandemic in the U.S., I do not use data on confirmed cases and rely solely on public fatality data to estimate model parameters. Fits to the first 20 days of data determine a degenerate combination of the basic reproduction number, $R_0$, and the mean time to removal from the infectious population, $\gamma^{-1}$, with $\gamma (R_0-1) = 0.25 (0.21)$ inverse days for NYC (Cook County). Equivalently, the initial doubling time was $t_d = 2.8 (3.4)$ days for NYC (Cook). The early fatality data suggest that both locations had infections in early February. I model the mitigation measures implemented in mid-March in both locations (distancing, quarantine, isolation, etc) via a time-dependent reproduction number $R_t$ that declines monotonically from $R_0$ to a sma ller asymptotic value, with a parameterized functional form. The timing (mid-March) and duration (several days) of the transitions in $R_t$ appear well determined by the data. However, the fatality data determine only a degenerate combination of the parameters $R_0$, the percentage reduction in social contact due to mitigation measures, $X$, and the infection fatality rate (IFR), $f$. With flat priors, based on simulations the NYC model parameters have 95.45\% credible intervals of $R_0 = 3.0-5.4$, $X= 80-99.9\%$ and $f=2-6\%$, with $5-13\%$ of the population asymptotically infected. A strong external prior indicating a lower value of $f$ or of $\gamma^{-1}$ would imply lower values of $R_0$ and $X$ and higher percentage infection of the population. For Cook County, the evolution was qualitatively different: after mitigation measures were implemented, the daily fatality counts reached a plateau for about a month before tailing off. This is consistent with an $SIR$ model that exhibits ``critical slowing-down", in which $R_t$ plateaus ! at a value just above unity. For Cook County, the 95.45\% credible intervals for the model parameters are much broader and shifted downward, $R_0=1.4-4.7$, $X=26-54\%$, and $f=0.1-0.6\%$ with $15-88\%$ of the population asymptotically infected. Despite the apparently lower efficacy of its social contact reduction measures, Cook County has had significantly fewer fatalities per population than NYC, $D_\infty/N=100$ vs. 270 per 100,000. In the model, this is attributed to the lower inferred IFR for Cook; an external prior pointing to similar values of the IFR for the two locations would instead chalk up the difference in $D/N$ to differences in the relative growth rate of the disease. I derive a model-dependent threshold, $X^{crit}_e$, for `safe' re-opening, that is, for easing of contact reduction that would not trigger a second wave; for NYC, the models predict that increasing social contact by more than 20\% from post-mitigation levels will lead to renewed spread, while for Cook County the threshold value is very uncertain, given the parameter degeneracies. The timing of 2nd-wave growth will depend on the amplitude of contact increase relative to $X^{crit}_e$ and on the asymptotic growth rate, and the impact in terms of fatalities will depend on the parameter $f$.

Frieman, Joshua↗

Code Comparison Study Fosters Confidence in the Numerical Simulation of Enhanced Geothermal Systems

Numerical simulation has become a standard analytical tool for scientists and engineers to evaluate the potential and performance of enhanced geothermal systems. A variety of numerical simulators developed by industry, universities, and national laboratories are currently available and being applied to better understand enhanced geothermal systems at the field scale. To yield credible predictions and be of value to site operators, numerical simulators must be able to accurately represent the complex coupled processes induced by producing geothermal systems, such as fracture aperture changes due to thermal stimulation, fracture shear displacement with fluid injection, rate of thermal depletion of reservoir rocks, and permeability alteration with mineral precipitation or dissolution. A suite of numerical simulators was exercised on a series of test problems that considered coupled thermal, hydraulic, geomechanical, and geochemical (THMC) processes. Problems were selected and designed to isolate selected coupled processes, to be executed on workstation class computers, and have simple but illustrative metrics for result comparisons. This paper summarizes the initial suite of seven benchmark problems, describes the code comparison activities, provides example results for problems and documents the capabilities of currently available numerical simulation codes to represent coupled processes that occur during the production of geothermal resources. Code comparisons described in this paper use the ISO (International Organization for Standardization) standard ISO-13538 for proficiency testing of numerical simulators. This approach was adopted for a recent code comparison study within the radiation transfer-modeling field of atmospheric sciences, which was focused on canopy reflectance models. This standard specifies statistical methods for analyzing laboratory data from proficiency testing schemes to demonstrate that the measurement results do not exhibit evidence of an unacceptable level of bias.

benchmark problems↗

Design and execution of a verification, validation, and uncertainty quantification plan for a numerical model of left ventricular flow after LVAD implantation

Left ventricular assist devices (LVADs) are implantable pumps that act as a life support therapy for patients with severe heart failure. Despite improving the survival rate, LVAD therapy can carry major complications. Particularly, the flow distortion introduced by the LVAD in the left ventricle (LV) may induce thrombus formation. While previous works have used numerical models to study the impact of multiple variables in the intra-LV stagnation regions, a comprehensive validation analysis has never been executed. The main goal of this work is to present a model of the LV-LVAD system and to design and follow a verification, validation and uncertainty quantification (VVUQ) plan based on the ASME V&V40 and V&V20 standards to ensure credible predictions. The experiment used to validate the simulation is the SDSU cardiac simulator, a bench mock-up of the cardiovascular system that allows mimicking multiple operation conditions for the heart-LVAD system. The numerical model is based on Alya, the BSC’s in-house platform for numerical modelling. Alya solves the Navier-Stokes equation with an Arbitrary Lagrangian-Eulerian (ALE) formulation in a deformable ventricle and includes pressure-driven valves, a 0D Windkessel model for the arterial output and a LVAD boundary condition modeled through a dynamic pressure-flow performance curve. The designed VVUQ plan involves: (a) a risk analysis and the associated credibility goals; (b) a verification stage to ensure correctness in the numerical solution procedure; (c) a sensitivity analysis to quantify the impact of the inputs on the four quantities of interest (QoIs) (average aortic root flow $Q^{avg}_{Ao}$, maximum aortic root flow $Q^{max}_{Ao}$, average LVAD flow $Q^{avg}_{VAD}$, and maximum LVAD flow $Q^{max}_{VAD}$; (d) an uncertainty quantification using six validation experiments that include extreme operating conditions. Numerical code verification tests ensured correctness of the solution procedure and numerical calculation verification showed a grid convergence index (GCI)95% <3.3%. The total Sobol indices obtained during the sensitivity analysis demonstrated that the ejection fraction, the heart rate, and the pump performance curve coefficients are the most impactful inputs for the analysed QoIs. The Minkowski norm is used as validation metric for the uncertainty quantification. It shows that the midpoint cases have more accurate results when compared to the extreme cases. The total computational cost of the simulations was above 100 [core-years] executed in around three weeks time span in Marenostrum IV supercomputer. This work details a novel numerical model for the LV-LVAD system, that is supported by the design and execution of a VVUQ plan created following recognised international standards. We present a methodology demonstrating that stringent VVUQ according to ASME standards is feasible but computationally expensive.

59 BASIC BIOLOGICAL SCIENCES↗

Amundsen Sea Embayment ice-sheet mass-loss predictions to 2050 calibrated using observations of velocity and elevation change

Mass loss from the Amundsen Sea Embayment of the West Antarctic Ice Sheet is a major contributor to global sea-level rise (SLR) and has been increasing over recent decades. Predictions of future SLR are increasingly modelled using ensembles of simulations within which model parameters and external forcings are varied within credible ranges. Accurately reporting the uncertainty associated with these predictions is crucial in enabling effective planning for, and construction of defences against, rising sea levels. Calibrating model simulations against current observations of ice-sheet behaviour enables the uncertainty to be reduced. Here we calibrate an ensemble of BISICLES ice-sheet model simulations of ice loss from the Amundsen Sea Embayment using remotely sensed observations of surface elevation and ice speed. Each calibration type is shown to be capable of reducing the 90% credibility bounds of predicted contributions to SLR by 34 and 43% respectively.

54 ENVIRONMENTAL SCIENCES↗

Sensitivity and Uncertainty Workflow of Full System SIERRA Models Supporting High Consequence Applications

Credibility of end-to-end CompSim (Computational Simulation) models and their agile execution requires an expressive framework to describe, communicate and execute complex computational tool chains representing the model. All stakeholders from system engineering and customers through model developers and V&V partners need views and functionalities of the workflow representing the model in a manner that is natural to their discipline. In the milestone and in this report we define workflow as a network of computation simulation activities executed autonomously on a distributed set of computational platforms. The FY19 ASC L2 Milestone (6802) for the Integrated Workflow (IWF) project was designed to integrate and improve existing capabilities or develop new functionalities to provide a wide range of stakeholders a coherent and intuitive platform capable of defining and executing CompSim modeling from analysis workflow definition to complex ensemble calculations. The main goal of the milestone was to advance the integrated workflow capabilities to support the weapon system analysts with a production deployment in FY20. Ensemble calculations supporting program decisions include sensitivity analysis, optimization and uncertainty quantification. The goal of the L2 milestone aligned with the ultimate goal of the IWF project is to foster cultural and technical shift toward and integrated CompSim capability based on automated workflows. Specific deliverables were defined in five broad categories: 1) Infrastructure, including development of distributed-computing workflow capability, 2) integration of Dakota (Sandia's sensitivity, optimization and UQ engine) with SAW (Sandia Analysis Workbench), 3) ARG (Automatic Report Generator introspecting analysis artifacts and generating human-readable extensible and archivable reports), 4) Libraries and Repositories aiding capability reuse, and 5) Exemplars to support training, capturing best practices and stress testing of the platform. A set of exemplars was defined to represent typical weapon system qualification CompSim projects. Analyzing the required capabilities and using the findings to plan implementation of required capabilities ensured optimal allocation of development resources focused on production deployment after the L2 is completed. It was recognized early that the end-to-end modeling applications pose a considerable number of diverse risks, and a formal risk tracking process was implemented. The project leveraged products, capabilities and development tasks of IWF partners. SAW, Dakota, Cubit, Sierra, Slycat, and NGA (NexGen Analytics, a small business) contributed to the integrated platform developed during this milestone effort. New products delivered include: a) NGW (Next Generation Workflow) for robust workflow definition and execution, b) Dakota wizards, editor and results visualization, and c) the automatic report generator ARG. User engagement was initiated early in the development process eliciting concrete requirements and actionable feedback to assure that the integrated CompSim capability will have high user acceptance and impact. The current integrated capabilities have been demonstrated and are continually being tested by a set of exemplars ranging from training scenarios to computationally demanding uncertainty analyses. The integrated workflow platform has been deployed on both SRN (Sandia Restricted Network) and SCN (Sandia Classified Network). Computational platforms where the system has been demonstrated span from Windows (Creo the CAD platform chosen by Sandia) to Trinity HPC (Sierra and CTH solvers). Follow up work will focus on deployment at SNL and other sites in the nuclear enterprise (LLNL, KCNSC), training and consulting support to democratize the analysis agility, process health and knowledge management benefits the NGW platform provides.

97 MATHEMATICS AND COMPUTING↗

Modeling personnel turnover in the parametric organization

A model is developed for simulating the dynamics of a newly formed organization, credible during all phases of organizational development. The model development process is broken down into the activities of determining the tasks required for parametric cost analysis (PCA), determining the skills required for each PCA task, determining the skills available in the applicant marketplace, determining the structure of the model, implementing the model, and testing it. The model, parameterized by the likelihood of job function transition, has demonstrated by the capability to represent the transition of personnel across functional boundaries within a parametric organization using a linear dynamical system, and the ability to predict required staffing profiles to meet functional needs at the desired time. The model can be extended by revisions of the state and transition structure to provide refinements in functional definition for the parametric and extended organization.

Dean, Edwin B.↗

A framework for strategic discovery of credible neural network surrogate models under uncertainty

The widespread integration of deep neural networks in developing data-driven surrogate models for high-fidelity simulations of complex physical systems highlights the critical necessity for robust uncertainty quantification techniques and credibility assessment methodologies, ensuring the reliable deployment of surrogate models in consequential decision-making. Here, this study presents the Occam Plausibility Algorithm for surrogate models (OPAL-surrogate), providing a systematic framework to uncover predictive neural network-based surrogate models within the large space of potential models, including various neural network classes and choices of architecture and hyperparameters. The framework is grounded in hierarchical Bayesian inferences and employs model validation tests to evaluate the credibility and prediction reliability of the surrogate models under uncertainty. Leveraging these principles, OPAL-surrogate introduces a systematic and efficient strategy for balancing the trade-off between model complexity, accuracy, and prediction uncertainty. The effectiveness of OPAL-surrogate is demonstrated through two modeling problems, including the deformation of porous materials for building insulation and turbulent combustion flow for ablation of solid fuels within hybrid rocket motors.

42 ENGINEERING↗

Expanded verification and validation studies of hypersonic aerodynamics with multiple physics-fidelity models

Hypersonic aerothermodynamics is an important domain of modern multiphysics simulation. The Multi-Fidelity Toolkit is a simulation tool being developed at Sandia National Laboratories to predict aerodynamic properties for compressible flows from a range of physics fidelities and computational speeds. These models include the Reynolds-averaged Navier–Stokes (RANS) equations, the Euler equations with momentum-energy integral technique (MEIT), and modified Newtonian aerodynamics with flat-plate boundary layer (MNA+FPBL) equations, and they can be invoked independently or coupled with hierarchical Kriging to interpolate between high-fidelity simulations using lower-fidelity data. However, as with any new simulation capability, verification and validation are necessary to gather credibility evidence. This work describes formal code- and solution-verification activities, as well as model validation with uncertainty considerations. Code verification activities on the MNA+FPBL model build on previous work by focusing on the viscous portion of the model. Viscous quantities of interest are compared against those from an analytical solution for flat-plate, inclined-plate, and cone geometries. The code verification methodology for the MEIT model is also presented. Test setup and results of code verification tests on the laminar and turbulent models within MEIT are shown. Solution-verification activities include grid-refinement studies on simulations that model the HIFiRE-1 wind tunnel experiments. These experiments are used for validation of all model fidelities. A thorough validation comparison with prediction error and uncertainty is also presented. Three additional HIFiRE-1 experimental runs are simulated in this study, and the solution verification and validation work examines the effects of the associated parameter changes on model performance. Finally, a study is presented that compares the computational costs and fidelities from each of the different models.

42 ENGINEERING↗

An Uncertainty Structure Matrix for Models and Simulations

Software that is used for aerospace flight control and to display information to pilots and crew is expected to be correct and credible at all times. This type of software is typically developed under strict management processes, which are intended to reduce defects in the software product. However, modeling and simulation (M&S) software may exhibit varying degrees of correctness and credibility, depending on a large and complex set of factors. These factors include its intended use, the known physics and numerical approximations within the M&S, and the referent data set against which the M&S correctness is compared. The correctness and credibility of an M&S effort is closely correlated to the uncertainty management (UM) practices that are applied to the M&S effort. This paper describes an uncertainty structure matrix for M&S, which provides a set of objective descriptions for the possible states of UM practices within a given M&S effort. The columns in the uncertainty structure matrix contain UM elements or practices that are common across most M&S efforts, and the rows describe the potential levels of achievement in each of the elements. A practitioner can quickly look at the matrix to determine where an M&S effort falls based on a common set of UM practices that are described in absolute terms that can be applied to virtually any M&S effort. The matrix can also be used to plan those steps and resources that would be needed to improve the UM practices for a given M&S effort.

Green, Lawrence L.↗

NREL's Field Data Repository Supports Accurate Home Energy Analysis (Fact Sheet)

This fact sheet discusses NREL's work to develop a repository of research-level residential building characteristics and historical energy use data to support ongoing efforts to improve the accuracy of residential energy analysis tools and the efficiency of energy assessment processes. The objective of this project is to create a robust empirical data source to support the research goals of the Department of Energy's Building America program, which is to improve the efficiency of existing U.S. homes by 30% to 50%. Researchers can use this data source to test the accuracy of building energy simulation software and energy audit procedures, ultimately leading to more credible and less expensive energy analysis.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Quantifying the agreement between observed and simulated extratropical modes of interannual variability

Using historical simulations of the Coupled Model Intercomparison Project-5 (CMIP5) and multiple observationally-based datasets, we employ skill metrics to analyze the fidelity of the simulated Northern Annular Mode, the North Atlantic Oscillation, the Pacific North America pattern, the Southern Annular Mode, the Pacific Decadal Oscillation, the North Pacific Oscillation, and the North Pacific Gyre Oscillation. We assess the benefits of a unified approach to evaluate these modes of variability, which we call the common basis function (CBF) approach, based on projecting model anomalies onto observed empirical orthogonal functions (EOFs). The CBF approach circumvents issues with conventional EOF analysis, eliminating, for example, corrections of arbitrarily assigned, but inconsistent, signs of the EOF’s/PC’s being compared. It also avoids the problem that sometimes the first observed EOF is more similar to a higher order model EOF, particularly if the simulated EOFs are not well separated. Compared to conventional EOF analysis of models, the CBF approach indicates that models compare significantly better with observations in terms of pattern correlation and root-mean-squared-error (RMSE) than heretofore suggested. In many cases, models are doing a credible job at capturing the observationally-based estimates of patterns; however, errors in simulated amplitudes can be large and more egregious than pattern errors. In the context of the broad distribution of errors in the CMIP5 ensemble, sensitivity tests demonstrate that our results are relatively insensitive to methodological considerations (CBF vs. conventional approach), observational uncertainties in pattern (as determined by using multiple datasets), and internal variability (when multiple realizations from the same model are compared). The skill metrics proposed in this study can provide a useful summary of the ability of models to reproduce the observed EOF patterns and amplitudes. Additionally, the skill metrics can be used as a tool to objectively highlight where potential model improvements might be made. Finally, we advocate more systematic and objective testing of simulated extratropical variability, especially during the non-dominant seasons of each mode, when many models are performing relatively poorly.

58 GEOSCIENCES↗