Search NASA⌕ Search

SEARCH · Search NASA

Results for “Task complexity”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,333 records · Page 74
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Enhanced Oblique Decision Tree Enabled Policy Extraction for Deep Reinforcement Learning in Power System Emergency Control

Deep reinforcement learning (DRL) algorithms have successfully solved many challenging problems in various power system control scenarios. However, their decision-making process is usually regarded as black-boxes. Furthermore, how DRL models interact with human intelligence remains an open problem. Thus, this paper proposes a policy extraction framework to extract a complex DRL model into an explainable policy. This framework includes three parts: 1) DRL training and data generation. We train an agent for a specific control task and generate data, which contains the control policy of the agent. 2) Policy extraction. We propose an information gain rate based weighted oblique decision tree (IGR-WODT) for DRL policy extraction. 3) Policy evaluation. We define three metrics to evaluate the performance of the proposed approach. A case study for the under-voltage load shedding problem shows that the IGR-WODT presents a performance enhancement compared with DRL, weighted oblique decision tree, and univariate decision tree. The proposed policy extraction method could provide an intuitive explanation of the neural network decision-making process to the dispatchers when making final decisions on power grid operation. Also, the resulted rule-based controller could replace the deep neural network-based controller in many field edge devices with limited computing resources, providing comparable performance.

deep reinforcement learning↗
DOE OSTI2020

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Godoy, William↗
DOE OSTI2021

Neural-adjoint method for the inverse design of all-dielectric metasurfaces

All-dielectric metasurfaces exhibit exotic electromagnetic responses, similar to those obtained with metal-based metamaterials. Research in all-dielectric metasurfaces currently uses relatively simple unit-cell designs, but increased geometrical complexity may yield even greater scattering states. Although machine learning has recently been applied to the design of metasurfaces with impressive results, the much more challenging task of finding a geometry that yields a desired spectra remains largely unsolved. We propose and demonstrate a method capable of finding accurate solutions to ill-posed inverse problems, where the conditions of existence and uniqueness are violated. A specific example of finding the metasurface geometry which yields a radiant exitance matching the external quantum efficiency of gallium antimonide is demonstrated. We also show how the neural-adjoint method can intelligently grow the design search space to include designs that increasingly and accurately approximate the desired scattering response. The neural-adjoint method is not restricted to the case demonstrated and may be applied to plasmonics, photonic crystal, and other artificial electromagnetic materials.

42 ENGINEERING↗
DOE OSTI2018

The MCNP6 Book On Unstructured Mesh Geometry: User's Guide for MCNP 6.2.1

Los Alamos National Laboratory’s (LANL) Monte Carlo N-Particle R (MCNP R ) transport code has a more general geometry capability than has been available in most combinatorial geometry codes [1]. In addition to the capability of combining several predefined geometric bodies, as in a combinatorial scheme, MCNP6 [2] gives the user the added flexibility of defining geometric regions from all the first and second degree surfaces of analytical geometry and elliptical tori and then combining them with Boolean operators. This decades-old constructive solid geometry (CSG) capability has been well-tested and verified. However, it has long been recognized that as the model complexity increases, this process of describing the geometry is difficult, tedious, time-consuming, and error prone[3, 4, 5]. Consequently, innovators have taken on the task of developing a better way to construct geometries, not only in MCNP6, but other particle transport codes as well

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Quantum-Inspired Bayesian Sampling for Uncertainty Quantification and Machine Learning (Final Technical Report)

With increasing simulation and measurement data, machine learning and artificial intelligence have been widely used in computational decision-making of complex engineering systems. The resulting tools, such as uncertainty quantification solvers, reinforcement learning, and physics-informed machine learning, have achieved great success in critical DOE tasks such as material discovery and design, energy system modeling and control, and numerical weather and climate prediction. A core topic in scientific machine learning and artificial intelligence is Bayesian inference: given an observed data set, people want to estimate the posterior distribution of a (possibly large) number of hidden parameters. Due to the flexibility and weak assumptions, Bayesian sampling has been the mainstream Bayesian inference solvers despite the rapid progress of approximate Bayesian inference. Classical Bayesian sampling methods such as Markov-chain Monte Carlo suffer from a low-acceptance rate due to the random walk nature, therefore state-of-the-art techniques use Hamiltonian Monte Carlo and its variants to efficiently draw posterior samples in a high dimension. The key idea of Hamiltonian Monte Carlo and its variants is to simulate the Hamiltonian dynamics of a classical particle with a fixed mass, and their performance significantly degrades when the posterior distribution is highly spiky or has multiple modes. Leveraging the idea of quantum physics, this project has investigated new theory, algorithms and applications of Bayesian inference (especially Bayesian sampling). The main results include: (1) novel quantum-inspired Bayesian sampling methods that can lead to better accuracy for challenging multi-modal or spiky distributions, (2) more scalable machine learning framework leveraging tensor-compressed Bayesian inference, and (3) Bayesian and sampling approaches for verifying the robustness of continuous and binary neural networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Quantifying automated vehicle benefits in reducing driving stress: a simulation experiment approach

Driving is a stressful activity because of the mental workload required to maneuver a vehicle in certain travel contexts, such as congested traffic, multi-modal networks requiring complex interaction with surrounding vehicles, and aggressive driving. Autonomous vehicles (AVs), on the other hand, can reduce the mental workload by performing most of the driving tasks and providing users with a comfortable ride. This study develops a pathway model to relate different health determinants, including travel reliability, safety, driving comfort, and value of time, to Autonomous vehicles driving and studies their impact on the value of driving stress. A case study example of Autonomous vehicles simulation is used to determine the impact of these health determinants. The value of driving stress in Autonomous vehicles is estimated as a function of the value of these individual health determinants. The results show that the perception of safe or unsafe driving in Autonomous vehicles is the most important factor in changing the perception of driving stress in Autonomous vehicles. Similarly, perceptions of comfortable driving in Autonomous vehicles and reduced workload with a higher value of time also reduce driving stress in Autonomous vehicles. These results allow Autonomous vehicles adoption models to explicitly consider driving stress reduction as a benefit and can improve understanding of Autonomous vehicles adoption, which may require quantitative analysis of underlying motivating benefits, including driving stress reduction.

Khattak, Zulqarnain H.↗
DOE OSTI2020

Laboratory Validation of Integrated Lighting Systems Retrofit Performance and Energy Savings

Light-emitting diodes (LED) fixtures and lamps have emerged as leading technologies for general illumination and are a well-established energy efficiency retrofit measure in commercial buildings (from around 2% of installed fixtures and lamps in 2013 to 28% by 2020). Retrofit approaches that integrate elements, such as networked controls, daylight dimming, and advanced shade technologies lag in comparison. Integrated retrofits have been shown to increase savings over single end-use retrofits, but are perceived as higher complexity and risk. More validation of integrated lighting system performance is needed. This study presents results from laboratory testing of three packages combining fixtures, networked controls, task tuning, and daylight dimming, advanced shades, and lighting layout changes. We characterize performance in perimeter open-office zones, finding energy savings from 20% for daylight dimming and automated shades (no LED retrofit) to over 70% for LED retrofits with advanced controls and shades or lighting layout changes. We present some implementation details, including lessons learned from installation and commissioning in the laboratory setting. We also discuss cost-benefit analysis approaches for the types of packages presented, including the need to quantify and incorporate energy and non-energy benefits for advanced shades packages, which enhance occupant comfort but add significant cost.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1974

Atmospheric effects on the Space Shuttle approach and landing

The present work describes piloted simulations of Space Shuttle approach and landing where the pilot either flies the aircraft or monitors a computer-controlled flight with the option to take control. The simulated runs encompassed approach to the airport, acquisition of the glide slope and localizer, and accomplishing final flare and touchdown. The degree of difficulty associated with the flight task was imposed through modeled atmospheric winds and superimposed atmospheric turbulence. The simulator models the Orbiter vehicle with the aid of an all-digital computer complex and a fixed-base crew station. Comparison between piloted and computer-controlled responses to unexpected atmospheric conditions was obtained by recording data from instrument displays available to the pilot.

Ragsdale, W. A.↗
NASA NTRS1982

The pilot's role in manned space flight

The present investigation regarding the pilot's role in manned space flight takes into account space missions conducted with the Mercury capsule, Gemini, Apollo, Skylab, and the Shuttle. It is concluded that advancements in digital systems and automation technology have made many of the space pilot's tasks easier. However, these advancements have also made the space pilot's training more complicated. He must be familiar with the interrelated failure effects in highly complex vehicle systems. The nominal performance of the Shuttle fly-by-wire entry control system depends, for instance, on nominal electrical power from three fuel cells, nominal performance of three hydraulic auxiliary power units, five computers, other equipment, and microwave landing systems. The pilot must monitor and manage failures in these systems, and, in addition, must be prepared to intervene if an abort situation creates off-nominal conditions.

North, W. J.↗
NASA NTRS1990

Challenges in the 1990's for astronaut training simulators

New challenges for the simulation community at the Johnson Space Center both in near and long terms are considered. In the near term, the challenges of supporting an increasing flight rate, maintaining operations while replacing obsolete subsystems, and incorporating forthcoming changes to the Space Shuttle are discussed, and focus is placed on a change of forward flight-deck instruments from electro-mechanical devices to electronic displays. Training astronauts for complex concurrent missions involving multiple spacecraft and geographically dispersed ground facilities is considered to be foremost of the long-term challenges, in addition to the tasks of improving the simulator reliability and the operational efficiency of the facilities.

Brown, Patrick M.↗
NASA NTRS2000

Performance Assessment of OVERFLOW on Distributed Computing Environment

The aerodynamic computer code, OVERFLOW, with a multi-zone overset grid feature, has been parallelized to enhance its performance on distributed and shared memory paradigms. Practical application benchmarks have been set to assess the efficiency of code's parallelism on high-performance architectures. The code's performance has also been experimented with in the context of the distributed computing paradigm on distant computer resources using the Information Power Grid (IPG) toolkit, Globus. Two parallel versions of the code, namely OVERFLOW-MPI and -MLP, have developed around the natural coarse grained parallelism inherent in a multi-zonal domain decomposition paradigm. The algorithm invokes a strategy that forms a number of groups, each consisting of a zone, a cluster of zones and/or a partition of a large zone. Each group can be thought of as a process with one or multithreads assigned to it and that all groups run in parallel. The -MPI version of the code uses explicit message-passing based on the standard MPI library for sending and receiving interzonal boundary data across processors. The -MLP version employs no message-passing paradigm; the boundary data is transferred through the shared memory. The -MPI code is suited for both distributed and shared memory architectures, while the -MLP code can only be used on shared memory platforms. The IPG applications are implemented by the -MPI code using the Globus toolkit. While a computational task is distributed across multiple computer resources, the parallelism can be explored on each resource alone. Performance studies are achieved with some practical aerodynamic problems with complex geometries, consisting of 2.5 up to 33 million grid points and a large number of zonal blocks. The computations were executed primarily on SGI Origin 2000 multiprocessors and on the Cray T3E. OVERFLOW's IPG applications are carried out on NASA homogeneous metacomputing machines located at three sites, Ames, Langley and Glenn. Plans for the future will exploit the distributed parallel computing capability on various homogeneous and heterogeneous resources and large scale benchmarks. Alternative IPG toolkits will be used along with sophisticated zonal grouping strategies to minimize the communication time across the computer resources.

Djomehri, M. Jahed↗
NASA NTRS2004

Clarissa Spoken Dialogue System for Procedure Reading and Navigation

Speech is the most natural modality for humans use to communicate with other people, agents and complex systems. A spoken dialogue system must be robust to noise and able to mimic human conversational behavior, like correcting misunderstandings, answering simple questions about the task and understanding most well formed inquiries or commands. The system aims to understand the meaning of the human utterance, and if it does not, then it discards the utterance as being meant for someone else. The first operational system is Clarissa, a conversational procedure reader and navigator, which will be used in a System Development Test Objective (SDTO) on the International Space Station (ISS) during Expedition 10. In the present environment one astronaut reads the procedure on a Manual Procedure Viewer (MPV) or paper, and has to stop to read or turn pages, shifting focus from the task. Clarissa is designed to read and navigate ISS procedures entirely with speech, while the astronaut has his eyes and hands engaged in performing the task. The system also provides an MPV like graphical interface so the procedure can be read visually. A demo of the system will be given.

Hieronymus, James↗
NASA NTRS2005

Autonomic Cluster Management System (ACMS): A Demonstration of Autonomic Principles at Work

Cluster computing, whereby a large number of simple processors or nodes are combined together to apparently function as a single powerful computer, has emerged as a research area in its own right. The approach offers a relatively inexpensive means of achieving significant computational capabilities for high-performance computing applications, while simultaneously affording the ability to. increase that capability simply by adding more (inexpensive) processors. However, the task of manually managing and con.guring a cluster quickly becomes impossible as the cluster grows in size. Autonomic computing is a relatively new approach to managing complex systems that can potentially solve many of the problems inherent in cluster management. We describe the development of a prototype Automatic Cluster Management System (ACMS) that exploits autonomic properties in automating cluster management.

Baldassari, James D.↗
NASA NTRS1999

Assessment of Human Factors

Human Factors Engineering, often referred to as Ergonomics, is a science that applies a detailed understanding of human characteristics, capabilities, and limitations to the design, evaluation, and operation of environments, tools, and systems for work and daily living. Human Factors is the investigation, design, and evaluation of equipment, techniques, procedures, facilities, and human interfaces, and encompasses all aspects of human activity from manual labor to mental processing and leisure time enjoyments. In spaceflight applications, human factors engineering seeks to: (1) ensure that a task can be accomplished, (2) maintain productivity during spaceflight, and (3) ensure the habitability of the pressurized living areas. DSO 904 served as a vehicle for the verification and elucidation of human factors principles and tools in the microgravity environment. Over six flights, twelve topics were investigated. This study documented the strengths and limitations of human operators in a complex, multifaceted, and unique environment. By focusing on the man-machine interface in space flight activities, it was determined which designs allow astronauts to be optimally productive during valuable and costly space flights. Among the most promising areas of inquiry were procedures, tools, habitat, environmental conditions, tasking, work load, flexibility, and individual control over work.

Mount, Frances↗
DOE OSTI2024

DECOVALEX-2023: Task C Final Report

The Full-scale Emplacement (FE) heater experiment at the Mont Terri Underground Rock Laboratory (URL) was designed and conducted by Nagra to replicate an emplacement tunnel of Nagra’s reference repository design at 1:1 scale. Alongside testing the technical feasibility of constructing disposal tunnels, emplacing waste containers in the tunnels and then backfilling them, the main goals of the FE experiment are (1) to obtain a better understanding of the coupled effects of induced thermo-hydro-mechanical (THM) processes that may occur and (2) to validate existing coupled THM models (Müller et al., 2017). A key aspect of ensuring safety for repositories located in low-permeability rock involves minimizing any damage to the rock itself, thereby preserving its integrity and promoting a stable environment Amongst a number of processes that could damage the rock is the increase in pore pressure due to thermal loading caused by heat emitted from the waste. To reduce the potential damage of the rock, it is important to analyse the evolution of heat over time due to the heat load of the containers and assess possible consequences by coupled THM models. The aim of Task C of DECOVALEX-2023 was to build 3D numerical models of the FE experiment, focussing in particular on the heating induced pore pressure change in the Opalinus Clay. Data from a large number of sensors were available from the FE experiment for model comparison. These sensors measured temperature and relative humidity in the bentonite around the heaters, and temperature, pressure and displacement/strain in the surrounding Opalinus clay. Data were available from the start of excavation (April 2012) up to August 2020 for most sensors (more than 5 years from the start of heating in December 2014). To fulfil the overall aim of the task, the work was broken down into a number of steps, starting with simpler models to build confidence in each team’s approach and then moving to more complex models that better represent the FE experiment. Step 0 consisted of 2D benchmark models, gradually increasing the number of processes that are represented from thermal (T) only models in Step 0a, to coupled thermal hydraulic (TH) models in Step 0b with a representation of changing porosity, to coupled thermo-hydro-mechanical (THM) models in Step 0c, where porosity changes are calculated by the mechanical model. A detailed specification of processes, parameters, initial and boundary conditions was provided for this step, with the ambition that all teams would work towards close agreement in their model results, thus building confidence in the model implementations. vi It was not straightforward to achieve agreement between the teams, so additional steps (Step 0b2, 0b3, 0c2, 0c3) were added along with derivation of some analytical solutions against which the models could be compared. The reasons for the differences between teams were investigated and found to be caused primarily by different conceptual model assumptions (including temperature dependence of the thermal expansion of water), different model formulations (including porosity evolution) and differences in modelled domain sizes, boundary conditions and grid discretisation. This demonstrates that comparisons between multiple modelling teams and/or comparison with analytical results and experimental data are highly beneficial in providing an indication of uncertainty in model predictions. At the conclusion of Step 0, almost all teams had achieved a close agreement in model results and those that had not achieved an agreement knew the reason for this. Step 1 moved from 2D models to 3D models of the FE experiment without adding technical features like shotcrete or EDZ, and only considering the heating phase. Initially the 3D model was tightly specified to continue to build confidence in the model implementations (Step 1a). The results of Step 1a were compared to the data from the FE-experiment without the teams seeing the data. The teams were then provided with a sub-set of the data from the FE-experiment and invited to consider how best to use the large dataset for model comparison (Step 1b). Teams were then asked to use the data provided to calibrate their models, only changing material property values rather than adding features or processes to their models (Step 1c). In Step 1, teams were asked to only model the heating phase of the experiment, so pressure in the Opalinus Clay was reported as change in pressure since the initial conditions were specified rather than modelled. The change from 2D to 3D models was accompanied by an increase in the dispersion of results between the teams. Some of this was resolved during the task, but some remained and is potentially due to model discretisation. Calibration of parameters was useful in improving the fit of the models to the data but the remaining differences indicated that the models were missing features or processes. In Step 2, the teams were asked to update their models with additional features and processes as well as calibrating parameters to try and improve the fit of the models to the data. Teams were encouraged to represent ventilation of the open FE tunnel prior to backfilling with heaters and bentonite and in Step 2, the absolute pressure in the Opalinus Clay was compared between the teams. Teams took different approaches, but there was consideration of adding shotcrete and an EDZ into the model, representing stress change during excavation and different approaches to modelling ventilation of the FE tunnel. Overall, the documented results showed a very good agreement for temperature. The results for porewater pressure evolution showed a significant improvement for most teams compared to Step 1c with a good agreement to the measurements for several teams whereas some teams overpredicted the pressure increase and others overpredicted the drainage effect especially for the sensors close to the heater. Step 3 was an opportunity for teams to use the models developed in Step 1 and Step 2 to make predictions about the temperature and pressure changes that will be expected at the FE experiment over the next few years in light of the planned changes in thermal output of the heaters.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS1998

Preliminary Work for Examining the Scalability of Reinforcement Learning

Researchers began studying automated agents that learn to perform multiple-step tasks early in the history of artificial intelligence (Samuel, 1963; Samuel, 1967; Waterman, 1970; Fikes, Hart & Nilsonn, 1972). Multiple-step tasks are tasks that can only be solved via a sequence of decisions, such as control problems, robotics problems, classic problem-solving, and game-playing. The objective of agents attempting to learn such tasks is to use the resources they have available in order to become more proficient at the tasks. In particular, each agent attempts to develop a good policy, a mapping from states to actions, that allows it to select actions that optimize a measure of its performance on the task; for example, reducing the number of steps necessary to complete the task successfully. Our study focuses on reinforcement learning, a set of learning techniques where the learner performs trial-and-error experiments in the task and adapts its policy based on the outcome of those experiments. Much of the work in reinforcement learning has focused on a particular, simple representation, where every problem state is represented explicitly in a table, and associated with each state are the actions that can be chosen in that state. A major advantage of this table lookup representation is that one can prove that certain reinforcement learning techniques will develop an optimal policy for the current task. The drawback is that the representation limits the application of reinforcement learning to multiple-step tasks with relatively small state-spaces. There has been a little theoretical work that proves that convergence to optimal solutions can be obtained when using generalization structures, but the structures are quite simple. The theory says little about complex structures, such as multi-layer, feedforward artificial neural networks (Rumelhart & McClelland, 1986), but empirical results indicate that the use of reinforcement learning with such structures is promising. These empirical results make no theoretical claims, nor compare the policies produced to optimal policies. A goal of our work is to be able to make the comparison between an optimal policy and one stored in an artificial neural network. A difficulty of performing such a study is finding a multiple-step task that is small enough that one can find an optimal policy using table lookup, yet large enough that, for practical purposes, an artificial neural network is really required. We have identified a limited form of the game OTHELLO as satisfying these requirements. The work we report here is in the very preliminary stages of research, but this paper provides background for the problem being studied and a description of our initial approach to examining the problem. In the remainder of this paper, we first describe reinforcement learning in more detail. Next, we present the game OTHELLO. Finally we argue that a restricted form of the game meets the requirements of our study, and describe our preliminary approach to finding an optimal solution to the problem.

Clouse, Jeff↗
NASA NTRS1991

The exploration metaphor

NASA's experience in planetary exploration has demonstrated that the desktop workstation is inadequate for many visualization situations. The primary mission displays for the unmanned Surveyor missions to the moon during the mid-1960's, for example, were environmental images assembled on the inside surfaces of spherical shells. Future exploration missions will greatly benefit from advances in digital computer and display technology, but there remain unmet user interface needs. Alternative user interfaces and metaphors are needed for planetary exploration and other interactions with complex spatial environments. These interfaces and metaphors would enable the user to directly explore environments and naturally manipulate objects in those environments. Personal simulators, virtual workstations, and telepresence user interfaces are systems capable of providing this integration of user space and task space. The Exploration Metaphor is a useful concept for guiding the design of user interfaces for virtual environments and telepresence. To apply the Exploration Metaphor is to assert that computing is like exploration, and to support objects, operations, and contexts comparable to those encountered in the exploration of natural environments. The Exploration Metaphor, under development for user interfaces in support of NASA's planetary exploration missions and goals, will also benefit other applications where complex spatial information must be visualized. Visualization methods and systems for planetary exploration are becoming increasingly integrated and interactive as computing technology improves. These advances will benefit from virtual environment and telepresence interface technology. A key development has been the processing of multiple images and other sensor data to create detailed digital models of the planets and moons. Data from images of the Earth, Mars, and Miranda, for example, have been converted into 3D models, and dynamic virtual fly-overs have been computed as demonstrations. Similar processing of lower altitude photography and the use of computer aided design tools promise to produce very detailed models in the future.

Mcgreevy, Michael W.↗
NASA NTRS1992

Comprehensive system models: Strategies for evaluation

The task of evaluating comprehensive earth system models is vast involving validations of every model component at every scale of organization, as well as tests of all the individual linkages. Even the most detailed evaluation of each of the component processes and the individual links among them should not, however, engender confidence in the performance of the whole. The integrated earth system is so rich with complex feedback loops, often involving components of the atmosphere, oceans, biosphere, and cryosphere, that it is certain to exhibit emergent properties very difficult to predict from the perspective of a narrow focus on any individual component of the system. Therefore, a substantial share of the task of evaluating comprehensive earth system models must reside at the level of whole system evaluations. Since complete, integrated atmosphere/ ocean/ biosphere/ hydrology models are not yet operational, questions of evaluation must be addressed at the level of the kinds of earth system processes that the models should be competent to simulate, rather than at the level of specific performance criteria. Here, we have tried to identify examples of earth system processes that are difficult to simulate with existing models and that involve a rich enough suite of feedbacks that they are unlikely to be satisfactorily described by highly simplified or toy models. Our purpose is not to specify a checklist of evaluation criteria but to introduce characteristics of the earth system that may present useful opportunities for model testing and, of course, improvement.

Field, Christopher↗