Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23

Development of aerosol activation in the double-moment Unified Model and evaluation with CLARIFY measurements

Representing the number and mass of cloud and aerosol particles independently in a climate, weather prediction or air quality model is important in order to simulate aerosol direct and indirect effects on radiation balance. Here we introduce the first configuration of the UK Met Office Unified Model in which both cloud and aerosol particles have “double-moment” representations with prognostic number and mass. The GLObal Model of Aerosol Processes (GLOMAP) aerosol microphysics scheme, already used in the Hadley Centre Global Environmental Model version 3 (HadGEM3) climate configuration, is coupled to the Cloud AeroSol Interacting Microphysics (CASIM) cloud microphysics scheme. We demonstrate the performance of the new configuration in high-resolution simulations of a case study defined from the CLARIFY aircraft campaign in 2017 near Ascension Island in the tropical southern Atlantic. We improve the physical basis of the activation scheme by representing the effect of existing cloud droplets on the activation of new aerosol, and we also discuss the effect of unresolved vertical velocities. We show that neglect of these two competing effects in previous studies led to compensating errors but realistic droplet concentrations. While these changes lead only to a modest improvement in model performance, they reinforce our confidence in the ability of the model microphysics code to simulate the aerosol–cloud microphysical interactions it was designed to represent. Capturing these interactions accurately is critical to simulating aerosol effects on climate.

54 ENVIRONMENTAL SCIENCES↗

Toward a chemical reanalysis in a coupled chemistry-climate model: An evaluation of MOPITT CO assimilation and its impact on tropospheric composition

Here, we examine in detail a 1 year global reanalysis of carbon monoxide (CO) that is based on joint assimilation of conventional meteorological observations and Measurement of Pollution in The Troposphere (MOPITT) multispectral CO retrievals in the Community Earth System Model (CESM). Our focus is to assess the impact to the chemical system when CO distribution is constrained in a coupled full chemistry-climate model like CESM. To do this, we first evaluate the joint reanalysis (MOPITT Reanalysis) against four sets of independent observations and compare its performance against a reanalysis with no MOPITT assimilation (Control Run). We then investigate the CO burden and chemical response with the aid of tagged sectoral CO tracers. We estimate the total tropospheric CO burden in 2002 (from ensemble mean and spread) to be 371 ± 12% Tg for MOPITT Reanalysis and 291 ± 9% Tg for Control Run. Our multispecies analysis of this difference suggests that (a) direct emissions of CO and hydrocarbons are too low in the inventory used in this study and (b) chemical oxidation, transport, and deposition processes are not accurately and consistently represented in the model. Increases in CO led to net reduction of OH and subsequent longer lifetime of CH 4 (Control Run: 8.7 years versus MOPITT Reanalysis: 9.3 years). Yet at the same time, this increase led to 5-10% enhancement of Northern Hemisphere O 3 and overall photochemical activity via HO x recycling. Such nonlinear effects further complicate the attribution to uncertainties in direct emissions alone. This has implications to chemistry-climate modeling and inversion studies of longer-lived species.

54 ENVIRONMENTAL SCIENCES↗

An Overview of Atmospheric Chemistry and Air Quality Modeling

This presentation will include my personal research experience and an overview of atmospheric chemistry and air quality modeling to the participants of the NASA Student Airborne Research Program (SARP 2017). The presentation will also provide examples on ways to apply airborne observations for chemical transport (CTM) and air quality (AQ) model evaluation. CTM and AQ models are important tools in understanding tropospheric-stratospheric composition, atmospheric chemistry processes, meteorology, and air quality. This presentation will focus on how NASA scientist currently apply CTM and AQ models to better understand these topics. Finally, the importance of airborne observation in evaluating these topics and how in situ and remote sensing observations can be used to evaluate and improve CTM and AQ model predictions will be highlighted.

Atmospheric↗

Modern sensitivity analysis - 15651

Sensitivity analysis has been applied to complex models, such as performance assessment models, since at least the advent of computational tools to support such an approach to model evaluation. Initially, sensitivity analysis methods were applied to deterministic models and focused on changing one input variable at a time. Probabilistic modeling opened the door to evaluating changes in all input variables simultaneously. Standardized rank regression methods were primarily used in the last century to evaluate probabilistic models. These methods assume linearity in the rank response of the model to the inputs. At the very least this assumes monotonicity in the response, but for the most part the response needs to be near-linear. This is highly unlikely for performance assessment models that are based on differential equations, multiplication and division of variables that are usually not correlated, and conservatism that is not well understood. There are better statistical methods for evaluating highly non-linear and dynamic probabilistic models. Some of these methods were founded in the data mining industry. This century has seen a rapid development of the internet that led directly to the development of data mining techniques for massive databases. Although the goals were different, these techniques can be adapted to sensitivity analysis for complex probabilistic models, which, also, form large, or even massive, databases. These modern sensitivity analysis procedures assess the importance of every explanatory variable (input parameters) used in a probabilistic model. In practice, this means that for each endpoint of interest, every explanatory variable in the model is associated with a quantitative measure of importance. For a given explanatory variable, the importance value depends on the endpoint of interest. This global approach to sensitivity analysis allows all levels of interactions to be evaluated, so that the variation in the output can be measured at every level of possible interaction. The effects are collected together during sensitivity analysis processing, and then separated out to attribute the overall contribution of each input parameter to the model output. These global sensitivity analysis procedures have the ability to characterize non-linear and non-monotonic relationships between explanatory variables and endpoints of interest. This is critically important because of the need to characterize complex interactions among multiple explanatory variables in PA models that often have non-linear and non-monotonic effects. The options for sensitivity analysis are fully explored, including an evaluation of several methods when applying them to a well-known complex non-linear non-monotonic model. The results clearly demonstrate the relative benefit of using data mining methods over simple statistical approaches such as standardized rank regression. (authors)

97 MATHEMATICS AND COMPUTING↗

Evaluation and Improvement of Cloud and Convective Parameterizations from Analyses of ARM Observations and Models

Over this period the PI and his performed a broad range of data analysis, model evaluation, and model improvement studies using ARM data. These included cloud regimes in the TWP and their evolution over the MJO; M-PACE IOP SCM-CRM intercomparisons; simulations of convective updraft strength and depth during TWP-ICE; evaluation of convective entrainment parameterizations using TWP-ICE simulations; evaluation of GISS GCM cloud behavior vs. long-term SGP cloud statistics; classification of aerosol semi-direct effects on cloud cover; depolarization lidar constraints on cloud phase; preferred states of the winter Arctic atmosphere, surface, and sub-surface; sensitivity of convection to tropospheric humidity; constraints on the parameterization of mesoscale organization from TWP-ICE WRF simulations; updraft and downdraft properties in TWP-ICE simulated convection; insights from long-term ARM records at Manus and Nauru.

54 ENVIRONMENTAL SCIENCES↗

Integrated Modeling Driven Evaluation of Opportunities for Climate‐Resilient Perennial Biomass Crop Plantings in Flood‐Prone Agricultural Landscapes

Adapting to future climate change in flood-prone landscapes will require climate-resilient agricultural systems. Planting perennial crops, like switchgrass and willow, along river corridors can mitigate future flooding while supporting bioenergy markets. We developed an integrated assessment linking climate, hydrologic, and inundation model results to assess future flood risk to river-adjacent agricultural lands in the Mid-Atlantic Region (MAR) and explore this opportunity. We produced ensemble streamflow projections for every MAR stream using a hydrologic model driven by a suite of downscaled and bias-corrected Coupled Model Intercomparison Project Phase 6 climate projections. We then conducted high-resolution inundation mapping based on projected flood frequencies for baseline and future periods. Results show that in the near-term future, at least two-thirds of the streams will experience 100-year floods more severe than the baseline 200-year floods. Riparian zones are projected to face a median rise of inundation by 9.5%–24.1%. Results show that there is an opportunity to mitigate flooding in over half of MAR's counties with the quantities of switchgrass and willow plantings anticipated for mature bioenergy markets, even under the most extreme (200-year) flood events. Our integrated modeling framework can guide similar regions to evaluate opportunities for flood-resilient agricultural systems under climate change.

60 APPLIED LIFE SCIENCES↗

Computer simulation incorporating a helicopter model for evaluation of aircraft avionics systems

A computer program was developed to integrate avionics research in navigation, guidance, controls, and displays with a realistic aircraft model. A user oriented program is described that allows a flexible combination of user supplied models to perform research in any avionics area. A preprocessor technique for selecting various models without significantly changing the memory storage is included. Also included are mathematical models for several avionics error models and for the CH-47 helicopter used in this program.

Ostroff, A. J.↗

Status of yield estimation technology: A review of second-generation model development and evaluation

Multiple regression models were studied in order to determine their yield estimation capability for any arbitrary unit area and to obtain greater responsiveness and accuracy through the use of additional data sources applied at smaller spatial and temporal scales. It was concluded that data base inadequacy was the factor limiting performance in the models studied and that each of the models has more yield predicting capability than was reached during LACIE.

Stuff, R. G.↗

A simple model for straggling evaluation

Some straggling models had largely been abandoned in favor of Monte Carlo simulations of straggling which are accurate but time consuming, limiting their application in practice. The difficulty of simple analytic models is the failure to give accurate values past 85% of the particle range. A simple model is derived herein based on a second order approximation upon which rapid analysis tools are developed for improved understanding of material charged particle transmission properties. Published by Elsevier Science B.V.

Models, Statistical↗

A Simple Model for Straggling Evaluation

Simple straggling models had largely been abandoned in favor of Monte Carlo simulations of straggling which are accurate but time consuming, limiting their application in practice. The difficulty of simple analytic models is the failure to give accurate values past 85% of the particle range. A simple model is derived herein based on a second order approximation upon which rapid analysis tools are developed for improved understanding of material charged particle transmission properties.

Wilson, J. W.↗

Bridging the Gap Between Requirements and Model Analysis: Evaluation on Cyber-Physical Challenge Problems

We present a framework for introducing high-level requirement specifications in the automated analysis of dataflow models. By integrating the FRET requirements elicitation tool with the CoCoSim analyzer, our framework enables the analysis of hierarchical Simulink models against requirements written in a restricted English language. More precisely, we support: automatic extraction of Simulink model information and association of high-level requirements with target model signals and components; translation of temporal logic formulas into synchronous dataflow CoCoSpec specifications as well as Simulink monitors; and interpretation of counterexamples produced by the analysis both at the requirement and model level. The features provided by our framework are generic and can be used to integrate other requirements elicitation and Simulink/Lustre analysis tools. We report on the lessons learned from the application of our approach to the Lockheed Martin Cyber-Physical, aerospace-inspired challenge problems. For the analysis, we used the Kind2, Zustre, and Simulink Design Verifier (SLDV) tools.

Simulink↗

Development and Implementation of Dynamic Scripts to Support Local Model Verification at National Weather Service Weather Forecast Offices

Local modeling with a customized configuration is conducted at National Weather Service (NWS) Weather Forecast Offices (WFOs) to produce high-resolution numerical forecasts that can better simulate local weather phenomena and complement larger scale global and regional models. The advent of the Environmental Modeling System (EMS), which provides a pre-compiled version of the Weather Research and Forecasting (WRF) model and wrapper Perl scripts, has enabled forecasters to easily configure and execute the WRF model on local workstations. NWS WFOs often use EMS output to help in forecasting highly localized, mesoscale features such as convective initiation, the timing and inland extent of lake effect snow bands, lake and sea breezes, and topographically-modified winds. However, quantitatively evaluating model performance to determine errors and biases still proves to be one of the challenges in running a local model. Developed at the National Center for Atmospheric Research (NCAR), the Model Evaluation Tools (MET) verification software makes performing these types of quantitative analyses easier, but operational forecasters do not generally have time to familiarize themselves with navigating the sometimes complex configurations associated with the MET tools. To assist forecasters in running a subset of MET programs and capabilities, the Short-term Prediction Research and Transition (SPoRT) Center has developed and transitioned a set of dynamic, easily configurable Perl scripts to collaborating NWS WFOs. The objective of these scripts is to provide SPoRT collaborating partners in the NWS with the ability to evaluate the skill of their local EMS model runs in near real time with little prior knowledge of the MET package. The ultimate goal is to make these verification scripts available to the broader NWS community in a future version of the EMS software. This paper provides an overview of the SPoRT MET scripts, instructions for how the scripts are run, and example use cases.

Zavordsky, Bradley↗

Development and Implementation of Dynamic Scripts to Support Local Model Verification at National Weather Service Weather Forecast Offices

Local modeling with a customized configuration is conducted at National Weather Service (NWS) Weather Forecast Offices (WFOs) to produce high-resolution numerical forecasts that can better simulate local weather phenomena and complement larger scale global and regional models. The advent of the Environmental Modeling System (EMS), which provides a pre-compiled version of the Weather Research and Forecasting (WRF) model and wrapper Perl scripts, has enabled forecasters to easily configure and execute the WRF model on local workstations. NWS WFOs often use EMS output to help in forecasting highly localized, mesoscale features such as convective initiation, the timing and inland extent of lake effect snow bands, lake and sea breezes, and topographically-modified winds. However, quantitatively evaluating model performance to determine errors and biases still proves to be one of the challenges in running a local model. Developed at the National Center for Atmospheric Research (NCAR), the Model Evaluation Tools (MET) verification software makes performing these types of quantitative analyses easier, but operational forecasters do not generally have time to familiarize themselves with navigating the sometimes complex configurations associated with the MET tools. To assist forecasters in running a subset of MET programs and capabilities, the Short-term Prediction Research and Transition (SPoRT) Center has developed and transitioned a set of dynamic, easily configurable Perl scripts to collaborating NWS WFOs. The objective of these scripts is to provide SPoRT collaborating partners in the NWS with the ability to evaluate the skill of their local EMS model runs in near real time with little prior knowledge of the MET package. The ultimate goal is to make these verification scripts available to the broader NWS community in a future version of the EMS software. This paper provides an overview of the SPoRT MET scripts, instructions for how the scripts are run, and example use cases.

Zavodsky, Bradley↗

Chapter 10 - Remote Sensing Measurements of Aerosol Properties

Satellite instruments have proven especially capable at monitoring the quantity of airborne particles in columns of atmosphere, globally. This chapter describes the principles of satellite measurements and retrieval algorithms, and surveys current instruments and their capabilities. We outline the issues associated with retrieval algorithms, such as surface characterization and aerosol proximity to clouds, and the challenges with interpretation of the results. The relationship between measured aerosol properties and climate-relevant aerosol properties simulated in models is outlined, as well as how measurements are used to evaluate models. Most space-based aerosol instruments are passive sensors that measure reflected sunlight at multiple wavelengths, some at multiple viewing angles. A few are active sensors that send out their own laser light and measure the returned signal. Except when clouds are present, the excess amount of light scattered back to space, beyond that expected from the surface and atmospheric gas, is attributed to aerosol. Satellite measurements are used in many ways in aerosol research. They often provide the only method for monitoring hazardous phenomena such as major wildfire and volcanic eruption plumes, especially in remote areas. Stable, long-term, near-global-scale satellite data records make it possible to identify regional and global aerosol trends. Aerosol radiative effects on climate can be quantified on a near-global scale and used to estimate the strength of aerosol–radiation and aerosol–cloud interactions as well as to evaluate climate model simulations of these interactions. Aerosol-type mapping from satellite imagery is helpful for source attribution, model validation, and to constrain particle light-absorption properties that are essential for radiative forcing calculations. The range of aerosol properties retrieved from satellite observations has grown considerably since the first global estimates of aerosol optical depth (τ a) over ocean were made in the late 1970s. Methods for retrieving particle size and light-absorption properties were explored in the 1990s using multispectral, multi-angle observations, and polarization in visible and near-infrared wavelengths. Sensitivity to particle light absorption, primarily from black or brown carbon content, improved with the inclusion of UV channels, and sensitivity to very thin aerosol layers in the upper troposphere and lower stratosphere was advanced with the use of limb-sounding instruments and active sensors. There are limitations to every measurement technique, including satellite aerosol remote sensing. For wide-swath, passive instruments, aerosol retrievals near clouds can present substantial challenges as far as 15 km away due to cloud-scattered light contaminating the signal. In nearly all cases, retrievals over bright snow and ice surfaces are precluded because surface reflectance uncertainties can overwhelm the aerosol signal. Similarly, meteorological cloud is identified and masked out where possible. Data from passive sensors also lack vertical resolution except those that view toward the limb or where multi-angle imagery is acquired over plumes from wildfires, erupting volcanoes, and wind-blown dust. Yet, passive sensors provide vastly more coverage than the active instruments that mitigate these issues. Particle microphysical information is qualitative from all remote sensing techniques, relying on proxies to infer particle composition, hygroscopicity, and the amount of light-absorbing material. Further, particles smaller than about 200 nm diameter cannot be distinguished from atmospheric gas molecules with remote sensing, which hinders studies of cloud condensation nuclei and their effects on clouds. Most satellite instruments dedicated to aerosol observations are in low-Earth, near-polar, sun-synchronous orbits, which means they cross the equator at the same local time each day. Most are set on cycles that repeat approximately every 16 days, which makes it difficult to monitor aerosol evolution locally. Geostationary satellites make it possible to observe changes occurring from minutes to hours over regions up to 8000 km in size, but lack coverage of high latitudes, and often provide more limited constraints on aerosol properties. Ground-truth data are vital for satellite aerosol-retrieval validation. The AErosol RObotic NETwork (AERONET) of sun photometers was created in 1993 and has become an established global network of over 350 instruments for validating satellite measurements. The network, as well as global networks of ground-based lidars, solar flux radiometers and other sun photometers, are widely used for evaluating global satellite retrievals and model simulations. NASA's Earth Observing System (EOS) program beginning in 1999 led to improvements in reliability, spatial resolution, and spectral resolution (and hence, to improved particle size discrimination and light absorption properties). Satellite payloads include advanced broad-swath and multi-angle imagers, along with the first space-based active sensor focused largely on long-term aerosol monitoring. Since about 2002, Europe's SENTINEL and operational meteorological satellite fleets are also providing sustained aerosol observations, with planned continuation until at least 2030. Satellite remote sensing instruments offer valuable data for evaluating aerosol representations in global climate models. They have been used to assess aerosol optical and physical properties, trends and distributions, and are applied increasingly as direct model constraints in data assimilation to create global aerosol reanalysis products. Aerosol optical depth is the most common quantity adopted for routine model evaluation, including multiwavelength data to loosely constrain particle-size distributions. These evaluations of multiple models have revealed general biases in their regional aerosol amounts and seasonal patterns of transport and removal. Although satellite measurements have near-global coverage, substantial errors can be introduced into the model observation comparison unless attention is paid to spatial and temporal collocation, cloud screening, subgrid-scale variability, and measurement uncertainties that vary with retrieval conditions.

aerosol properties↗

Modeling and evaluating 239 Pu and 235 U PFNS and average prompt-neutron multiplicity [Slides]

The following are currently in VIII.1 LANL and IAEA test files: 239 Pu nu-bar including CGMF modeling and CEA data, 239 Pu PFNS including Chi-Nu & CEA data, 239 Pu(n,f) cross section including fission TPC data. The following are currently being tested: 235 U nu-bar including CGMF modeling and 235 U PFNS including Chi-Nu data. Upcoming tasks include the correction of 235 U Chi-Nu PFNS at higher E inc , benchmarking 235 U PFNS and nu-bar evaluations, and getting 238 U nu-bar.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

TPSAS-NF1676L-33231-DND

It is well established that clouds have a profound influence on Earth's energy budget and that cloud feedbacks are responsible for much of the uncertainty in climate model projections of global warming in response to radiative forcing by greenhouse gases. While most model evaluation efforts focus on model representation of the observed mean state. For example, the regional mean climatology of key climate variables there is also tremendous value in testing how a model represents observed internal variations of the climate system. Here we use CERES observations during and after the so-called global warming hiatus to evaluate a subset of CMIP6 climate models. The CERES data show a marked 0.83 Wm-2 reduction in global mean reflected SW TOA flux during the three years following the hiatus that results in an increase in net energy into the climate system. The primary driver of the decrease in SW TOA flux is a reduction in low cloud cover over the eastern Pacific Ocean that occurs in response to an unprecedented increase in SSTs in that region following the hiatus. The CMIP6 simulations consist of AMIP-style experiments in which SSTs and sea-ice boundary conditions are constrained by observations. For each model, the AMIP runs were extended through the end of 2017 from the official CMIP6 AMIP end period in 2014. Results show that the models do a good job at capturing the pattern of observed changes in TOA flux observed by CERES, but tend to underestimate the magnitude.

Norman G Loeb↗

Multi-model Mean Nitrogen and Sulfur Deposition from the Atmospheric Chemistry and Climate Model Intercomparison Project (ACCMIP): Evaluation of Historical and Projected Future Changes

We present multi-model global datasets of nitrogen and sulfate deposition covering time periods from 1850 to 2100, calculated within the Atmospheric Chemistry and Climate Model Intercomparison Project (ACCMIP). The computed deposition fluxes are compared to surface wet deposition and ice-core measurements. We use a new dataset of wet deposition for 2000-2002 based on critical assessment of the quality of existing regional network data. We show that for present-day (year 2000 ACCMIP time-slice), the ACCMIP results perform similarly to previously published multi-model assessments. The analysis of changes between 1980 and 2000 indicates significant differences between model and measurements over the United States, but less so over Europe. This difference points towards misrepresentation of 1980 NH3 emissions over North America. Based on ice-core records, the 1850 deposition fluxes agree well with Greenland ice cores but the change between 1850 and 2000 seems to be overestimated in the Northern Hemisphere for both nitrogen and sulfur species. Using the Representative Concentration Pathways to define the projected climate and atmospheric chemistry related emissions and concentrations, we find large regional nitrogen deposition increases in 2100 in Latin America, Africa and parts of Asia under some of the scenarios considered. Increases in South Asia are especially large, and are seen in all scenarios, with 2100 values more than double 2000 in some scenarios and reaching >1300 mgN/m2/yr averaged over regional to continental scale regions in RCP 2.6 and 8.5, ~30-50% larger than the values in any region currently (2000). Despite known issues, the new ACCMIP deposition dataset provides novel, consistent and evaluated global gridded deposition fields for use in a wide range of climate and ecological studies.

ACCMIP↗