Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Mesoscale to microscale wind farm flow modeling and evaluation

The increasing size of wind turbines, with rotors already spanning more than 150 m diameter and hub heights above 100 m, requires proper modeling of the atmospheric boundary layer (ABL) from the surface to the free atmosphere. Furthermore, large wind farm arrays create their own boundary layer structure with unique physics. This poses significant challenges to traditional wind engineering models that rely on surface-layer theories and engineering wind farm models to simulate the flow in and around wind farms. However, adopting an ABL approach offers the opportunity to better integrate wind farm design tools and meteorological models. The challenge is how to build the bridge between atmospheric and wind engineering model communities and how to establish a comprehensive evaluation process that identifies relevant physical phenomena for wind energy applications with modeling and experimental requirements. A framework for model verification, validation, and uncertainty quantification is established to guide this process by a systematic evaluation of the modeling system at increasing levels of complexity. In terms of atmospheric physics, ‘building the bridge’ means developing models for the so-called ‘terra incognita,’ a term used to designate the turbulent scales that transition from mesoscale to microscale. This range of scales within atmospheric research deals with the transition from parameterized to resolved turbulence and the improvement of surface boundary-layer parameterizations. The coupling of meteorological and wind engineering flow models and the definition of a formal model evaluation methodology, is a strong area of research for the next generation of wind conditions assessment and wind farm and wind turbine design tools. Some fundamental challenges are identified in order to direct future research in this area.

17 WIND ENERGY↗
NASA NTRS2010

Space Weather Products at the Community Coordinated Modeling Center

The Community Coordinated Modeling Center (CCMC) is a US inter-agency activity aiming at research in support of the generation of advanced space weather models. As one of its main functions, the CCMC provides to researchers the use of space science models, even if they are not model owners themselves. The second CCMC activity is to support Space Weather forecasting at national Space Weather Forecasting Centers. This second activity involves model evaluations, model transitions to operations, and the development of space weather forecasting tools. Owing to the pace of development in the science community, new model capabilities emerge frequently. Consequently, space weather products and tools involve not only increased validity, but often entirely new capabilities. This presentation will review the present state of space weather tools as well as point out emerging future capabilities.

Hesse, Michael↗
NASA NTRS2009

Between the Rock and a Hard Place: The CCMC as a Transit Station Between Modelers and Forecasters

The Community Coordinated Modeling Center (CCMC) is a US inter-agency activity aiming at research in support of the generation of advanced space weather models. As one of its main functions, the CCMC provides to researchers the use of space science models, even if they are not model owners themselves. The second CCMC activity is to support Space Weather forecasting at national Space Weather Forecasting Centers. This second activity involved model evaluations, model transitions to operations, and the development of draft Space Weather forecasting tools. This presentation will focus on the latter element. Specifically, we will discuss the process of transition research models, or information generated by research models, to Space Weather Forecasting organizations. We will analyze successes as well as obstacles to further progress, and we will suggest avenues for increased transitioning success.

Hesse, Michael↗
NASA NTRS2009

Tools and Products of Real-Time Modeling: Opportunities for Space Weather Forecasting

The Community Coordinated Modeling Center (CCMC) is a US inter-agency activity aiming at research in support of the generation of advanced space weather models. As one of its main functions, the CCMC provides to researchers the use of space science models, even if they are not model owners themselves. The second CCMC activity is to support Space Weather forecasting at national Space Weather Forecasting Centers. This second activity involves model evaluations, model transitions to operations, and the development of draft Space Weather forecasting tools. This presentation will focus on the last element. Specifically, we will discuss present capabilities, and the potential to derive further tools. These capabilities will be interpreted in the context of a broad-based, bootstrapping activity for modern Space Weather forecasting.

Hesse, Michael↗
DOE OSTI2016

Model Test Bed for Evaluating Wave Models and Best Practices for Resource Assessment and Characterization

A wave model test bed is established to benchmark, test and evaluate spectral wave models and modeling methodologies (i.e., best practices) for predicting the wave energy resource parameters recommended by the International Electrotechnical Commission, IEC TS 62600-101Ed. 1.0 ©2015. Among other benefits, the model test bed can be used to investigate the suitability of different models, specifically what source terms should be included in spectral wave models under different wave climate conditions and for different classes of resource assessment. The overarching goal is to use these investigations to provide industry guidance for model selection and modeling best practices depending on the wave site conditions and desired class of resource assessment. Modeling best practices are reviewed, and limitations and knowledge gaps in predicting wave energy resource parameters are identified.

13 HYDRO ENERGY↗
DOE OSTI2020

The ARM Data-oriented Metrics and Diagnostics Package for Climate Models - A New Tool for Evaluating Climate Models with Field Data

The U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) program User Facility produces ground-based long-term continuous unique measurements for atmospheric state, precipitation, turbulent fluxes, radiation, aerosol, cloud and the land surface, which are collected at multiple sites. These comprehensive datasets have been widely used to calibrate climate models and are proven to be invaluable for climate model development and improvement. This article introduces an evaluation package to facilitate the use of ground-based ARM measurements in climate model evaluation. The ARM data-oriented metrics and diagnostics package (ARM-DIAGS) includes both ARM observational datasets and a Python-based analysis toolkit for computation and visualization. The observational datasets are compiled from multiple ARM data products and specifically tailored for use in climate model evaluation. In addition, ARM-DIAGS also includes simulation data from models participating the Coupled Model Inter-comparison Project (CMIP), which will allow climate-modeling groups to compare a new, candidate version of their model to existing CMIP models. The analysis toolkit is designed to make the metrics and diagnostics quickly available to the model developers.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2019

Beyond Static Benchmarking: Using Experimental Manipulations to Evaluate Land Model Assumptions

Land models are often used to simulate terrestrial responses to future environmental changes, but these models are not commonly evaluated with data from experimental manipulations. Results from experimental manipulations can identify and evaluate model assumptions that are consistent with appropriate ecosystem responses to future environmental change. We conducted simulations using three coupled carbon-nitrogen versions of the Community Land Model (CLM, versions 4, 4.5, and—the newly developed—5), and compared the simulated response to nitrogen (N) and atmospheric carbon dioxide (CO 2 ) enrichment with meta-analyses of observations from similar experimental manipulations. In control simulations, successive versions of CLM showed a poleward increase in gross primary productivity and an overall bias reduction, compared to FLUXNET-MTE observations. Simulations with N and CO 2 enrichment demonstrate that CLM transitioned from a model that exhibited strong nitrogen limitation of the terrestrial carbon cycle (CLM4) to a model that showed greater responsiveness to elevated concentrations of CO 2 in the atmosphere (CLM5). Overall, CLM5 simulations showed better agreement with observed ecosystem responses to experimental N and CO 2 enrichment than previous versions of the model. These simulations also exposed shortcomings in structural assumptions and parameterizations. Specifically, no version of CLM captures changes in plant physiology, allocation, and nutrient uptake that are likely important aspects of terrestrial ecosystems' responses to environmental change. These highlight priority areas that should be addressed in future model developments. Moving forward, incorporating results from experimental manipulations into model benchmarking tools that are used to evaluate model performance will help increase confidence in terrestrial carbon cycle projections.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Modeling and Evaluation of Cyber-Attacks on Grid-Interactive Efficient Buildings

Grid-interactive efficient buildings (GEBs) are not only exposed to passive threats (e.g., physical faults) but also active threats such as cyber-attacks launched on the network-based control systems. The impact of cyber-attacks on GEB operation are not yet fully understood, especially as regards the performance of grid services. To quantify the consequences of cyber-attacks on GEBs, this paper proposes a modeling and simulation framework that includes different cyber-attack models and key performance indexes to quantify the performance of GEB operation under cyber-attacks. The framework is numerically demonstrated to model and evaluate cyber-attacks such as data intrusion attacks and Denial-of-Service attacks on a typical medium-sized office building that uses the BACnet/IP protocol for communication networks. Simulation results show that, while different types of attacks could compromise the building systems to different extents, attacks via the remote control of a chiller yield the most significant consequences on a building system’s operation, including both the building service and the grid service. It is also noted that a cyber-attack impacts the building systems during the attack period as well as the post-attack period, which suggests that both periods should be considered to fully evaluate the consequences of a cyber-attack.

Fu, Yanyang↗
DOE OSTI2021

Simulation-based Performance Evaluation of Model Predictive Control for Building Energy Systems

The performance of model predictive control (MPC) can be significantly affected by different choices of controller parameters such as the time intervals for model discretization and control sampling. Due to the lack of a systematic understanding on how these parameters affect control performance, they are usually selected arbitrarily in practice.In this paper, the combined impacts of selected time intervals for model discretization and control sampling on the performance of MPC are comprehensively investigated for the first time through detailed simulations. Specifically, a typical MPC strategy is first designed to improve building operations based on a reduced-order model of building dynamics. Then, the performance of the designed MPC is evaluated against different choices of time intervals for model discretization and control sampling on a simulated office building. The detailed simulation results reveal that the time interval for model discretization has a much greater influence on the performance of MPC than the time interval for control sampling. Although the time interval for control sampling usually receives more attentions in practice, it turns out that the time interval for model discretization affects the prediction performance, cost saving, and computation time simultaneously and more significantly. Therefore, the simulation-based performance evaluation presented here sheds light on the impacts of different time intervals and facilitates their selection for practical applications of MPC to building operations

Huang, Sen↗
DOE OSTI2023

Evaluation of native Earth system model output with ESMValTool v2.6.0

Earth system models (ESMs) are state-of-the-art climate models that allow numerical simulations of the past, present-day, and future climate. To extend our understanding of the Earth system and improve climate change projections, the complexity of ESMs heavily increased over the last decades. As a consequence, the amount and volume of data provided by ESMs has increased considerably. Innovative tools for a comprehensive model evaluation and analysis are required to assess the performance of these increasingly complex ESMs against observations or reanalyses. One of these tools is the Earth System Model Evaluation Tool (ESMValTool), a community diagnostic and performance metrics tool for the evaluation of ESMs. Input data for ESMValTool needs to be formatted according to the CMOR (Climate Model Output Rewriter) standard, a process that is usually referred to as “CMORization”. While this is a quasi-standard for large model intercomparison projects like the Coupled Model Intercomparison Project (CMIP), this complicates the application of ESMValTool to non-CMOR-compliant climate model output. In this paper, we describe an extension of ESMValTool introduced in v2.6.0 that allows seamless reading and processing of “native” climate model output, i.e., operational output produced by running the climate model through the standard workflow of the corresponding modeling institute. This is achieved by an extension of ESMValTool's preprocessing pipeline that performs a CMOR-like reformatting of the native model output during runtime. Thus, the rich collection of diagnostics provided by ESMValTool is now fully available for these models. For models that use unstructured grids, a further preprocessing step required to apply many common diagnostics is regridding to a regular latitude–longitude grid. Extensions to ESMValTool's regridding functions described here allow for more flexible interpolation schemes that can be used on unstructured grids. Currently, ESMValTool supports nearest-neighbor, bilinear, and first-order conservative regridding from unstructured grids to regular grids. Example applications of this new native model support are the evaluation of new model setups against predecessor versions, assessing of the performance of different simulations against observations, CMORization of native model data for contributions to model intercomparison projects, and monitoring of running climate model simulations. For the latter, new general-purpose diagnostics have been added to ESMValTool that are able to plot a wide range of variable types. Currently, five climate models are supported: CESM2 (experimental; at the moment, only surface variables are available), EC-Earth3, EMAC, ICON, and IPSL-CM6. As the framework for the CMOR-like reformatting of native model output described here is implemented in a general way, support for other climate models can be easily added.

58 GEOSCIENCES↗
DOE OSTI2016

The GEWEX LandFlux project: Evaluation of model evaporation using tower-based and globally gridded forcing data

Determining the spatial distribution and temporal development of evaporation at regional and global scales is required to improve our understanding of the coupled water and energy cycles and to better monitor any changes in observed trends and variability of linked hydrological processes. With recent international efforts guiding the development of long-term and globally distributed flux estimates, continued product assessments are required to inform upon the selection of suitable model structures and also to establish the appropriateness of these multi-model simulations for global application. In support of the objectives of the Global Energy and Water Cycle Exchanges (GEWEX) LandFlux project, four commonly used evaporation models are evaluated against data from tower-based eddy-covariance observations, distributed across a range of biomes and climate zones. The selected schemes include the Surface Energy Balance System (SEBS) approach, the Priestley–Taylor Jet Propulsion Laboratory (PT-JPL) model, the Penman–Monteith-based Mu model (PM-Mu) and the Global Land Evaporation Amsterdam Model (GLEAM). Here we seek to examine the fidelity of global evaporation simulations by examining the multi-model response to varying sources of forcing data. To do this, we perform parallel and collocated model simulations using tower-based data together with a global-scale grid-based forcing product. Through quantifying the multi-model response to high-quality tower data, a better understanding of the subsequent model response to the coarse-scale globally gridded data that underlies the LandFlux product can be obtained, while also providing a relative evaluation and assessment of model performance. Using surface flux observations from 45 globally distributed eddy-covariance stations as independent metrics of performance, the tower-based analysis indicated that PT-JPL provided the highest overall statistical performance (0.72; 61 W m –2 ; 0.65), followed closely by GLEAM (0.68; 64 W m –2 ; 0.62), with values in parentheses representing the R 2 , RMSD and Nash–Sutcliffe efficiency (NSE), respectively. PM-Mu (0.51; 78 W m –2 ; 0.45) tended to underestimate fluxes, while SEBS (0.72; 101 W m –2 ; 0.24) overestimated values relative to observations. A focused analysis across specific biome types and climate zones showed considerable variability in the performance of all models, with no single model consistently able to outperform any other. Results also indicated that the global gridded data tended to reduce the performance for all of the studied models when compared to the tower data, likely a response to scale mismatch and issues related to forcing quality. Rather than relying on any single model simulation, the spatial and temporal variability at both the tower- and grid-scale highlighted the potential benefits of developing an ensemble or blended evaporation product for global-scale LandFlux applications. Hence, challenges related to the robust assessment of the LandFlux product are also discussed.

58 GEOSCIENCES↗
DOE OSTI2018

Experiments, Statistical Analysis, and Modeling to Evaluate the Porosity Influence in SPS Coatings

Suspension plasma spray (SPS) is far more complicated than conventional plasma spray and requires a deep knowledge about the influence of process parameters and their correlations. In this study, YSZ coatings were manufactured by SPS with six different process parameters such as plasma power, suspension mass load, original powder size, substrate surface topology, spray distance, and spray step. Then, the porosity of as-prepared coatings was investigated by image method and X-ray transmission technique. A multivariate analysis on the collected experimental data was carried out by employing mathematical statistics methods. The results showed that: 1) Coating porosity has a negative correlation with plasma power and suspension mass load and a positive correlation with the original powder size, spray distance, spray step, and substrate roughness; 2) Spraying distance is the main factor affecting to coating porosity, followed by suspension mass load and substrate surface roughness, respectively. A linear model for porosity prediction was developed and was verified by experiments. Furthermore, the mechanism by which process parameters influence coating porosity is additionally discussed.

36 MATERIALS SCIENCE↗
DOE OSTI2022

A guideline to document occupant behavior models for advanced building controls

The availability of computational power, and a wealth of data from sensors have boosted the development of model-based predictive control for smart and effective control of advanced buildings in the last decade. More recently occupant-behavior models have been developed for including people in the building control loops. However, while important objectives of scientific research are reproducibility and replicability of results, not all information is available from published documents. Therefore, the aim of this paper is to propose a guideline for a thorough and standardized occupant-behavior model documentation. For that purpose, the literature screening for the existing occupant behavior models in building control was conducted, and the occupant behavior modeling processes were studied to extract practices and gaps for each of the following phases: problem statement, data collection, and preprocessing, model development, model evaluation, and model implementation. Here, the literature screening pointed out that the current state-of-the-art on model documentation shows little unification, which poses a particular burden for the model application and replication in field studies. In addition to the standardized model documentation, this work presented a model-evaluation schema that enabled benchmarking of different models in field settings as well as the recommendations on how OB models are integrated with the building system.

Building control↗
NASA NTRS2003

A Dynamic Risk Model for Evaluation of Space Shuttle Abort Scenarios

The Space Shuttle is an advanced manned launch system with a respectable history of service and a demonstrated level of safety. Recent studies have shown that the Space Shuttle has a relatively low probability of having a failure that is instantaneously catastrophic during nominal flight as compared with many US and international launch systems. However, since the Space Shuttle is a manned. system, a number of mission abort contingencies exist to primarily ensure the safety of the crew during off-nominal situations and to attempt to maintain the integrity of the Orbiter. As the Space Shuttle ascends to orbit it transverses various intact abort regions evaluated and planned before the flight to ensure that the Space Shuttle Orbiter, along with its crew, may be returned intact either to the original launch site, a transoceanic landing site, or returned from a substandard orbit. An intact abort may be initiated due to a number of system failures but the highest likelihood and most challenging abort scenarios are initiated by a premature shutdown of a Space Shuttle Main Engine (SSME). The potential consequences of such a shutdown vary as a function of a number of mission parameters but all of them may be related to mission time for a specific mission profile. This paper focuses on the Dynamic Abort Risk Evaluation (DARE) model process, applications, and its capability to evaluate the risk of Loss Of Vehicle (LOV) due to the complex systems interactions that occur during Space Shuttle intact abort scenarios. In addition, the paper will examine which of the Space Shuttle subsystems are critical to ensuring a successful return of the Space Shuttle Orbiter and crew from such a situation.

Henderson, Edward M.↗
DOE OSTI2022

New Hybrid Model for Evaluating the Frequency-Dependent Leakage Inductance of a Variable Inductance Transformer (VIT)

Skin and proximity effects can cause a significant drop in the effective leakage inductance of a transformer when the operating frequency is increased. Although the magnetic image method-based double-2-D model can calculate the lowfrequency leakage inductance with sufficient accuracy, it is inherently a frequency-independent model. While Dowell’s 1-D model uses frequency-dependent relations to account for both skin and proximity effects, its accuracy is severely affected by the assumed winding geometry. In this paper, a hybrid model is proposed that uses superposition to combine a modified Dowell’s model with the double-2-D model. The proposed model is investigated on a variable inductance transformer (VIT)—a partially-filled transformer whose leakage inductance can be varied by moving one of the windings mechanically. The frequency-dependent leakage inductances of the VIT evaluated using the hybrid model are in excellent agreement with the corresponding finite element method (FEM) simulated and experimentally measured values, thereby validating the proposed hybrid model.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2018

Using the Atmospheric Radiation Measurement (ARM) Datasets to Evaluate Climate Models in Simulating Diurnal and Seasonal Variations of Tropical Clouds

We use the long-term Atmospheric Radiation Measurement (ARM) datasets collected at the three Tropical Western Pacific (TWP) sites as a tropical testbed to evaluate the ability of the Community Atmosphere Model (CAM5) to simulate the various types of clouds, their seasonal and diurnal variations, and their impact on surface radiation. We conducted a series of CAM5 simulations at various horizontal grid spacing (around 2°, 1°, 0.5°, and 0.25°) with meteorological constraints from reanalysis. Model biases in the seasonal cycle of cloudiness are found to be weakly dependent on model resolution. Positive biases (up to 20%) in the annual mean total cloud fraction appear mostly in stratiform ice clouds. Higher-resolution simulations do reduce the positive bias in the frequency of ice clouds, but they inadvertently increase the negative biases in convective clouds and low-level liquid clouds, leading to a positive bias in annual mean shortwave fluxes at the sites, as high as 65 W m-2 in the 0.25° simulation. Such resolution-dependent biases in clouds can adversely lead to biases in ambient thermodynamic properties and, in turn, feedback on clouds. Both the CAM5 model and ARM observations show distinct diurnal cycles in total, stratiform and convective cloud fractions; however, they are out-of-phase by 12 hours and the biases vary by site. Our results suggest that biases in deep convection affect the vertical distribution and diurnal cycle of stratiform clouds through the transport of vapor and/or the detrainment of liquid and ice. We also found that the modelled gridmean surface longwave fluxes are systematically larger than site measurements when the grid that the ARM sites reside in is partially covered by ocean. The modeled longwave fluxes at such sites also lack a discernable diurnal cycle because the ocean part of the grid is warmer and less sensitive to radiative heating/cooling compared to land. Higher spatial resolution is more helpful is this regard. Our testbed approach can be easily adapted for the evaluation of new parameterizations being developed for CAM5 or other global or regional model simulations at high spatial resolutions.

Wang, Hailong↗