Evaluation of modeled global vegetation carbon dynamics: Analysis based on global carbon flux and above-ground biomass data
Not provided.
Environmental Sciences & Ecology↗
SEARCH · Search NASA
Results for “model evaluation”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Evaluation of the Spaceflight Cardiovascular Risk Using the Clinical Prediction Model Astro-CHARM and the NASA Radiation Risk Model
Evaluation of the spaceflight cardiovascular risk using the clinical prediction model Astro-CHARM and the NASA Radiation Risk Model
Ianik Plante↗
Evaluating mixture adsorption models using molecular simulation
Explore the source record for details and available documents.
membrane, carbon capture, materials and chemistry ↗
Design, modeling, and evaluation of a doublet heat extraction model in enhanced geothermal systems
Not Available
Xia, Yidong (ORCID:0000000219557330)↗
Surrogate Models for TREAT Transient Calculations
The Transient Reactor Test Facility (TREAT) at Idaho National Laboratory (INL) is currently preparing for resumption of operation after more than 20 years in a long-term standby state. In parallel with preparation for operations and a new transient testing program, the DOE-NE Advanced Modeling and Simulation program (NEAMS) has been supporting the development of three-dimensional transient multi-physics modeling capabilities within the reactor physics application MAMMOTH [1]. Recently, the MAMMOTH reactor physics team has begun development of uncertainty and sensitivity analysis for the TREAT core. This research will be used to support the analysis of reactor operation and experiment planning. Traditional approaches, i.e., Monte Carlo simulations, for such tasks require a large number of model evaluations under both steady state and transient conditions. However, exhaustive sets of sampling runs are simply not practical here since evaluations of each model can take a substantial amount of time. A solution to this problem is to build a surrogate model employing a smaller number of model evaluations. The surrogate model is designed to provide an accurate and fast approximation to the detailed TREAT model. In this manner, the surrogate model can be used in place of the TREAT transient model for uncertainty and sensitivity analysis at a fraction of the computational cost. The high dimensional model representation (HDMR) [2] method is employed here for approximating the responses of interest in terms of functions of lower dimensions. This method has been implemented in RAVEN [3] and all TREAT transient calculations are performed using the multiphysics code MAMMOTH. In addition, a code interface has also been developed in order to couple RAVEN with MAMMOTH. The surrogate model is then tested and compared for the MAMMOTH model of the TREAT startup testing program Transient Test #15 core [4], and its accuracy is measured by its ability to estimate the mean, coefficient of variance, skewness, excess kurtosis and probability density function of the outputs of Transient Test #15 model, as well as root mean square error metric.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Surrogate Models for TREAT Transient Calculations
The Transient Reactor Test Facility (TREAT) at Idaho National Laboratory (INL) is currently preparing for resumption of operation after more than 20 years in a long-term standby state. In parallel with preparation for operations and a new transient testing program, the DOENE Advanced Modeling and Simulation program (NEAMS) has been supporting the development of three-dimensional transient multi-physics modeling capabilities within the reactor physics application MAMMOTH. Recently, the MAMMOTH reactor physics team has begun development of uncertainty and sensitivity analysis for the TREAT core. This research will be used to support the analysis of reactor operation and experiment planning. Traditional approaches, i.e., Monte Carlo simulations, for such tasks require a large number of model evaluations under both steady state and transient conditions. However, exhaustive sets of sampling runs are simply not practical for TREAT since evaluations of each model can take a substantial amount of time. A solution to this problem is to build a surrogate model employing a smaller number of model evaluations. The surrogate model is designed to provide an accurate and fast approximation to the detailed TREAT model. In this manner, the surrogate model can be used in place of the TREAT transient model for uncertainty and sensitivity analysis at a fraction of the computational cost. The high dimensional model representation (HDMR) method is employed here for approximating the responses of interest in terms of functions of lower dimensions. This method has been implemented in RAVEN and all TREAT transient calculations are performed using the multiphysics code MAMMOTH. In addition, a code interface has also been developed in order to couple RAVEN with MAMMOTH. The surrogate model is then tested and compared for TREAT startup testing program Transient Test no. 15 core, and its accuracy is measured by its ability to estimate the mean, coefficient of variance, skewness, excess kurtosis and probability density function of the outputs of Transient Test no. 15 model, as well as root mean square error metric.
21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
Advancing Methodologies for Applying Machine Learning and Evaluating Spatiotemporal Models of Fine Particulate Matter (PM 2.5 ) Using Satellite Data Over Large Regions
Reconstructing the distribution of fine particulate matter (PM 2.5 ) in space and time, even far from ground monitoring sites, is an important exposure science contribution to epidemiologic analyses of PM 2.5 health impacts. Flexible statistical methods for prediction have demonstrated the integration of satellite observations with other predictors, yet these algorithms are susceptible to overfitting the spatiotemporal structure of the training datasets. We present a new approach for predicting PM 2.5 using machine-learning methods and evaluating prediction models for the goal of making predictions where they were not previously available. We apply extreme gradient boosting (XGBoost) modeling to predict daily PM 2.5 on a 1 x 1 km 2 resolution for a 13 state region in the Northeastern USA for the years 2000–2015 using satellite-derived aerosol optical depth and implement a recursive feature selection to develop a parsimonious model. We demonstrate excellent predictions of withheld observations but also contrast an RMSE of 3.11 μg/m 3 in our spatial cross-validation withholding nearby sites versus an overfit RMSE of 2.10 μg/m 3 using a more conventional random ten-fold splitting of the dataset. As the field of exposure science moves forward with the use of advanced machine-learning approaches for spatiotemporal modeling of air pollutants, our results show the importance of addressing data leakage in training, overfitting to spatiotemporal structure, and the impact of the predominance of ground monitoring sites in dense urban sub-networks on model evaluation. The strengths of our resultant modeling approach for exposure in epidemiologic studies of PM 2.5 include improved efficiency, parsimony, and interpretability with robust validation while still accommodating complex spatiotemporal relationships.
air pollution↗
Nudging allows direct evaluation of coupled climate models with in situ observations: a case study from the MOSAiC expedition
Comparing the output of general circulation models to observations is essential for assessing and improving the quality of models. While numerical weather prediction models are routinely assessed against a large array of observations, comparing climate models and observations usually requires long time series to build robust statistics. Here, we show that by nudging the large-scale atmospheric circulation in coupled climate models, model output can be compared to local observations for individual days. We illustrate this for three climate models during a period in April 2020 when a warm air intrusion reached the MOSAiC (Multidisciplinary drifting Observatory for the Study of Arctic Climate) expedition in the central Arctic. Radiosondes, cloud remote sensing and surface flux observations from the MOSAiC expedition serve as reference observations. The climate models AWI-CM1/ECHAM and AWI-CM3/IFS miss the diurnal cycle of surface temperature in spring, likely because both models assume the snowpack on ice to have a uniform temperature. CAM6, a model that uses three layers to represent snow temperature, represents the diurnal cycle more realistically. During a cold and dry period with pervasive thin mixed-phase clouds, AWI-CM1/ECHAM only produces partial cloud cover and overestimates downwelling shortwave radiation at the surface. AWI-CM3/IFS produces a closed cloud cover but misses cloud liquid water. Our results show that nudging the large-scale circulation to the observed state allows a meaningful comparison of climate model output even to short-term observational campaigns. We suggest that nudging can simplify and accelerate the pathway from observations to climate model improvements and substantially extends the range of observations suitable for model evaluation.
54 ENVIRONMENTAL SCIENCES↗
Evaluating Climate Models’ Cloud Feedbacks Against Expert Judgment
Abstract The persistent and growing spread in effective climate sensitivity (ECS) across global climate models necessitates rigorous evaluation of their cloud feedbacks. Here we evaluate several cloud feedback components simulated in 19 climate models against benchmark values determined via an expert synthesis of observational, theoretical, and high‐resolution modeling studies. We find that models with smallest feedback errors relative to these benchmark values generally have moderate total cloud feedbacks (0.4–0.6 W m −2 K −1 ) and ECS (3–4 K). Those with largest errors generally have total cloud feedback and ECS values that are too large or too small. Models tend to achieve large positive total cloud feedbacks by having several cloud feedback components that are systematically biased high rather than by having a single anomalously large component, and vice versa. In general, better simulation of mean‐state cloud properties leads to stronger but not necessarily better cloud feedbacks. The Python code base provided herein could be applied to developmental versions of models to assess cloud feedbacks and cloud errors and place them in the context of other models and of expert judgment in real‐time during model development.
58 GEOSCIENCES↗
Denoising Seismic Waveforms Using a Wavelet-Transform-Based Machine-Learning Method
Seismic waveform data recorded at stations can be thought of as a superposition of the signal from a source of interest and noise from other sources. Frequency‐based filtering methods for waveform denoising do not result in desired outcomes when the targeted signal and noise occupy similar frequency bands. Recently, denoising techniques based on deep‐learning convolutional neural networks (CNNs), in which a recorded waveform is decomposed into signal and noise components, have led to improved results. These CNN methods, which use short‐time Fourier transform representations of the time series, provide signal and noise masks for the input waveform. These masks are used to create denoised signal and designaled noise waveforms, respectively. However, advancements in the field of image denoising have shown the benefits of incorporating discrete wavelet transforms (DWTs) into CNN architectures to create multilevel wavelet CNN (MWCNN) models. The MWCNN model preserves the details of the input due to the good time–frequency localization of the DWT. In this report we use a data set of over 382,000 constructed seismograms recorded by the University of Utah Seismograph Stations network to compare the performance of CNN and MWCNN‐based denoising models. Evaluation of both models on constructed test data shows that the MWCNN model outperforms the CNN model in the ability to recover the ground‐truth signal component in terms of both waveform similarity and preservation of amplitude information. Model evaluation of real‐world data shows that both the CNN and MWCNN models outperform standard band‐pass filtering (BPF; average improvement in signal‐to‐noise ratio of 9.6 and 19.7 dB, respectively, with respect to BPF). Evaluation of continuous data suggests the MWCNN denoiser can improve both signal detection capabilities and phase arrival time estimates.
58 GEOSCIENCES↗
Beyond the Thickness Distribution: Modeling and Evaluating Sea Ice Thickness in Earth System Models
Explore the source record for details and available documents.
Computer Science↗
Stochastic performance modeling and evaluation of obstacle detectability with imaging range sensors
Statistical modeling and evaluation of the performance of obstacle detection systems for Unmanned Ground Vehicles (UGVs) is essential for the design, evaluation, and comparison of sensor systems. In this report, we address this issue for imaging range sensors by dividing the evaluation problem into two levels: quality of the range data itself and quality of the obstacle detection algorithms applied to the range data. We review existing models of the quality of range data from stereo vision and AM-CW LADAR, then use these to derive a new model for the quality of a simple obstacle detection algorithm. This model predicts the probability of detecting obstacles and the probability of false alarms, as a function of the size and distance of the obstacle, the resolution of the sensor, and the level of noise in the range data. We evaluate these models experimentally using range data from stereo image pairs of a gravel road with known obstacles at several distances. The results show that the approach is a promising tool for predicting and evaluating the performance of obstacle detection with imaging range sensors.
Matthies, Larry↗
A sampled - data pursuit tracking model.
Evaluation of sampled data pursuit tracking model
SAMPLED DATA SYSTEM↗
Evaluation of semiempirical atmospheric density models for orbit determination applications
This paper presents the results of an investigation of the orbit determination performance of the Jacchia-Roberts (JR), mass spectrometer incoherent scatter 1986 (MSIS-86), and drag temperature model (DTM) atmospheric density models. Evaluation of the models was performed to assess the modeling of the total atmospheric density. This study was made generic by using six spacecraft and selecting time periods of study representative of all portions of the 11-year cycle. Performance of the models was measured for multiple spacecraft, representing a selection of orbit geometries from near-equatorial to polar inclinations and altitudes from 400 kilometers to 900 kilometers. The orbit geometries represent typical low earth-orbiting spacecraft supported by the Goddard Space Flight Center (GSFC) Flight Dynamics Division (FDD). The best available modeling and orbit determination techniques using the Goddard Trajectory Determination System (GTDS) were employed to minimize the effects of modeling errors. The latest geopotential model available during the analysis, the Goddard earth model-T3 (GEM-T3), was employed to minimize geopotential model error effects on the drag estimation. Improved-accuracy techniques identified for TOPEX/Poseidon orbit determination analysis were used to improve the Tracking and Data Relay Satellite System (TDRSS)-based orbit determination used for most of the spacecraft chosen for this analysis. This paper shows that during periods of relatively quiet solar flux and geomagnetic activity near the solar minimum, the choice of atmospheric density model used for orbit determination is relatively inconsequential. During typical solar flux conditions near the solar maximum, the differences between the JR, DTM, and MSIS-86 models begin to become apparent. Time periods of extreme solar activity, those in which the daily and 81-day mean solar flux are high and change rapidly, result in significant differences between the models. During periods of high geomagnetic activity, the standard JR model was outperformed by DTM. Modification of the JR model to use a geomagnetic heating delay of 3 hours, as used in DTM, instead of the 6.7-hour delay produced results comparable to or better than the DTM performance, reducing definitive orbit solution ephermeris overlap differences by 30 to 50 percent. The reduction in the overlap differences would be useful for mitigating the impact of geomagnetic storms on orbit prediction.
Cox, C. M.↗
Modeling and Evaluating Pilot Performance in NextGen: Review of and Recommendations Regarding Pilot Modeling Efforts, Architectures, and Validation Studies
NextGen operations are associated with a variety of changes to the national airspace system (NAS) including changes to the allocation of roles and responsibilities among operators and automation, the use of new technologies and automation, additional information presented on the flight deck, and the entire concept of operations (ConOps). In the transition to NextGen airspace, aviation and air operations designers need to consider the implications of design or system changes on human performance and the potential for error. To ensure continued safety of the NAS, it will be necessary for researchers to evaluate design concepts and potential NextGen scenarios well before implementation. One approach for such evaluations is through human performance modeling. Human performance models (HPMs) provide effective tools for predicting and evaluating operator performance in systems. HPMs offer significant advantages over empirical, human-in-the-loop testing in that (1) they allow detailed analyses of systems that have not yet been built, (2) they offer great flexibility for extensive data collection, (3) they do not require experimental participants, and thus can offer cost and time savings. HPMs differ in their ability to predict performance and safety with NextGen procedures, equipment and ConOps. Models also vary in terms of how they approach human performance (e.g., some focus on cognitive processing, others focus on discrete tasks performed by a human, while others consider perceptual processes), and in terms of their associated validation efforts. The objectives of this research effort were to support the Federal Aviation Administration (FAA) in identifying HPMs that are appropriate for predicting pilot performance in NextGen operations, to provide guidance on how to evaluate the quality of different models, and to identify gaps in pilot performance modeling research, that could guide future research opportunities. This research effort is intended to help the FAA evaluate pilot modeling efforts and select the appropriate tools for future modeling efforts to predict pilot performance in NextGen operations.
pilot human performance models↗
Best-Estimate-Plus-Uncertainty-Informed Deterministic Safety Evaluation
Nuclear power plant (NPP) utilities operate contingent on receiving a favorable safety assessment from their national regulatory and safety authority. Among the many facets of NPP safety addressed is verification of integral design-basis performance - a demonstration of a plant's response to a suite of challenging transient and accident scenarios. Thermal-hydraulic system computer codes like RELAP5, TRAC, CATHARE, and ATHLET are used as the simulation engine as part of a broader evaluation model that accounts for a variety of uncertainties. It is common for these evaluation models to apply either a super-bounding/ deterministic or best-estimate plus uncertainty (BEPU) approach. The trend over the past two decades has been away from the classical deterministic methods towards methods explicitly incorporating parametric uncertainty models. These methods center upon the evaluation of parametric and system response uncertainties, where these uncertainties are derived from the convolution of one or more parametric uncertainty models. Acceptance of BEPU methods has provided the nuclear industry benefits regarding operational flexibility and, in some cases, power output. The advantage of BEPU for new NPPs is less obvious. Generally, these new designs include engineered safety features that practically eliminate fuel damage resulting from design-basis events. To substantiate such claims, applying super-bounding assumptions on uncertainties may be a more expedient demonstration of a plant's safety case than those based on BEPU. In addition, the requirements on design inputs for a deterministic evaluation model are substantially less than that for a BEPU evaluation model. As such, applying a deterministic evaluation model is expected to have the advantage of reducing the duration of the licensing review process. For the current operating fleet of light-water reactors (LWRs), the nuclear safety authorities in most countries have followed the U.S. Nuclear Regulatory Commission's rules for the preparation of safety analysis. These rules are specified in the U.S. Code of Federal Regulations, most notably in, Part 10, Section 50.46, entitled 'acceptance criteria for emergency core cooling systems for light-water nuclear power reactors' and its Appendix K. Notably, Appendix K specifies deterministic requirements for analysis of the loss-of-coolant accident (LOCA), which is generally regarded as the maximum credible accident for LWRs. Projecting 10 CFR 50, Appendix K, to new plants is not obviously applicable as it addresses light water fluid flow and heat transfer phenomena associated with core uncovery. Preventing core uncovery is a requirement for several designs in the class of small modular reactors (SMR). In doing so, the majority of phenomenological uncertainties prescribed in Appendix K are eliminated. Similarly, the impact of other phenomena to effect a change in various safety measures, i.e., figures-of-merit, is expected to be different. The implication is that development of a deterministic evaluation model must be derived with an original technical basis rather than solely rely on Appendix K. While a logical conclusion, the expectation of original bases, regardless of the treatment of uncertainties, is embodied in the U.S. NRC's Regulatory Guide (RG) 1.203, Transient and Accident Analysis. Within the evaluation model development and application process framework of RG 1.203, this paper describes a BEPU-informed deterministic evaluation model for NPPs with large inherent safety margins. (authors)
21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
A global model–measurement evaluation of particle light scattering coefficients at elevated relative humidity
The uptake of water by atmospheric aerosols has a pronounced effect on particle light scattering properties, which in turn are strongly dependent on the ambient relative humidity (RH). Earth system models need to account for the aerosol water uptake and its influence on light scattering in order to properly capture the overall radiative effects of aerosols. Here we present a comprehensive model–measurement evaluation of the particle light scattering enhancement factor f(RH), defined as the particle light scattering coefficient at elevated RH (here set to 85 %) divided by its dry value. The comparison uses simulations from 10 Earth system models and a global dataset of surface-based in situ measurements. In general, we find a large diversity in the magnitude of predicted f(RH) amongst the different models, which can not be explained by the site types. Based on our evaluation of sea salt scattering enhancement and simulated organic mass fraction, there is a strong indication that differences in the model parameterizations of hygroscopicity and model chemistry are driving at least some of the observed diversity in simulated f(RH). Additionally, a key point is that defining dry conditions is difficult from an observational point of view and, depending on the aerosol, may influence the measured f(RH). The definition of dry also impacts our model evaluation, because several models exhibit significant water uptake between RH = 0 % and 40 %. The multisite average ratio between model outputs and measurements is 1.64 when RH = 0 % is assumed as the model dry RH and 1.16 when RH = 40 % is the model dry RH value. The overestimation by the models is believed to originate from the hygroscopicity parameterizations at the lower RH range which may not implement all phenomena taking place (i.e., not fully dried particles and hysteresis effects). This will be particularly relevant when a location is dominated by a deliquescent aerosol such as sea salt. Our results emphasize the need to consider the measurement conditions in such comparisons and recognize that measurements referred to as dry may not be dry in model terms. Recommendations for future model–measurement evaluation and model improvements are provided.
54 ENVIRONMENTAL SCIENCES↗
Market Evaluation of Energy Storage Systems Incorporating Technology-Specific Nonlinear Models.
Abstract not provided.
Nguyen, Tu Anh↗