Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Performance Evaluation and Modeling Techniques for Parallel Processors

In practice, the performance evaluation of supercomputers is still substantially driven by singlepoint estimates of metrics (e.g., MFLOPS) obtained by running characteristic benchmarks or workloads. With the rapid increase in the use of time-shared multiprogramming in these systems, such measurements are clearly inadequate. This is because multiprogramming and system overhead, as well as other degradations in performance due to time varying characteristics of workloads, are not taken into account. In multiprogrammed environments, multiple jobs and users can dramatically increase the amount of system overhead and degrade the performance of the machine. Performance techniques, such as benchmarking, which characterize performance on a dedicated machine ignore this major component of true computer performance. Due to the complexity of analysis, there has been little work done in analyzing, modeling, and predicting the performance of applications in multiprogrammed environments. This is especially true for parallel processors, where the costs and benefits of multi-user workloads are exacerbated. While some may claim that the issue of multiprogramming is not a viable one in the supercomputer market, experience shows otherwise. Even in recent massively parallel machines, multiprogramming is a key component. It has even been claimed that a partial cause of the demise of the CM2 was the fact that it did not efficiently support time-sharing. In the same paper, Gordon Bell postulates that, multicomputers will evolve to multiprocessors in order to support efficient multiprogramming. Therefore, it is clear that parallel processors of the future will be required to offer the user a time-shared environment with reasonable response times for the applications. In this type of environment, the most important performance metric is the completion of response time of a given application. However, there are a few evaluation efforts addressing this issue.

Dimpsey, Robert Tod↗
NASA NTRS1992

A statistical model for evaluating GOPEX uplink performance

This article describes a statistical model to analyze the signal intensity received at the solid-state imaging (SSI) camera of the Galileo optical communications system from an Earth-based transmitter (GOPEX) demonstration. The analytical model assumes that the optical beam possesses a Gaussian profile and the communication channel has a log-normal scattering characteristic. The atmospheric-induced jitter is modelled as two independent zero mean Gaussian random variables. By modelling the system parameters as a set of independent and identically distributed (iid) random variables, the combined impact of uncertainties due to system parameters and the turbulent atmosphere can be approximated by a log-normal distributed signal intensity at the spacecraft. A Monte-Carlo software simulation package was also developed to compute the confidence interval probabilities for general optical beam profiles. Numerical results show that the approximation is valid for a wide range of operation scenarios.

Kiesaleh, K.↗
DOE OSTI2016

Column Testing and 1D Reactive Transport Modeling to Evaluate Uranium Plume Persistence Processes

Motivation for Study: Natural flushing of contaminants at various U.S. Department of Energy Office of Legacy Management sites is not proceeding as quickly as predicted (plume persistence) Objectives: Help determine natural flushing rates using column tests. Use 1D reactive transport modeling to better understand the major processes that are creating plume persistence Approach: Core samples from under a former mill tailings area Tailings have been removed. Column leaching using lab-prepared water similar to nearby Gunnison River water. 1D reactive transport modeling to evaluate processes

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Multifidelity importance sampling

Estimating statistics of model outputs with the Monte Carlo method usually requires a large number of model evaluations. This leads to long runtimes if the model is expensive to evaluate. Importance sampling is one method that can lead to a reduction in the number of model evaluations. Importance sampling uses a biasing distribution to sample the model more efficiently, but generating such a biasing distribution can be difficult and usually also requires model evaluations. A different strategy to speed up Monte Carlo sampling is to replace the computationally expensive high-fidelity model with a computationally cheap surrogate model; however, because the surrogate model outputs are only approximations of the high-fidelity model outputs, the estimate obtained using a surrogate model is in general biased with respect to the estimate obtained using the high-fidelity model. We introduce a multifidelity importance sampling (MFIS) method, which combines evaluations of both the high-fidelity and a surrogate model. It uses a surrogate model to facilitate the construction of the biasing distribution, but relies on a small number of evaluations of the high-fidelity model to derive an unbiased estimate of the statistics of interest. We prove that the MFIS estimate is unbiased even in the absence of accuracy guarantees on the surrogate model itself. The MFIS method can be used with any type of surrogate model, such as projection-based reduced-order models and data-fit models. Moreover, the MFIS method is applicable to black-box models, i.e., where only inputs and the corresponding outputs of the high-fidelity and the surrogate model are available but not the details of the models themselves. We demonstrate on nonlinear and time-dependent problems that our MFIS method achieves speedups of up to several orders of magnitude compared to Monte Carlo with importance sampling that uses the high-fidelity model only.

42 ENGINEERING↗
DOE OSTI2020

Observations for Model Intercomparison Project (Obs4MIPs): status for CMIP6

The Observations for Model Intercomparison Project (Obs4MIPs) was initiated in 2010 to facilitate the use of observations in climate model evaluation and research, with a particular target being the Coupled Model Intercomparison Project (CMIP), a major initiative of the World Climate Research Programme (WCRP). To this end, Obs4MIPs (1) targets observed variables that can be compared to CMIP model variables; (2) utilizes dataset formatting specifications and metadata requirements closely aligned with CMIP model output; (3) provides brief technical documentation for each dataset, designed for nonexperts and tailored towards relevance for model evaluation, including information on uncertainty, dataset merits, and limitations; and (4) disseminates the data through the Earth System Grid Federation (ESGF) platforms, making the observations searchable and accessible via the same portals as the model output. Taken together, these characteristics of the organization and structure of obs4MIPs should entice a more diverse community of researchers to engage in the comparison of model output with observations and to contribute to a more comprehensive evaluation of the climate models. At present, the number of obs4MIPs datasets has grown to about 80; many are undergoing updates, with another 20 or so in preparation, and more than 100 are proposed and under consideration. A partial list of current global satellite-based datasets includes humidity and temperature profiles; a wide range of cloud and aerosol observations; ocean surface wind, temperature, height, and sea ice fraction; surface and top-of-atmosphere longwave and shortwave radiation; and ozone (O 3 ), methane (CH 4 ), and carbon dioxide (CO 2 ) products. A partial list of proposed products expected to be useful in analyzing CMIP6 results includes the following: alternative products for the above quantities, additional products for ocean surface flux and chlorophyll products, a number of vegetation products (e.g., FAPAR, LAI, burned area fraction), ice sheet mass and height, carbon monoxide (CO), and nitrogen dioxide (NO 2 ). While most existing obs4MIPs datasets consist of monthly-mean gridded data over the global domain, products with higher time resolution (e.g., daily) and/or regional products are now receiving more attention. Along with an increasing number of datasets, obs4MIPs has implemented a number of capability upgrades including (1) an updated obs4MIPs data specifications document that provides additional search facets and generally improves congruence with CMIP6 specifications for model datasets, (2) a set of six easily understood indicators that help guide users as to a dataset's maturity and suitability for application, and (3) an option to supply supplemental information about a dataset beyond what can be found in the standard metadata. With the maturation of the obs4MIPs framework, the dataset inclusion process, and the dataset formatting guidelines and resources, the scope of the observations being considered is expected to grow to include gridded in situ datasets as well as datasets with a regional focus, and the ultimate intent is to judiciously expand this scope to any observation dataset that has applicability for evaluation of the types of Earth system models used in CMIP.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1979

Evaluation of models to predict insolation on tilted surfaces

An empirical study was performed to evaluate the validity of various insolation models which employ either an isotropic or an anisotropic distribution approximation for sky light when predicting insolation on tilted surfaces. Data sets of measured hourly insolation values were obtained over a 6-month period using pyranometers which received diffuse and total solar radiation on a horizontal plane and total radiation on surfaces tilted toward the equator at 37 degrees and 60 degrees angles above the horizon. Data on the horizontal surfaces were used in the insolation models to predict insolation on the tilted surface; comparisons of measured vs calculated insolation on the tilted surface were examined to test the validity of the sky light approximations. It was found that the Liu-Jordan isotropic distribution model provides a good fit to empirical data under overcast skies but underestimates the amount of solar radiation incident on tilted surfaces under clear and partly cloudy conditions.

Klucher, T. M.↗
DOE OSTI2022

Characterizing vertical upper ocean temperature structures in the European Arctic through unsupervised machine learning

In-situ observations of subsurface ocean temperatures are, in many regions, inconsistently distributed in time and space. These spatio-temporal inconsistencies in the observational network lead to difficulties in utilizing those observations effectively for ocean model evaluation or understanding larger-scale ocean characteristics. Model accuracy of subsurface ocean characteristics is especially important within regions that contain complex ocean structures. One such region is the European Arctic which not only contains several types of water masses with unique characteristics, but also wintertime sea ice coverage and complex bathymetry. This study presents an unsupervised neural networking technique that can be used in combination with traditional ocean model evaluation techniques to provide additional information on the accuracy of modeled vertical ocean temperature profiles. Self-organizing maps is an unsupervised machine learning technique that we apply to approximately twenty thousand Argo and CTD temperature profiles from 2012 to 2020 in the European Arctic to categorize the observed vertical ocean temperature structures in the top 150 m. The observed ocean profile categories, or neurons, defined by the self-organizing map show strong spatial and temporal dependencies. We then use the neuron weights, or the learned temperature profile structure of each neuron, to validate the spatial and temporal variability of modeled vertical temperature structures. This analysis gives us new insights about the model’s capabilities to reproduce specific vertical structures of the top-most ocean layer within different regions and seasons. Mapping modeled ocean temperature profiles onto the neuron-space of the observationally-defined self organized map highlights the potential of this method to advance our understanding of model deficiencies in that region.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1985

Multiscale modeling to evaluate proposed space-based Doppler lidar sampling strategies

A proposal has been made to place a pulsed Doppler lidar on a space platform (Huffaker, et al., 1980; Emmitt, 1982) in a low earth orbit (200 to 800 km) to measure the atmospheric winds with a spatial resolution commensurate with the current continental rawinsonde network density - i.e., 300 to 500 km resolution. In the case of the space-based doppler lidar, the full range of space scales applies. Single shot pulses with dimensions of 10 x 1000 meters are used to sample areas 100,000 x 100,000 meters to resolved mass flow structure with wavelength of 1 million meters. Simulation studies, therefore, require an equally broad range of atmospheric models. A general circulation model is appropriate to answer questions regarding the impact of a global wind measuring system upon synoptic forecasts. Since the nominal resolution of the spacebased system is expected to be a few 100's of kilometers, then a numerical model with mesoscale dynamics is required. The meaning of an average Doppler shift within a laser pulse volume must be evaluated with models of turbulent/convective scale motions and aerosol gradients. Examples of how models on all these scales have been applied in an ongoing simulation study are presented. In particular, the uncertainties in a mesoscale wind estimate are separated into those arising from pulse scale variances and those due to sample distribution within a prescribed resolution volume. Trade-offs between accuracy and representativeness are discussed in terms of the model results.

Emmitt, G. D.↗
NASA NTRS2016

Evaluation of Modeled Vertical Distribution of Atmospheric SO2 and Sulfate in UTLS

We evaluated the vertical profiles of both SO2 and sulfate in the AEROCOM (Aerosol Model Intercomparison) Phase II participating models. SO2 and sulfate show significant concentration gradient in both horizontal and vertical directions. Both online and offline aerosol transport models show large difference in the vertical distribution of sulfur species from surface all the way up to lower stratosphere. Comparison with available aircraft measurements suggests models agree with observations well when SO2 concentration is high. For the volcanic plumes, the injection height and magnitude determines initial SO2 plume distribution and following transport pattern. At high altitude, where the background concentration of SO2 is often below the detection limit of the current aircraft instruments and satellite retrievals, modeled SO2 and sulfate concentration, lifetime, and budget, as well as their uncertainties can be difficult to be accurately quantified.

volcanic plumes↗
DOE OSTI2019

Visual Analytics for Deep Embeddings of Large Scale Molecular Dynamics Simulations

Molecular Dynamics (MD) simulation have been emerging as an excellent candidate for understanding complex atomic and molecular scale mechanism of bio-molecules that control essential bio-physical phenomenon in a living organism. But this MD technique produces large-size and long-timescale data that are inherently high-dimensional and occupies many terabytes of data. Processing this immense amount of data in a meaningful way is becoming increasingly difficult. Therefore, specific dimensionality reduction algorithm using deep learning technique has been employed here to embed the high-dimensional data in a lower-dimension latent space that still preserves the inherent molecular characteristics i.e. retains biologically meaningful information. Subsequently, the results of the embedding models are visualized for model evaluation and analysis of the extracted underlying features. However, most of the existing visualizations for embeddings have limitations in evaluating the embedding models and understanding the complex simulation data. We propose an interactive visual analytics system for embeddings of MD simulations to not only evaluate and explain an embedding model but also analyze various characteristics of the simulations. Our system enables exploration and discovery of meaningful and semantic embedding results and supports the understanding and evaluation of results by the quantitatively described features of the MD simulations (even without specific labels).

Chae, Junghoon↗
DOE OSTI2023

Dynamic Model Agnostic Reliability Evaluation of Machine-Learning Models Integrated in Instrumentation & Control Systems

In recent years, the field of data-driven neural network-based machine learning (ML) algorithms has grown significantly and spurred research in its applicability to instrumentation and control systems. While they are promising in operational contexts, the trustworthiness of such algorithms is not adequately assessed. Failures of ML-integrated systems are poorly understood; the lack of comprehensive risk modeling can degrade the trustworthiness of these systems. In recent reports by the National Institute for Standards and Technology, trustworthiness in ML is a critical barrier to adoption and will play a vital role in intelligent systems' safe and accountable operation. Thus, in this work, we demonstrate a real-time model-agnostic method to evaluate the relative reliability of ML predictions by incorporating out-of-distribution detection on the training dataset. It is well documented that ML algorithms excel at interpolation (or near-interpolation) tasks but significantly degrade at extrapolation. This occurs when new samples are "far" from training samples. The method, referred to as the Laplacian distributed decay for reliability (LADDR), determines the difference between the operational and training datasets, which is used to calculate a prediction's relative reliability. LADDR is demonstrated on a feedforward neural network-based model used to predict safety significant factors during different loss-of-flow transients. LADDR is intended as a "data supervisor" and determines the appropriateness of well-trained ML models in the context of operational conditions. Ultimately, LADDR illustrates how training data can be used as evidence to support the trustworthiness of ML predictions when utilized for conventional interpolation tasks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Fuel Thermo-physical Characterization Project: Evaluation of Models to Calculate Thermal Diffusivity of Layered Composites

The Office of Material Management and Minimization Fuel Thermo-physical Characterization Project at Pacific Northwest National Laboratory (PNNL) is tasked with using PNNL facilities and processes to receive irradiated low enriched uranium-molybdenum fuel plate samples and perform analyses in support of the Office of Material Management and Minimization Reactor Conversion Program. This work is in support of the Fuel Development Pillar that is managed by Idaho National Laboratory. A key portion of the scope associated with this project was to measure the thermal properties of fuel segments harvested from plates that were irradiated in the Advanced Test Reactor. Thermal diffusivity of samples prepared from the fuel segments was measured using laser flash analysis. Two models, one developed by PNNL and the other developed by the Technische Universität München (TUM), were evaluated to extract the thermal diffusivity of the uranium-molybdenum alloy from measurements made on the irradiated, layered composites. The experimental data of the “TC” irradiated fuel segment was evaluated using both models considering a three-layer and five-layer system. Both models are in acceptable agreement with one another and indicate that the zirconium diffusion barrier has a minimal impact on the overall thermal diffusivity of the monolithic U-Mo fuel.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
NASA NTRS2012

Using Remotely Sensed Data and Hydrologic Models to Evaluate the Effects of Climate Change on Shallow Aquatic Ecosystems in the Mobile Bay, AL Estuary

Coastal systems in the northern Gulf of Mexico, including the Mobile Bay, AL estuary, are subject to increasing pressure from a variety of activities including climate change. Climate changes have a direct effect on the discharge of rivers that drain into Mobile Bay and adjacent coastal water bodies. The outflows change water quality (temperature, salinity, and sediment concentrations) in the shallow aquatic areas and affect ecosystem functioning. Mobile Bay is a vital ecosystem that provides habitat for many species of fauna and flora. Historically, submerged aquatic vegetation (SAV) and seagrasses were found in this area of the northern Gulf of Mexico; however the extent of vegetation has significantly decreased over the last 60 years. The objectives of this research are to determine: how climate changes affect runoff and water quality in the estuary and how these changes will affect habitat suitability for SAV and seagrasses. Our approach is to use watershed and hydrodynamic modeling to evaluate the impact of climate change on shallow water aquatic ecosystems in Mobile Bay and adjacent coastal areas. Remotely sensed Landsat data were used for current land cover land use (LCLU) model input and the data provided by Intergovernmental Panel on Climate Change (IPCC) of the future changes in temperature, precipitation, and sea level rise were used to create the climate scenarios for the 2025 and 2050 model simulations. Project results are being shared with Gulf coast stakeholders through the Gulf of Mexico Data Atlas to benefit coastal policy and climate change adaptation strategies.

Estes, M. G.↗
NASA NTRS2001

Evaluation of Stratospheric Transport in New 3D Models Using the Global Modeling Initiative Grading Criteria

The Global Modeling Initiative (GMI) Team developed objective criteria for model evaluation in order to identify the best representation of the stratosphere. This work created a method to quantitatively and objectively discriminate between different models. In the original GMI study, 3 different meteorological data sets were used to run an offline chemistry and transport model (CTM). Observationally-based grading criteria were derived and applied to these simulations and various aspects of stratospheric transport were evaluated; grades were assigned. Here we report on the application of the GMI evaluation criteria to CTM simulations integrated with a new assimilated wind data set and a new general circulation model (GCM) wind data set. The Finite Volume Community Climate Model (FV-CCM) is a new GCM developed at Goddard which uses the NCAR CCM physics and the Lin and Rood advection scheme. The FV-Data Assimilation System (FV-DAS) is a new data assimilation system which uses the FV-CCM as its core model. One year CTM simulations of 2.5 degrees longitude by 2 degrees latitude resolution were run for each wind data set. We present the evaluation of temperature and annual transport cycles in the lower and middle stratosphere in the two new CTM simulations. We include an evaluation of high latitude transport which was not part of the original GMI criteria. Grades for the new simulations will be compared with those assigned during the original GMT evaluations and areas of improvement will be identified.

Strahan, Susan E.↗
DOE OSTI2021

Quantitative resilience evaluation on recovery from emergency situations in nuclear power plants

Here, this paper mainly introduces how to develop a resilience evaluation model by quantifying the relationship between resilience and resilience components in regard to recovering from emergency situations in nuclear power plants (NPPs). It is an extension of the author’s previous researches. To develop the model, the first step was to analyze event reports published in Republic of Korea in terms of resilience factors from different safety perspectives. Second, statistical methods such as factor analysis, principal component analysis, and multivariate regression analysis were applied to the dataset to identify relations among resilience factors and develop a quantitative resilience evaluation model for recovery from emergency situations in NPPs. Third, the model was validated using the recent event report data released in Korea. Lastly, we present a discussion on how to apply the quantitative resilience evaluation model to results from stress tests performed in Korea.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2012

A model for evaluating the social performance of construction waste management

Highlights: Black-Right-Pointing-Pointer Scant attention is paid to social performance of construction waste management (CWM). Black-Right-Pointing-Pointer We develop a model for assessing the social performance of CWM. Black-Right-Pointing-Pointer With the model, the social performance of CWM can be quantitatively simulated. - Abstract: It has been determined by existing literature that a lot of research efforts have been made to the economic performance of construction waste management (CWM), but less attention is paid to investigation of the social performance of CWM. This study therefore attempts to develop a model for quantitatively evaluating the social performance of CWM by using a system dynamics (SD) approach. Firstly, major variables affecting the social performance of CWM are identified and a holistic system for assessing the social performance of CWM is formulated in line with feedback relationships underlying these variables. The developed system is then converted into a SD model through the software iThink. An empirical case study is finally conducted to demonstrate application of the model. Results of model validation indicate that the model is robust and reasonable to reflect the situation of the real system under study. Findings of the case study offer helpful insights into effectively promoting the social performance of CWM of the project investigated. Furthermore, the model exhibits great potential to function as an experimental platform for dynamically evaluating effects of management measures on improving the social performance of CWM of construction projects.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS1993

Programs Help Create And Evaluate Markov Models

Pade Approximation With Scaling (PAWS) and Scaled Taylor Exponential Matrix (STEM) computer programs provide flexible, user-friendly, language-based interface for creation and evaluation of Markov models describing behaviors of fault-tolerant reconfigurable computer systems. Produce exact solution for probabilities of system failures and provide conservative estimates of numbers of significant digits in solutions. Also offer as part of bundled package with SURE and ASSIST, two other reliable analysis programs developed by Systems Validation Methods group at Langley Research Center.

Butler, Ricky W.↗