Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Socially-aware evaluation framework for transportation

Technological advancements are rapidly changing traffic management in cities. Navigation applications, in particular, have impacted cities in many ways by rerouting traffic. As different routing strategies distribute traffic differently, understanding these disparities across multiple city-relevant dimensions is extremely important for decision-makers. We develop a multi-themed framework called Socially- Aware Evaluation Framework for Transportation (SAEF), which assists in understanding how traffic routing and the resultant dynamics affect cities. The framework is presented for four Bay Area cities, for which we compare three routing strategies - user equilibrium travel time, system optimal travel time, and system optimal fuel. The results demonstrate that many neighborhood impacts, such as traffic load on residential streets and around minority schools, degraded with the system-optimal travel time and fuel routing in comparison to the user-equilibrium travel time routing. The findings also show that all routing strategies subject the city's disadvantaged neighborhoods to disproportionate traffic exposure. Our intent with this work is to provide an evaluation framework that enables reflection on the consequences of traffic routing and management strategies, allowing city planners to recognize the trade-offs and potential unintended consequences.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2021

Bayesian Monte Carlo Evaluation Framework for Imperfect Nuclear Data

Bayesian evaluation of resolved resonance region (RRR) nuclear data has historically been carried out using the generalized least squares (GLS) formalism, as implemented in, e.g., SAMMY. We have recently developed a prototype of Bayesian Monte Carlo (BMC) evaluation framework, implemented using a Markov Chain Monte Carlo (MCMC) method with a Metropolis-Hastings (MH) acceptance criterion. This was done in order to remove the approximations underlying the conventional GLS evaluations, namely, the linear approximation, and the approximation that all probability density functions (PDFs) are of the normal kind. Recent works by others have used similar stochastic approaches to quantify cross section uncertainties from ENDF evaluated co-variances, and/or, from integral benchmark data, but those have not been conceived as an evaluation framework like the one presented here.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Building a Clean Energy Workforce: An Evaluation Framework for State Energy Office Workforce Programs

This fact sheet is a summary of the workforce evaluation framework that was first proposed in the white paper, An Evaluation Framework for State Energy Offices' Energy Efficiency and Clean Energy Workforce Program. This evaluation framework was developed to guide State Energy Offices (SEOs) in assessing and enhancing the workforce development programs the manage, fund, or partner on. By following this step-by-step framework, SEOs can evaluate program components, identify areas for improvement, and optimize program outcomes for greater impact.

clean energy↗
DOE OSTI2024

Data-Informed Evaluation Framework for Integrated Energy Systems: Insights from Power, Process Heat, and Hydrogen Production Applications

The multi-criteria decision analysis (MCDA) framework provides a systematic evaluation of the diverse preferences and performance metrics associated with alternative solutions. This approach is advantageous over a single-criterion methodology, which are only valid under conditions that assume ceteris paribus or an "apples-to-apples" comparison. However, selecting suitable technologies for integrated energy systems (IES) can be likened to an "apples-to-oranges" comparison, given the heterogeneous factors at stake. These factors include economics and performance parameters, geological compatibility, and environmental impacts. Consequently, past research has often employed a mixture of qualitative and quantitative criteria tailored to the specific interests of each study. While the method proves effective in handling the intricate interplay of criteria, the resulting rankings and scores can vary from study to study. This inconsistency is introduced from the use of subjectively defined thresholds and weights. As a result, decision-makers frequently find it challenging to establish clear connections between specific criteria and the resulting scores, as the transformation of criteria into ordinal scores results in a substantial loss of information. To address this challenge, we introduce a data-informed IES evaluation framework that offers comprehensive, interpretable, and traceable evaluations backed by quantifiable rationale. First, we identified key IES evaluation criteria from a decade of literature, focusing on relevant IES applications in power, process heat, and hydrogen production. We leveraged state-of-the-art cost estimates from the Idaho National Laboratory (INL) and technical data from 78 reactor designs from the International Atomic Energy Agency (IAEA) and the Organization for Economic Co-operation and Development - Nuclear Energy Agency (OECD-NEA). Lastly, we established thresholds by analyzing the mean, variance, root mean square, and slope of values across alternatives, categorizing the preferences of decision-makers into distinct utility functions, such as linear, saturating, exponential, and stepwise. Our approach yielded two main outcomes: (1) it provided consistent assessments across different stakeholder groups and (2) it visualized uncertainties in the decision-making context via comprehensive sensitivity analysis. To demonstrate the impact of our framework, we conducted case studies on 6 reactor designs (AP1000, NuScale, BWRX-300, Xe-100, eVinci, iMSR) for the three applications. Our data-driven framework proved to be highly effective in addressing heterogenous uncertainties faced by varied decision-makers? preferences and IES applications, as well as cost and technical estimates of advanced reactors.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

Data-Informed Evaluation Framework for Integrated Energy Systems: Insights from Power, Process Heat, and Hydrogen Production Applications

The multi-criteria decision analysis (MCDA) framework provides a systematic evaluation of the diverse preferences and performance metrics associated with alternative solutions. This approach is advantageous over a single-criterion methodology, which are only valid under conditions that assume ceteris paribus or an "apples-to-apples" comparison. However, selecting suitable technologies for integrated energy systems (IES) can be likened to an "apples-to-oranges" comparison, given the heterogeneous factors at stake. These factors include economics and performance parameters, geological compatibility, and environmental impacts. Consequently, past research has often employed a mixture of qualitative and quantitative criteria tailored to the specific interests of each study. While the method proves effective in handling the intricate interplay of criteria, the resulting rankings and scores can vary from study to study. This inconsistency is introduced from the use of subjectively defined thresholds and weights. As a result, decision-makers frequently find it challenging to establish clear connections between specific criteria and the resulting scores, as the transformation of criteria into ordinal scores results in a substantial loss of information. To address this challenge, we introduce a data-informed IES evaluation framework that offers comprehensive, interpretable, and traceable evaluations backed by quantifiable rationale. First, we identified key IES evaluation criteria from a decade of literature, focusing on relevant IES applications in power, process heat, and hydrogen production. We leveraged state-of-the-art cost estimates from the Idaho National Laboratory (INL) and technical data from 78 reactor designs from the International Atomic Energy Agency (IAEA) and the Organization for Economic Co-operation and Development - Nuclear Energy Agency (OECD-NEA). Lastly, we established thresholds by analyzing the mean, variance, root mean square, and slope of values across alternatives, categorizing the preferences of decision-makers into distinct utility functions, such as linear, saturating, exponential, and stepwise. Our approach yielded two main outcomes: (1) it provided consistent assessments across different stakeholder groups and (2) it visualized uncertainties in the decision-making context via comprehensive sensitivity analysis. To demonstrate the impact of our framework, we conducted case studies on 6 reactor designs (AP1000, NuScale, BWRX-300, Xe-100, eVinci, iMSR) for the three applications. Our data-driven framework proved to be highly effective in addressing heterogenous uncertainties faced by varied decision-makers? preferences and IES applications, as well as cost and technical estimates of advanced reactors.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1992

Proposed evaluation framework for assessing operator performance with multisensor displays

Despite aggressive work on the development of sensor fusion algorithms and techniques, no formal evaluation procedures have been proposed. Based on existing integration models in the literature, an evaluation framework is developed to assess an operator's ability to use multisensor, or sensor fusion, displays. The proposed evaluation framework for evaluating the operator's ability to use such systems is a normative approach: The operator's performance with the sensor fusion display can be compared to the models' predictions based on the operator's performance when viewing the original sensor displays prior to fusion. This allows for the determination as to when a sensor fusion system leads to: 1) poorer performance than one of the original sensor displays (clearly an undesirable system in which the fused sensor system causes some distortion or interference); 2) better performance than with either single sensor system alone, but at a sub-optimal (compared to the model predictions) level; 3) optimal performance (compared to model predictions); or, 4) super-optimal performance, which may occur if the operator were able to use some highly diagnostic 'emergent features' in the sensor fusion display, which were unavailable in the original sensor displays. An experiment demonstrating the usefulness of the proposed evaluation framework is discussed.

Foyle, David C.↗
DOE OSTI2023

Open Source Evaluation Framework for Solar Forecasting

The Solar Forecast Arbiter is an open-source evaluation framework for solar forecasting. The framework enables evaluations of solar irradiance, solar power, and net-load forecasts that are impartial, repeatable and auditable. The Solar Forecast Arbiter addresses stakeholder-informed use cases including evaluation of forecast skill, comparisons to reference data sets, private forecast trials, and evaluation of probabilistic forecast skill. The framework includes a data validation toolkit, reference data sources, data privacy protocols, and benchmark forecast capabilities for intra-hour and day ahead forecast horizons. Reports and metrics communicate the relative merits of the test and benchmark forecasts. The reports are created from standardized templates and include graphics for qualitatively evaluating deterministic and probabilistic forecasts and standard metrics for quantitatively evaluating forecasts. The Solar Forecast Arbiter is designed to support all solar forecasting stakeholders, including Solar Forecasting 2 Topic Area 2 and Topic Area 3 teams.

14 SOLAR ENERGY↗
NASA NTRS1999

Evaluation Framework for NASA's Educational Outreach Programs

The objective of the proposed work is to develop an evaluation framework for NASA's educational outreach efforts. We focus on public (rather than technical or scientific) dissemination efforts, specifically on Internet-based outreach sites for children.The outcome of this work is to propose both methods and criteria for evaluation, which would enable NASA to do a more analytic evaluation of its outreach efforts. The proposed framework is based on IRL's ethnographic and video-based observational methods, which allow us to analyze how these sites are actually used.

Berg, Rick↗
NASA NTRSDate not supplied

A Comprehensive eVTOL Performance Evaluation Framework in Urban Air Mobility

In this paper, we developed an open-source simulation framework for the evaluation of electric vertical takeoff and landing vehicles (eVTOLs) in the context of Unmanned Traffic Management (UTM) and under the concept of Urban Air Mobility (UAM). Unlike most existing studies, the proposed framework combines the utilization of UTM and eVTOLs to develop a realistic UAM testing platform. For this purpose, we first develop an UTM simulator to simulate the real-world UAM environment. Then, instead of using a simplified eVOTL model, a high-fidelity eVTOL design tool, namely SUAVE, is employed and an dilation sub-module is introduced to bridge the gap between the UTM simulator and SUAVE eVTOL performance evaluation tool to elaborate the complete mission profile. Based on the developed simulation framework, experiments are conducted and the results are presented to analyze the performance of eVTOLs in the UAM environment.

Mrinmoy Sarkar↗
DOE OSTI2026

Rapid Evaluation Framework for the CMIP7 Assessment Fast Track

As Earth system models (ESMs) grow in complexity and in volume of output data, there is an increasing need for rapid, comprehensive evaluation of their scientific performance. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require expeditious response for model analyses designed to inform and drive integrated Earth system assessments. To meet this challenge, the Rapid Evaluation Framework (REF), a community-driven platform for benchmarking and performance assessment of ESMs, was designed and developed. The initial implementation of the REF, constructed to meet the near-term needs of the CMIP7 Assessment Fast Track, builds upon four disparate community evaluation and benchmarking tools that are coupled together using the Coordinated Model Evaluation Capabilities (CMEC) framework. The REF runs within a containerized workflow for portability and reproducibility and is aimed at generating and organizing diagnostics covering a variety of model variables. The REF leverages well documented observational datasets to provide assessments of model fidelity across a collection of diagnostics. All diagnostics were identified and selected with community involvement and consultation. Operational integration with the Earth System Grid Federation (ESGF) will permit automated execution of the REF for selected diagnostics as soon as model output data are published on ESGF by the originating modeling centers. The REF is designed to be portable across a range of current computational platforms to facilitate use by modeling centers for assessing the evolution of model versions or gauging the relative performance of CMIP simulations before being published on ESGF. When integrated into production simulation workflows, results from the REF provide immediate quantitative feedback that allows model developers and scientists to quickly identify model biases and performance issues. After the REF is released to the community, its subsequent development and support will be prioritized by an international consortium of scientists and engineers, enabling a broader impact across Earth science disciplines. For instance, the REF will facilitate improvements to models and will enhance confidence in model projections through process-based selection of models based on their performance with respect to observations. Production of reproducible diagnostics and community-based assessments are key features of the REF. Furthermore, providing interoperability with existing evaluation packages assures that contributions from previous community efforts will be available for use in future model intercomparison projects.

Hoffman, Forrest [ORNL] (ORCID:0000000158024134)↗
DOE OSTI2024

A Proposed Evaluation Framework for New and Emerging Low Embodied-Carbon Concrete Technologies

New opportunities for carbon reductions in buildings create a strong need for a common framework and method for those who design, build and influence construction to evaluate lifecycle carbon reductions from design decisions and technology choices. These opportunities include a wide range of low-embodied-carbon concrete materials being rapidly developed and introduced to the market. How to evaluate these newer materials and technologies has become critical for both public- and private-sector actors seeking to decarbonize building constructions by leveraging the Infrastructure Investment and Jobs Act (IIJA) and Inflation Reduction Act (IRA) funds. We propose an evaluation framework to assess the lifecycle carbon reductions from adoption of these technologies, including a subset of key “must have” (1) technical criteria (embodied carbon level, technology development stage); (2) market criteria (market size, scalability); and (3) financial criteria (cost of technology implementation compared to businessas-usual) from a range of options. We discuss how to use the framework and illustrate it using a “heatmap,” rating score and short case study of a promising technology. We also propose a plan to implement this framework that includes (1) standardized measurement and validation methods for verifying emission reductions from these technologies, and (2) avenues to implement real world demonstrations. We conclude with recommendations for next steps on framework refinement and commercialization strategy development.

Singh, Reshma↗
NASA NTRS2017

Assessment of Intralaminar Progressive Damage and Failure Analysis Using an Efficient Evaluation Framework

Reducing the timeline for development and certification for composite structures has been a long standing objective of the aerospace industry. This timeline can be further exacerbated when attempting to integrate new fiber-reinforced composite materials due to the large number of testing required at every level of design. computational progressive damage and failure analysis (PDFA) attempts to mitigate this effect; however, new PDFA methods have been slow to be adopted in industry since material model evaluation techniques have not been fully defined. This study presents an efficient evaluation framework which uses a piecewise verification and validation (V&V) approach for PDFA methods. Specifically, the framework is applied to evaluate PDFA research codes within the context of intralaminar damage. Methods are incrementally taken through various V&V exercises specifically tailored to study PDFA intralaminar damage modeling capability. Finally, methods are evaluated against a defined set of success criteria to highlight successes and limitations.

Hyder, Imran↗
DOE OSTI2024

An Evaluation Framework for State Energy Offices' Energy Efficiency and Clean Energy Workforce Programs

Historic levels of funding for energy efficiency and clean energy projects from the Bipartisan Infrastructure Law and Inflation Reduction Act will be flowing through State Energy Offices (SEOs) in the coming years, including funds specifically for supporting workforce development efforts. This report offers a workforce evaluation framework that they can use to track and measure the progress, performance, and effectiveness of energy efficiency and clean energy workforce programs that they fund and/or implement. This evaluation step is essential to ensuring that SEOs' workforce efforts remain relevant to the evolving needs of the energy efficiency and clean energy sector and advance a workforce that is inclusive and diverse thus maximizing the success of their programs.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

BISON-FIPD integration enhanced low-burnup SFR metallic fuel swelling model evaluation framework

Experiments indicated that metallic fuel in sodium-cooled fast reactors (SFRs) rapidly swells radially and axially at low burnup. Despite that, prior studies have been focused on describing high burnup axial fuel elongation. With recent conventional and non-conventional metallic fuel concepts being considered for license applications, understanding multidimensional fuel swelling at a wide range of burnup levels is important to fuel analysis and qualification activities. Here, we report the development and demonstration efforts of a low-burnup SFR metallic fuel swelling model evaluation framework using the BISON advanced fuel performance code. The framework leverages the Integral Fast Reactor (IFR) program X423 experiment data through the ongoing integration project to enable standardized and automated use of legacy metallic fuel irradiation data maintained in the SFR fuel irradiation and physics database (FIPD) for BISON metallic fuel model verification and validation. In conclusion, the performance of the framework was demonstrated using the two representative metallic fuel swelling model sets implemented in BISON, with a series of insights about future advanced swelling model development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2021

Bayesian Monte-Carlo Evaluation Framework for Imperfect Data [Slides]

BMC evaluation is a tool to address imperfect data & models, non-linear models, and non-normal PDFs. New posterior PDFs may need new storage formats to allow storage of non-normal PDFs. Storing posterior sets allows for: variance, covariance, skewness, etc.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Bayesian Monte Carlo Evaluation Framework for Imperfect Data and Models [Abstract]

Nuclear data evaluation methods conventionally make the following assumptions: prior and posterior probability distribution functions (PDFs) of all model parameters and data are normal (Gaussian); the linear approximation is sufficiently accurate for minimization of a cost function (even for non-linear models); and that both the model (of, e.g., neutron cross section) and experimental data (including their covariance data) are perfect. These assumptions are inherent to the well-known generalized linear least squares (GLLS) minimization method commonly used for evaluations of resolved resonance region (RRR) neutron cross sections. However, these assumptions are often not justified due to the presence of non-normal PDFs, non-linear models (e.g. R -matrix formalism), and inherent imperfections in data and models (e.g. discrepant data sets, discrepancies between the previous evaluation and newly measured data, or imperfect covariance data). We remove the said assumptions in a mathematical framework of Bayes’ theorem, and implement it using the Metropolis-Hastings Monte Carlo method. Parameters of a new kind are introduced to parameterize inherent imperfections, e.g. , any discrepancies between the theoretical model and measured data. These new parameters enable evaluators to quantify their expert judgement about any discrepancies or imperfections in a reproducible manner. We demonstrate the framework with an ongoing evaluation of 233 U in the eV region using the ENDF-B/VIII library and transmission data measured by Guber, et al. , and compare the posterior parameters to those obtained by conventional evaluation methods.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Bayesian Monte-Carlo Evaluation Framework for Imperfect Nuclear Data [Slides]

BMC evaluation is a tool used to address imperfect data and models, non-linear models, and non-normal PDFs. ENDF-6 format does not allow non-normal parameter PDFs. Storing posterior sets allows for variance, covariance, skewness, etc. To better predict criticality, we should document non-normal parameter PDFs (i.e. asymmetric uncertainty) and consider non-linear sensitivity of $k_{\text{eff}}$ to resonance parameters.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗