Search NASA⌕ Search

SEARCH · Search NASA

Results for “Expert Judgment”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.

Expert Judgment↗

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.

Expert Judgement↗

Criteria for Retention of 3013 S1 Containers Based on Relative Risk and Expert Judgment

An evaluation was performed to assess the suitability of thirty-three 3013 containers proposed for retention. These containers have moisture levels greater than 0.08 wt.% – the S1 population. The remainder of the S1 population stored at SRS will be down blended and disposed of by the end of 2028. Based on field surveillance and shelf-life data available to date as well as informed technical judgment, no container is currently expected to fail in its 50-year storage period. However, corrosion risk varies across the S1 population. Relative risks were evaluated using predicted Consensus Scores and their 95% Upper Prediction Limits (UPLs). The predicted values are based on a statistical model of Consensus Score as a function of moisture, chloride, and whether the packaged material was electrorefining scrap packaged at Hanford. Consensus Score has been shown to be a useful indicator of corrosion potential, and the UPL captures uncertainty in the model predictions, providing a conservative indicator of corrosion potential. Using UPLs to determine relative risks, together with expert review, three containers were identified as not suitable for retention, and the remainder were determined to be suitable.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Development of an Expert Judgement Elicitation and Calibration Methodology for Risk Analysis in Conceptual Vehicle Design

A comprehensive expert-judgment elicitation methodology to quantify input parameter uncertainty and analysis tool uncertainty in a conceptual launch vehicle design analysis has been developed. The ten-phase methodology seeks to obtain expert judgment opinion for quantifying uncertainties as a probability distribution so that multidisciplinary risk analysis studies can be performed. The calibration and aggregation techniques presented as part of the methodology are aimed at improving individual expert estimates, and provide an approach to aggregate multiple expert judgments into a single probability distribution. The purpose of this report is to document the methodology development and its validation through application to a reference aerospace vehicle. A detailed summary of the application exercise, including calibration and aggregation results is presented. A discussion of possible future steps in this research area is given.

Unal, Resit↗

Safety Risk Knowledge Elicitation in Support of Aeronautical R and D Portfolio Management: A Case Study

Aviation is a problem domain characterized by a high level of system complexity and uncertainty. Safety risk analysis in such a domain is especially challenging given the multitude of operations and diverse stakeholders. The Federal Aviation Administration (FAA) projects that by 2025 air traffic will increase by more than 50 percent with 1.1 billion passengers a year and more than 85,000 flights every 24 hours contributing to further delays and congestion in the sky (Circelli, 2011). This increased system complexity necessitates the application of structured safety risk analysis methods to understand and eliminate where possible, reduce, and/or mitigate risk factors. The use of expert judgments for probabilistic safety analysis in such a complex domain is necessary especially when evaluating the projected impact of future technologies, capabilities, and procedures for which current operational data may be scarce. Management of an R&D product portfolio in such a dynamic domain needs a systematic process to elicit these expert judgments, process modeling results, perform sensitivity analyses, and efficiently communicate the modeling results to decision makers. In this paper a case study focusing on the application of an R&D portfolio of aeronautical products intended to mitigate aircraft Loss of Control (LOC) accidents is presented. In particular, the knowledge elicitation process with three subject matter experts who contributed to the safety risk model is emphasized. The application and refinement of a verbal-numerical scale for conditional probability elicitation in a Bayesian Belief Network (BBN) is discussed. The preliminary findings from this initial step of a three-part elicitation are important to project management practitioners as they illustrate the vital contribution of systematic knowledge elicitation in complex domains.

Shih, Ann T.↗

A mathematical approach to using the forgetting curve to evaluate experience and training factors in human reliability analysis

Traditional human reliability analysis (HRA) methods have difficulty dealing with the dynamic nature of factors such as time and rely on static and expert-judgment-based assessments of performance-shaping factors (PSFs) across limited levels. In this study, we introduce a mathematical approach for dynamically evaluating the experience and training PSF. Our proposed method integrates the psychological concept of the “forgetting curve” to evaluate how PSFs are impacted by the number of trainings and the time elapsed since training. To confirm the validity of the model, we provide experimental data fitted by identifying the quantitative relationship between training and human performance. This research enables dynamic and objective assessments, thus reducing reliance on subjective expert judgment and improving the accuracy of HRA.

99 - GENERAL AND MISCELLANEOUS↗

A Step-Wise Approach to Elicit Triangular Distributions

Adapt/combine known methods to demonstrate an expert judgment elicitation process that: 1.Models expert's inputs as a triangular distribution, 2.Incorporates techniques to account for expert bias and 3.Is structured in a way to help justify expert's inputs. This paper will show one way of "extracting" expert opinion for estimating purposes. Nevertheless, as with most subjective methods, there are many ways to do this.

Greenberg, Marc W.↗

Examining Cloud Feedback Components in the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM)

Cloud feedback remains the main source of uncertainty in climate sensitivity estimated by global climate models (GCMs), largely because subgrid cloud responses are parameterized in GCMs due to their coarse resolution. Here, this study examines cloud feedback in the global 3.25-km Simple Cloud-Resolving Energy Exascale Earth System Model (E3SM) Atmosphere Model (SCREAM 3 km) through a pair of 1-yr atmosphere-only simulations with control and +4-K sea surface temperature perturbations. SCREAM 3 km produces a positive cloud feedback that falls within but at the upper end of the range of Coupled Model Intercomparison Project phase 5 (CMIP5) and CMIP phase 6 (CMIP6) models and expert judgment. The positive cloud feedback arises from positive contributions from both high- and low-level clouds, with increases in high-cloud altitude and decreases in low-cloud amount and optical depth playing key roles. The stronger-than-CMIP-average feedback is mainly attributable to the high-cloud altitude feedback, owing to cloud tops rising nearly isothermally in SCREAM 3 km. The positive low-cloud amount feedback is weaker in SCREAM than in GCMs because estimated inversion strength (EIS) increases more dramatically with warming. A coarser 12-km resolution version of SCREAM exhibits a weaker positive cloud feedback than SCREAM 3 km, mainly because its low-cloud-radiative flux is more sensitive to EIS, leading to a stronger negative low-cloud amount feedback. With this process-level assessment of cloud feedback, this study reveals where SCREAM aligns with and diverges from conventional GCMs and expert assessment, providing insights to inform further model improvement and future expert assessment.

Cloud radiative effects↗

Virtual refrigerant charge sensor for variable-speed heat pumps based on feature selection

The refrigerant charge level in heat pump systems significantly impacts their energy efficiency. Virtual refrigerant charge (VRC) sensing technology has been comprehensively investigated and well-established due to its lower cost compared to physical sensors. However, the previous VRC research often relied on expert judgment and physical reasoning for their variable selection, which can potentially select redundant (or highly correlated) or insignificant features, and it is also primarily focused on single-speed systems. To address these challenges, this study proposes a VRC algorithm for variable-speed heat pumps that selects features through a rigorous feature selection method in combination with physical insights. We also propose a piecewise linear model structure segmented by subcooling temperature to accurately predict charge levels, particularly when subcooling temperatures are substantially low. The proposed algorithm was evaluated using experimental data of a residential R410A heat pump, and the performance was compared with two baseline VRC algorithms. The results are: (1) The proposed algorithm outperforms for the case with subcooling temperature less than 1 °C. (2) The proposed algorithm achieves a tested mean absolute percentage error (MAPE) of 4.23%, and improves the overall accuracy for cooling conditions by approximately 60%, compared with the two baseline algorithms. (3) The proposed algorithm uses two fewer features and improves the accuracy for undercharge cooling conditions by 68.0%, compared with baseline algorithm 2. These improvements enhance prediction accuracy and prevent overfitting, providing a more reliable refrigerant charge level prediction and helping improve the heat pump energy efficiency.

Liang, Chenjiyu↗

Impact of representative ground motion level on seismic PSA with the boundary between overestimation and underestimation

One commonly used approach in seismic probabilistic safety assessment (PSA) is the discrete method. This method follows the standard PSA framework and can be applied to various models, such as multi-unit models, while reducing computational costs using standard software. However, due to the inability to subdivide intervals infinitely, the discrete method approximates with a finite number of subintervals. In practice, different numbers of subintervals are applied, and the representative ground motion level is selected based on expert judgment. When employing a smaller number of subintervals, it is important to take caution to prevent underestimation. This study analyzes the impact of the representative ground motion level on seismic risk. It confirms that underestimation can occur with a small number of subintervals depending on the representative ground motion level. This study also proposes a method for determining the boundary of underestimation and overestimation. The method is demonstrated through examples, providing a mathematical foundation for selecting appropriate representative ground motion levels. By avoiding underestimation, this research helps prevent the oversight of significant risk contributors and enhances the understanding of seismic risk.

99 - GENERAL AND MISCELLANEOUS↗

Understanding Decision-Relevant Regional Data Products: Workshop Report

A broad community of climate adaptation practitioners, stakeholders and policymakers rely on historical reconstructions and future projections of local to regional climate. To be of value to these users, climate data must be credible, salient, and authoritative (Cash et al. 2002). Namely, data must be consistent with our physical understanding of the global Earth system, must be relevant for informing the decision-making process, and must be backed by expert judgment. As more and more data products have become available, multiple challenges have emerged around the production, evaluation, selection, and use of these data products. Consequently, to ensure crucial decisions leverage the best possible historical and future physical climate data, there is a pressing need to develop a coordinated national climate data strategy that is inclusive of all relevant communities of practice.

54 ENVIRONMENTAL SCIENCES↗

Assessing and Enabling Trustworthy Predictions for High-Consequence Decisions

Predictions from physics-based computational models provide critical information to inform high consequence decisions, e.g., engineering design decisions. The ability to assess the reliability of such predictions is therefore critical. However, to date, reliability assessment rely heavily on expert judgment and qualitative arguments. This report details the efforts of LDRD 233072 to develop quantitative methods to assess reliability of model predictions, especially in the context of simplifying assumptions that can impact their reliability.

42 ENGINEERING↗

Battery Life Prediction Using Reduced-Order Physics Models and Machine Learning (CRADA Final Report)

Phase 1 (Original CRADA, plus no-cost extension modifications #1-3, 6/1/2017 to 3/13/2021): The Australian Department of Defence (AUDoD) is performing accelerated aging tests of Li-ion batteries to benchmark their reliability and degradation characteristics. Using its previously developed battery lifetime predictive model framework, the National Laboratory of the Rockies (NLR) will develop analytical models based the AUDoD data to predict lifetime of the multiple Li-ion battery chemistries under real-world use scenarios of interest to AUDoD. The NLR model is based on physical degradation mechanisms encountered by Li-ion batteries and has been previously validated. Phase 2 (CRADA modification #4, plus no-cost extension modification #5, 2/22/2021 to 3/30/2025): Train and support Australian Department of Defence personnel to use NLR software for model-based estimation of Li-ion battery lifetime using accelerated battery aging data collected by the Australian Department of Defence. Under separate DOE funding from 2019 to 2021, NLR enhanced its battery life-prediction software using machine learning algorithms to automate portions of the model-fitting process, requiring significantly less labor and expert judgment and also adding uncertainty quantification, increasing statistical rigor. Under Phase 2, NLR will customize NLR Software and provide it to AuDoD. NLR will enhance its NLR Model to capture aging modes of AuDoD's multi-cell modules, including cell-balancing effects. NLR will develop example single-cell and multi-cell models based on one AuDoD battery aging dataset. NLR will train AuDoD personnel on NLR Software. By the conclusion of the project, NLR will have provided AuDoD the training materials, a user manual and software needed to perform their own analysis of additional and/or future battery aging datasets.

33 ADVANCED PROPULSION SYSTEMS↗

Automated shaker placement and regularized input estimation for MIMO testing.

Multi-input, multi-output (MIMO) testing is used in component qualification to reproduce operational responses in the laboratory. It is often preferred to single-input and base-shake testing because of the potential for equivalent or better tests using smaller actuators and shorter test suites. Given a target response, two key steps in MIMO test design are selecting actuator locations and solving for input loads. Actuator locations are often manually selected using expert judgment. If an automatic method is used, locations are usually determined by simulating the vibration control problem and minimizing a combination of the input energy and control residuals. To select a configuration, the relative importance of input energy and residuals must be specified. Specifying relative weights is, in general, a manual and subjective process. This paper develops an objective function that compares actuator configurations based on control accuracy and required input energy without any manual parameter tuning. The objective function uses an optimally selected tradeoff parameter for each candidate configuration. To choose actuator locations using the new objective function, a pivoting algorithm for integer programming problems is developed. Starting with an initial configuration (such as the one generated by a greedy algorithm), the pivoting algorithm guarantees an objective function decrease in each iteration until convergence is reached. In a simulation featuring a structure excited by a diffuse acoustic field, electrodynamic shaker locations and regularized inputs are solved for without any analyst-specified parameters. Simulations are performed in MIMO configurations where the number of target responses is less than, equal to, and greater than the number of actuators.

Multi-input multi-output↗

Eucalyptus – An Analysis Suite for Fault Trees with Uncertainty Quantification

Eucalyptus is a novel code developed at Lawrence Livermore National Laboratory to incorporate uncertainty quantification into Fault Tree Analysis (FTA). This tool addresses the challenge of imperfect knowledge in “grey-box” systems by allowing analysts to incorporate and propagate uncertainty from component-level assessments to system-level effects. Eucalyptus facilitates a consistent evaluation of the impact of subject matter expert judgment and knowledge gaps on overall system response by Monte Carlo generation of possible system fault trees, sampling probabilities of the existence of subsystems and components. Here, the code supports the specification of fault trees through text and allows export to various formats, including auto-generated images, easing analysis and reducing errors. It has undergone extensive verification testing, demonstrating its reliability and readiness for deployment, and leverages on-node parallelism for rapid analysis. Example analyses are shown that include the identification of system failure paths and quantification of the value of further information about system components.

Fault Tree Analysis↗

Decision paths in complex tasks

Complex real world action and its prediction and control has escaped analysis by the classical methods of psychological research. The reason is that psychologists have no procedures to parse complex tasks into their constituents. Where such a division can be made, based say on expert judgment, there is no natural scale to measure the positive or negative values of the components. Even if we could assign numbers to task parts, we lack rules i.e., a theory, to combine them into a total task representation. We compare here two plausible theories for the amalgamation of the value of task components. Both of these theories require a numerical representation of motivation, for motivation is the primary variable that guides choice and action in well-learned tasks. We address this problem of motivational quantification and performance prediction by developing psychophysical scales of the desireability or aversiveness of task components based on utility scaling methods (Galanter 1990). We modify methods used originally to scale sensory magnitudes (Stevens and Galanter 1957), and that have been applied recently to the measure of task 'workload' by Gopher and Braune (1984). Our modification uses utility comparison scaling techniques which avoid the unnecessary assumptions made by Gopher and Braune. Formula for the utility of complex tasks based on the theoretical models are used to predict decision and choice of alternate paths to the same goal.

Galanter, Eugene↗

Applications of Principled Search Methods in Climate Influences and Mechanisms

Forest and grass fires cause economic losses in the billions of dollars in the U.S. alone. In addition, boreal forests constitute a large carbon store; it has been estimated that, were no burning to occur, an additional 7 gigatons of carbon would be sequestered in boreal soils each century. Effective wildfire suppression requires anticipation of locales and times for which wildfire is most probable, preferably with a two to four week forecast, so that limited resources can be efficiently deployed. The United States Forest Service (USFS), and other experts and agencies have developed several measures of fire risk combining physical principles and expert judgment, and have used them in automated procedures for forecasting fire risk. Forecasting accuracies for some fire risk indices in combination with climate and other variables have been estimated for specific locations, with the value of fire risk index variables assessed by their statistical significance in regressions. In other cases, the MAPSS forecasts [23, 241 for example, forecasting accuracy has been estimated only by simulated data. We describe alternative forecasting methods that predict fire probability by locale and time using statistical or machine learning procedures trained on historical data, and we give comparative assessments of their forecasting accuracy for one fire season year, April- October, 2003, for all U.S. Forest Service lands. Aside from providing an accuracy baseline for other forecasting methods, the results illustrate the interdependence between the statistical significance of prediction variables and the forecasting method used.

Glymour, Clark↗