Eliciting Experts' Judgments about Uncertain Probabilities: A Brief Primer to Using Expert Elicitation Well.
Abstract not provided.
Butler, Karin↗
SEARCH · Search NASA
Results for “expert elicitation”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Expert elicitation survey predicts 37% to 49% declines in wind energy costs by 2050
Wind energy has experienced accelerated cost reduction over the past five years—far greater than predicted in a 2015 expert elicitation. Here we report results from a new survey on wind costs, compare those with previous results and discuss the accuracy of the earlier predictions. In this work, we show that experts in 2020 expect future onshore and offshore wind costs to decline 37–49% by 2050, resulting in costs 50% lower than predicted in 2015. This is due to cost reductions witnessed over the past five years and expected continued advancements. If realized, these costs might allow wind to play a larger role in energy supply than previously anticipated. Considering both surveys, we also conclude that there is considerable uncertainty about future costs. Our results illustrate the importance of considering cost uncertainty, highlight the value and limits of using experts to reveal those uncertainties, and yield possible lessons for energy modellers and expert elicitation.
17 WIND ENERGY↗
The Utility in Conjoint Analysis as a Fast Expert Elicitation Technique
This paper presents an interface and analysis technique for quickly conducting expert elicitation with the goal of determining entity importance. Our interface deploys a two-alternative choice experiment that is capable of representing knowledge graphs in an easy to interpret fashion for users with limited experience with knowledge graphs. Our analysis methodology takes advantage of conjoint analysis techniques and provides entity weights for many SMEs simultaneously. The results largely align with individual participant fits.
Conjoint Analysis, interface, Expert Elicitation, ↗
Expert Elicitation for Tidal Energy Levelized Cost of Energy: Present and Future
In accordance with the Government Performance and Results Act (GPRA), H2O annually assesses marine energy technology development resulting from government-funded research and development programs and strategy. For GPRA reporting, H2O uses the levelized cost of energy (LCOE) - which represents the total system cost per unit of energy produced - to measure the progression of marine energy technology development, assess the impact of their research and development programs, and identify future research priorities. To support H2O's GPRA reporting requirements and inform future strategy, the National Laboratory of the Rockies conducted a tidal energy LCOE expert elicitation study to estimate present and future LCOE[AB2.1]. This report describes the motivation and background for the elicitation study, the methodology used to conduct the study, and the study results. It also provides future recommendations for accelerated tidal energy LCOE reduction based on feedback from study participants.
16 TIDAL AND WAVE POWER↗
Machine Learning for the Validation of Expert-Elicited Causal Risk Diagrams
Exposure to spaceflight poses risk to human health in complex ways. To help manage this risk, the Human Systems Risk Board (HSRB) at the National Aeronautics and Space Administration (NASA) maintains a set of causal diagrams that attempt to explain how spaceflight hazards generate health risks and lead to adverse outcomes both in-mission, immediately post-mission, and over the long term. These causal risk diagrams are formulated as directed acyclic graphs (DAGs) and can function as knowledge graphs of connected risks and outcomes. These DAGs have proven useful for communication, and, through network analysis, have allowed for the identification of structurally important factors in the risk network. However, the utility these DAGs provide is directly proportional to their verisimilitude, making assessment of this trait using empirical data – whether from actual human spaceflight or various spaceflight analogue exposures and model organisms – a high priority. In this research we explore the use of machine learning algorithms to learn DAG structure from empirical data as a means of evaluating human-elicited DAG structures. To do so, we test several different graph structure-learning algorithms on data concerning changes in the bones of rats and mice after exposure to either spaceflight or a spaceflight analogue. We explore potential methods for indexing the similarity between each algorithm’s output DAG with all the others and with that of the expert-elicited DAG. We discuss next steps in this ongoing line of research and open science initiatives underway to complete them.
directed acyclic graphs↗
Quantifying Earth System Interactions for Sustainable Food Production Via Expert Elicitation
Several safe boundaries of critical Earth system processes have already been crossed due to human perturbations; not accounting for their interactions may further narrow the safe operating space for humanity. Using expert knowledge elicitation, we explored interactions among seven variables representing Earth system processes relevant to food production, identifying many interactions little explored in Earth system literature. We found that green water and land system change affect other Earth system processes strongly, while land, freshwater and ocean components of biosphere integrity are the most impacted by other Earth system processes, most notably blue water and biogeochemical flows. We also mapped a complex network of mechanisms mediating these interactions and created a future research prioritization scheme based on interaction strengths and existing knowledge gaps. Our study improves the understanding of Earth system interactions, with sustainability implications including improved Earth system modelling and more explicit biophysical limits for future food production.
Expert knowledge elicitation↗
A Quantitative Approach to Perform Expert Elicitation for Space Communication System Design and Risk Analysis: Methodology and Experimental Results
No abstract available
expert elicitation↗
Expert Elicitation on Wind Farm Control
Wind farm control is an active and growing field of research in which the control actions of individual turbines in a farm are coordinated, accounting for inter-turbine aerodynamic interaction, to improve the overall performance of the wind farm and to reduce costs. The primary objectives of wind farm control include increasing power production, reducing turbine loads, and providing electricity grid support services. Additional objectives include improving reliability or reducing external impacts to the environment and communities. In 2019, a European research project (FarmConners) was started with the main goal of providing an overview of the state-of-the-art in wind farm control, identifying consensus of research findings, data sets, and best practices, providing a summary of the main research challenges, and establishing a roadmap on how to address these challenges. Complementary to the FarmConners project, an IEA Wind Topical Expert Meeting (TEM) and two rounds of surveys among experts were performed. From these events we can clearly identify an interest in more public validation campaigns. Additionally, a deeper understanding of the mechanical loads and the uncertainties concerning the effectiveness of wind farm control are considered two major research gaps.
17 WIND ENERGY↗
Expert Elicitation for Wave Energy LCOE Futures
As the world faces increasing threats from climate change, the importance of developing renewable energy technologies and reducing their costs have similarly increased. Marine energy technologies (which include wave, tidal, ocean current, ocean thermal and salinity gradient resources) are often referred to as the most nascent and newest suite of renewable technologies under development. There are vast marine energy resources available around the world and within U.S. territorial waters, and as the technologies have continued to develop, the long-term trajectory of cost reductions and performance improvements is of particular interest. This study specifically investigates the long-term cost reduction potential for commercial wave energy technologies, as wave energy represents the largest marine energy resource available to the continental U.S. Similar studies focused on other resources and technologies may follow in the future.
16 TIDAL AND WAVE POWER↗
An Expert Elicitation Method for Estimating Probabilistic Schedule Risk During Conceptual System Design.
Abstract not provided.
Pedersen, Robert Douglas↗
Reliability Prediction using FMEA, FTA, and Related Techniques [Slides]
Summary: We presented a reliability analysis framework. We made point estimates of reliability using reliability block diagrams, fault trees, success trees and estimates with uncertainty using expert elicitation, Monte Carlo simulation, Bayesian analysis. Expert elicitation of failure modes and probabilities is labor-intensive, but critical. Bayesian analysis updates information from expert elicitation with data from reliability and aging tests (aging/compatibility data are needed to estimate lower-bound reliabilities at end of life). Estimation by more than one method helps insure consistency and accuracy.
97 MATHEMATICS AND COMPUTING↗
Development of an Expert Judgement Elicitation and Calibration Methodology for Risk Analysis in Conceptual Vehicle Design
A comprehensive expert-judgment elicitation methodology to quantify input parameter uncertainty and analysis tool uncertainty in a conceptual launch vehicle design analysis has been developed. The ten-phase methodology seeks to obtain expert judgment opinion for quantifying uncertainties as a probability distribution so that multidisciplinary risk analysis studies can be performed. The calibration and aggregation techniques presented as part of the methodology are aimed at improving individual expert estimates, and provide an approach to aggregate multiple expert judgments into a single probability distribution. The purpose of this report is to document the methodology development and its validation through application to a reference aerospace vehicle. A detailed summary of the application exercise, including calibration and aggregation results is presented. A discussion of possible future steps in this research area is given.
Unal, Resit↗
A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation
The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.
Expert Judgement↗
A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation
The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.
Expert Judgment↗
The Potential of Hydrogeodesy to Address Water-Related and Sustainability Challenges
Increasing climatic and human pressures are changing the world's water resources and hydrological processes at unprecedented rates. Understanding these changes requires comprehensive monitoring of water resources. Hydrogeodesy, the science that measures the Earth's solid and aquatic surfaces, gravity field, and their changes over time, delivers a range of novel monitoring tools that are complementary to traditional hydrological methods. It encompasses geodetic technologies such as Altimetry, Interferometric Synthetic Aperture Radar (InSAR), Gravimetry, and Global Navigation Satellite Systems (GNSS). Beyond quantifying these changes, there is a need to understand how hydrogeodesy can contribute to more ambitious goals dealing with water-related and sustainability sciences. Addressing this need, we combine a meta-analysis of over 3,000 articles to chart the range, trends, and applications of satellite-based hydrogeodesy with an expert elicitation that systematically assesses the potential of hydrogeodesy. We find a growing body of literature relating to the advancements in hydrogeodetic methods, their accuracy and precision, and their inclusion in hydrological modeling, with a considerably smaller portion related to understanding hydrological processes, water management, and sustainability sciences. The meta-analysis also shows that while lakes, groundwater and glaciers are commonly monitored by these technologies, wetlands or permafrost could benefit from a wider range of applications. In turn, the expert elicitation envisages the potential of hydrogeodesy to help solve the 23 Unsolved Questions of the International Association of Hydrological Sciences and advance knowledge as guidance toward a safe operating space for humanity. It also highlights how this potential can be maximized by combining hydrogeodetic technologies simultaneously, exploiting artificial intelligence, and accurately integrating other Earth science disciplines. Finally, we call for a coordinated way forward to include hydrogeodesy in tertiary education and broaden its application to water-related and sustainability sciences in order to exploit its full potential.
54 ENVIRONMENTAL SCIENCES↗
Advancing Wildland Fire Response with NASA’s Second Shift Capabilities
Novel “Second-Shift” capabilities—leveraging Uncrewed Aerial Systems and Optionally Piloted Vehicles—are identified that could extend wildfire aerial logistics support into night or low-visibility conditions. Expert elicitations with subject-matter experts informed the development of these conceptual capabilities and identified key challenges in wildfire logistics operations. Furthermore, the potential of these capabilities could extend beyond aerial logistics support, encompassing aerial suppression, observation, and emergency extraction support for wildland firefighting. An approach to development, implementation and validation of the novel capabilities is described.
Wildfire logistics↗
Advancing Wildland Fire Response with NASA’s Second Shift Capabilities
Novel “Second-Shift” capabilities—leveraging Uncrewed Aerial Systems and Optionally Piloted Vehicles—are identified that could extend wildfire aerial logistics support into night or low-visibility conditions. Expert elicitations with subject-matter experts informed the development of these conceptual capabilities and identified key challenges in wildfire logistics operations. Furthermore, the potential of these capabilities could extend beyond aerial logistics support, encompassing aerial suppression, observation, and emergency extraction support for wildland firefighting. An approach to development, implementation and validation of the novel capabilities is described.
Wildfire logistics↗
Methods for R&D Portfolio Analysis and Evaluation (Workshop Report)
The Workshop on Methods for R&D Portfolio Analysis and Evaluation convened on 17–18 July 2019 at the National Renewable Energy Laboratory in Golden, Colorado, and examined strengths and weaknesses of the various methodologies applicable to R&D portfolio modeling, analysis, and decision support, given pragmatic constraints such as data availability, uncertainties in estimating the impact of R&D spending, and practical operational overheads. Participants employed their deep expertise in approaches such as stochastic optimization, real options, Monte-Carlo analysis, Bayesian networks, decision theory, complex systems analysis, deep uncertainty, and technology-evolution modeling to critique the initial example models developed by the project’s core team and to conduct thought experiments grounded in real-life technology models, progress data, expert elicitation, and portfolio information. This engagement of participants’ methodological expertise with the practical requirements of real-life portfolio decision support yielded ideas for improved approaches, alternative methodological hypotheses, and hybridization of methodologies that are well-grounded theoretically, computationally sound, and realistically executable given data availability and other practical constraints.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗