Search NASA⌕ Search

SEARCH · Search NASA

Results for “Expert opinion elicitation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

Final Technical Report on Quantifying Dependability Attributes of Software Based Safety Critical Instrumentation and Control Systems in Nuclear Power Plants

With the current transition from analog to digital instrumentation and control systems in nuclear power plants, the number and variety of software-based systems have significantly increased. The sophisticated nature and increasing complexity of software raises trust in these systems as a significant challenge. The trust placed in a software system is typically termed software dependability. Software dependability analysis faces uncommon challenges since software systems’ characteristics differ from those of hardware systems. The lack of systematic science-based methods for quantifying the dependability attributes in software-based instrumentation as well as control systems in safety critical applications has proved itself to be a significant inhibitor to the expanded use of modern digital technology in the nuclear industry. Dependability refers to the ability of a system to deliver a service that can be trusted. Dependability is commonly considered as a general concept that encompasses different attributes, e.g., reliability, safety, security, availability and maintainability. Dependability research has progressed significantly over the last few decades. For example, various assessment models and/or design approaches have been proposed for software reliability, software availability and software maintainability. Advances have also been made to integrate multiple dependability attributes, e.g., integrating security with other dependability attributes, measuring availability and maintainability, modeling reliability and availability, quantifying reliability and security, exploring the dependencies between security and safety and developing integrated analysis models. However, there is still a lack of understanding of the dependencies between various dependability attributes as a whole and of how such dependencies are formed. To address the need for quantification and give a more objective basis to the review process -- therefore reducing regulatory uncertainty -- measures and methods are needed to assess dependability attributes early on, as well as throughout the life-cycle process of software development. In this research, extensive expert opinion elicitation is used to identify the measures and methods for assessing software dependability. Semi-structured questionnaires were designed to elicit expert knowledge. A new notation system, Causal Mechanism Graphing, was developed to extract and represent such knowledge. The Causal Mechanism Graphs were merged, thus, obtaining the consensus knowledge shared by the domain experts. In this report, we focus on how software contributes to dependability. However, software dependability is not discussed separately from the context of systems or socio-technical systems. Specifically, this report focuses on software dependability, reliability, safety, security, availability, and maintainability. Our research was conducted in the sequence of stages found below. Each stage is further examined in its corresponding chapter. Stage 1 (Chapter 2): Elicitation of causal maps describing the dependencies between dependability attributes. These causal maps were constructed using expert opinion elicitation. This chapter describes the expert opinion elicitation process, the questionnaire design, the causal map construction method and the causal maps obtained. Stage 2 (Chapter 3): Elicitation of the causal map describing the occurrence of the event of interest for each dependability attribute. The causal mechanisms for the “event of interest” were extracted for each of the software dependability attributes. The “event of interest” for a dependability attribute is generally considered to be the “attribute failure”, e.g. security failure. The extraction was based on the analysis of expert elicitation results obtained in Stage 1. Stage 3 (Chapter 4): Identification of relevant measurements. Measures for the “events of interest” and their causal mechanisms were obtained from expert opinion elicitation for each of the software dependability attributes. The measures extracted are presented in this chapter. Stage 4 (Chapter 5): Assessment of the coverage of the causal maps via measures. Coverage was assessed to determine whether the measures obtained were sufficient to quantify software dependability, and what measures are further required. Stage 5 (Chapter 6): Identification of “missing” measures and measurement approaches for concepts not covered. New measures, for concepts that had not been covered sufficiently as determined in Stage 4, were identified using supplementary expert opinion elicitation as well as literature reviews. Stage 6 (Chapter 7): Building of a detailed quantification model based on the causal maps and measurements obtained. Ability to derive such a quantification model shows that the causal models and measurements derived from the previous stages (Stage 1 to Stage 5) can form the technical basis for developing dependability quantification models. Scope restrictions have led us to prioritize this demonstration effort. The demonstration was focused on a critical system, i.e. the reactor protection system. For this system, a ranking of the software dependability attributes by nuclear stakeholders was developed. As expected for this application, the stakeholder ranking identified safety as the most critical attribute to be quantified. A safety quantification model limited to the requirements phase of development was built. Two case studies were conducted for verification. A preliminary control gate for software safety for the requirements stage was proposed and applied to the first case study. The control gate allows a cost effective selection of the duration of the requirements phase.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Advanced Sodium Fast Reactor Accident Source Terms: Research Needs

An expert opinion elicitation has been used to evaluate phenomena that could affect releases of radionuclides during accidents at sodium-cooled fast reactors. The intent was to identify research needed to develop a mechanistic model of radionuclide release for licensing and risk assessment purposes. Experts from the USA, France, the European Union, and Japan identified phenomena that could affect the release of radionuclides under hypothesized accident conditions. They qualitatively evaluated the importance of these phenomena and the need for additional experimental research. The experts identified seven phenomena that are of high importance and have a high need for additional experimental research: high temperature release of radionuclides from fuel during an energetic event, energetic interactions between molten reactor fuel and sodium coolant and associated transfer of radionuclides from the fuel to the coolant, entrainment of fuel and sodium bond material during the depressurization of a fuel rod with breached cladding, rates of radionuclide leaching from fuel by liquid sodium, surface enrichment of sodium pools by dissolved and suspended radionuclides, thermal decomposition of sodium iodide in the containment atmosphere, and reactions of iodine species in the containment to form volatile organic iodides. Other issues of high importance were identified that might merit further research as development of the mechanistic model of radionuclide release progressed.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Sodium Fast Reactor Safety and Licensing Research Plan – Volume II

Expert panels comprised of subject matter experts identified at the U.S. National Laboratories (SNL, ANL, INL, ORNL, LBL, and BNL), universities (University of Wisconsin and Ohio State University), international agencies (IRSN, CEA, JAEA, KAERI, and JRC-IE) and private consultation companies (Radiation Effects Consulting) were assembled to perform a gap analysis for sodium fast reactor licensing. Expert-opinion elicitation was performed to qualitatively assess the current state of sodium fast reactor technologies. Five independent gap analyses were performed resulting in the following topical reports: (1) Accident Initiators and Sequences (i.e., Initiators/Sequences Technology Gap Analysis), (2) Sodium Technology Phenomena (i.e., Advanced Burner Reactor Sodium Technology Gap Analysis), (3) Fuels and Materials (i.e., Sodium Fast Reactor Fuels and Materials: Research Needs), (4) Source Term Characterization (i.e., Advanced Sodium Fast Reactor Accident Source Terms: Research Needs), and (5) Computer Codes and Models (i.e., Sodium Fast Reactor Gaps Analysis of Computer Codes and Models for Accident Analysis and Reactor Safety). Volume II of the Sodium Research Plan consolidates the five gap analysis reports produced by each expert panel, wherein the importance of the identified phenomena and necessities of further experimental research and code development were addressed. The findings from these five reports comprised the basis for the analysis in Sodium Fast Reactor Research Plan Volume I.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

The Appropriateness of Renal Angioplasty. The ANPARIA Software: A Multidisciplinary Expert Panel Approach

Percutaneous transluminal renal angioplasty (PTRA) is an invasive technique that is costly and involves the risk of complications and renal failure. The ability of PTRA to reduce the administration of antihypertensive drugs has been demonstrated. A potentially greater benefit, which nevertheless remains to be proven, is the deferral of the need for chronic dialysis. The aim of the study (ANPARIA) was to assess the appropriateness of PTRA to impact on the evolution of renal function. A standardized expert panel method was used to assess the appropriateness of medical treatment alone or medical treatment with revascularization in various clinical situations. The choice of revascularization by either PTRA or surgery was examined for each clinical situation. Analysis was based on a detailed literature review and on systematically elicited expert opinion, which were obtained during a two-round modified Delphi process. The study provides detailed responses on the appropriateness of PTRA for 1848 distinct clinical scenarios. Depending on the major clinical presentation, appropriateness of revascularization varied from 32% to 75% for individual scenarios (overal 48%). Uncertainty as to revascularization was 41% overall. When revascularization was appropriate, PTRA was favored over surgery in 94% of the scenarios, except in certain cases of aortic atheroma where sugery was the preferred choice. Kidney size >7 cm, absence of coexisting disease, acute renal failure, a high degree of stenosis ({>=}70%), and absence of multiple arteries were identified as predictive variables of favorable appropriateness ratings. Situations such as cardiac failure with pulmonary edema or acute thrombosis of the renal artery were defined as indications for PTRA. This study identified clinical situations in which PTRA or surgery are appropriate for renal artery disease. We built a decision tree which can be used via Internet: the ANPARIA software (http://www.chu-clermontferrand.fr/anpariahttp://www.chu-clermontferrand.fr/anparia/). In numerous clinical situations uncertainty remains as to whether PTRA prevents deterioration of renal function.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Sodium fast reactor gaps analysis of computer codes and models for accident analysis and reactor safety.

This report summarizes the results of an expert-opinion elicitation activity designed to qualitatively assess the status and capabilities of currently available computer codes and models for accident analysis and reactor safety calculations of advanced sodium fast reactors, and identify important gaps. The twelve-member panel consisted of representatives from five U.S. National Laboratories (SNL, ANL, INL, ORNL, and BNL), the University of Wisconsin, the KAERI, the JAEA, and the CEA. The major portion of this elicitation activity occurred during a two-day meeting held on Aug. 10-11, 2010 at Argonne National Laboratory. There were two primary objectives of this work: (1) Identify computer codes currently available for SFR accident analysis and reactor safety calculations; and (2) Assess the status and capability of current US computer codes to adequately model the required accident scenarios and associated phenomena, and identify important gaps. During the review, panel members identified over 60 computer codes that are currently available in the international community to perform different aspects of SFR safety analysis for various event scenarios and accident categories. A brief description of each of these codes together with references (when available) is provided. An adaptation of the Predictive Capability Maturity Model (PCMM) for computational modeling and simulation is described for use in this work. The panel's assessment of the available US codes is presented in the form of nine tables, organized into groups of three for each of three risk categories considered: anticipated operational occurrences (AOOs), design basis accidents (DBA), and beyond design basis accidents (BDBA). A set of summary conclusions are drawn from the results obtained. At the highest level, the panel judged that current US code capabilities are adequate for licensing given reasonable margins, but expressed concern that US code development activities had stagnated and that the experienced user-base and the experimental validation base was decaying away quickly.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Causal Mechanism Graph – A new notation for capturing cause-effect knowledge in software dependability

Understanding cause-effect relations between concepts in software dependability engineering is fundamental to various research or industrial activities. Cognitive maps are traditionally used to elicit and represent such knowledge; however they seem incapable of accurately representing complex causal mechanisms in dependability engineering. Here, this paper proposes a new notation called Causal Mechanism Graph (CMG) to elicit and represent the cause-effect domain knowledge embedded in experts’ minds or described in the literature. CMG contains a new set of symbols elicited from domain experts to capture the recurring interaction mechanisms between multiple concepts in software dependability engineering. Furthermore, compared to major existing graphic methods, CMG is particularly robust and suitable for mental knowledge elicitation: it allows one to represent the full range of cause-effect knowledge, accurately or fuzzily as one sees fit depending on the depth of knowledge he/she has. This feature combined with excellent reliability and validity poses CMG as a promising method that has the potential to be used in various areas, such as software dependability requirement elicitation, software dependability assessment and dependability risk control.

42 ENGINEERING↗

Identifying Adversarial Cyber-Activity in Operational Technology Environments Using Bayesian Networks

Critical infrastructure and other operational technology (OT) environments face increasing cybersecurity risks from adversarial behavior. This paper describes the development of a risk model using a Bayesian network to enhance the comprehension of observable cyber events caused by malicious activity in OT environments. The core of the Bayesian network is a process model that describes the stages of adversary behavior. The remainder of the model is based on the MITRE ATT&CK® for Industrial Control Systems (ICS) taxonomy, which includes tactics and techniques that may be used by the adversary. The observables provide evidence for adversary behavior through the intermediary technique and tactic nodes. One challenge in constructing this model is a lack of open-source data from cyber-attacks on OT systems. This paper discusses learning from limited data, the elicitation of expert opinion to construct the conditional probability tables when data is scarce, and the refinement of the most difficult conditional probabilities tables using several forms of sensitivity analyses. Finally, the Bayesian network is demonstrated using two historical case studies: the DarkSide ransomware attack on the Colonial Pipeline and the destructive cyberattack targeting the ThyssenKrupp blast furnace. Index Terms—Cybersecurity, industrial control systems, operational technology

97 - MATHEMATICS AND COMPUTING↗

Molten Salt Reactor Fundamental Safety Function PIRT

The phenomenon identification and ranking table (PIRT) process was employed to evaluate the current capability to assess the ability of liquid-salt–fueled molten salt reactors to achieve their fundamental safety functions (FSFs). The PIRT process provides a structured mechanism to elicit and document expert opinions on the most important phenomena and the corresponding level of knowledge with regard to achieving the FSFs. The PIRT panel included reactor developers, accident progression evaluation tool developers, US Department of Energy (DOE) national laboratory technical staff, university researchers, and US Nuclear Regulatory Commission (NRC) staff. The information-gathering process concentrated on phenomena related to accidents in which the fuel salt—including any in the cover gas—has been released from the first barrier layer. This information-gathering process included identifying all potential accidents that could result in the release of a substantial amount of radionuclides that could breach or bypass the first barrier. The PIRT did not identify any previously unrecognized systemic vulnerabilities. However, the elicitation process identified several areas with low levels of knowledge and significant potential impact on accident progression. These areas either require additional research to improve the state of knowledge or additional design conservatism to accommodate the remaining unknowns.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Parameterizing Bayesian network Representations of Social-Behavioral Models by Expert Elicitation

Bayesian networks provide a general framework with which to model many natural phenomena. The mathematical nature of Bayesian networks enables a plethora of model validation and calibration techniques: e.g parameter estimation, goodness of fit tests, and diagnostic checking of the model assumptions. However, they are not free of shortcomings. Parameter estimation from relevant extant data is a common approach to calibrating the model parameters. In practice it is not uncommon to find oneself lacking adequate data to reliably estimate all model parameters. In this paper we present the early development of a novel application of conjoint analysis as a method for eliciting and modeling expert opinions and using the results in a methodology for calibrating the parameters of a Bayesian network.

99 GENERAL AND MISCELLANEOUS↗

Cost estimation: An expert-opinion approach

A methodology is outlined which can be used to estimate the costs of research and development projects. The approach uses the Delphi technique a method developed by the Rand Corporation for systematically eliciting and evaluating group judgments in an objective manner. The use of the Delphi allows for the integration of expert opinion into the cost-estimating process in a consistent and rigorous fashion. This approach can also signal potential cost-problem areas. This result can be a useful tool in planning additional cost analysis or in estimating contingency funds. A Monte Carlo approach is also examined.

Buffalano, C.↗

Expert Elicitation Method Selection Process and Method Comparison

Research on integrative modeling has gained considerable attention in recent years and expert opinion has been increasingly recognized as an important data source and modeling contributor. However, little research has systematically compared and evaluated expert elicitation methods in terms of their ability to link to computational models that capture human behavior and social phenomena. In this paper, we describe a decision-making process we used for evaluating and selecting a task specific elicitation method within the framework of integrative computational social-behavioral modeling. From the existing literature, we identified the characteristics of problems that each candidate method is well suited to address. A small-scale expert elicitation was also conducted to evaluate the comparative strength and weaknesses of the methods against a number of consensus-based decision criteria. By developing a set of explicit method evaluation criteria and a description characterizing decision problems for the candidate methods, we seek to gain a better understanding of the feasibility and cost-effectiveness of integrating elicitation methods with computational modeling techniques. This serves an important first step toward expanding our research effort and trajectory toward greater interdisciplinary modeling research of human behavior.

99 GENERAL AND MISCELLANEOUS↗

Uncertainty and Risk Assessment in the Design Process for Wind

This report summarizes the concepts and opinions that emerged from an initial study on the subject of uncertainty in wind design that included expert elicitation during a workshop held at the National Wind Technology Center at the National Renewable Energy Laboratory July 12-13, 2016. In this paper, five major categories of uncertainties are identified. The first category is associated with direct impacts on turbine loads, (i.e., the inflow including extreme events, aero-hydro-servo-elastic response, soil-structure inter- action, and load extrapolation). The second category encompasses material behavior and strength. Site suitability and due-diligence aspects pertain to the third category. Calibration of partial safety factors and optimal reliability levels make up the fourth one. And last but not least, is the category associated with uncertainties in computational modeling. The main sections of this paper follow this organization.

17 WIND ENERGY↗

Application of Analytical Hierarchy Process for Narrowing Down Nep Candidate Reactor Designs

Nuclear electric propulsion (NEP)-powered vehicles have been contemplated for human Mars missions. The nuclear power system contemplates using a high temperature light-weight nuclear reactor for the production of electrical power in the range of 2-5 MWe with a 3-10 year service life. A myriad of technology options exist for achieving these mission objectives. The analytic hierarchy process (AHP), a multi-attribute decision method, is being used to narrow down the candidate designs. The AHP is a structured decision process that fuses model-supplied quantitative data with subjective assessments to facilitate decisions that involve multiple competing criteria. Proven end-to-end nuclear design and systems analysis tools will be used to provide quantitative performance data such as end-to-end system reliability, system robustness to recover from off-normal conditions, system specific weight (α in kg/kWe), and the ability to meet the service life-time and power level requirements. Three high assay low enriched uranium (HALEU) reactor concepts, namely, gas-, heat pipe- and pumped liquid-cooled nuclear cores coupled to a He/Xe gas or supercritical CO2- Brayton power conversion system are being modeled using the AHP to understand the trade-offs associated with these design combinations. In addition to the reactor and power conversion options, there are a myriad of additional components and subsystems – e.g., radiators, heat exchangers, and recuperators – that also figure into the evaluation process. The choice of the right overall system is a multidimensional problem that has to include not only quantitative data, but also the so called “external factors”, examples of which include the component and subsystem Technology Readiness Levels (TRLs), the associated Advancement Degrees of Difficulty (AD2), the cost and schedule required to achieve a technical maturity consistent with mission infusion, alignment with the priorities of NASA’s Space Nuclear Propulsion program, and alignment with other ongoing government and commercial investments in micro reactors. Focused expert elicitations form the basis for qualitative data set. As a final step, decision-makers individually express their opinions regarding the relative importance of the criteria and preferences among the alternatives through pairwise comparisons. The paper will describe the AHP approach, progress to-date applying it to the NEP human Mars mission problem, and preliminary results. Use of AHP provides sufficient flexibility for incorporating industry input at different stages as the technologies evolve through additional research and development. It is expected that the decision process will be ongoing and expanded to examine additional options and technology choices, culminating in a defensible set of candidate reactor concepts that will form the basis for developing a multi-year NEP technology maturation strategy. .

Dasari V Rao↗

Development of an Expert Judgement Elicitation and Calibration Methodology for Risk Analysis in Conceptual Vehicle Design

A comprehensive expert-judgment elicitation methodology to quantify input parameter uncertainty and analysis tool uncertainty in a conceptual launch vehicle design analysis has been developed. The ten-phase methodology seeks to obtain expert judgment opinion for quantifying uncertainties as a probability distribution so that multidisciplinary risk analysis studies can be performed. The calibration and aggregation techniques presented as part of the methodology are aimed at improving individual expert estimates, and provide an approach to aggregate multiple expert judgments into a single probability distribution. The purpose of this report is to document the methodology development and its validation through application to a reference aerospace vehicle. A detailed summary of the application exercise, including calibration and aggregation results is presented. A discussion of possible future steps in this research area is given.

Unal, Resit↗

A Step-Wise Approach to Elicit Triangular Distributions

Adapt/combine known methods to demonstrate an expert judgment elicitation process that: 1.Models expert's inputs as a triangular distribution, 2.Incorporates techniques to account for expert bias and 3.Is structured in a way to help justify expert's inputs. This paper will show one way of "extracting" expert opinion for estimating purposes. Nevertheless, as with most subjective methods, there are many ways to do this.

Greenberg, Marc W.↗

Community participation in Health Impact Assessment. A scoping review of the literature

Currently, the engagement of local communities in Health Impact Assessment is becoming more and more important. A scoping review was performed to take stock of visions, methods and experiences in this field. A combined Scopus and Medline search yielded 100 articles in scientific journals. The final selection consisted of 43 papers, including case studies, evaluation studies, reviews, and opinion papers. After analysis, consultation of four experts was performed to check preliminary study outcomes. A grey literature web search was performed to check and complement the results. Results show that community participation is generally considered a core element in HIA. Views as expressed in the papers concern, firstly, the need for and value of local knowledge, secondly, the adherence to or application of democratic values and, thirdly, empowerment of communities. Three categories of methods are used in relation to community participation, often in combination: methods to facilitate knowledge elicitation, to ensure the inclusion of communities in the HIA process, and to build community capacity to participate in policy development. However, the theoretical or practical underpinning of the choice for specific methods is mostly not presented. The experiences described in the papers mainly focus on the access to local knowledge and its usability as a source of evidence in the HIA process. Described effects of community participation are (improved) relations between communities and local agencies, policy makers and professionals and the empowerment of community members. Although these effects are ascribed to community participation, many papers do not provide support for this conclusion beyond the retrospective perception of participants. Expert consultation and additional analysis of the grey literature supported the results derived from the scientific literature and provided more in-depth knowledge. In the grey literature theoretical frameworks, methods and tools for community participation in HIA were more extensively reported as compared to the scientific literature. We conclude that the visions, methods and experiences concerning community participation show that a participative approach may contribute to better, context specific knowledge. It appears that participative HIA has health promotion potential as it helps develop responsive policies. To accomplish this, HIA should, firstly, be better embedded in broader health promotion programmes. Secondly, the methods and approaches for community participation applied in HIA should be theory-informed and well described. The grey literature offers entry points. Finally, more robust and systematic evaluation and research is needed to assess the impact of HIAs on communities and policies.

54 ENVIRONMENTAL SCIENCES↗