Search NASA⌕ Search

SEARCH · Search NASA

Results for “Expert Judgement”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2004

Development of an Expert Judgement Elicitation and Calibration Methodology for Risk Analysis in Conceptual Vehicle Design

A comprehensive expert-judgment elicitation methodology to quantify input parameter uncertainty and analysis tool uncertainty in a conceptual launch vehicle design analysis has been developed. The ten-phase methodology seeks to obtain expert judgment opinion for quantifying uncertainties as a probability distribution so that multidisciplinary risk analysis studies can be performed. The calibration and aggregation techniques presented as part of the methodology are aimed at improving individual expert estimates, and provide an approach to aggregate multiple expert judgments into a single probability distribution. The purpose of this report is to document the methodology development and its validation through application to a reference aerospace vehicle. A detailed summary of the application exercise, including calibration and aggregation results is presented. A discussion of possible future steps in this research area is given.

Unal, Resit↗
NASA NTRS2020

Burning Embers: Towards More Transparent and Robust Climate-change Risk Assessments

The Intergovernmental Panel on Climate Change (IPCC) reports provide policy-relevant insights about climate impacts, vulnerabilities and adaptation through a process of peer-reviewed literature assessments underpinned by expert judgement. An iconic output from these assessments is the burning embers diagram, first used in the Third Assessment Report to visualize reasons for concern, which aggregate climate-change-related impacts and risks to various systems and sectors. These burning embers use colour transitions to show changes in the assessed level of risk to humans and ecosystems as a function of global mean temperature. In this Review, we outline the history and evolution of the burning embers and associated reasons for concern framework, focusing on the methodological approaches and advances. While the assessment framework and figure design have been broadly retained over time, refinements in methodology have occurred, including the consideration of different risks, use of confidence statements, more formalized protocols and standardized metrics. Comparison across reports reveals that the risk level at a given temperature has generally increased with each assessment cycle, reflecting accumulating scientific evidence. For future assessments, an explicit, transparent and systematic process of expert elicitation is needed to enhance comparability, quality and credibility of burning embers.

burning embers diagram↗
DOE OSTI2025

Laser Confocal Microscopy Uncertainty Quantification Study

At Los Alamos National Laboratory (LANL), the Storage Safety and Engineering (SSE) team completes annual surveillance on a subset of in-use interim nuclear material storage containers in fulfilment of requirements outlined in DOE Manual M 441.1-1. The containers are selected through several methods, such as subject matter expert judgement, random selection, and trending items. Following these selections, the SSE team has the capacity to complete surveillance on 15-20 containers each fiscal year, composed of a combination of SAVY-4000 and Hagan storage containers. Through previous work, the stainless-steel components of the containers have been identified as life limiting components, with an emphasis on the thin-walled bodies. The team is focused on understanding the extent of general and pitting corrosion, due to observations of extensive corrosion from stored contents and bag-out-bag degradation. Quantifying corrosion effects on the thin-walled stainless steel container bodies, and understanding potential impacts to the respective design release rates and design qualification release rates is paramount to the team. To date, destructive examination (DE) has proven to be the most insightful method for developing an understanding on the extent of corrosion on used containers. To standardize this process, the SSE team developed a destructive examination guide for analyzing stainless steel components of the containers. Corroded containers of interest are identified during surveillance activities and set aside for sectioning and characterization. Following sectioning, a major step in the DE workflow is the utilization of laser confocal microscopy for scanning corroded samples of interest and extracting data on pits, such as count, depth, and equivalent diameter. Adhering to the techniques outlined in the DE guide, analysis has been completed on two Hagans and one SAVY-4000 container, with the maximum pit depth recorded as 139.1 ± 22.82 μm on a 17.5 year old Hagan. The findings from the completed destructive examinations will be utilized to support lifetime extension efforts of the SAVY-4000 as the team can better estimate corrosion rates and effects over time based on stored contents and age. Due to the implications of observing extreme pit depths that approach the nominal container body thickness of .0299 inches (0.759 mm) or minimum container thickness of 0.236” (0.6 mm), high confidence in the LCM measurements is desired. Through testing outlined in, it was concluded that the total error ascribed to the 20x objective when conducting large image mapping on the Keyence VK-X3050 laser confocal microscope (LCM) relative to a 50x objective (reference) is 16.4% (± 8.73%). For shallow features on the order of pristine SAVY surface defects (i.e. 5 μm), this uncertainty is appropriate. However, this conservative estimate of total error poses a fundamental concern for pit depths that approach the thickness of the measured samples. That is, with the measurement uncertainty currently employed on all measurements, the LCM would be unable to resolve if a pit with a depth of 515 μm is through wall. Standard step height samples were procured and used in the present study to assess the resolution and repeatability of height measurements. Understanding the resolution and repeatability of height measurements was the first focus of the team as it relates directly to pit depth, which is of primary concern. Calibration gratings were procured to evaluate the resolution and repeatability of measurements in the X and Y axes of the LCM stage. The results of the depth uncertainty study were conducted first and presented in the subsequent sections. The planar uncertainty study is appended to the depth study with conclusions from both summarized at the end of the report.

36 MATERIALS SCIENCE↗
NASA NTRS2002

A Life-Cycle Cost Estimating Methodology for NASA-Developed Air Traffic Control Decision Support Tools

This paper describes the development of a life-cycle cost (LCC) estimating methodology for air traffic control Decision Support Tools (DSTs) under development by the National Aeronautics and Space Administration (NASA), using a combination of parametric, analogy, and expert opinion methods. There is no one standard methodology and technique that is used by NASA or by the Federal Aviation Administration (FAA) for LCC estimation of prospective Decision Support Tools. Some of the frequently used methodologies include bottom-up, analogy, top-down, parametric, expert judgement, and Parkinson's Law. The developed LCC estimating methodology can be visualized as a three-dimensional matrix where the three axes represent coverage, estimation, and timing. This paper focuses on the three characteristics of this methodology that correspond to the three axes.

Wang, Jianzhong Jay↗
NASA NTRS2019

Tropospheric Ozone Assessment Report: Tropospheric Ozone From 1877 to 2016, Observed Levels, Trends and Uncertainties

From the earliest observations of ozone in the lower atmosphere in the 19th century, both measurement methods and the portion of the globe observed have evolved and changed. These methods have different uncertainties and biases, and the data records differ with respect to coverage (space and time), information content, and representativeness. In this study, various ozone measurement methods and ozone datasets are reviewed and selected for inclusion in the historical record of background ozone levels, based on relationship of the measurement technique to the modern UV absorption standard, absence of interfering pollutants, representativeness of the well-mixed boundary layer and expert judgement of their credibility. There are significant uncertainties with the 19th and early 20th-century measurements related to interference of other gases. Spectroscopic methods applied before 1960 have likely underestimated ozone by as much as 11% at the surface and by about 24% in the free troposphere, due to the use of differing ozone absorption coefficients. There is no unambiguous evidence in the measurement record back to 1896 that typical mid-latitude background surface ozone values were below about 20 nmol mol–1, but there is robust evidence for increases in the temperate and polar regions of the northern hemisphere of 30–70%, with large uncertainty, between the period of historic observations, 1896–1975, and the modern period (1990–2014). Independent historical observations from balloons and aircraft indicate similar changes in the free troposphere. Changes in the southern hemisphere are much less. Regional representativeness of the available observations remains a potential source of large errors, which are difficult to quantify. The great majority of validation and intercomparison studies of free tropospheric ozone measurement methods use ECC ozonesondes as reference. Compared to UV-absorption measurements they show a modest (~1–5% ±5%) high bias in the troposphere, but no evidence of a change with time. Umkehr, lidar, and FTIR methods all show modest low biases relative to ECCs, and so, using ECC sondes as a transfer standard, all appear to agree to within one standard deviation with the modern UV-absorption standard. Other sonde types show an increase of 5–20% in sensitivity to tropospheric ozone from 1970–1995. Biases and standard deviations of satellite retrieval comparisons are often 2–3 times larger than those of other free tropospheric measurements. The lack of information on temporal changes of bias for satellite measurements of tropospheric ozone is an area of concern for long-term trend studies.

UV absorption standard↗
DOE OSTI2024

Investigation of Correlation Methods for Use in Criticality Safety

Although their adoption by practitioners has been limited, the introduction of similarity indices in criticality safety was a major step forward in reducing the reliance on expert judgement in discerning applicable experiments for the validation of new appliations in criticality safety analyses. Similarity indices have been successfully employed in bias trending and data assimilation techniques, but it is often unclear which acceptance criteria should be used. In their 2004 paper, Broadhead et al. specify the most widely used similarity parameter, ck, as an acceptance cutoff at 0.9. (Broadhead et al., ”Sensitivity and Uncertainty-Based Criticality Safety Validation Techniques,” Nucl. Sci. Eng. 146, 340–366, 2004). Experiments with a ck < 0.9 are often not considered applicable for code validation. This heuristic is based on quantitative studies and engineering judgement, but in some cases, experiments with ck < 0.9 can be used to accurately estimate computational bias. This suggests that further analysis is needed to determine what components of ck are driving applicability and accuracy in bias estimation. For cases in which applicable experiments may not be available (as is the case with UF6 transport canisters), understanding what distinguishes experiments in providing adequate bias estimates aside from just the similarity index is particularly necessary. To further the goal to better interpret ck values, several visualization tools were developed to assist in the investigation of which components of ck are driving applicability.

ck↗
NASA NTRSDate not supplied

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.

Expert Judgement↗
DOE OSTI2024

Inferring Reliability Model Parameters from Expert Opinion

Here, we propose a method for constructing bathtub models of reliability from opinion. The method is intended for reliability studies early in the design and prototyping of a new system, before data concerning reliability has become available. A stylized bathtub curve is presented for soliciting best engineering judgement from technical experts. By pooling these stylized curves, we produce data that can be used to infer parameters for a piece-wise Weibull model of reliability. A numerical example demonstrates the practicality of the method while also highlighting potential pitfalls when working with subjective data.

42 ENGINEERING↗
NASA NTRS1990

General method of pattern classification using the two-domain theory

Human beings judge patterns (such as images) by complex mental processes, some of which may not be known, while computing machines extract features. By representing the human judgements with simple measurements and reducing them and the machine extracted features to a common metric space and fitting them by regression, the judgements of human experts rendered on a sample of patterns may be imposed on a pattern population to provide automatic classification.

Rorvig, Mark E.↗
NASA NTRS1993

General method of pattern classification using the two-domain theory

Human beings judge patterns (such as images) by complex mental processes, some of which may not be known, while computing machines extract features. By representing the human judgements with simple measurements and reducing them and the machine extracted features to a common metric space and fitting them by regression, the judgements of human experts rendered on a sample of patterns may be imposed on a pattern population to provide automatic classification.

Rorvig, Mark E.↗
NASA NTRSDate not supplied

Application of Analytical Hierarchy Process for Narrowing Down Nep Candidate Reactor Designs

NASA is currently studying the possibility of human exploration of Mars in the late 2030s timeframe using a hybrid nuclear electric propulsion (NEP)/chemical propulsion system. Present analyses indicate such a mission could require a reactor power system in the range of 2 to 6 MWe operating for approximately 3 years. At present, the Technology Readiness Levels (TRL) of key NEP technologies are still low, implying that significant technology development activities must be undertaken prior to making informed design selections for an integrated propulsion system. NASA’s Space Nuclear Propulsion (SNP) project has recently undertaken an in-depth effort to identify NEP candidate technologies and determine the required developmental work to mature these technologies to the point where they could realistically support the design and assembly of a full NEP system. A version of the Analytical Hierarchy Process is being employed to narrow the technology candidates for SNP investment planning. This multi-attribute decision-making process relies on quantified technical inputs and the judgement of Subject Matter Experts (SMEs) to evaluate technology options against key technical and programmatic Figures-of-Merit (FOM) at the subsystem and optimized system level. The process and an example evaluation of nuclear fuel/moderator combinations in a sodium heat pipe honeycomb-geometry reactor concept are described. The evaluation and analysis methodology described in this paper can be employed for pairwise comparisons between reactor concepts and technology choices as new data become available.

Dasari V Rao↗
NASA NTRS1989

Neural networks in support of manned space

Many lobbyists in Washington have argued that artificial intelligence (AI) is an alternative to manned space activity. In actuality, this is the opposite of the truth, especially as regards artificial neural networks (ANNs), that form of AI which has the greatest hope of mimicking human abilities in learning, ability to interface with sensors and actuators, flexibility and balanced judgement. ANNs and their relation to expert systems (the more traditional form of AI), and the limitations of both technologies are briefly reviewed. A Few highlights of recent work on ANNs, including an NSF-sponsored workshop on ANNs for control applications are given. Current thinking on ANNs for use in certain key areas (the National Aerospace Plane, teleoperation, the control of large structures, fault diagnostics, and docking) which may be crucial to the long term future of man in space is discussed.

Werbos, Paul J.↗
NASA NTRS2020

Orbit Selection for the Proposed Lynx Observatory Mission

The Advanced Concepts Office design team performed several analyses and trades in support of orbit selection for the proposed Lynx mission, an x-ray observatory being submitted to the Astro2020 Decadal Survey. Though the descriptions in this Technical Memorandum (TM) focus on the Lynx mission, the approach and process for selecting the final orbit is applicable to a variety of proposed science and exploration missions. To select the best orbit for the Lynx science, mission designers assembled a team of subsystem and discipline experts, in addition to mission analysts, to evaluate several candidate orbits. These discipline experts included members of the science and instrument team, power and avionics, thermal, propulsion, and environments. The goal was to clearly show the benefits and weaknesses of each orbit in the trade space and provide sound justification for the final selection. Discipline experts conducted trades and evaluated the results using a variety of methods including engineering judgement, rough estimates, and detailed calculations, and rolled the results into a final grade using a weighted grading method. The orbit options could then be ranked. The principal investigator (PI) for the mission, along with the science team, was given the task of final orbit selection. The result of the trades indicated that a halo orbit about the second Sun-Earth Lagrange point (SE-L2), similar to the planned orbit for the James Webb Space Telescope (JWST), was the best choice for the Lynx mission. Details of how the team arrived at this selection are below.

Hopkins, R. C.↗
NASA NTRSDate not supplied

Methods for Developing Successful Systems Engineers

Systems Engineering (SE) is a complex and challenging field that incorporates the knowledge of systems engineering processes, the ability to synthesize a wide-range of engineering disciplines, and the ability to lead a team of people to successfully accomplish the goals of a project. It requires hard technical skills and soft-skill leadership savvy. As a result, three main development needs are identified: 1) Knowledge of SE processes, the benefits of these processes to a project and their tailored application 2) Knowledge of a wide-range of engineering disciplines, how they interrelate in a system, and the development of sound technical judgement 3) Team leadership to direct and motivate a team of subsystem and discipline experts This paper describes the establishment of a comprehensive training and development program for Systems Engineers at NASA Ames Research Center that addresses in part each of these three areas from the perspective of the implementing manager. A variety of methods have been utilized including the establishment of a SE Community of Practice, a unique and innovative web tool, on-line videos, classroom training in NASA’s 17 Common Technical Processes, guidance on the tailored application of these processes, monthly technical talks, mentoring in both technical judgment and team leadership, and NASA’s Leadership Development Programs. While much of professional SE development must come through project experience, the approaches listed above can accelerate development. The diversity of skills required of Systems Engineering demands a multi-faceted approach to successfully train and develop this critical skill.

development↗
NASA NTRSDate not supplied

Methods for Developing Successful Systems Engineers

Systems Engineering (SE) is a complex and challenging field that incorporates the knowledge of systems engineering processes, the ability to synthesize a wide-range of engineering disciplines, and the ability to lead a team of people to successfully accomplish the goals of a project. It requires hard technical skills and soft-skill leadership savvy. As a result, three main development needs are identified: 1. Knowledge of SE processes, the benefits of these processes to a project and their tailored application 2. Knowledge of a wide-range of engineering disciplines, how they interrelate in a system, and the development of sound technical judgement 3. Team leadership to direct and motivate a team of subsystem and discipline experts This paper describes the establishment of a comprehensive training and development program for Systems Engineers at NASA Ames Research Center that addresses in part each of these three areas from the perspective of the implementing manager. A variety of methods have been utilized including the establishment of a SE Community of Practice, a unique and innovative web tool, on-line videos, classroom training in NASA’s 17 Common Technical Processes, guidance on the tailored application of these processes, monthly technical talks, mentoring in both technical judgment and team leadership, and NASA’s Leadership Development Programs. While much of professional SE development must come through project experience, the approaches listed above can accelerate development. The diversity of skills required of Systems Engineering demands a multi-faceted approach to successfully train and develop this critical skill.

development↗
NASA NTRS1972

Roles of the ground and flight crew in Apollo operations.

The functions performed by ground personnel and flight crews of Apollo missions are reviewed, covering vehicle systems operation and management, Saturn launch vehicle, command-service and lunar modules, lunar rover, communication relay unit and ground-commanded TV assembly. Details of the flight path, scientific experiments, photographic equipment and crew health monitoring are also discussed in terms of these functions. The coordinated combination of large numbers of experts and data processing equipment on the ground with the crew's capabilities to operate the spacecraft and exercise on-the-spot judgements is viewed as the backbone of the successful fulfillment of Apollo missions.

Wolfer, B. M.↗
NASA NTRS1991

Neural nets for aligning optical components in harsh environments: Beam smoothing spatial filter as an example

The goal is to develop an approach to automating the alignment and adjustment of optical measurement, visualization, inspection, and control systems. Classical controls, expert systems, and neural networks are three approaches to automating the alignment of an optical system. Neural networks were chosen for this project and the judgements that led to this decision are presented. Neural networks were used to automate the alignment of the ubiquitous laser-beam-smoothing spatial filter. The results and future plans of the project are presented.

Decker, Arthur J.↗
NASA NTRS2000

Multimodality Instrument for Tissue Characterization

A system with multimodality instrument for tissue identification includes a computer-controlled motor driven heuristic probe with a multisensory tip is discussed. For neurosurgical applications, the instrument is mounted on a stereotactic frame for the probe to penetrate the brain in a precisely controlled fashion. The resistance of the brain tissue being penetrated is continually monitored by a miniaturized strain gauge attached to the probe tip. Other modality sensors may be mounted near the probe tip to provide real-time tissue characterizations and the ability to detect the proximity of blood vessels, thus eliminating errors normally associated with registration of pre-operative scans, tissue swelling, elastic tissue deformation, human judgement, etc., and rendering surgical procedures safer, more accurate, and efficient. A neural network, program adaptively learns the information on resistance and other characteristic features of normal brain tissue during the surgery and provides near real-time modeling. A fuzzy logic interface to the neural network program incorporates expert medical knowledge in the learning process. Identification of abnormal brain tissue is determined by the detection of change and comparison with previously learned models of abnormal brain tissues. The operation of the instrument is controlled through a user friendly graphical interface. Patient data is presented in a 3D stereographics display. Acoustic feedback of selected information may optionally be provided. Upon detection of the close proximity to blood vessels or abnormal brain tissue, the computer-controlled motor immediately stops probe penetration.

Mah, Robert W.↗