Search NASA⌕ Search

SEARCH · Search NASA

Results for “Confidence”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Multimodal MSI in Conjunction with Broad Coverage Spatially Resolved MS2 Increases Confidence in Both Molecular Identification and Localization

One critical aspect of mass spectrometry imaging (MSI) is the need to con?dently identify detected analytes. While orthogonal tandem MS (e.g., LC-MS2) experiments from sample extracts can assist in annotating ions, the spatial information about these molecules is lost. Accordingly, this could cause mislead conclusions, especially in cases where isobaric species exhibit di?erent distributions within a sample. In this Technical Note, we employed a multimodal imaging approach, using matrix assisted laser desorption/ionization (MALDI)-MSI and liquid extraction surface analysis (LESA)-MS2I, to con?dently annotate and One critical aspect of mass spectrometry imaging (MSI) is the need to con?dently identify detected analytes. While orthogonal tandem MS (e.g., LC-MS2) experiments from sample extracts can assist in annotating ions, the spatial information about these molecules is lost. Accordingly, this could cause mislead conclusions, especially in cases where isobaric species exhibit di?erent distributions within a sample. In this Technical Note, we employed a multimodal imaging approach, using matrix assisted laser desorption/ionization (MALDI)-MSI and liquid extraction surface analysis (LESA)-MS2I, to con?dently annotate and localize a broad range of metabolites involved in a tripartite symbiosis system of moss, cyanobacteria, and fungus. We found that the combination of these two imaging modalities generated very congruent ion images, providing the link between highly accurate structural information onfered by LESA and high spatial resolution attainable by MALDI. These results demonstrate how this combined methodology could be very useful in di?erentiating metabolite routes in complex systems.

Velickovic, Dusan↗

Relating confidence to information uncertainty in qualitative reasoning

Qualitative reasoning makes use of qualitative assessments provided by subject matter experts to model factors such as security risk. Confidence in a result is important and useful when comparing competing security risk results. Quantifying the confidence in an evidential reasoning result must be consistent and based on the available information. A novel method is proposed to determine a qualitative measure of confidence in a qualitative reasoning result from the available information uncertainty in the result using membership values in the fuzzy sets of confidence. In this study information uncertainty is quantified through measures of non-specificity and conflict. Fuzzy values for confidence are established from information uncertainty values that lie between the measured minimum and maximum information uncertainty values. Measured values of information uncertainty in each result is used to obtain the confidence. The determined confidence values are used to compare competing scenarios and understand the influences on the desired result.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Confidence Assessment for Automatic Target Recognition

Confidence assessment is critical for effective automatic target recognition (ATR). Productive use and interpretation of ATR results by analysts or downstream algorithms requires not only algorithmic declarations of target presence and identity, but also algorithmic assessment of the certainty of those declarations in comparison to the certainties of alternative target-identity possibilities. Unfortunately, despite its importance, confidence assessment is an understudied, underdeveloped, and often-neglected function of ATR systems. This lack of regard stems not only from the difficulty of accurate algorithmic determination of target-identity certainty, but also from a general lack of understanding and careful consideration about what confidence should actually represent. We present a framework for confidence assessment that establishes a clear definition of confidence and provides a straightforward theoretical basis for its calculation. This framework is grounded in a hypothesis-theoretic consideration of ATR and it springs from from a handful of axiomatic principles concerning the nature and meaning of confidence in this context. This framework establishes a rigorous mathematical definition of confidence and it provides equations relating confidence to other information that is almost always provided by ATRs. We present an approach for computing confidence within this framework, using an advance process of ATR characterization followed by a simple computation at the time of ATR execution. We discuss practical difficulties with our approach, and we suggest methods for effective mitigation of these difficulties in implemented systems.

97 MATHEMATICS AND COMPUTING↗

An analysis of confidence limit calculations used in AAPM Task Group No. 119

Purpose: The report issued by AAPM Task Group No. 119 outlined a procedure for evaluating the effectiveness of IMRT commissioning. The procedure involves measuring gamma pass-rate indices for IMRT plans of standard phantoms and determining if the results fall within a confidence limit set by assuming normally distributed data. As stated in the TG report, the assumption of normally distributed gamma pass rates is a convenient approximation for commissioning purposes, but may not accurately describe the data. Here the authors attempt to better describe gamma pass-rate data by fitting it to different distributions. The authors then calculate updated confidence limits using those distributions and compare them to those derived using TG No. 119 method. Methods: Gamma pass-rate data from 111 head and neck patients are fitted using the TG No. 119 normal distribution, a truncated normal distribution, and a Weibull distribution. Confidence limits to 95% are calculated for each and compared. A more general analysis of the expected differences between the TG No. 119 method of determining confidence limits and a more time-consuming curve fitting method is performed. Results: The TG No. 119 standard normal distribution does not fit the measured data. However, due to the small range of measured data points, the inaccuracy of the fit has only a small effect on the final value of the confidence limits. The confidence limits for the 111 patient plans are within 0.1% of each other for all distributions. The maximum expected difference in confidence limits, calculated using TG No. 119's approximation and a truncated distribution, is 1.2%. Conclusions: A three-parameter Weibull probability distribution more accurately fits the clinical gamma index pass-rate data than the normal distribution adopted by TG No. 119. However, the sensitivity of the confidence limit on distribution fit is low outside of exceptional circumstances.

60 APPLIED LIFE SCIENCES↗

The effect of terrorism on public confidence : an exploratory study.

A primary goal of terrorism is to instill a sense of fear and vulnerability in a population and to erode confidence in government and law enforcement agencies to protect citizens against future attacks. In recognition of its importance, the Department of Homeland Security includes public confidence as one of the metrics it uses to assess the consequences of terrorist attacks. Hence, several factors--including a detailed understanding of the variations in public confidence among individuals, by type of terrorist event, and as a function of time--are critical to developing this metric. In this exploratory study, a questionnaire was designed, tested, and administered to small groups of individuals to measure public confidence in the ability of federal, state, and local governments and their public safety agencies to prevent acts of terrorism. Data were collected from the groups before and after they watched mock television news broadcasts portraying a smallpox attack, a series of suicide bomber attacks, a refinery bombing, and cyber intrusions on financial institutions that resulted in identity theft and financial losses. Our findings include the following: (a) the subjects can be classified into at least three distinct groups on the basis of their baseline outlook--optimistic, pessimistic, and unaffected; (b) the subjects make discriminations in their interpretations of an event on the basis of the nature of a terrorist attack, the time horizon, and its impact; (c) the recovery of confidence after a terrorist event has an incubation period and typically does not return to its initial level in the long-term; (d) the patterns of recovery of confidence differ between the optimists and the pessimists; and (e) individuals are able to associate a monetary value with a loss or gain in confidence, and the value associated with a loss is greater than the value associated with a gain. These findings illustrate the importance the public places in their confidence in government and law enforcement and also indicate that the level of importance is clearly of a magnitude on the order of other major terrorist event consequences, such as loss of human life and impacts to the economy.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Rats use memory confidence to guide decisions

Memory enables access to past experiences to guide future behavior. Humans can determine which memories to trust (high confidence) and which to doubt (low confidence). How memory retrieval, memory confidence, and memory-guided decisions are related, however, is not understood. In particular, how confidence in memories is used in decision making is unknown. We developed a spatial memory task in which rats were incentivized to gamble their time: betting more following a correct choice yielded greater reward. Rat behavior reflected memory confidence, with higher temporal bets following correct choices. We applied machine learning to identify a memory decision variable and built a generative model of memories evolving over time that accurately predicted both choices and confidence reports. Our results reveal in rats an ability thought to exist exclusively in primates and introduce a unified model of memory dynamics, retrieval, choice, and confidence.

60 APPLIED LIFE SCIENCES↗

Characterization of Strategies for Obtaining Confident Identifications in Bottom-Up Proteomics Measurements Using Hybrid FTMS instruments

Hybrid FTMS instruments, such as the LTQ-FTTM and LTQ-OrbitrapTM, are capable of generating fast duty cycle linear ion trap MS/MS data along with high resolution information without compromising the overall throughput of measurements. Combined with online LC separations, these instruments represent powerful and flexible tools for proteomics research. In the present work, we explore strategies for high throughput, high coverage proteomics measurements using hybrid FTMS instruments. Our accurate mass and time tag (AMT tag) strategy enables identification of thousands of peptides in a single LC-FTMS analysis by comparing accurate molecular mass and LC elution time information from the analysis to a reference database. An alternative strategy considered here employs linear ion trap (low resolution) MS/MS identifications generated by an appropriate search engine, such as SEQUEST; the high resolution precursor ion spectra were used to refine the MS/MS identifications, an approach termed Accurate Precursor Mass Filter (APMF). The APMF results can be additionally filtered using the LC elution time information from the AMT tag database, which constitutes a Precursor Mass and Time Filter (PMTF), the third approach implemented in this study. Both the APMF and the PMTF approaches are evaluated for coverage and confidence of peptide identifications and contrasted with the current AMT tag strategy. Two separate methodologies were used to reliably quantify identification confidence: a commonly used decoy database method and an alternative method based on the mass accuracy histogram. The two methodologies produced consistent results, confirming the validity of the identification confidence evaluations. Comparison of the three approaches has shown that the AMT tag data analysis approach may be preferential for studies giving a priority to the highest achievable coverage. The APMF approach by itself does not require AMT tag database and provides a moderate coverage combined with acceptable confidence values of ~99%.The PMTF approach yielded a significantly better peptide identification confidence, >99.9%, that essentially excluded any false peptide identifications. The results suggest that even with a perfect peptide ID (0% FDR in the peptide MS/MS database), the peak matching FDR is a function of the database size, so smaller high confidence databases are the goal. Thus a combined strategy can implement multi-pass APMF approach to generate high confidence AMT tag databases, which can be then validated using PMTF approach; the compact high quality databases will be used for subsequent high-throughput, high coverage AMT tag studies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Method for Generating Expert Derived Confidence Scores

We executed a pilot demonstration of a methodology for developing a new confidence metric to help operators calibrate their trust in ML event classifiers. This confidence metric was derived from domain expert judgment and was accompanied with a qualitative description describing the reason for each confidence rating. After learning the boundaries of an ML’s performance by studying a subset of events an SME rated his confidence in the ML’s ability to classify similar events and provided an explanation for his ratings. To demonstrate this methodology, we developed our expert driven confidence scores for the ML event classifier within the ESAMS. Next, we assessed the accuracy of the human expert confidence scores relative to the ML’s uncertainty quantification scores. This report includes a description of our methodology, summary of our findings and future directions.

97 MATHEMATICS AND COMPUTING↗

Calculation of narrower confidence intervals for tree mortality rates when we know nothing but the location of the death/survival events

Abstract Many ecological applications, like the study of mortality rates, require the estimation of proportions and confidence intervals for them. The traditional way of doing this applies the binomial distribution, which describes the outcome of a series of Bernoulli trials. This distribution assumes that observations are independent and the probability of success is the same for all the individual observations. Both assumptions are obviously false in many cases. I show how to apply bootstrap and the Poisson binomial distribution (a generalization of the binomial distribution) to the estimation of proportions. Any information at the individual level would result in better (narrower) confidence intervals around the estimation of proportions. As a case study, I applied this method to the calculation of mortality rates in a forest plot of tropical trees in Lambir Hills National Park, Malaysia. I calculated central estimates and 95% confidence intervals for species‐level mortality rates for 1,007 tree species. I used a very simple model of spatial dependence in survival to estimate individual‐level risk of mortality. The results obtained by accounting for heterogeneity in individual‐level risk of mortality were comparable to those obtained with the binomial distribution in terms of central estimates, but the precision increased in virtually all cases, with an average reduction in the width of the confidence interval of ~20%. Spatial information allows the estimation of individual‐level probabilities of survival, and this increases the precision in the estimates of mortality rates. The general method described here, with modifications, could be applied to reduce uncertainty in the estimation of proportions related to any spatially structured phenomenon with two possible outcomes. More sophisticated approaches can yield better estimates of individual‐level mortality and thus narrower confidence intervals.

59 BASIC BIOLOGICAL SCIENCES↗

A National Radiation Oncology Medical Student Clerkship Survey: Didactic Curricular Components Increase Confidence in Clinical Competency

Purpose: Students applying to radiation oncology residency programs complete 1 or more radiation oncology clerkships. This study assesses student experiences and perspectives during radiation oncology clerkships. The impact of didactic components and number of clerkship experiences in relation to confidence in clinical competency and preparation to function as a first-year radiation oncology resident are evaluated. Methods and Materials: An anonymous, Internet-based survey was sent via direct e-mail to all applicants to a single radiation oncology residency program during the 2012-2013 academic year. The survey was composed of 3 main sections including questions regarding baseline demographic information and prior radiation oncology experience, rotation experiences, and ideal clerkship curriculum content. Results: The survey response rate was 37% (70 of 188). Respondents reported 191 unique clerkship experiences. Of the respondents, 27% (19 of 70) completed at least 1 clerkship with a didactic component geared towards their level of training. Completing a clerkship with a didactic component was significantly associated with a respondent's confidence to function as a first-year radiation oncology resident (Wilcoxon rank–sum P=.03). However, the total number of clerkships completed did not correlate with confidence to pursue radiation oncology as a specialty (Spearman ρ P=.48) or confidence to function as a first year resident (Spearman ρ P=.43). Conclusions: Based on responses to this survey, rotating students perceive that the majority of radiation oncology clerkships do not have formal didactic curricula. Survey respondents who completed a clerkship with a didactic curriculum reported feeling more prepared to function as a radiation oncology resident. However, completing an increasing number of clerkships does not appear to improve confidence in the decision to pursue radiation oncology as a career or to function as a radiation oncology resident. These results support further development of structured didactic curricula for the radiation oncology clerkship.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Maximum-confidence discrimination among symmetric qudit states

We study the maximum-confidence (MC) measurement strategy for discriminating among nonorthogonal symmetric qudit states. Restricting to linearly dependent and equally likely pure states, we find the optimal positive operator valued measure (POVM) that maximizes our confidence in identifying each state in the set and minimizes the probability of obtaining inconclusive results. The physical realization of this POVM is completely determined and it is shown that after an inconclusive outcome, the input states may be mapped into a new set of equiprobable symmetric states, restricted, however, to a subspace of the original qudit Hilbert space. By applying the MC measurement again onto this new set, we can still gain some information about the input states, although with less confidence than before. This leads us to introduce the concept of sequential maximum-confidence (SMC) measurements, where the optimized MC strategy is iterated in as many stages as allowed by the input set, until no further information can be extracted from an inconclusive result. Within each stage of this measurement our confidence in identifying the input states is the highest possible, although it decreases from one stage to the next. In addition, the more stages we accomplish within the maximum allowed, the higher will be the probability of correct identification. We will discuss an explicit example of the optimal SMC measurement applied in the discrimination among four symmetric qutrit states and propose an optical network to implement it.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Correction of confidence intervals in excess relative risk models using Monte Carlo dosimetry systems with shared errors

Introduction: Measurement errors are ubiquitous in epidemiological studies, especially for doses arising from environmental and occupational exposures, which are difficult to assess. Correction methods for independent measurement errors, have been comprehensively described elsewhere, including regression calibration, simulation extrapolation (SIMEX), etc. However, when exposure estimates are constructed using complicated physical and biological models, uncertainties in the dosimetry system can become very complex. In such systems, uncertainties in some parameters may affect a large group of study participants simultaneously, and hence are “shared” as opposed to independent dosimetric uncertainties. Representing the structure of the uncertainty in the dosimetry by providing multiple realizations of possible dose has been described for several studies. The goal of this paper is to implement, and explore the properties of, a generalized estimating equation-based approach to shared uncertainties based on principles described in an earlier paper. Methods: The effects of shared and unshared dosimetry error on parameter estimates and confidence intervals in a generalized excess relative risk model are examined. A post-hoc correction method is implemented that constructs corrected confidence intervals based on an underlying variance structure described in an earlier paper. In this correction method it is assumed that the estimated slope parameter in a linear excess relative risk model is distributed as a mixture of normal and lognormal components. A simulation study of the behavior of this correction was developed based on multiple realizations provided by the Monte Carlo dosimetry system used for the plutonium exposures in the Mayak Worker Cohort and a risk model with close similarity to that used in analyses of lung cancer data in that cohort. Results: In the simulation study we found that shared dose errors severely degraded the coverage probabilities of confidence limits for the slope parameter in the linear excess relative risk model relating uncertain exposure to lung cancer risk. Using our correction method to adjust the confidence intervals provided much improved coverage probabilities for this parameter. A smaller amount of degradation in confidence interval coverage for other parameters (e.g. those in the baseline and dose modification terms) was noted, and the proposed correction method also improved coverage for these parameters as well. Discussion: Compared to other recently described methods our proposed correction method relies on relatively familiar software to fit dose response models. The basic modeling philosophy described here first uses existing Poisson regression methods to estimate parameters in the excess relative risk model; in this first stage mean dose (computed from the multiple realizations) is used as the independent variable in the regression. Once the model is fit we then correct (widen) the standard errors for all parameters using the proposed correction method. The simulation studies indicate that the methods described here are an important contribution to the available tools for dealing with shared measurement errors in complex risk analysis.

61 RADIATION PROTECTION AND DOSIMETRY↗

Confidence Calibration Metrics

This technical report serves to summarize a literature search conducted that covered confidence calibration. This report is meant to serve as a solid starting reference for individuals interested in learning more about the confidence calibration domain as well as for individuals more familiar with this work – as a summarizing document for calibration metrics is notably lacking in the literature. This report is not meant to serve as a comprehensive review of everything that has been done in this field – in fact, the reader is encouraged to look further into this domain. We describe confidence and calibration and discuss properties of good calibration metrics. We detail various calibration and calibration-tangential metrics, presenting equations, algorithms, parameters, and an analysis of strengths and weaknesses. We apply a subset of these metrics to eight proxy confidence assessment datasets. We examine the various metrics in the context of model confidence. Finally, we discuss promising future directions and outstanding questions.

99 GENERAL AND MISCELLANEOUS↗

Random selection as a confidence building tool

Any verification measurement performed on potentially classified nuclear material must satisfy two seemingly contradictory constraints. First and foremost, no classified information can be released. At the same time, the monitoring party must have confidence in the veracity of the measurement. The first concern can be addressed by performing the measurements within the host facility using instruments under the host's control. Because the data output in this measurement scenario is also under host control, it is difficult for the monitoring party to have confidence in that data. One technique for addressing this difficulty is random selection. The concept of random selection can be thought of as four steps: (1) The host presents several 'identical' copies of a component or system to the monitor. (2) One (or more) of these copies is randomly chosen by the monitors for use in the measurement system. (3) Similarly, one or more is randomly chosen to be validated further at a later date in a monitor-controlled facility. (4) Because the two components or systems are identical, validation of the 'validation copy' is equivalent to validation of the measurement system. This procedure sounds straightforward, but effective application may be quite difficult. Although random selection is often viewed as a panacea for confidence building, the amount of confidence generated depends on the monitor's continuity of knowledge for both validation and measurement systems. In this presentation, we will discuss the random selection technique, as well as where and how this technique might be applied to generate maximum confidence. In addition, we will discuss the role of modular measurement-system design in facilitating random selection and describe a simple modular measurement system incorporating six small {sup 3}He neutron detectors and a single high-purity germanium gamma detector.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Achieving reasonable confidence in the validation of control room designs and modifications: A summary of the 2015 NEA experts' workshop - 124

Defining and achieving reasonable confidence in the validation of nuclear power plant control room designs is a complex challenge for many reasons; not the least of which is that validation is a test of strength rather than a test of truth. In 2015, the Nuclear Energy Agency, Committee for Safety of Nuclear Installations' Working Group on Human and Organizational Factors (WGHOF) hosted an experts' workshop to identify and critically examine potential means to address the challenge of achieving reasonable confidence in control room design validations. The experts' principal task was to answer four challenge questions. (1) What are the critical considerations in defining validation objectives and how do these impact achieving reasonable confidence? (2) What methods, approaches, resources, or rationales might be used for deriving performance requirements, selecting measures, and establishing acceptance criteria so as to support reasonable confidence? (3) What methods might be used to develop scenarios that maximize the amount and relevance of information in support of the validation conclusions and achieving reasonable confidence? (4) How should the validation results be aggregated and analyzed to determine the final validation conclusions? This paper provides an overview of the experts' responses to these challenges and briefly touches upon their implications for future directions. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

A Statistical Method for Assessing Peptide Identification Confidence in Accurate Mass and Time Tag Proteomics

High-throughput proteomics is rapidly evolving to require high mass measurement accuracy for a variety of different applications. Increased mass measurement accuracy in bottom-up proteomics specifically allows for an improved ability to distinguish and characterize detected MS features, which may in turn be identified by, e.g., matching to entries in a database for both precursor and fragmentation mass identification methods. Many tools exist with which to score the identification of peptides from LC-MS/MS measurements or to assess matches to an accurate mass and time (AMT) tag database, but these two calculations remain distinctly unrelated. Here we present a statistical method, Statistical Tools for AMT tag Confidence (STAC), which extends our previous work incorporating prior probabilities of correct sequence identification from LC-MS/MS, as well as the quality with which LC-MS features match AMT tags, to evaluate peptide identification confidence. Compared to existing tools, we are able to obtain significantly more high-confidence peptide identifications at a given false discovery rate and additionally assign confidence estimates to individual peptide identifications. Freely available software implementations of STAC are available in both command line and as a Windows graphical application.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Application of a Metal Artifact Reduction Algorithm for C-Arm Cone-Beam CT: Impact on Image Quality and Diagnostic Confidence for Bronchial Artery Embolization

Objective: The objective of this study was to evaluate the potential benefit of a dedicated cone-beam-CT streak metal artifact removal technique (SMART) in terms of both image quality and diagnostic confidence in patients undergoing bronchial artery embolization. Methods: A total of 17 patients were included in this retrospective study. The SMART algorithm was applied to images containing streak artifacts generated by a radiopaque intra-arterial catheter tip. Quantitative evaluation of artifact severity was performed via measurement of the Hounsfield units along a closed loop surrounding the catheter tip and was conducted in the frequency domain following the application of the discrete Fourier transform to the measured data. A high proportion of power in the low frequencies of the resulting spectrum indicated a high level of streak artifacts. Qualitative evaluation of diagnostic confidence was performed using a 4-point Likert scale. Results: Both quantitative and qualitative evaluation demonstrated a significant reduction in artifact severity using the SMART algorithm. Quantitative evaluation demonstrated a mean artifact reduction of 22.5% using SMART compared to non-SMART images (p < 0.001). Qualitative evaluation demonstrated the greatest artifact reduction at the inner and outer aortic curvature, as well as immediately surrounding the tip of the catheter. In 6 of 17 cases, the use of the SMART algorithm yielded additional clinical information, increasing mean diagnostic confidence from 3.17 to 3.78 (p < 0.001). Conclusion: The SMART algorithm allows for efficient reduction of metal artifacts introduced by radiopaque catheter tips during cone-beam CT. Using this algorithm, diagnostic images of the aortic arch were significantly improved both quantitatively and qualitatively, yielding clinically relevant levels of enhanced diagnostic confidence. These results demonstrate that the SMART algorithm improves diagnostic and clinical characterization of the course of bronchial arteries on CBCT images, potentially improving the accuracy and clinical efficacy of bronchial artery embolization. Level of Evidence: 3.

62 RADIOLOGY AND NUCLEAR MEDICINE↗