Search NASA⌕ Search

SEARCH · Search NASA

Results for “false discovery rate”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Enter Gaussian Mixture Modeling Extensions for Improved False Discovery Rate Estimation in GC-MS Metabolomics

Identifying small molecules (e.g., metabolites) is key towards driving scientific advancement in metabolomics, and gas chromatography–mass spectrometry (GC-MS) is an analytic method that may be applied to facilitate this process. The typical GC-MS identification workflow involves quantifying the similarity of an observed sample spectrum and other features (e.g. retention index) to that of several references, noting the compound of the best-matching reference spectrum as the identified metabolite. While a deluge of similarity metrics exists, none characterize the error rate of generated identifications, thereby presenting an unknown risk of false identification or discovery. To quantify this unknown risk, we propose a model-based framework for estimating the false discovery rate (FDR) among a set of identifications. Extending the traditional mixture modeling framework, our method incorporates both similarity score and experimental information in estimating the FDR. We apply these models to identification lists derived from across 548 samples of varying complexity and sample type (e.g., fungal species, standard mixtures, etc.), comparing their performance to that of the traditional Gaussian mixture model (GMM). Through simulation, we additionally assess the impact of reference library size on the accuracy of FDR estimates. In comparing the best performing model extensions to the GMM, our results indicate relative decreases in median absolute estimation error (MAE) ranging from 12% to 70%, based on comparisons of the median MAEs across all hit-lists. Results indicate that these relative performance improvements generally hold despite library size, however FDR estimation error typically worsens as the set of reference compounds diminishes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Target–decoy false discovery rate estimation using Crema

Assigning statistical confidence estimates to discoveries produced by a tandem mass spectrometry proteomics experiment is critical to enabling principled interpretation of the results and assessing the cost/benefit ratio of experimental follow-up. The most common technique for computing such estimates is to use target-decoy competition (TDC), in which observed spectra are searched against a database of real (target) peptides and a database of shuffled or reversed (decoy) peptides. TDC procedures for estimating the false discovery rate (FDR) at a given score threshold have been developed for application at the level of spectra, peptides, or proteins. Although these techniques are relatively straightforward to implement, it is common in the literature to skip over the implementation details or even to make mistakes in how the TDC procedures are applied in practice. Here we present Crema, an open-source Python tool that implements several TDC methods of spectrum-, peptide- and protein-level FDR estimation. Crema is compatible with a variety of existing database search tools and provides a straightforward way to obtain robust FDR estimates.

59 BASIC BIOLOGICAL SCIENCES↗

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗

Evaluation of In Silico Multifeature Libraries for Providing Evidence for the Presence of Small Molecules in Synthetic Blinded Samples

The current gold standard for unambiguous molecular identification in metabolomics analysis is comparing two or more orthogonal properties from the analysis of authentic reference materials (standards) to experimental data acquired in the same laboratory with the same analytical methods. This represents a significant limitation for comprehensive chemical identification of small molecules in complex samples. The process is time-consuming and costly, and the majority of molecules are not yet represented by standards. Thus, there is a need to assemble evidence for the presence of small molecules in complex samples through the use of libraries containing calculated chemical properties. To address this need, we developed a Multi-Attribute Matching Engine (MAME) and a library derived in-part from our in silico Chemical Library Engine (ISiCLE). Here, we describe an initial evaluation of these methods in a blinded analysis of synthetic chemical mixtures as part of the U.S. Environmental Protection Agency’s (EPA) Non-Targeted Analysis Collaborative Trial (ENTACT, Phase 1). For molecules in all mixtures, the initial blinded false negative rate (FNR), false discovery rate (FDR), and accuracy were 57%, 77%, and 91%, respectively. For high evidence scores, the FDR was 35%. After unblinding of the sample compositions, we optimized the scoring parameters to better exploit the available evidence, and increased the accuracy for molecules suspected as present. The final FNR, FDR, and accuracy were 67%, 53%, and 96%, respectively. For high evidence scores, the FDR was 10%. This study demonstrates that multi-attribute matching methods in conjunction with in silico libraries may one day enable reduced reliance on experimentally-derived libraries for building evidence for the presence of molecules in complex samples.

Nunez, Jamie↗

Significant Association between Sulfate-Reducing Bacteria and Uranium-Reducing Microbial Communities as Revealed by a Combined Massively Parallel Sequencing-Indicator Species Approach

Massively parallel sequencing has provided a more affordable and high-throughput method to study microbial communities, although it has mostly been used in an exploratory fashion. We combined pyrosequencing with a strict indicator species statistical analysis to test if bacteria specifically responded to ethanol injection that successfully promoted dissimilatory uranium(VI) reduction in the subsurface of a uranium contamination plume at the Oak Ridge Field Research Center in Tennessee. Remediation was achieved with a hydraulic flow control consisting of an inner loop, where ethanol was injected, and an outer loop for flow-field protection. This strategy reduced uranium concentrations in groundwater to levels below 0.126 M and created geochemical gradients in electron donors from the inner-loop injection well toward the outer loop and downgradient flow path. Our analysis with 15 sediment samples from the entire test area found significant indicator species that showed a high degree of adaptation to the three different hydrochemical-created conditions. Castellaniella and Rhodanobacter characterized areas with low pH, heavy metals, and low bioactivity, while sulfate-, Fe(III)-, and U(VI)-reducing bacteria (Desulfovibrio, Anaeromyxobacter, and Desulfosporosinus) were indicators of areas where U(VI) reduction occurred. The abundance of these bacteria, as well as the Fe(III) and U(VI) reducer Geobacter, correlated with the hydraulic connectivity to the substrate injection site, suggesting that the selected populations were a direct response to electron donor addition by the groundwater flow path. A false-discovery-rate approach was implemented to discard false-positive results by chance, given the large amount of data compared.

54 ENVIRONMENTAL SCIENCES↗

Are 100 Ensemble Members Enough to Capture the Remote Atmospheric Response to +2°C Arctic Sea Ice Loss?

This study presents results from the Polar Amplification Multimodel Intercomparison Project (PAMIP) single-year time-slice experiments that aim to isolate the atmospheric response to Arctic sea ice loss at global warming levels of +2°C. Using two general circulation models (GCMs), the ensemble size is increased up to 300 ensemble members, beyond the recommended 100 members. After partitioning the response in groups of 100 ensemble members, the reproducibility of the results is evaluated, with a focus on the response of the midlatitude jet streams in the North Atlantic and North Pacific. Both atmosphere-only and coupled ocean–atmosphere PAMIP experiments are analyzed. Substantial differences in the midlatitude response are found among the different experiment subsets, suggesting that 100-member ensembles are still significantly influenced by internal variability, which can mislead conclusions. Despite an overall stronger response, the coupled ocean–atmosphere runs exhibit greater spread due to additional ENSO-related internal variability when the ocean is interactive. The lack of consistency in the response is true for anomalies that are statistically significant according to Student’s t and false discovery rate tests. This is problematic for the multimodel assessment of the response, as some of the spread may be attributed to different model sensitivities whereas it is due to internal variability. Here, we propose a method to overcome this consistency issue that allows for more robust conclusions when only 100 ensemble members are used.

58 GEOSCIENCES↗

Correction of quantification errors in pelvic and spinal lesions caused by ignoring higher photon attenuation of bone in [{sup 18}F]NaF PET/MR

Purpose: MR-based attenuation correction (MRAC) in routine clinical whole-body positron emission tomography and magnetic resonance imaging (PET/MRI) is based on tissue type segmentation. Due to lack of MR signal in cortical bone and the varying signal of spongeous bone, standard whole-body segmentation-based MRAC ignores the higher attenuation of bone compared to the one of soft tissue (MRAC{sub nobone}). The authors aim to quantify and reduce the bias introduced by MRAC{sub nobone} in the standard uptake value (SUV) of spinal and pelvic lesions in 20 PET/MRI examinations with [{sup 18}F]NaF. Methods: The authors reconstructed 20 PET/MR [{sup 18}F]NaF patient data sets acquired with a Philips Ingenuity TF PET/MRI. The PET raw data were reconstructed with two different attenuation images. First, the authors used the vendor-provided MRAC algorithm that ignores the higher attenuation of bone to reconstruct PET{sub nobone}. Second, the authors used a threshold-based algorithm developed in their group to automatically segment bone structures in the [{sup 18}F]NaF PET images. Subsequently, an attenuation coefficient of 0.11 cm{sup −1} was assigned to the segmented bone regions in the MRI-based attenuation image (MRAC{sub bone}) which was used to reconstruct PET{sub bone}. The automatic bone segmentation algorithm was validated in six PET/CT [{sup 18}F]NaF examinations. Relative SUV{sub mean} and SUV{sub max} differences between PET{sub bone} and PET{sub nobone} of 8 pelvic and 41 spinal lesions, and of other regions such as lung, liver, and bladder, were calculated. By varying the assigned bone attenuation coefficient from 0.11 to 0.13 cm{sup −1}, the authors investigated its influence on the reconstructed SUVs of the lesions. Results: The comparison of [{sup 18}F]NaF-based and CT-based bone segmentation in the six PET/CT patients showed a Dice similarity of 0.7 with a true positive rate of 0.72 and a false discovery rate of 0.33. The [{sup 18}F]NaF-based bone segmentation worked well in the pelvis and spine. However, it showed artifacts in the skull and in the extremities. The analysis of the 20 [{sup 18}F]NaF PET/MRI examinations revealed relative SUV{sub max} differences between PET{sub nobone} and PET{sub bone} of (−8.8% ± 2.7%, p = 0.01) and (−8.1% ± 1.9%, p = 2.4 × 10{sup −8}) in pelvic and spinal lesions, respectively. A maximum SUV{sub max} underestimation of −13.7% was found in lesion in the third cervical spine. The averaged SUV{sub mean} differences in volumes of interests in lung, liver, and bladder were below 3%. The average SUV{sub max} differences in pelvic and spinal lesions increased from −9% to −18% and −8% to −17%, respectively, when increasing the assigned bone attenuation coefficient from 0.11 to 0.13 cm{sup −1}. Conclusions: The developed automatic [{sup 18}F]NaF PET-based bone segmentation allows to include higher bone attenuation in whole-body MRAC and thus improves quantification accuracy for pelvic and spinal lesions in [{sup 18}F]NaF PET/MRI examinations. In nonbone structures (e.g., lung, liver, and bladder), MRAC{sub nobone} yields clinically acceptable accuracy.

60 APPLIED LIFE SCIENCES↗

Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events

The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

54 ENVIRONMENTAL SCIENCES↗

A Statistical Method for Assessing Peptide Identification Confidence in Accurate Mass and Time Tag Proteomics

High-throughput proteomics is rapidly evolving to require high mass measurement accuracy for a variety of different applications. Increased mass measurement accuracy in bottom-up proteomics specifically allows for an improved ability to distinguish and characterize detected MS features, which may in turn be identified by, e.g., matching to entries in a database for both precursor and fragmentation mass identification methods. Many tools exist with which to score the identification of peptides from LC-MS/MS measurements or to assess matches to an accurate mass and time (AMT) tag database, but these two calculations remain distinctly unrelated. Here we present a statistical method, Statistical Tools for AMT tag Confidence (STAC), which extends our previous work incorporating prior probabilities of correct sequence identification from LC-MS/MS, as well as the quality with which LC-MS features match AMT tags, to evaluate peptide identification confidence. Compared to existing tools, we are able to obtain significantly more high-confidence peptide identifications at a given false discovery rate and additionally assign confidence estimates to individual peptide identifications. Freely available software implementations of STAC are available in both command line and as a Windows graphical application.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

ABRF Proteome Informatics Research Group (iPRG) 2016 Study: Inferring Proteoforms from Bottom-up Proteomics Data

This report presents the results from the 2016 ABRF Proteome Informatics Research Group (iPRG) study on proteoform inference and FDR estimation from bottom-up proteomics data. For this study, three replicate Q Exactive Orbitrap LC-MS/MS datasets were generated from each of four E. coli samples spiked with different equimolar mixtures of small recombinant proteins selected to mimic pairs of homologous proteins. Participants were given raw data and a sequence file, and asked to identify the proteins and provide estimates on the false discovery rate at the proteoform level. As part of this study, we tested a new submission system with a format validator running on a virtual private server (VPS) and allowed methods to be provided as executable R Markdown or IPython Notebooks. The task was perceived as difficult, and only eight unique submissions were received, though those who participated did well, with no one method performing best on all samples. However, none of the submissions included a complete Markdown or Notebook, even though examples were provided. Future iPRG studies need to be more successful in promoting and encouraging participation. The VPS and submission validator easily scale to much larger numbers of participants in these types of studies. The unique “ground-truth” dataset for proteoform identification generated for this study is now available to the research community, as are the server-side scripts for validating and managing submissions.

Proteomics, proteoforms, inference, false discover↗

Improving Peptide-Level Mass Spectrometry Analysis via Double Competition

A critical statistical task in the analysis of shotgun proteomics data involves controlling the false discovery rate (FDR) among the reported set of discoveries. This task is most commonly solved at the peptide-spectrum match (PSM) level by using target-decoy competition (TDC), in which a set of observed spectra are searched against a database containing a mixture of real (target) and decoy peptides. The PSM-level procedure can be adapted to the peptide level by selecting the top-scoring PSM per peptide prior to FDR estimation. Here, we investigate both PSM-level and peptide-level FDR control methods and come to two conclusions. First, although the TDC procedure is provably correct under certain assumptions, we observe that one of these assumptions - that incorrect PSMs are independent of one another - is frequently violated. Hence, we empirically demonstrate that TDC-based PSM-level FDR estimates can be liberally biased. Further, we propose that researchers avoid PSM-level results and instead focus on peptide-level analysis. Second, we investigate three ways to carry out peptide level TDC and show that the most common method ("PSM-only") offers the lowest statistical power in practice. The most powerful method, peptide-level FDR with PSM competition ("PSM-and-peptide"), carries out competition first at the PSM level and then again at the peptide level. In our experiments, this approach yields an average increase of 17% more discovered peptides at a1% FDR threshold relative to the PSM-only method.

59 BASIC BIOLOGICAL SCIENCES↗

Validation tests of an improved kernel density estimation method for identifying disease clusters

The spatial filter method, which belongs to the class of kernel density estimation methods, has been used to make morbidity and mortality maps in several recent studies. We propose improvements in the method that include a spatial basis of support designed to give a constant standard error for the standardized mortality/morbidity rate; a stair-case weight method for weighting observations to reduce estimation bias; and a method for selecting parameters to control three measures of performance of the method: sensitivity, specificity and false discovery rate. We test the performance of the method using Monte Carlo simulations of hypothetical disease clusters over a test area of four counties in Iowa. The simulations include different types of spatial disease patterns and high resolution population distribution data. Results confirm that the new features of the spatial filter method do substantially improve its performance in realistic situations comparable to those where the method is likely to be used.

59 BASIC BIOLOGICAL SCIENCES↗

Reduced evolutionary rates in HIV-1 reveal extensive latency periods among replicating lineages

Background HIV-1 can persist for the duration of a patient's life due in part to its ability to hide from the immune system, and from antiretroviral drugs, in long-lived latent reservoirs. Latent forms of HIV-1 may also be disproportionally involved in transmission. Thus, it is important to detect and quantify latency in the HIV-1 life cycle. Results We developed a novel molecular clock–based phylogenetic tool to investigate the prevalence of HIV-1 lineages that have experienced latency. The method removes alternative sources that may affect evolutionary rates, such as hypermutation, recombination, and selection, to reveal the contribution of generation-time effects caused by latency. Our method was able to recover latent lineages with high specificity and sensitivity, and low false discovery rates, even on relatively short branches on simulated phylogenies. Applying the tool to HIV-1 sequences from 26 patients, we show that the majority of phylogenetic lineages have been affected by generation-time effects in every patient type, whether untreated, elite controller, or under effective or failing treatment. Furthermore, we discovered extensive effects of latency in sequence data ( gag, pol , and env ) from reservoirs as well as in the replicating plasma population. To better understand our phylogenetic findings, we developed a dynamic model of virus-host interactions to investigate the proportion of lineages in the actively replicating population that have ever been latent. Assuming neutral evolution, our dynamic modeling showed that under most parameter conditions, it is possible for a few activated latent viruses to propagate so that in time, most HIV-1 lineages will have been latent at some time in their past. Conclusions These results suggest that cycling in and out of latency plays a major role in the evolution of HIV-1. Thus, no aspect of HIV-1 evolution can be fully understood without considering latency - including treatment, drug resistance, immune evasion, transmission, and pathogenesis.

Virology↗

Large Improvements in MS/MS Based Peptide Identification Rates using a Hybrid Analysis

We have developed a hybrid method for identifying peptides from global proteomics studies that significantly increases sensitivity and specificity in matching peptides to tandem mass spectra using database searches. The method increased the number of spectra that can be assigned to a peptide in a global proteomics study by 57-147% at an estimated false discovery rate of 5%, with clear room for even greater improvements. The approach combines the general utility of using consensus model spectra typical of database search methods1-3 with the accuracy of the intensity information contained in spectral libraries4-6. This hybrid approach is made possible by recent developments that elucidated the statistical framework common to both data analysis and statistical thermodynamics, resulting in a chemically inspired approach to incorporating fragment intensity information into both database searches and spectral library searches. We applied this approach to proteomics analysis of Synechococcus sp. PCC 7002, a cyanobacterium that is a model organism for studies of photosynthetic carbon fixation and biofuels development. The increased specificity and sensitivity of this approach allowed us to identify many more peptides involved in the processes important for photoautotrophic growth.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Modeling inter-signal arrival times for accurate detection of CAN bus signal injection attacks

Modern vehicles rely on hundreds of on-board electronic control units (ECUs) communicating over in-vehicle networks. As external interfaces to the car control networks (such as the on-board diagnostic (OBD) port, auxiliary media ports, etc.) become common, and vehicle-to-vehicle / vehicle-to-infrastructure technology is in the near future, the attack surface for vehicles grows, exposing control networks to potentially life-critical attacks. This paper addresses the need for securing the CAN bus by detecting anomalous traffic patterns via unusual refresh rates of certain commands. While previous works have identified signal frequency as an important feature for CAN bus intrusion detection, this paper provides the first such algorithm with experiments on five attack scenarios. Our data-driven anomaly detection algorithm requires only five seconds of training time (on normal data) and achieves true positive / false discovery rates of 0.9998/0.00298, respectively (micro-averaged across the five experimental tests).

anomaly detection↗

ROCker: accurate detection and quantification of target genes in short-read metagenomic data sets by modeling sliding-window bitscores

Functional annotation of metagenomic and metatranscriptomic data sets relies on similarity searches based on e-value thresholds resulting in an unknown number of false positive and negative matches. To overcome these limitations, we introduce ROCker, aimed at identifying position-specific, most-discriminant thresholds in sliding windows along the sequence of a target protein, accounting for non-discriminative domains shared by unrelated proteins. ROCker employs the receiver operating characteristic (ROC) curve to minimize false discovery rate (FDR) and calculate the best thresholds based on how simulated shotgun metagenomic reads of known composition map onto well-curated reference protein sequences and thus, differs from HMM profiles and related methods. We showcase ROCker using ammonia monooxygenase (amoA) and nitrous oxide reductase (nosZ) genes, mediating oxidation of ammonia and the reduction of the potent greenhouse gas, N 2 O, to inert N 2 , respectively. ROCker typically showed 60-fold lower FDR when compared to the common practice of using fixed e-values. Previously uncounted ‘atypical’ nosZ genes were found to be two times more abundant, on average, than their typical counterparts in most soil metagenomes and the abundance of bacterial amoA was quantified against the highly-related particulate methane monooxygenase (pmoA). Therefore, ROCker can reliably detect and quantify target genes in short-read metagenomes.

59 BASIC BIOLOGICAL SCIENCES↗