Search NASA⌕ Search

SEARCH · Search NASA

Results for “False Positive Rate”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

pyFLANK, a graph neural network based null distribution inference model for F ST outlier detection

Detecting genomic regions under selection is essential for understanding how populations adapt to different environments, yet it remains challenging due to the confounding effects of demographic history and linkage disequilibrium (LD). Fixation index (F ST ) is a widely used statistic to identify genomic regions under adaptation. However, identifying genes under selection by defining F ST outliers often remains challenging, owing to confounding effects of underlying demographic history. Traditional methods assume independence among loci and rely on simple demographic models, while newer models perform much better but are computationally expensive and not easily scalable. Here, we present pyFLANK, an open-source and automated Python implementation which detects F ST outliers using a null distribution inferred from quasi-independent loci. Our tool integrates three approaches to identify loci obeying a null distribution: graph neural network (GNN) inference, linkage disequilibrium (LD)-based inference, and user-defined input. Because pyFLANK uses GNN-based inference of quasi-independent loci, it yields a more accurate null model with less need for user parameter input. In simulation experiments, pyFLANK achieved lower false positive rates than current methods while maintaining comparable detection power, indicating that its refined null model better distinguishes true adaptive loci from background variation. The GNN-based model, in particular, detected additional loci associated with phenotypic variance that were not identified by existing methods. Assessments of simulation and real data from different species demonstrate that pyFLANK achieves lower false positive rates compared with other commonly used F ST outlier detectors, while maintaining comparable detection power and excellent computational performance, providing a robust and user-friendly tool for identifying loci under divergent selection. It extends existing F ST outlier frameworks by incorporating explicit LD-aware strategies for null model calibration. The method is intended as a practical and scalable complement to existing genome scan approaches.

FST↗

On the Performance of Detecting Injection of Fabricated Messages into the CAN Bus

There have been several public demonstrations of attacks on connected vehicles showing the ability of an attacker to take control of a targeted vehicle by injecting messages into their Controller Area Network (CAN) bus. In this article, using injected speed reading and Revolutions Per Minute (RPM) reading messages in in-motion vehicle, we examine the ability of the Pearson correlation and the unsupervised learning methods k-means clustering and Hidden Markov Model (HMM) to differentiate ’no-attack’ and ’under-attack’ states of the given vehicle. We found that the Pearson correlation distinguishes the two states, the k-means clustering method has an acceptable accuracy but high false positive rate and HMM detects attacks with acceptable detection rate but has a high false positive in detecting attacks from speed readings when there is no attack. The accuracy of these unsupervised learning methods are comparable to the ones of the supervised learning methods used by CAN bus Intrusion Detection System (IDS) suppliers. In addition, the article shows that studying CAN anomaly detection techniques using off-vehicle test facilities may not properly evaluate the performance of the detection techniques. Finally, the results suggest using other features besides the data content of the CAN messages and integrate knowledge about how the Electronic Control Units (ECUs) collaborate in building effective techniques for the detection of injection of fabricated message attacks.

97 MATHEMATICS AND COMPUTING↗

Via Machinae 2.0 : Full-sky, model-agnostic search for stellar streams in Gaia DR2

ABSTRACT We present an update to Via Machinae, an automated stellar stream-finding algorithm based on the deep learning anomaly detector ANODE. Via Machinae identifies stellar streams within Gaia, using only angular positions, proper motions, and photometry, without reference to a model of the Milky Way potential for orbit integration or stellar distances. This new version, Via Machinae 2.0, includes many improvements and refinements to nearly every step of the algorithm, that altogether result in more robust and visually distinct stream candidates than our original formulation. In this work, we also provide a quantitative estimate of the false positive rate of Via Machinae 2.0 by applying it to a simulated Gaia-mock catalogue based on galaxia, a smooth model of the Milky Way that does not contain substructure or stellar streams. Finally, we perform the first full-sky search for stellar streams with Via Machinae 2.0, identifying 102 streams at high significance within the Gaia Data Release 2, of which only 10 have been previously identified. While follow-up observations for further confirmation are required, taking into account the false positive rate presented in this work, we expect approximately 90 of these stream candidates to correspond to real stellar structures.

Astronomy & Astrophysics↗

AviSense: A Real-time System for Detection, Classification, and Analysis of Aviation Signals

Wireless systems are an integral part of aviation. Apart from their apparent use in air-to-ground communication, wireless systems play a crucial role in avionic functions including navigation and landing. An interference-free wireless environment is therefore critical for the uninterrupted operation and safety of an aircraft. Hence, there is an urgency for airport facilities to acquire the capability to continuously monitor aviation frequency bands for real-time detection of interference and anomalies. To meet this critical need, we design and build AviSense, an SDR-based real-time, versatile system for monitoring aviation bands. AviSense detects and characterizes signal activities to enable practical and effective anomaly detection. We identify and tackle the challenges posed by a diverse set of critical aviation bands and technologies. We evaluate our methodology with real-world aviation signal measurements and two custom datasets of anomalous signals. We find that our signal classification capability achieves a true positive rate of ~99%, with few exceptions, and a false positive rate of less than 4%. We also demonstrate that AviSense can effectively distinguish between different types of anomalies. We build and evaluate a prototype implementation of AviSense that supports distributed monitoring.

42 ENGINEERING↗

Real Time Intelligent Target Detection and Analysis with Machine Vision

We present an algorithm for detecting a specified set of targets for an Automatic Target Recognition (ATR) application. ATR involves processing images for detecting, classifying, and tracking targets embedded in a background scene. We address the problem of discriminating between targets and nontarget objects in a scene by evaluating 40x40 image blocks belonging to an image. Each image block is first projected onto a set of templates specifically designed to separate images of targets embedded in a typical background scene from those background images without targets. These filters are found using directed principal component analysis which maximally separates the two groups. The projected images are then clustered into one of n classes based on a minimum distance to a set of n cluster prototypes. These cluster prototypes have previously been identified using a modified clustering algorithm based on prior sensed data. Each projected image pattern is then fed into the associated cluster's trained neural network for classification. A detailed description of our algorithm will be given in this paper. We outline our methodology for designing the templates, describe our modified clustering algorithm, and provide details on the neural network classifiers. Evaluation of the overall algorithm demonstrates that our detection rates approach 96% with a false positive rate of less than 0.03%.

Howard, Ayanna↗

Long-Term field testing of the accuracy and HVAC energy savings potential of occupancy presence sensors in A Single-Family home

The energy-saving potential of occupancy-centric smart thermostats has been extensively explored in simulations but lacked field testing for energy savings quantification and sensor performance assessment in real buildings. This paper presents a long-term field study conducted in a single-family home in Texas, U.S. to evaluate the performance of occupancy-centric controls (OCC) of HVAC (heating, ventilation, and air-conditioning) system in terms of energy savings, sensor accuracy, and impact on electric peak demand. The test site was equipped with a commercial off-the-shelf (COTS) smart thermostat and multiple occupancy presence sensors for OCC implementation. Additionally, a sub-metering system was installed to monitor electricity consumption of various end-use equipment, including the HVAC system. A supplementary device was installed to track the ground-truth occupancy for the accuracy evaluation of the occupancy presence sensor. Scenarios of baseline and OCC controls were alternated weekly over the 20-month testing period. The results indicated an effective OCC execution, as evidenced by indoor temperature profiles. During the 2023 cooling season, OCC achieved total energy savings of 1,958 kWh, corresponding to a 17.6% energy savings ratio. Under certain conditions, daily HVAC energy savings reached as high as 17 kWh, with a savings ratio of 35%. Sensor performance showed an overall accuracy of 83.8%, a False Positive Rate (FPR) of 12.8%, and a False Negative Rate (FNR) of 47.4%. A key limitation was the sensor’s inability to detect stationary occupants during sleep, leading to a midnight FNR of nearly 100% and significantly compromising thermal comfort. Additionally, the implementation of OCC resulted in extended periods of high electricity demand on summer afternoons, affecting occupant’s thermal comfort and posing potential challenges to community-level grid operations if OCC were widely adopted. Furthermore, this study addresses a critical research gap by empirically investigating energy-saving potential and occupancy sensor performance in residential buildings. Through a comprehensive field-testing study, the research examines the interrelationship between sensor accuracy, energy savings, and thermal comfort, an area that has received limited attention in the current literature.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗

Data-driven landslide nowcasting at the global scale

Landslides affect nearly every country in the world each year. To better understand this global hazard, the Landslide Hazard Assessment for Situational Awareness (LHASA) model was developed previously. LHASA version 1 combines satellite precipitation estimates with a global landslide susceptibility map to produce a gridded map of potentially hazardous areas from 60° North-South every 3 h. LHASA version 1 categorizes the world’s land surface into three ratings: high, moderate, and low hazard with a single decision tree that first determines if the last seven days of rainfall were intense, then evaluates landslide susceptibility. LHASA version 2 has been developed with a data-driven approach. The global susceptibility map was replaced with a collection of explanatory variables, and two new dynamically varying quantities were added: snow and soil moisture. Along with antecedent rainfall, these variables modulated the response to current daily rainfall. In addition, the Global Landslide Catalog (GLC) was supplemented with several inventories of rainfall-triggered landslide events. These factors were incorporated into the machine-learning framework XGBoost, which was trained to predict the presence or absence of landslides over the period 2015–2018, with the years 2019–2020 reserved for model evaluation. As a result of these improvements, the new global landslide nowcast was twice as likely to predict the occurrence of historical landslides as LHASA version 1, given the same global false positive rate. Furthermore, the shift to probabilistic outputs allows users to directly manage the trade-off between false negatives and false positives, which should make the nowcast useful for a greater variety of geographic settings and applications. In a retrospective analysis, the trained model ran over a global domain for 5 years, and results for LHASA version 1 and version 2 were compared. Due to the importance of rainfall and faults in LHASA version 2, nowcasts would be issued more frequently in some tropical countries, such as Colombia and Papua New Guinea; at the same time, the new version placed less emphasis on arid regions and areas far from the Pacific Rim. LHASA version 2 provides a nearly real-time view of global landslide hazard for a variety of stakeholders.

XGBoos↗

DeepSAT: A Deep Learning Approach to Tree-Cover Delineation in 1-m NAIP Imagery for the Continental United States

High resolution tree cover classification maps are needed to increase the accuracy of current land ecosystem and climate model outputs. Limited studies are in place that demonstrates the state-of-the-art in deriving very high resolution (VHR) tree cover products. In addition, most methods heavily rely on commercial softwares that are difficult to scale given the region of study (e.g. continents to globe). Complexities in present approaches relate to (a) scalability of the algorithm, (b) large image data processing (compute and memory intensive), (c) computational cost, (d) massively parallel architecture, and (e) machine learning automation. In addition, VHR satellite datasets are of the order of terabytes and features extracted from these datasets are of the order of petabytes. In our present study, we have acquired the National Agriculture Imagery Program (NAIP) dataset for the Continental United States at a spatial resolution of 1-m. This data comes as image tiles (a total of quarter million image scenes with ~60 million pixels) and has a total size of ~65 terabytes for a single acquisition. Features extracted from the entire dataset would amount to ~8-10 petabytes. In our proposed approach, we have implemented a novel semi-automated machine learning algorithm rooted on the principles of "deep learning" to delineate the percentage of tree cover. Using the NASA Earth Exchange (NEX) initiative, we have developed an end-to-end architecture by integrating a segmentation module based on Statistical Region Merging, a classification algorithm using Deep Belief Network and a structured prediction algorithm using Conditional Random Fields to integrate the results from the segmentation and classification modules to create per-pixel class labels. The training process is scaled up using the power of GPUs and the prediction is scaled to quarter million NAIP tiles spanning the whole of Continental United States using the NEX HPC supercomputing cluster. An initial pilot over the state of California spanning a total of 11,095 NAIP tiles covering a total geographical area of 163,696 sq. miles has produced true positive rates of around 88 percent for fragmented forests and 74 percent for urban tree cover areas, with false positive rates lower than 2 percent for both landscapes.

Imagery↗

Application of Fault Management Theory to the Quantitative Selection of a Launch Vehicle Abort Trigger Suite

The theory of System Health Management (SHM) and of its operational subset Fault Management (FM) states that FM is implemented as a "meta" control loop, known as an FM Control Loop (FMCL). The FMCL detects that all or part of a system is now failed, or in the future will fail (that is, cannot be controlled within acceptable limits to achieve its objectives), and takes a control action (a response) to return the system to a controllable state. In terms of control theory, the effectiveness of each FMCL is estimated based on its ability to correctly estimate the system state, and on the speed of its response to the current or impending failure effects. This paper describes how this theory has been successfully applied on the National Aeronautics and Space Administration's (NASA) Space Launch System (SLS) Program to quantitatively estimate the effectiveness of proposed abort triggers so as to select the most effective suite to protect the astronauts from catastrophic failure of the SLS. The premise behind this process is to be able to quantitatively provide the value versus risk trade‐off for any given abort trigger, allowing decision makers to make more informed decisions. All current and planned crewed launch vehicles have some form of vehicle health management system integrated with an emergency launch abort system to ensure crew safety. While the design can vary, the underlying principle is the same: detect imminent catastrophic vehicle failure, initiate launch abort, and extract the crew to safety. Abort triggers are the detection mechanisms that identify that a catastrophic launch vehicle failure is occurring or is imminent and cause the initiation of a notification to the crew vehicle that the escape system must be activated. While ensuring that the abort triggers provide this function, designers must also ensure that the abort triggers do not signal that a catastrophic failure is imminent when in fact the launch vehicle can successfully achieve orbit. That is, the abort triggers must have low false negative rates to be sure that real crew‐threatening failures are detected, and also low false positive rates to ensure that the crew does not abort from non‐crew‐threatening launch vehicle behaviors. The analysis process described in this paper is a compilation of over six years of lessons learned and refinements from experiences developing abort triggers for NASA's Constellation Program (Ares I Project) and the SLS Program, as well as the simultaneous development of SHM/FM theory. The paper will describe the abort analysis concepts and process, developed in conjunction with SLS Safety and Mission Assurance (S&MA) to define a common set of mission phase, failure scenario, and Loss of Mission Environment (LOME) combinations upon which the SLS Loss of Mission (LOM) Probabilistic Risk Assessment (PRA) models are built. This abort analysis also requires strong coordination with the Multi‐Purpose Crew Vehicle (MPCV) and SLS Structures and Environments (STE) to formulate a series of abortability tables that encapsulate explosion dynamics over the ascent mission phase. The design and assessment of abort conditions and triggers to estimate their Loss of Crew (LOC) Benefits also requires in‐depth integration with other groups, including Avionics, Guidance, Navigation and Control(GN&C), the Crew Office, Mission Operations, and Ground Systems. The outputs of this analysis are a critical input to SLS S&MA's LOC PRA models. The process described here may well be the first full quantitative application of SHM/FM theory to the selection of a sensor suite for any aerospace system.

Lo, Yunnhon↗

Follow-up Imaging of Disk Candidates from the Disk Detective Citizen Science Project: New Discoveries and False Positives in WISE Circumstellar Disk Surveys

The Disk Detective citizen science project aims to find new stars with excess 22 m emission from circumstellar dust in the All WISE data release from the Wide-field Infrared Survey Explorer. We evaluated 261 Disk Detective objects of interest with imaging with the Robo-AO adaptive optics instrument on the 1.5 m telescope at Palomar Observatory and with RetroCam on the 2.5 m du Pont Telescope at Las Campanas Observatory to search for background objects at 0 15–12 separations from each target. Our analysis of these data leads us to reject 7% of targets. Combining this result with statistics from our online image classification efforts implies that at most7.9%±0.2% of All WISE-selected infrared excesses are good disk candidates. Applying our false-positive rates to other surveys, we find that the infrared excess searches of McDonald et al. and Marton et al. all have false-positiverates >70%. Moreover, we find that all 13 disk candidates in Theissen & West with W4 signal-to-noise ratio >3are false positives. We present 244 disk candidates that have survived vetting by follow-up imaging. Of these,213 are newly identified disk systems. Twelve of these are candidate members of comoving pairs based on Gaia astrometry, supporting the hypothesis that warm dust is associated with binary systems. We also note the discovery of 22 m excess around two known members of the Scorpius–Centaurus association, and we identifyknown disk host WISEA J164540.79-310226.6 as a likely Sco-Cen member. Thirty of these disk candidates arecloser than 125 pc (including 26 debris disks), making them good targets for both direct-imaging exoplanetsearches.

Wide-field Infrared Survey Explorer↗

Graph Neural Networks for low-energy event classification & reconstruction in IceCube

IceCube, a cubic-kilometer array of optical sensors built to detect atmospheric and astrophysical neutrinos between 1 GeV and 1 PeV, is deployed 1.45 km to 2.45 km below the surface of the ice sheet at the South Pole. The classification and reconstruction of events from the in-ice detectors play a central role in the analysis of data from IceCube. Reconstructing and classifying events is a challenge due to the irregular detector geometry, inhomogeneous scattering and absorption of light in the ice and, below 100 GeV, the relatively low number of signal photons produced per event. To address this challenge, it is possible to represent IceCube events as point cloud graphs and use a Graph Neural Network (GNN) as the classification and reconstruction method. The GNN is capable of distinguishing neutrino events from cosmic-ray backgrounds, classifying different neutrino event types, and reconstructing the deposited energy, direction and interaction vertex. Based on simulation, we provide a comparison in the 1 GeV–100 GeV energy range to the current state-of-the-art maximum likelihood techniques used in current IceCube analyses, including the effects of known systematic uncertainties. For neutrino event classification, the GNN increases the signal efficiency by 18% at a fixed background rate, compared to current IceCube methods. Alternatively, the GNN offers a reduction of the background (i.e. false positive) rate by over a factor 8 (to below half a percent) at a fixed signal efficiency. For the reconstruction of energy, direction, and interaction vertex, the resolution improves by an average of 13%–20% compared to current maximum likelihood techniques in the energy range of 1 GeV–30 GeV. The GNN, when run on a GPU, is capable of processing IceCube events at a rate nearly double of the median IceCube trigger rate of 2.7 kHz, which opens the possibility of using low energy neutrinos in online searches for transient events.

47 OTHER INSTRUMENTATION↗

The IPAC Image Subtraction and Discovery Pipeline for the Intermediate Palomar Transient Factory

We describe the near real-time transient-source discovery engine for the intermediate Palomar Transient Factory (iPTF), currently in operations at the Infrared Processing and Analysis Center (IPAC), Caltech. We coin this system the IPAC/iPTF Discovery Engine (or IDE). We review the algorithms used for PSF-matching, image subtraction, detection, photometry, and machine-learned (ML) vetting of extracted transient candidates. We also review the performance of our ML classifier. For a limiting signal-to-noise ratio of 4 in relatively unconfused regions, bogus candidates from processing artifacts and imperfect image subtractions outnumber real transients by approximately equal to 10:1. This can be considerably higher for image data with inaccurate astrometric and/or PSF-matching solutions. Despite this occasionally high contamination rate, the ML classifier is able to identify real transients with an efficiency (or completeness) of approximately equal to 97% for a maximum tolerable false-positive rate of 1% when classifying raw candidates. All subtraction-image metrics, source features, ML probability-based real-bogus scores, contextual metadata from other surveys, and possible associations with known Solar System objects are stored in a relational database for retrieval by the various science working groups. We review our efforts in mitigating false-positives and our experience in optimizing the overall system in response to the multitude of science projects underway with iPTF.

methods: analytical – methods: data analysis –↗

Error Correction Decoding for Underwater Acoustic Tag Detection

This paper presents a decoding method applied to a set of underwater acoustic tags that employ very short signal beacons periodically transmitted. The tags use the CRC8- Dallas linear block code for error detection originally. Though the CRC8-Dallas is quite weak, syndrome-based error correction is applied to correct the decoding errors. Meanwhile, multi-channel array receivers are utilized to find the common decodes of the tag IDs, and the list of the true tag IDs is used to test the detection accuracy and false alarm rate. The proposed syndrome decoding method is applied to over 2 million files recorded in field experiments. The results demonstrate that the syndrome method yields around 9.85% more true positive decodes at the cost of a 2 - 5 times increase in false positive rates. Further insights are also gained to enable future improvement of the detection algorithms

Hall, Joseph↗

A Bayesian approach to strong lens finding in the era of wide-area surveys

ABSTRACT The arrival of the Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST), Euclid-Wide and Roman wide-area sensitive surveys will herald a new era in strong lens science in which the number of strong lenses known is expected to rise from $\mathcal {O}(10^3)$ to $\mathcal {O}(10^5)$. However, current lens-finding methods still require time-consuming follow-up visual inspection by strong lens experts to remove false positives which is only set to increase with these surveys. In this work, we demonstrate a range of methods to produce calibrated probabilities to help determine the veracity of any given lens candidate. To do this we use the classifications from citizen science and multiple neural networks for galaxies selected from the Hyper Suprime-Cam survey. Our methodology is not restricted to particular classifier types and could be applied to any strong lens classifier which produces quantitative scores. Using these calibrated probabilities, we generate an ensemble classifier, combining citizen science, and neural network lens finders. We find such an ensemble can provide improved classification over the individual classifiers. We find a false-positive rate of 10−3 can be achieved with a completeness of 46 per cent, compared to 34 per cent for the best individual classifier. Given the large number of galaxy–galaxy strong lenses anticipated in LSST, such improvement would still produce significant numbers of false positives, in which case using calibrated probabilities will be essential for population analysis of large populations of lenses and to help prioritize candidates for follow-up.

79 ASTRONOMY AND ASTROPHYSICS↗

Hyper Spectral Anomaly Detection

Anomaly detection is a common machine learning (ML) task with growing importance in the fields of imaging, quality assurance, and multiple security related disciplines. Anomaly detection is more difficult than traditional machine learning methods due to the inherent unlabeled nature of the datasets. Existing anomaly detection architectures commonly face challenges with explainability, retaining information related to the relational structure of the data, and false positive rates. Hyperspectral Imaging Anomaly Detection (HSI) is a statistical model that employs vertex and edge weighted graphs to preserve the data’s relationships on different topographical scales. The model is able to generalize from anomaly detection in 2D images to novel datasets related to cyber-security. Furthermore, the use of multi-spectral and other filtering methods results in fewer false positives and increases the explainability of model predictions. When applying HSI to cyber-security datasets, we are able to successfully detect malicious activity with a relatively high degree of accuracy.

97 - MATHEMATICS AND COMPUTING↗