Search NASA⌕ Search

SEARCH · Search NASA

Results for “false positive analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Fault Management Algorithm Risk Assessment for the NASA Space Launch System

This paper presents the false positive (FP) and false negative (FN) risk assessment process currently being conducted for the Space Launch System (SLS) Artemis II Fault Management (FM) detection functions. The analysis scope, general assumptions and guide rules, and key modeling concepts were discussed to establish the basis of the risk assessments conducted. Initial analyses indicated a dominance in the total risk by software and firmware failures. This paper presents efforts applied to refine the software risks and the overall impact of implementing those modifications. Current analyses conducted on the detection functions implemented for the SLS Artemis II mission indicate primary risk drivers for the individual FM detection functions are flight software failures, firmware design failures, and hardware Common Cause Failures (CCFs). There still remains issues of how to account for time and redundancy in the software risk estimations.

probability risk analysis↗
NASA NTRSDate not supplied

Fault Management Algorithm Risk Assessment for the NASA Space Launch System

This presentation describes the false positive (FP) and false negative (FN) risk assessment process currently being conducted for the Space Launch System (SLS) Artemis II Fault Management (FM) detection functions. The analysis scope, general assumptions and guide rules, and key modeling concepts were discussed to establish the basis of the risk assessments conducted. Initial analyses indicated a dominance in the total risk by software and firmware failures. This paper presents efforts applied to refine the software risks and the overall impact of implementing those modifications. Current analyses conducted on the detection functions implemented for the SLS Artemis II mission indicate primary risk drivers for the individual FM detection functions are flight software failures, firmware design failures, and hardware Common Cause Failures (CCFs). There still remains issues of how to account for time and redundancy in the software risk estimations.

probability risk analysis↗
NASA NTRS2002

Reducing False Positives in Runtime Analysis of Deadlocks

This paper presents an improvement of a standard algorithm for detecting dead-lock potentials in multi-threaded programs, in that it reduces the number of false positives. The standard algorithm works as follows. The multi-threaded program under observation is executed, while lock and unlock events are observed. A graph of locks is built, with edges between locks symbolizing locking orders. Any cycle in the graph signifies a potential for a deadlock. The typical standard example is the group of dining philosophers sharing forks. The algorithm is interesting because it can catch deadlock potentials even though no deadlocks occur in the examined trace, and at the same time it scales very well in contrast t o more formal approaches to deadlock detection. The algorithm, however, can yield false positives (as well as false negatives). The extension of the algorithm described in this paper reduces the amount of false positives for three particular cases: when a gate lock protects a cycle, when a single thread introduces a cycle, and when the code segments in different threads that cause the cycle can actually not execute in parallel. The paper formalizes a theory for dynamic deadlock detection and compares it to model checking and static analysis techniques. It furthermore describes an implementation for analyzing Java programs and its application to two case studies: a planetary rover and a space craft altitude control system.

Bensalem, Saddek↗
NASA NTRS2015

Abort Trigger False Positive and False Negative Analysis Methodology for Threshold-Based Abort Detection

This paper describes a quantitative methodology for bounding the false positive (FP) and false negative (FN) probabilities associated with a human-rated launch vehicle abort trigger (AT) that includes sensor data qualification (SDQ). In this context, an AT is a hardware and software mechanism designed to detect the existence of a specific abort condition. Also, SDQ is an algorithmic approach used to identify sensor data suspected of being corrupt so that suspect data does not adversely affect an AT's detection capability. The FP and FN methodologies presented here were developed to support estimation of the probabilities of loss of crew and loss of mission for the Space Launch System (SLS) which is being developed by the National Aeronautics and Space Administration (NASA). The paper provides a brief overview of system health management as being an extension of control theory; and describes how ATs and the calculation of FP and FN probabilities relate to this theory. The discussion leads to a detailed presentation of the FP and FN methodology and an example showing how the FP and FN calculations are performed. This detailed presentation includes a methodology for calculating the change in FP and FN probabilities that result from including SDQ in the AT architecture. To avoid proprietary and sensitive data issues, the example incorporates a mixture of open literature and fictitious reliability data. Results presented in the paper demonstrate the effectiveness of the approach in providing quantitative estimates that bound the probability of a FP or FN abort determination.

risk assessment↗
DOE OSTI2024

Physics-inspired spatiotemporal-graph AI ensemble for the detection of higher order wave mode signals of spinning binary black hole mergers

We present a new class of AI models for the detection of quasi-circular, spinning, non-precessing binary black hole mergers whose waveforms include the higher order gravitational wave modes ($\ell$, |m|) = {(2,2), (2,1), (3,3), (3,2), (4,4)}, and mode mixing effects in the $\ell$ = 3, |m| = 2 harmonics. These AI models combine hybrid dilated convolution neural networks to accurately model both short- and long-range temporal sequential information of gravitational waves; and graph neural networks to capture spatial correlations among gravitational wave observatories to consistently describe and identify the presence of a signal in a three detector network encompassing the Advanced LIGO and Virgo detectors. We first trained these spatiotemporal-graph AI models using synthetic noise, using 1.2 million modeled waveforms to densely sample this signal manifold, within 1.7 h using 256 NVIDIA A100 GPUs in the Polaris supercomputer at the Argonne Leadership Computing Facility. This distributed training approach exhibited optimal classification performance, and strong scaling up to 512 NVIDIA A100 GPUs. With these AI ensembles we processed data from a three detector network, and found that an ensemble of 4 AI models achieves state-of-the-art performance for signal detection, and reports two misclassifications for every decade of searched data. We distributed AI inference over 128 GPUs in the Polaris supercomputer and 128 nodes in the Theta supercomputer, and completed the processing of a decade of gravitational wave data from a three detector network within 3.5 h. Finally, we fine-tuned these AI ensembles to process the entire month of February 2020, which is part of the O3b LIGO/Virgo observation run, and found 6 gravitational waves, concurrently identified in Advanced LIGO and Advanced Virgo data, and zero false positives. This analysis was completed in one hour using one NVIDIA A100 GPU.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

DriveSense: A Noise-Resilient Framework for Driving Mode Identification

Accurate drive mode classification is essential for enhancing the reliability and predictive maintenance of heavy-duty electric trucks. This study proposes a novel fuzzy logic-based framework, DriveSense, for real-time drive mode classification, addressing key challenges such as sensor noise, transitional behaviors, and computational efficiency. The proposed approach integrates a two-stage filtering pipeline, combining adaptive outlier removal and a dynamic Kalman filter to enhance data quality. A fuzzy inference system with smoothened trapezoidal membership functions is then applied to classify driving modes into standstill, constant speed, acceleration, and deceleration while mitigating the effects of noise and edge cases. Performance evaluation using real-world and simulated drive cycles demonstrates significant improvements in classification accuracy (up to 97.8%), F1-score (up to 0.97), and robustness against noise, while reducing false positives. Comparative analysis against baseline models, demonstrates DriveSense’s superior accuracy and generalizability across diverse driving patterns. The framework’s lightweight and interpretable fuzzy inference engine operates with low computational latency, ensuring compatibility with real-time embedded systems typical of heavy-duty electric trucks. Moreover, DriveSense models transitional behaviors through overlapping fuzzy sets and adaptive borderline classification logic, enabling smooth identification of subtle shifts such as rolling stops or gradual deceleration. These results highlight DriveSense’s potential to enhance predictive maintenance strategies, reduce downtime, and support scalable, fleet-wide diagnostics.

Kumar, Praveen [Oak Ridge National Laboratory (ORN↗
NASA NTRS2023

SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model

As aviation systems continue to operate with high traffic, large amounts of documents containing safety-relevant data continue to be generated via reporting systems such as the ASRS. Advanced natural language processing techniques, specifically pre-trained language models, have shown great success in domain-specific applications; however, the text in aviation safety reports is inundated with jargon and thus not fully utilized by general pre-trained models. In this research, we work towards developing a safety-informed aerospace-specific language model by pre-training a Bidirectional Encoder Representations from Transformer (BERT) model on reports from the Aviation Safety Reporting System and the National Transportation Safety Board. The resulting model, called SafeAeroBERT, is fine-tuned for the specific task of document classification, and can be further tuned for named-entity recognition, relation detection, information retrieval, and summarization. Results from the classification task are compared between SafeAeroBERT, the base BERT, and SciBERT models and show SafeAeroBERT outperforms the general BERT and SciBERT on classifying reports about human factors, aircraft, and procedure. SafeAeroBERT can be used on custom tasks, not limited to document classification, and is intended to aid an intelligent knowledge manager for safety report repositories.

Aviation↗
NASA NTRS2019

Assessing Reliability of NDE Flaw Detection Using Smaller Number of Demonstration Data Points

The paper provides an engineering analysis approach for assessing reliability of NDE flaw detection using smaller number of demonstration data points. It explores dependence of probability of detection (POD), probability of false positive (POF), on contrast-to-noise ratio, and net decision threshold-to-noise ratio in a simulated data; and draws some generically applicable inferences to devise the approach. ASTM nondestructive evaluation standards provide requirements on signal-to-noise ratio and/or contrast-to-noise ratio in order to provide reliable flaw detection and limit false positive calls. POD analysis of inspection test data results in an estimated flaw size, denoted by 𝑎90/95. This flaw size has 90% POD and minimum 95% confidence. POF is also estimated in the analysis. POD demonstration requires specimens with flaws of known size. In many situations, it is very expensive to produce the large number of flaws required for the POD analysis. In some situations, only real flaws can truly represent the flaws for demonstration. Real flaws of correct size and location in part configuration specimen may be difficult to produce, if not impossible. Here, an engineering analysis approach is devised using simulation to assess reliability of NDE technique when a limited number of flaws are available for demonstration. In this simulation, a technique is considered reliable, if it provides flaw detectability size equal to or better than the theoretical 𝑎90𝑡ℎ used in simulation and also provides a POF less than or equal to a chosen value. The paper uses simulated signal response versus flaw size data to devise the approach. Linear correlation is used between the signal response data and flaw size. POD software mh1823 uses generalized linear model (GLM) in POD analysis after transforming the flaw size and signal response, if needed, using logarithm. Therefore, this approach is in agreement with the linear signal correlation used in mh1823. Using the POD analysis of data, generic conditions on contrast-to-noise ratio and net decision threshold-to-noise ratio are derived for reliable flaw detection. In order to assess technique reliability using the engineering approach, signal response-to-flaw size correlation about the flaw size of concern is needed. In addition, measurement of noise is also needed. If the technique meets the above requirements, assumption of linear signal-to-flaw size correlation and conditions on noise, then the technique can be assessed using this analysis as it fits the underlying POD model used here. The approach is conservative and is designed to provide a larger flaw size compared to the POD approach. Such NDE technique assessment approach, although, not as rigorous as POD, can be cost effective if the larger flaw size can be tolerated. Typically, this is a situation for all quality control NDE inspections. Here, an NDE technique needs to be reliable and 𝑎90/95 is not estimated, but the assessed flaw size is assumed to be larger than the unknown a90 due to conservative factors or margins. Applicability of the approach for assessing reliability of flaw detection in x-ray radiography and 2D imaging in general is also explored.

Koshti, Ajay M.↗
DOE OSTI2026

Mono-mix strategy enables comparative proteomics of a cross-kingdom microbial symbiosis

Cross-kingdom microbial symbioses, such as those between algae and bacteria, are key players in biogeochemical cycles. The molecular changes during initiation and establishment of symbiosis are of great interest, but quantitatively monitoring such changes can be challenging, particularly when the microorganisms differ greatly in size or are intimately associated. Here, we analyze output from label-free, data-dependent acquisition (DDA) LC-MS/MS proteomics experiments investigating the well-studied interaction between the alga Chlamydomonas reinhardtii and the heterotrophic bacterium Mesorhizobium japonicum. We found that detection of bacterial proteins decreased in coculture by 50% proteome-wide due to the abundance of algal proteins. As a result, standard differential expression analysis led to numerous false-positive reports of significantly downregulated proteins, where it was not possible to distinguish meaningful biological responses to symbiosis from artifacts of the reduced protein detection in coculture relative to monoculture. We show that data normalization alone does not eliminate the impact of altered detection on differential expression analysis of the cross-kingdom symbiosis. We assessed two additional strategies to overcome this methodological artifact inherent to DDA proteomics. In the first, we combined algal and bacterial monocultures at a relative abundance that mimicked the coculture, creating a “mono-mix” control to which the coculture could be compared. This approach enabled comparable detection of bacterial proteins in the coculture and the monoculture control. In the second strategy, we enhanced detection of lowly abundant bacterial proteins by using sample fractionation upstream of LC-MS/MS analysis. When these simple approaches were combined, they allowed for meaningful comparisons of nearly 10,000 algal proteins and over 4,000 bacterial proteins in response to symbiosis by DDA. They successfully recovered expected changes in the bacterial proteome in response to algal coculture, including upregulation of sugar-binding proteins and transporters. They also revealed novel proteomic responses to coculture that guide hypotheses about algal-bacterial interactions.

Dupuis, Sunnyjoy [University of California, Berkel↗
DOE OSTI2024

Accelerating cavity fault prediction using deep learning at Jefferson Laboratory

Abstract Accelerating cavities are an integral part of the continuous electron beam accelerator facility (CEBAF) at Jefferson Laboratory. When any of the over 400 cavities in CEBAF experiences a fault, it disrupts beam delivery to experimental user halls. In this study, we propose the use of a deep learning model to predict slowly developing cavity faults. By utilizing pre-fault signals, we train a long short-term memory-convolutional neural network binary classifier to distinguish between radio-frequency (RF) signals during normal operation and RF signals indicative of impending faults. We optimize the model by adjusting the fault confidence threshold and implementing a multiple consecutive window criterion to identify fault events, ensuring a low false positive rate. Results obtained from analysis of a real dataset collected from the accelerating cavities simulating a deployed scenario demonstrate the model’s ability to identify normal signals with 99.99% accuracy and correctly predict 80% of slowly developing faults. Notably, these achievements were achieved in the context of a highly imbalanced dataset, and fault predictions were made several hundred milliseconds before the onset of the fault. Anticipating faults enables preemptive measures to improve operational efficiency by preventing or mitigating their occurrence.

43 PARTICLE ACCELERATORS↗
NASA NTRS2004

Applying Jlint to Space Exploration Software

Java is a very successful programming language which is also becoming widespread in embedded systems, where software correctness is critical. Jlint is a simple but highly efficient static analyzer that checks a Java program for several common errors, such as null pointer exceptions, and overflow errors. It also includes checks for multi-threading problems, such as deadlocks and data races. The case study described here shows the effectiveness of Jlint in find-false positives in the multi-threading warnings gives an insight into design patterns commonly used in multi-threaded code. The results show that a few analysis techniques are sufficient to avoid almost all false positives. These techniques include investigating all possible callers and a few code idioms. Verifying the correct application of these patterns is still crucial, because their correct usage is not trivial.

Artho, Cyrille↗
NASA NTRS2017

Reducing V&V Cost of Flight Critical Systems: Myth or Reality?

This paper presents an overview of NASA research program on the V&V of flight critical systems. Five years ago, NASA started an effort to reduce the cost and possibly increase the effectiveness of V&V for flight critical systems. It is the right time to take a look back and realize what progress has been made. This paper describes our overall approach and the tools introduced to address different phases of the software lifecycle. For example, we have improved testing by developing a statistical learning approach tor defining test cases. The tool automatically identifies possible unsafe conditions by analyzing outliers in output data; using an iterative learning process, it can then generate more test cases that represent potentially unsafe regions of operation. At the code level, we have developed and made available as open source a static analyzer for C and C++ programs called IKOS. We have shown that IKOS is very precise in the analysis of embedded C programs (very few false positives) and a bit less for regular C and C++ code. At the design level, in collaboration with our NRA partners, we have developed a suite of analysis tools for Simulink models. The analysis is done in a compositional framework for scalability.

Brat, Guillaume P.↗
DOE OSTI2026

Image processing pipeline for AI-driven nanoparticle megalibrary characterization

Recent innovations have made it possible to produce megalibraries, millions of structurally and compositionally distinct nanoparticles on a chip. These megalibraries yield vast volumes of data that are impossible to analyze manually, necessitating the development of automated tools. In previous work, we created a binary classification machine learning model to select quality nanoparticle images for downstream analysis. In this work, we show that adding a custom image processing step before training can produce significantly higher-performing models in a fraction of the time and make them more robust to different image noise levels and microscope acquisition settings. The image processing pipeline proposed here effectively cleans raw nanoparticle images, enhances key features, and allows us to use much lower resolution images and simpler neural network model architectures. These features result in higher performance and significant cost savings. Experiments demonstrate superior performance relative to baseline, including an 18.2% improvement in recall and a 13.1% increase in accuracy. Given the high cost of downstream analysis, it is critical to minimize false positives, and our best-performing model reaches a precision of 95.9% and a weighted F-score of 95.1% on an unseen test set. Additionally, model training time is reduced from hours to less than a minute. We also show that, using this custom image processing pipeline, model performance is significantly improved at lower pixel resolutions compared to downsizing alone. We expect that adopting this pipeline for AI-driven automated nanoparticle characterization will allow researchers to rapidly and accurately analyze much greater volumes of data, thereby accelerating materials discovery.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
NASA NTRSDate not supplied

Investigating the Use of Machine Learning (ML) to Assess Tropospheric Doppler Radar Wind Profiler (TDRWP) Data Quality

Manual Quality Control (MQC) of Tropospheric Doppler Radar Wind Profiler (TDRWP) data is essential for defining an accurate climatology for downstream aerospace vehicle assessments. MQC traditionally takes around 30.5 hours per year of radar data. The Marshall Space Flight Center Natural Environments Branch (MSFC NE) used machine learning (ML) to test the feasibility of automating the MQC process, showing a potential to reduce labor by 300%. However, analysis of the model showed some false positives. We compared a neural network to the model to validate it and develop a process for assessing comparable solutions in the future.

Corey Walker↗
NASA NTRS2010

Multi-Stage System for Automatic Target Recognition

A multi-stage automated target recognition (ATR) system has been designed to perform computer vision tasks with adequate proficiency in mimicking human vision. The system is able to detect, identify, and track targets of interest. Potential regions of interest (ROIs) are first identified by the detection stage using an Optimum Trade-off Maximum Average Correlation Height (OT-MACH) filter combined with a wavelet transform. False positives are then eliminated by the verification stage using feature extraction methods in conjunction with neural networks. Feature extraction transforms the ROIs using filtering and binning algorithms to create feature vectors. A feedforward back-propagation neural network (NN) is then trained to classify each feature vector and to remove false positives. The system parameter optimizations process has been developed to adapt to various targets and datasets. The objective was to design an efficient computer vision system that can learn to detect multiple targets in large images with unknown backgrounds. Because the target size is small relative to the image size in this problem, there are many regions of the image that could potentially contain the target. A cursory analysis of every region can be computationally efficient, but may yield too many false positives. On the other hand, a detailed analysis of every region can yield better results, but may be computationally inefficient. The multi-stage ATR system was designed to achieve an optimal balance between accuracy and computational efficiency by incorporating both models. The detection stage first identifies potential ROIs where the target may be present by performing a fast Fourier domain OT-MACH filter-based correlation. Because threshold for this stage is chosen with the goal of detecting all true positives, a number of false positives are also detected as ROIs. The verification stage then transforms the regions of interest into feature space, and eliminates false positives using an artificial neural network classifier. The multi-stage system allows tuning the detection sensitivity and the identification specificity individually in each stage. It is easier to achieve optimized ATR operation based on its specific goal. The test results show that the system was successful in substantially reducing the false positive rate when tested on a sonar and video image datasets.

Chao, Tien-Hsin↗
DOE OSTI2025

Improving Cyber Situational Understanding

Effective cybersecurity operations require the ability to analyze large amounts of information to assess security risks and formulate defensive strategies against adversaries. This has become more complex in recent years as the sprawl and interconnectivity of devices grows through implementation of virtualization, cloud computing, and Internet of Things (IoT). The amount of data and analysis required for effective cybersecurity command and control decisions far exceeds humans’ capacity to perform manually. We characterize the analysis problem as cyber situational understanding. The research presented to improve cyber situational understanding focuses on vulnerability analysis and threat intelligence. Regarding vulnerabilities, entities must analyze and plan work for between thousands and tens of thousands of software vulnerabilities annually. Entities heavily use network firewalls to limit vulnerability exposure. As a result, some of these vulnerabilities permit exposure to adversarial exploitation, whereas others are inaccessible and therefore present negligible risk of exploitation. Distinguishing between high and low risk software vulnerabilities requires a deep understanding of the vulnerability, network firewall protection, and characteristics of the targeted device. This problem is solved by extracting network service features from vulnerability data features using both machine-learning and natural language processing. Then, the network firewall topology is parsed to determine which vulnerabilities are reachable by adversaries. Ultimately, a state-based safety analysis ascertains which vulnerabilities are unsafe. A related vulnerability analysis problem occurs in cybersecurity operations when associating an entity’s hardware and software assets to public vulnerability databases. Assets often reveal hardware and software through installation artifacts and network service identification, and entities store these artifacts in inventory databases. However, software and hardware vendors apply a standard Common Platform Enumeration (CPE) naming convention when publicly reporting vulnerabilities. Associating these two datasets often requires many hours to days of manual inspection. The proposed solution automates the mapping approach of human analysts using fuzzy matching techniques, natural language processing, and, ultimately, machine learning to present a small set of recommendations for mapping the two datasets. The result significantly reduces human analysis time and reduces the occurrence of false positives in vulnerability notifications. Finally, cyber threat intelligence (CTI) requires associating cyber observable artifacts, such as IP addresses, URIs, and file hashes, with cyber threat tactics, techniques, and procedures. Unfortunately, most CTI data is compartmentalized across multiple organizations and cannot be shared due to the legal and reputational risk with cyber threat being associated with the entity. The approach to solving this problem inovlves using a distributed ledger with anonymous token spending and authentication. This allows a consortium of semi-trusted entities to share the workload of curating CTI for a threat sharing community’s cooperative benefit.

Huff, Philip↗
NASA NTRS2010

Ares I-X Ground Diagnostic Prototype

Automating prelaunch diagnostics for launch vehicles offers three potential benefits. First, it potentially improves safety by detecting faults that might otherwise have been missed so that they can be corrected before launch. Second, it potentially reduces launch delays by more quickly diagnosing the cause of anomalies that occur during prelaunch processing. Reducing launch delays will be critical to the success of NASA's planned future missions that require in-orbit rendezvous. Third, it potentially reduces costs by reducing both launch delays and the number of people needed to monitor the prelaunch process. NASA is currently developing the Ares I launch vehicle to bring the Orion capsule and its crew of four astronauts to low-earth orbit on their way to the moon. Ares I-X will be the first unmanned test flight of Ares I. It is scheduled to launch on October 27, 2009. The Ares I-X Ground Diagnostic Prototype is a prototype ground diagnostic system that will provide anomaly detection, fault detection, fault isolation, and diagnostics for the Ares I-X first-stage thrust vector control (TVC) and for the associated ground hydraulics while it is in the Vehicle Assembly Building (VAB) at John F. Kennedy Space Center (KSC) and on the launch pad. It will serve as a prototype for a future operational ground diagnostic system for Ares I. The prototype combines three existing diagnostic tools. The first tool, TEAMS (Testability Engineering and Maintenance System), is a model-based tool that is commercially produced by Qualtech Systems, Inc. It uses a qualitative model of failure propagation to perform fault isolation and diagnostics. We adapted an existing TEAMS model of the TVC to use for diagnostics and developed a TEAMS model of the ground hydraulics. The second tool, Spacecraft Health Inference Engine (SHINE), is a rule-based expert system developed at the NASA Jet Propulsion Laboratory. We developed SHINE rules for fault detection and mode identification. The prototype uses the outputs of SHINE as inputs to TEAMS. The third tool, the Inductive Monitoring System (IMS), is an anomaly detection tool developed at NASA Ames Research Center and is currently used to monitor the International Space Station Control Moment Gyroscopes. IMS automatically "learns" a model of historical nominal data in the form of a set of clusters and signals an alarm when new data fails to match this model. IMS offers the potential to detect faults that have not been modeled. The three tools have been integrated and deployed to Hangar AE at KSC where they interface with live data from the Ares I-X vehicle and from the ground hydraulics. The outputs of the tools are displayed on a console in Hangar AE, one of the locations from which the Ares I-X launch will be monitored. The full paper will describe how the prototype performed before the launch. It will include an analysis of the prototype's accuracy, including false-positive rates, false-negative rates, and receiver operating characteristics (ROC) curves. It will also include a description of the prototype's computational requirements, including CPU usage, main memory usage, and disk usage. If the prototype detects any faults during the prelaunch period then the paper will include a description of those faults. Similarly, if the prototype has any false alarms then the paper will describe them and will attempt to explain their causes.

Schwabacher, Mark↗
NASA NTRS2016

Evaluating Alerting and Guidance Performance of a UAS Detect-And-Avoid System

A key challenge to the routine, safe operation of unmanned aircraft systems (UAS) is the development of detect-and-avoid (DAA) systems to aid the UAS pilot in remaining "well clear" of nearby aircraft. The goal of this study is to investigate the effect of alerting criteria and pilot response delay on the safety and performance of UAS DAA systems in the context of routine civil UAS operations in the National Airspace System (NAS). A NAS-wide fast-time simulation study was conducted to assess UAS DAA system performance with a large number of encounters and a broad set of DAA alerting and guidance system parameters. Three attributes of the DAA system were controlled as independent variables in the study to conduct trade-off analyses: UAS trajectory prediction method (dead-reckoning vs. intent-based), alerting time threshold (related to predicted time to LoWC), and alerting distance threshold (related to predicted Horizontal Miss Distance, or HMD). A set of metrics, such as the percentage of true positive, false positive, and missed alerts, based on signal detection theory and analysis methods utilizing the Receiver Operating Characteristic (ROC) curves were proposed to evaluate the safety and performance of DAA alerting and guidance systems and aid development of DAA system performance standards. The effect of pilot response delay on the performance of DAA systems was evaluated using a DAA alerting and guidance model and a pilot model developed to support this study. A total of 18 fast-time simulations were conducted with nine different DAA alerting threshold settings and two different trajectory prediction methods, using recorded radar traffic from current Visual Flight Rules (VFR) operations, and supplemented with DAA-equipped UAS traffic based on mission profiles modeling future UAS operations. Results indicate DAA alerting distance threshold has a greater effect on DAA system performance than DAA alerting time threshold or ownship trajectory prediction method. Further analysis on the alert lead time (time in advance of predicted loss of well clear at which a DAA alert is first issued) indicated a strong positive correlation between alert lead time and DAA system performance (i.e. the ability of the UAS pilot to maneuver the unmanned aircraft to remain well clear). While bigger distance thresholds had beneficial effects on alert lead time and missed alert rate, it also generated a higher rate of false alerts. In the design and development of DAA alerting and guidance systems, therefore, the positive and negative effects of false alerts and missed alerts should be carefully considered to achieve acceptable alerting system performance by balancing false and missed alerts. The results and methodology presented in this study are expected to help stakeholders, policymakers and standards committees define the appropriate setting of DAA system parameter thresholds for UAS that ensure safety while minimizing operational impacts to the NAS and equipage requirements for its users before DAA operational performance standards can be finalized.

UAS Detect-and-Avoid (DAA) System↗