Statistical Outlier Detection (SOD): A computer program for detecting outliers in data
There are no author-identified significant results in this report.
Chhikara, R. S.↗
SEARCH · Search NASA
Results for “Outlier detection”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Robust statistical methods for automated outlier detection
The computational challenge of automating outlier, or blunder point, detection in radio metric data requires the use of nonstandard statistical methods because the outliers have a deleterious effect on standard least squares methods. The particular nonstandard methods most applicable to the task are the robust statistical techniques that have undergone intense development since the 1960s. These new methods are by design more resistant to the effects of outliers than standard methods. Because the topic may be unfamiliar, a brief introduction to the philosophy and methods of robust statistics is presented. Then the application of these methods to the automated outlier detection problem is detailed for some specific examples encountered in practice.
Jee, J. R.↗
pyFLANK, a graph neural network based null distribution inference model for F ST outlier detection
Detecting genomic regions under selection is essential for understanding how populations adapt to different environments, yet it remains challenging due to the confounding effects of demographic history and linkage disequilibrium (LD). Fixation index (F ST ) is a widely used statistic to identify genomic regions under adaptation. However, identifying genes under selection by defining F ST outliers often remains challenging, owing to confounding effects of underlying demographic history. Traditional methods assume independence among loci and rely on simple demographic models, while newer models perform much better but are computationally expensive and not easily scalable. Here, we present pyFLANK, an open-source and automated Python implementation which detects F ST outliers using a null distribution inferred from quasi-independent loci. Our tool integrates three approaches to identify loci obeying a null distribution: graph neural network (GNN) inference, linkage disequilibrium (LD)-based inference, and user-defined input. Because pyFLANK uses GNN-based inference of quasi-independent loci, it yields a more accurate null model with less need for user parameter input. In simulation experiments, pyFLANK achieved lower false positive rates than current methods while maintaining comparable detection power, indicating that its refined null model better distinguishes true adaptive loci from background variation. The GNN-based model, in particular, detected additional loci associated with phenotypic variance that were not identified by existing methods. Assessments of simulation and real data from different species demonstrate that pyFLANK achieves lower false positive rates compared with other commonly used F ST outlier detectors, while maintaining comparable detection power and excellent computational performance, providing a robust and user-friendly tool for identifying loci under divergent selection. It extends existing F ST outlier frameworks by incorporating explicit LD-aware strategies for null model calibration. The method is intended as a practical and scalable complement to existing genome scan approaches.
FST↗
System and Method for Outlier Detection via Estimating Clusters
An efficient method and system for real-time or offline analysis of multivariate sensor data for use in anomaly detection, fault detection, and system health monitoring is provided. Models automatically derived from training data, typically nominal system data acquired from sensors in normally operating conditions or from detailed simulations, are used to identify unusual, out of family data samples (outliers) that indicate possible system failure or degradation. Outliers are determined through analyzing a degree of deviation of current system behavior from the models formed from the nominal system data. The deviation of current system behavior is presented as an easy to interpret numerical score along with a measure of the relative contribution of each system parameter to any off-nominal deviation. The techniques described herein may also be used to "clean" the training data.
Iverson, David J.↗
Visualisation and outlier detection for probability density function ensembles
Abstract Exploratory data analysis (EDA) for functional data—data objects where observations are entire functions—is a difficult problem that has seen significant attention in recent literature. This surge in interest is motivated by the ubiquitous nature of functional data, which are prevalent in applications across fields such as meteorology, biology, medicine and engineering. Empirical probability density functions (PDFs) can be viewed as constrained functional data objects that must integrate to one and be nonnegative. They show up in contexts such as yearly income distributions, zooplankton size structure in oceanography and in connectivity patterns in the brain, among others. While PDF data are certainly common in modern research, little attention has been given to EDA specifically for PDFs. In this paper, we extend several methods for EDA on functional data for PDFs and compare them on simulated data that exhibit different types of variation, designed to mimic that seen in real‐world applications. We then use our new methods to perform EDA on the breakthrough curves observed in gas transport simulations for underground fracture networks.
97 MATHEMATICS AND COMPUTING↗
A simple data-driven level finding method of many-electron atoms and heavy nuclei based on statistical outlier detection
Here, we report a simple and pure data-driven method to find new energy levels of quantum many-body systems only from observed line wavelengths. In our method, all the possible combinations are computed from known energy levels and wavelengths of unidentified lines. As each excited state exhibits many transition lines to different lower levels, the true levels should be reconstructed coincidentally from many level-line combinations, while the wrong combinations distribute randomly. Such a coincidence can be easily detected statistically. We demonstrate this statistical method by finding new levels for various atomic and nuclear systems from unidentified line lists available online.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Machine Learning for analysis of speckle dynamics: quantification and outlier detection
Not Available
Konstantinova, Tatiana (ORCID:0000000197364246)↗
BROOD: Bilevel and Robust Optimization and Outlier Detection for Efficient Tuning of High-Energy Physics Event Generators
The parameters in Monte Carlo (MC) event generators are tuned on experimental measurements by evaluating the goodness of fit between the data and the MC predictions. The relative importance of each measurement is adjusted manually in an often time-consuming, iterative process to meet different experimental needs. In this work, we introduce several optimization formulations and algorithms with new decision criteria for streamlining and automating this process. These algorithms are designed for two formulations: bilevel optimization and robust optimization. Both formulations are applied to the datasets used in the ATLAS A14 tune and to the dedicated hadronization datasets generated by the SHERPA generator, respectively. The corresponding tuned generator parameters are compared using three metrics. We compare the quality of our automatic tunes to the published ATLAS A14 tune. Moreover, we analyze the impact of a pre-processing step that excludes data that cannot be described by the physics models used in the MC event generators.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Advancing Vision-based Feedback and Convolutional Neural Networks for Visual Outlier Detection
Machine learning has matured into a technology that has immediate applicability to the surveillance needs of nuclear material storage containers. These containers at LANL are the barrier preventing release of radioactive material to the workers, public, and environment during the storage period of the material. Annual surveillance activities can only provide coverage on a handful of containers. There is a significant need for surveillance tools to identify potential issues and precursors to containment failure that can be used during opportunistic inspections and, more generally, outside of annual surveillance activities. In this report we provide details on the advancement of our proposed embodiment that combines an automation system for taking pictures and a high-accuracy machine learning-driven object detection software. We further showcase the improvements on the software side with progress on extracting unique identification features and advances in detecting damage. The current state of the system captures subject matter expert training and a space-conscious design whose implementation is envisioned in the near future.
98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
Cluster-Based Flight Trajectory Outlier Detection
Given a set of flight trajectories, can we classify the trajectories that do not follow a normal path? By identifying abnormal trajectories, further analysis can be done to determine the reasoning for these actions. Addressing these scenarios can bring possible solutions for holding and rerouting problems when its time to incorporate UAM in the airspace.
Machine Learning↗
Outlier Detection and Removal in Multivariate Time Series for a More Robust Machine Learning–based Solar Flare Prediction
Explore the source record for details and available documents.
Junzhi Wen↗
Defining blood hematology reference values in female pig-tailed macaques ( Macaca nemestrina ) using the Isolation Forest algorithm
Background: Pig-tailed macaques (PTMs) are commonly used as preclinical models to assess antiretroviral drugs for HIV prevention research. Drug toxicities and disease pathologies are often preceded by changes in blood hematology. To better assess the safety profile of pharmaceuticals, we defined normal ranges of hematological values in PTMs using an Isolation Forest (iForest) algorithm. Methods: Eighteen female PTMs were evaluated. Blood was collected 1–24 times per animal for a total of 159 samples. Complete blood counts were performed, and iForest was used to analyze the hematology data to detect outliers. Results: Median, IQR, and ranges were calculated for 13 hematology parameters. From all samples, 22 outliers were detected. These outliers were excluded from the reference index. Conclusions: Using iForest, we defined a normal range for hematology parameters in female PTMs. This reference index can be a valuable tool for future studies evaluating drug toxicities in PTMs.
59 BASIC BIOLOGICAL SCIENCES↗
Identifying Outliers in AI-based Image Compression
Image compression using artificial intelligence (AI) is becoming increasingly prevalent across various fields, including scientific research. Scientific instruments can generate hundreds of images per second, and effectively compressing these images with high compression ratios is crucial for facilitating scientific discoveries. However, automatically detecting outlier cases, where compression may not have succeeded or where interesting scientific phenomena are present, poses a significant challenge. To address this, we have developed a methodology based on unsupervised machine learning techniques for detecting outlier compressed images. This methodology utilizes metrics such as peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), structural texture similarity index measure (STSIM), and deep image and structural texture similarity index (DISTS). We have evaluated our methodology on several unlabeled datasets, including microscopy and x-ray images, and have successfully identified multiple outlier images using our proposed approach. Furthermore, our approach has enabled us to identify image semantics that are valuable for post-experiment analysis by scientists.
Data Analysis↗
A statistical test procedure for detecting multiple outliers in a data set
There are no author-identified significant results in this report.
Chhikara, R. S.↗
Utilization of Machine Learning Techniques for Managing the Tracking and Data Relay Satellite Constellation
National Aeronautics and Space Administration’s (NASA) Goddard Space Flight Center (GSFC) operates a constellation of ten geosynchronous Tracking and Data Relay Satellites (TDRS). The TDRS constellation consists of multiple geosynchronous communication relay satellites located around the equator so they can provide continual coverage of any mission in low earth orbit. The TDRS are located primarily in three oceanic regions around the earth. NASA’s White Sands Complex provides the ground communication support for TDRS located over the Atlantic and Pacific Oceans. Another TDRS ground station in Guam supports the TDRS over the Indian Ocean. With these satellites the TDRS network can provide continuous coverage of satellites in low-earth orbit. The NASA Space Network (SN) project office at GSFC manages the constellation of spacecraft. Major customers of the TDRS constellation include, but are not limited to, the International Space Station and the Hubble Space Telescope. The TDRS constellation has three generations of satellites and has been active for over 30 years providing reliable communication links between customer satellites and corresponding ground stations. However, one of the major concerns for TDRS, and in any space mission, is to ensure the health and safety of the spacecraft. Generally, engineers use telemetry data to monitor and analyze the performance and state of health of the spacecraft. Telemetry data contains hundreds of parameters that monitor each important component in the spacecraft, which can be utilized to recognize and characterize the behavior of the spacecraft. Each parameter contains considerable information to represent time-dependent properties of each spacecraft subsystem and component. During the entire life of a TDRS spacecraft, thousands of gigabytes of telemetry data are transmitted in real-time from the spacecraft to the ground station at the White Sands Complex in Las Cruces, New Mexico, and recorded as historical data sets for engineers to process and analyze the events that occurred on-orbit. These parameters contain the function of multiple spacecraft subsystems, such as the attitude control system (ACS), Thermal, Electrical Power Subsystem (EPS), etc. . The first and second generations have exceeded their required lifetime and NASA is keen to manage these spacecrafts carefully in order to maximize the remaining life using the spacecraft telemetry. The challenge is to know when the risk of losing a spacecraft in geosynchronous orbit exceeds the benefit of continued operations for customer support. In the TDRS fleet, the EPS is the most critical subsystem related to spacecraft operations. Failure of the EPS would strand a spacecraft in geosynchronous orbit. Since EPS provides power to the spacecraft, component failures ultimately lead to the inability to support the spacecraft loads and the communications payload. For instance, TDRS-8 has several anomalies in EPS including the Bus Voltage Limiter (BVL) shunt current, solar array loss of circuits, and failed battery cells. Any of these anomalies can cause critical issues to the spacecraft. Therefore, developing a system to analyze and perform early detection of a potential anomaly is an important issue in telemetry data analysis. In recent years, Telemetry Mining (TM) has been proposed to process telemetry data by using Data Mining (DM) techniques such as classification, clustering, regression and anomaly detection. Anomaly detection, also known as outlier detection, has been widely used in many data mining areas such as remote sensing, medical data processing and digital image processing. The goal of anomaly detection is to detect abnormal data, which contains a relatively low probability of occurrence among the entire data set. Early detection of anomalies is one of the most significant issues in managing the spacecraft configuration. If anomalies can be detected early enough, then the redundant resources can be used to extend the life of the operational spacecraft. We present an unsupervised anomaly detection method to process the EPS data extracted from TDRS-8. This is different from traditional analytical methods, which use telemetry data to illustrate behavior and physical meaning of each spacecraft component. TM connects multiple parameters as a vector and then conducts data analysis on this high dimension telemetry vector. This method is looking at the properties of a high dimensional vector that is able to consider the relationship between different parameters in the anomaly detection problem. This kind of method performs much better than the traditional limit checking method. In addition, we propose a new approach of real-time anomaly detection to process telemetry data in real-time, which can then be applied to spacecraft monitoring with high reliability, low cost and high accuracy.
Machine Learning (ML)↗
Multi-area parameter error identification for large power systems
Power grid model parameters may contain errors due to various reasons. Detecting and correcting parameter errors typically requires significant computational effort due to the size and complexity of the parameter database. While the normalized Lagrange multiplier (NLM) method can effectively detect, identify and correct parameter errors, its computational burden could rapidly grow with increasing system size. This paper addresses this issue by proposing a multi-area parameter error identification method. Each area has its own outlier detection tool for detecting the incorrect parameters and measurements within the area. On the other hand, due to the reduced redundancy at area boundaries, parameter errors on branches incident to boundary buses may not be detected. Such errors are subsequently detected by a coordination level estimator completing the system-wide parameter detection procedure. In conclusion, performance of the developed method is demonstrated using the IEEE 118-bus and 2000-bus Texas synthetic systems.
24 POWER TRANSMISSION AND DISTRIBUTION↗
The Detection and Mitigation of RFI with the Aquarius L-Band Scatterometer
The Aquarius sea-surface salinity mission includes an L-band scatterometer to sense sea-surface roughness. This radar is subject to radio-frequency interference (RFI) in its passband from 1258 to 1262 MHz, a region also allocated for terrestrial radio location. Due to its received power sensitivity requirements, the expected RFI environment poses significant challenges. We present the results of a study evaluating the severity of terrestrial RFI sources on the operation of the Aquarius scatterometer, and propose a scheme to both detect and remove problematic RFI signals in the ocean backscatter measurements. The detection scheme utilizes the digital sampling of the ambient input power to detect outliers from the receiver noise floor which are statistically significant, and flags nearby radar echoes as potentially contaminated by RFI. This detection strategy, developed to meet tight budget and data downlink requirements, has been implemented and tested in hardware, and shows great promise for the detection and global mapping of L-band RFI sources.
Electromagnetic radiative interference↗
Efficient Mosaicking of Spitzer Space Telescope Images
A parallel version of the MOPEX software, which generates mosaics of infrared astronomical images acquired by the Spitzer Space Telescope, extends the capabilities of the prior serial version. In the parallel version, both the input image space and the output mosaic space are divided among the available parallel processors. This is the only software that performs the point-source detection and the rejection of spurious imaging effects of cosmic rays required by Spitzer scientists. This software includes components that implement outlier-detection algorithms that can be fine-tuned for a particular set of image data by use of a number of adjustable parameters. This software has been used to construct a mosaic of the Spitzer Infrared Array Camera Shallow Survey, which comprises more than 17,000 exposures in four wavelength bands from 3.6 to 8 m and spans a solid angle of about 9 square degrees. When this software was executed on 32 nodes of the 1,024-processor Cosmos cluster computer at NASA s Jet Propulsion Laboratory, a speedup of 8.3 was achieved over the serial version of MOPEX. The performance is expected to improve dramatically once a true parallel file system is installed on Cosmos.
Jacob, Joseph↗