Search NASA⌕ Search

SEARCH · Search NASA

Results for “outlier detection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Data-Driven Event Detection of Power Systems Based on Unequal-Interval Reduction of PMU Data and Local Outlier Factor

With the deployment of phasor measurement units (PMU) and wide area measurement system (WAMS), it is feasible to have an insight into the events occurred in power systems based on measured data. Thus, a novel data-driven algorithm based on local outlier factor (LOF) is proposed in this work to detect and locate events in power systems using reduced PMU data. First, the unequal-interval reduction method is presented to reduce the scale of PMU data in sub-stations and reconstruct it in master station of WAMS, which can relieve the burden of communication systems. Then, principle component analysis (PCA)-based similarity search method is proposed to measure the differences of operation state between any two buses. Next, LOF is presented to detect the abnormal events in power systems, and employed to determine the region of the event source. Finally, six cases from the Western electricity coordinating council (WECC) 179-bus power system, a case from the South China power system (SCPS), and a case from the Guangdong power system (GDPS) are utilized to demonstrate the effectiveness of the proposed algorithm. Overall, the results show that proposed algorithm is effective and can be applied to event detection, event location, and online monitoring, which can enhance the situation awareness ability of power system operators.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Mixture-Tuned, Clutter Matched Filter for Remote Detection of Subpixel Spectral Signals

Mapping localized spectral features in large images demands sensitive and robust detection algorithms. Two aspects of large images that can harm matched-filter detection performance are addressed simultaneously. First, multimodal backgrounds may thwart the typical Gaussian model. Second, outlier features can trigger false detections from large projections onto the target vector. Two state-of-the-art approaches are combined that independently address outlier false positives and multimodal backgrounds. The background clustering models multimodal backgrounds, and the mixture tuned matched filter (MT-MF) addresses outliers. Combining the two methods captures significant additional performance benefits. The resulting mixture tuned clutter matched filter (MT-CMF) shows effective performance on simulated and airborne datasets. The classical MNF transform was applied, followed by k-means clustering. Then, each cluster s mean, covariance, and the corresponding eigenvalues were estimated. This yields a cluster-specific matched filter estimate as well as a cluster- specific feasibility score to flag outlier false positives. The technology described is a proof of concept that may be employed in future target detection and mapping applications for remote imaging spectrometers. It is of most direct relevance to JPL proposals for airborne and orbital hyperspectral instruments. Applications include subpixel target detection in hyperspectral scenes for military surveillance. Earth science applications include mineralogical mapping, species discrimination for ecosystem health monitoring, and land use classification.

Thompson, David R.↗

Detecting Anomalies from End-to-End Internet Performance Measurements (PingER) Using Cluster Based Local Outlier Factor

PingER (Ping End-to-End Reporting) is a worldwide end-to-end Internet performance measurement framework. It was developed by the SLAC National Accelerator Laboratory, Stanford, USA and running from the last 20 years. It has more than 700 monitoring agents and remote sites which monitor the performance of Internet links around 170 countries of the world. At present, the size of the compressed PingER data set is about 60 GB comprising of 100,000 flat files. The data is publicly available for valuable Internet performance analyses. However, the data sets suffer from missing values and anomalies due to congestion, bottleneck links, queuing overflow, network software misconfiguration, hardware failure, cable cuts, and social upheavals. Therefore, the objective of this paper is to detect such performance drops or spikes labeled as anomalies or outliers for the PingER data set. In the proposed approach, the raw text files of the data set are transformed into a PingER dimensional model. The missing values are imputed using the k-NN algorithm. The data is partitioned into similar instances using the k-means clustering algorithm. Afterward, clustering is integrated with the Local Outlier Factor (LOF) using the Cluster Based Local Outlier Factor (CBLOF) algorithm to detect the anomalies or outliers from the PingER data. Lastly, anomalies are further analyzed to identify the time frame and location of the hosts generating the major percentage of the anomalies in the PingER data set ranging from 1998 to 2016.

97 MATHEMATICS AND COMPUTING↗

Groundspeed filtering for CTAS

Ground speed is one of the radar observables which is obtained along with position and heading from NASA Ames Center radar. Within the Center TRACON Automation System (CTAS), groundspeed is converted into airspeed using the wind speeds which CTAS obtains from the NOAA weather grid. This airspeed is then used in the trajectory synthesis logic which computes the trajectory for each individual aircraft. The time history of the typical radar groundspeed data is generally quite noisy, with high frequency variations on the order of five knots, and occasional 'outliers' which can be significantly different from the probable true speed. To try to smooth out these speeds and make the ETA estimate less erratic, filtering of the ground speed is done within CTAS. In its base form, the CTAS filter is a 'moving average' filter which averages the last ten radar values. In addition, there is separate logic to detect and correct for 'outliers', and acceleration logic which limits the groundspeed change in adjacent time samples. As will be shown, these additional modifications do cause significant changes in the actual groundspeed filter output. The conclusion is that the current ground speed filter logic is unable to track accurately the speed variations observed on many aircraft. The Kalman filter logic however, appears to be an improvement to the current algorithm used to smooth ground speed variations, while being simpler and more efficient to implement. Additional logic which can test for true 'outliers' can easily be added by looking at the difference in the a priori and post priori Kalman estimates, and not updating if the difference in these quantities is too large.

Slater, Gary L.↗

Evolutionary Analyses of Gene Expression Divergence in Panicum hallii : Exploring Constitutive and Plastic Responses Using Reciprocal Transplants

Abstract The evolution of gene expression is thought to be an important mechanism of local adaptation and ecological speciation. Gene expression divergence occurs through the evolution of cis- polymorphisms and through more widespread effects driven by trans-regulatory factors. Here, we explore expression and sequence divergence in a large sample of Panicum hallii accessions encompassing the species range using a reciprocal transplantation experiment. We observed widespread genotype and transplant site drivers of expression divergence, with a limited number of genes exhibiting genotype-by-site interactions. We used a modified FST–QST outlier approach (QPC analysis) to detect local adaptation. We identified 514 genes with constitutive expression divergence above and beyond the levels expected under neutral processes. However, no plastic expression responses met our multiple testing correction as QPC outliers. Constitutive QPC outlier genes were involved in a number of developmental processes and responses to abiotic environments. Leveraging earlier expression quantitative trait loci results, we found a strong enrichment of expression divergence, including for QPC outliers, in genes previously identified with cis and cis–environment interactions but found no patterns related to trans-factors. Population genetic analyses detected elevated sequence divergence of promoters and coding sequence of constitutive expression outliers but little evidence for positive selection on these proteins. Our results are consistent with a hypothesis of cis-regulatory divergence as a primary driver of expression divergence in P. hallii.

3′ TagSeq↗

Signatures of Selection for Resistance/Tolerance to Perkinsus olseni in Grooved Carpet Shell Clam ( Ruditapes decussatus ) Using a Population Genomics Approach

ABSTRACT The grooved carpet shell clam ( Ruditapes decussatus ) is a bivalve of high commercial value distributed throughout the European coast. Its production has suffered a decline caused by different factors, especially by the parasite Perkinsus olsenii . Improving production of R . decussatus requires genomic resources to ascertain the genetic factors underlying resistance/tolerance to P. olseni i . In this study, the first reference genome of R . decussatus was assembled through long‐ and short‐read sequencing (1677 contigs; 1.386 Mb) and further scaffolded at chromosome level with Hi‐C (19 superscaffolds; 95.4% of assembly). Repetitive elements were identified (32%) and masked for annotation of 38,276 coding‐ and 13,056 non‐coding genes. This genome was used as a reference to develop a 2bRAD‐Seq 13,438 SNP panel for a genomic screening on six shellfish beds distributed across the Atlantic Ocean and Mediterranean Sea. Beds were selected by perkinsosis prevalence and the infection level was individually evaluated in all the samples. Genetic diversity was significantly higher in the Mediterranean than in the Atlantic region. The main genetic breakage was detected between those regions (F ST = 0.224), being the Mediterranean more heterogeneous than the Atlantic. Several loci under divergent selection (394 outliers; 261 genomic windows) were detected across shellfish beds. Samples were also inspected to detect signals of selection for resistance/tolerance to P. olseni i by using infection‐level and population‐genomics approaches, and 90 common divergent outliers for resistance/tolerance to perkinsosis were identified and used for gene mining. Candidate genes and markers identified provide invaluable information for controlling perkinsosis and for improving production of the grooved carpet shell clam.

Sambade, Inés M. [Department of Zoology, Genetics ↗

Method for automatic correction of offset drift in online sensors

Abstract Successful operation and optimization of water treatment systems hinge on the availability of high-quality online sensor measurements. Ideally, the available measurements should be simultaneously accurate (i.e., unbiased and precise), representative, voluminous, and timely. This remains a pain-point in current water infrastructures, forming a barrier to a wider adoption of advanced and autonomous control systems. While short-lived symptoms, such as outliers and spikes, can be detected or corrected with state-of-the-art tools for fault detection and identification, it is much more difficult to detect, diagnose, and correct the symptoms of slow faults, such as changes in offset or sensitivity due to drift. The time scale of drift is often longer than the time scales of the system dynamics of interest. Moreover, sensor drift has been shown to occur at the same time and with similar rates when sensors are exposed to the same conditions. This challenges data quality management strategies based on redundancy. In this contribution, we develop a new method, including both a hands-off sensor calibration mechanism and an information-seeking control architecture that can handle the unique challenge of simultaneous and similar drift in online sensors.

Chowdhury, Dhrubajit↗

A three-year dataset supporting research on building energy management and occupancy analytics

Abstract This paper presents the curation of a monitored dataset from an office building constructed in 2015 in Berkeley, California. The dataset includes whole-building and end-use energy consumption, HVAC system operating conditions, indoor and outdoor environmental parameters, as well as occupant counts. The data were collected during a period of three years from more than 300 sensors and meters on two office floors (each 2,325 m 2 ) of the building. A three-step data curation strategy is applied to transform the raw data into research-grade data: (1) cleaning the raw data to detect and adjust the outlier values and fill the data gaps; (2) creating the metadata model of the building systems and data points using the Brick schema; and (3) representing the metadata of the dataset using a semantic JSON schema. This dataset can be used in various applications—building energy benchmarking, load shape analysis, energy prediction, occupancy prediction and analytics, and HVAC controls—to improve the understanding and efficiency of building operations for reducing energy use, energy costs, and carbon emissions.

24 POWER TRANSMISSION AND DISTRIBUTION↗

SeekAView: An Intelligent Dimensionality Reduction Strategy for Navigating High-Dimensional Data Spaces

Dealing with the curse of dimensionality is a key challenge in high-dimensional data visualization. We present SeekAView to address three main gaps in the existing research literature. First, automated methods like dimensionality reduction or clustering suffer from a lack of transparency in letting analysts interact with their outputs in real-time to suit their exploration strategies. The results often suffer from a lack of interpretability, especially for domain experts not trained in statistics and machine learning. Second, exploratory visualization techniques like scatter plots or parallel coordinates suffer from a lack of visual scalability: it is difficult to present a coherent overview of interesting combinations of dimensions. Third, the existing techniques do not provide a flexible workflow that allows for multiple perspectives into the analysis process by automatically detecting and suggesting potentially interesting subspaces. In SeekAView we address these issues using suggestion based visual exploration of interesting patterns for building and refining multidimensional subspaces. Compared to the state-of-the-art in subspace search and visualization methods, we achieve higher transparency in showing not only the results of the algorithms, but also interesting dimensions calibrated against different metrics. We integrate a visually scalable design space with an iterative workflow guiding the analysts by choosing the starting points and letting them slice and dice through the data to find interesting subspaces and detect correlations, clusters, and outliers. We present two usage scenarios for demonstrating how SeekAView can be applied in real-world data analysis scenarios.

Krause, Josua↗

Frequency Injection Based HVDC Attack-Defense Control Via Squeeze-Excitation Double CNN

Due to the independent controllability and fast power regulation capability, the High Voltage Direct Current (HVDC) system could be a prospective technology to provide multiple ancillary services to the system besides conventional bulk power transmission. However, with the increase of False Data Injection Attacks (FDIAs) on PMU data, the HVDC system could have the wrong response once the collected data that the HVDC system relied on is attacked, thus threatening the system operating security. How to ensure the security of the PMU-based HVDC ancillary service control become an urgent issue. To mitigate the risk, this paper proposed an HVDC attack-defense control based on the FDIAs detection method. Firstly, the Squeeze-Excitation based Double Convolutional Neural Networks (SE-DCNN) is proposed to realize fast identification of the attacking frequency type based on the time and frequency domain signals. Furthermore, the duration time of FDIAs is detected by the local outlier factor. Then, utilizing the results from SE-DCNN, HVDC ancillary service control framework is reorganized and an HVDC attack defense control is proposed for suppressing the potential influence of various types of FDIAs on the HVDC system ancillary service. Different experiments results demonstrate that the proposed method has the ability to significantly mitigate the frequency deviation and oscillation under the FDIA.

42 ENGINEERING↗

Decay curve analysis for data error quantification in time-domain induced polarization imaging

In recent years, the time-domain induced polarization (TDIP) imaging technique has emerged as a suitable method for the characterization and the monitoring of hydrogeologic and biogeochemical processes. However, one of the major challenges refers to the resolution of the electrical images. Hence, various studies have stressed the importance of data processing, error characterization, and the deployment of adequate inversion schemes. A widely accepted method to assess data error in electrical imaging relies on the analysis of the discrepancy between normal and reciprocal measurements. Nevertheless, the collection of reciprocals doubles the acquisition time and is only viable for a limited subset of commonly used electrode configurations (e.g., dipole-dipole [DD]). To overcome these limitations, we have developed a new methodology to quantify the data error in TDIP imaging, which is entirely based on the analysis of the recorded IP decay curve and does not require recollection of data (e.g., reciprocals). The first two steps of the methodology assess the general characteristics of the decay curves and the spatial consistency of the measurements for the detection and removal of outliers. In the third and fourth steps, we quantify the deviation of the measured decay curves from a smooth model for the estimation of random error of the total chargeability and transfer resistance measurement. The error models and imaging results obtained from this methodology - in the following referred to as "decay curve analysis" - are compared with those obtained following a conventional normal-reciprocal analysis revealing consistent results. As a result, we determine the applicability of our methodology with real field data collected at the floodplain scale (approximately 12 ha) using multiple gradient and DD configurations.

58 GEOSCIENCES↗

Coherent Uncertainty Analysis of Aerosol Measurements from Multiple Satellite Sensors

Aerosol retrievals from multiple spaceborne sensors, including MODIS (on Terra and Aqua), MISR, OMI, POLDER, CALIOP, and SeaWiFS altogether, a total of 11 different aerosol products were comparatively analyzed using data collocated with ground-based aerosol observations from the Aerosol Robotic Network (AERONET) stations within the Multi-sensor Aerosol Products Sampling System (MAPSS, http://giovanni.gsfc.nasa.gov/mapss/ and http://giovanni.gsfc.nasa.gov/aerostat/). The analysis was performed by comparing quality-screened satellite aerosol optical depth or thickness (AOD or AOT) retrievals during 2006-2010 to available collocated AERONET measurements globally, regionally, and seasonally, and deriving a number of statistical measures of accuracy. We used a robust statistical approach to detect and remove possible outliers in the collocated data that can bias the results of the analysis. Overall, the proportion of outliers in each of the quality-screened AOD products was within 12%. Squared correlation coefficient (R2) values of the satellite AOD retrievals relative to AERONET exceeded 0.6, with R2 for most of the products exceeding 0.7 over land and 0.8 over ocean. Root mean square error (RMSE) values for most of the AOD products were within 0.15 over land and 0.09 over ocean. We have been able to generate global maps showing regions where the different products present advantages over the others, as well as the relative performance of each product over different landcover types. It was observed that while MODIS, MISR, and SeaWiFS provide accurate retrievals over most of the landcover types, multi-angle capabilities make MISR the only sensor to retrieve reliable AOD over barren and snow / ice surfaces. Likewise, active sensing enables CALIOP to retrieve aerosol properties over bright-surface shrublands more accurately than the other sensors, while POLDER, which is the only one of the sensors capable of measuring polarized aerosols, outperforms other sensors in certain smoke-dominated regions, including broadleaf evergreens in Brazil and South-East Asia.

remote sensing↗

A New Load Residual Threshold Definition for the Evaluation of Wind Tunnel Strain-Gage Balance Data

A new definition of a threshold for the detection of load residual outliers of wind tunnel strain-gage balance data was developed. The new threshold is defined as the product between the inverse of the absolute value of the primary gage sensitivity and an empirical limit of the electrical outputs of a strain{gage. The empirical limit of the outputs is either 2.5 microV/V for balance calibration or check load residuals. A reduced limit of 0.5 microV/V is recommended for the evaluation of differences between repeat load points because, by design, the calculation of these differences removes errors in the residuals that are associated with the regression analysis of the data itself. The definition of the new threshold and different methods for the determination of the primary gage sensitivity are discussed. In addition, calibration data of a six-component force balance and a five-component semi-span balance are used to illustrate the application of the proposed new threshold definition to different types of strain{gage balances. During the discussion of the force balance example it is also explained how the estimated maximum expected output of a balance gage can be used to better understand results of the application of the new threshold definition.

load residuals↗

Aquarius RFI Detection and Mitigation Algorithm: Assessment and Examples

Aquarius is an L-band radiometer system designed to map sea surface salinity from space. This is a sensitive measurement, and protection from radio frequency interference (RFI) is important for success. An initial look at the performance of the Aquarius RFI detection and mitigation algorithm is reported together with examples of the global distribution of RFI at the L-band. To protect against RFI, Aquarius employs rapid sampling (10 ms) and a "glitch" detection algorithm that looks for outliers among the samples. Samples identified as RFI are removed, and the remainder is averaged to produce an RFI-free signal for the salinity retrieval algorithm. The RFI detection algorithm appears to work well over the ocean with modest rates for false alarms (5%) and missed detection. The global distribution of RFI coincides well with population centers and is consistent with observations reported by the Soil Moisture and Ocean Salinity mission.

remote sensing↗

Where’s Swimmy?: Mining unique color features buried in galaxies by deep anomaly detection using Subaru Hyper Suprime-Cam data

Abstract We present the Swimmy (Subaru WIde-field Machine-learning anoMalY) survey program, a deep-learning-based search for unique sources using multicolored (grizy) imaging data from the Hyper Suprime-Cam Subaru Strategic Program (HSC-SSP). This program aims to detect unexpected, novel, and rare populations and phenomena, by utilizing the deep imaging data acquired from the wide-field coverage of the HSC-SSP. This article, as the first paper in the Swimmy series, describes an anomaly detection technique to select unique populations as “outliers” from the data-set. The model was tested with known extreme emission-line galaxies (XELGs) and quasars, which consequently confirmed that the proposed method successfully selected $\sim\!\! 60\%$–$70\%$ of the quasars and $60\%$ of the XELGs without labeled training data. In reference to the spectral information of local galaxies at z = 0.05–0.2 obtained from the Sloan Digital Sky Survey, we investigated the physical properties of the selected anomalies and compared them based on the significance of their outlier values. The results revealed that XELGs constitute notable fractions of the most anomalous galaxies, and certain galaxies manifest unique morphological features. In summary, deep anomaly detection is an effective tool that can search rare objects, and, ultimately, unknown unknowns with large data-sets. Further development of the proposed model and selection process can promote the practical applications required to achieve specific scientific goals.

Astronomy & Astrophysics↗

Statistical Techniques For Real-time Anomaly Detection Using Spark Over Multi-source VMware Performance Data

Anomaly detection refers to the identi cation of an irregular or unusual pat- tern which deviates from what is standard, normal, or expected. Such deviated patterns typically correspond to samples of interest and are assigned different labels in different domains, such as outliers, anomalies, exceptions, or malware. Detecting anomalies in fast, voluminous streams of data is a formidable chal- lenge. This paper presents a novel, generic, real-time distributed anomaly detection framework for heterogeneous streaming data where anomalies appear as a group. We have developed a distributed statistical approach to build a model and later use it to detect anomaly. As a case study, we investigate group anomaly de- tection for a VMware-based cloud data center, which maintains a large number of virtual machines (VMs). We have built our framework using Apache Spark to get higher throughput and lower data processing time on streaming data. We have developed a window-based statistical anomaly detection technique to detect anomalies that appear sporadically. We then relaxed this constraint with higher accuracy by implementing a cluster-based technique to detect sporadic and continuous anomalies. We conclude that our cluster-based technique out- performs other statistical techniques with higher accuracy and lower processing time.

Apache Spark↗

Elastic depths for detecting shape anomalies in functional data

This this paper, we propose a new family of depth measures called the elastic depths that can be used to greatly improve shape anomaly detection in functional data. Shape anomalies are functions that have considerably different geometric forms or features from the rest of the data. Identifying them is generally more difficult than identifying magnitude anomalies because shape anomalies are often not distinguishable from the bulk of the data with visualization methods. The proposed elastic depths use the recently developed elastic distances to directly measure the centrality of functions in the amplitude and phase spaces. Measuring shape outlyingness in these spaces provides a rigorous quantification of shape which, in turn, gives the elastic depths a strong theoretical and practical advantage over other methods in detecting shape anomalies. A simple boxplot and thresholding method are introduced to identify shape anomalies using the elastic depths. We assess the elastic depth's detection skill on simulated shape outlier scenarios and compare them against popular shape anomaly detectors. Finally, bond yields, image outlines, and hurricane trajectories are used to demonstrate our method's applicability to functional data observed on three different manifolds.

97 MATHEMATICS AND COMPUTING↗

A method for separating seismo-ionospheric TEC outliers from heliogeomagnetic disturbances by using nu-SVR

Seismo-Ionospheric is a study of ionosphere disturbances associated with seismic activities. In many previous researches, heliogeomagnetic or strong earthquake activities can caused the disturbances in the ionosphere. However, it is difficult to separate these disturbances based on related sources. In this research, we proposed a method to separate these disturbances/outliers by using nu-SVR with the world-wide GPS data. TEC data related to the 26th December 2004 Sumatra and the 11th March 2011 Honshu earthquakes had been analyzed. After analyzed TEC data in several location around the earthquake epicenter and compared with geomagnetic data, the method shows a good result in the average to detect the source of these outliers. This method is promising to use in the future research.

58 GEOSCIENCES↗