Search NASA⌕ Search

SEARCH · Search NASA

Results for “outlier detection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

PAVC Gridded 20m Alaska NGEE Tier3 PFTs v1.0

These 20-meter spatial resolution gridded products provide per-pixel fractional cover (%) of Next Generation Ecosystem Experiments (NGEE) Arctic Plant Functional Types (PFTs) Tier 3 across Alaska, north of the boreal treeline. The products were developed for the NGEE Arctic project, which is improving Arctic vegetation representation and parameterization of the E3SM Land Model. This dataset includes 8 files containing fractional cover for NGEE Tier 3 PFTs (https://data.ess-dive.lbl.gov/view/doi:10.15485/2529470): (1) bryophytes; (2) lichens; (3) non-vascular plants, i.e., the sum of lichens and bryophytes; (4) deciduous shrubs, (5) evergreen shrubs, (6) forbs, (7) graminoids, and a non-PFT class, (8) litter. Each pixel contains the percent cover (expressed as a fraction of total ground cover) that was predicted by random-forest regression models. The random-forest models were trained on cover data collected at 978 plots from 2010 to 2021, of which are archived in the Pan-Arctic Vegetation Cover (PAVC) database (https://data.ess-dive.lbl.gov/datasets/doi:10.15485/2483557). The plot cover was linked to 20-meter spatial resolution, satellite-derived predictor variables: Sentinel-2 spectra and Sentinel-1 polarizations averaged over the 2019 growing season, as well as topographical features derived from ArcticDEM. Then, spatio-temporally anomalous plot data that introduced large variability to the regression outcomes were dropped using the Cook’s distance outlier detection method, and the models were re-created using high-quality plots and their associated satellite derived explanatory variables per each PFT. The correlations between plot-observed and satellite-derived fractional cover for all PFTs were well correlated (R2 = 0.69–0.95 and 0.5 for litter) and had low RMSE bias (0.02–0.11). This research was performed as a part of the NGEE Arctic project. The NGEE Arctic project was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.

54 ENVIRONMENTAL SCIENCES↗

Revisions to the SRCC Rating Process for Solar Water Heaters

In the United States, annual performance ratings for solar water heaters are computed with component-based simulation models driven by typical meteorological year weather and specified water draw. Changes in the process are being implemented to enhance credibility through increased transparency and accuracy. Changes to the process include using a graphical rather than text-based model-building tool, performing analytical tests on all components and systems, checking energy balances on every component, loop, and system at every time step, comparing the results to detect outliers and potential errors, and documenting the modeling process in detail. Examples of changes in ratings are shown, along with analytical and comparative testing results.

14 SOLAR ENERGY↗

User Behavior Analytics

User Behaviour Analytics is the tracking, collecting and assessing of user data and activities. The goal is to detect misuse of user credentials by developing models for the normal behaviour of user credentials within a computer network and detect outliers with respect to their baseline.

99 GENERAL AND MISCELLANEOUS↗

Analytical Functions for 200 West Pump-and-Treat SCADA Sensor Data

Historical operations at the U.S. Department of Energy’s Hanford Site included disposal of waste fluids to the subsurface in the 200 West Area on the Hanford Central Plateau. Subsequent infiltration of fluids has resulted in groundwater contamination with carbon tetrachloride, nitrate, uranium, technetium-99, and other contaminants. A pump-and-treat (P&T) system, with an extraction/injection well network and an aboveground treatment plant, was implemented as part of interim and final remedies in the 200 West area. The HYPATIA single-page web application (part of the SOCRATES suite) is being developed to provide access to and analysis of chemistry and treatment facility sensor data for this 200 West P&T system. For the web application, analytical algorithms were developed to perform summing, differencing, smoothing, outlier detection, change-point detection, mass flow rate, and injectivity calculations on the data. Candidate algorithms were identified and tested, with the best-performing algorithms then assembled for implementation in HYPATIA. Because HYPATIA is hosted on the Amazon Web Services (AWS) cloud computing platform, algorithms were implemented in a back-end AWS Lambda function that can be called by the HYPATIA front end. The Lambda function applies the requested data processing to specified data via functions written in R, Python, and JavaScript. Development, testing, and review of the data analysis algorithms was completed under an NQA-1 quality program. This new HYPATIA functionality will provide information to support site decisions regarding P&T system performance and optimization.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

High-dimensional Data-driven Energy optimization for Multi-Modal Transit Agencies (HD-EMMA) (Final Technical Report)

Public bus transit services in the U.S. are responsible for at least 19.7 million metric tons of CO 2 emission annually. Electric vehicles (EVs) can have a much lower environmental impact than comparable internal combustion engine vehicles (ICEVs), especially in urban areas. Unfortunately, EVs are also much more expensive than ICEVs. As a result, many public transit agencies can afford only mixed fleets of transit vehicles, consisting of EVs, hybrids (HEVs), and ICEVs. Transit agencies that operate such mixed fleets of vehicles face a challenging optimization problem: these agencies need to decide which vehicles are assigned to serving which transit trips. Since the advantage of EVs over ICEVs varies depending on the route and time of day (e.g., the benefit of EVs is higher in slower traffic with frequent stops and lower on highways), the assignment can have a significant effect on energy use and, hence, environmental impact. Through this project, we have developed reference data about energy collections and constructed a set of machine learning models that can accurately predict the energy consumption for the whole fleet at the level of each trip. We have used these models to develop a scheduling and assignment strategy that can rotate the different vehicle types across the transit agencies’ routes. The optimization algorithm ensures that the vehicles are matched to trips considering weather patterns, expected congestion, and road gradients to minimize the overall energy usage. We list the key observations from our project for other practitioners below. Details are available in the report, and the list of source code and our publications are included in the appendix. 1. We have demonstrated the feasibility of collecting, merging and analyzing large volumes of high-resolution real-world telemetry data from a mixed vehicle fleet. To mitigate the inherent noise of the recorded GPS points, the team developed an algorithm that filters data and maps the points onto a street. The algorithm considers previous and subsequent location measurements and different characteristics of nearby streets to determine how likely the vehicle travels on them. Then, the team segmented the time series into disjoint contiguous samples based on adjacent road segments and repeated the outlier detection and removal. For each data point, the team added features corresponding to elevation changes within the samples, weather features, such as temperature, and traffic data, such as speed ratio between actual speed and free-flow speed. 2. We have developed two forms of machine learning models that be used to understand and analyze the energy operations of a mixed vehicle transit fleet. The micro prediction model provides estimates of instantaneous energy prediction for all types of buses (diesel, hybrid, and electric). Such a model is important in evaluating the energy impacts of real-time bus operation strategies, but it is challenging due to diversified driving cycles of transit buses. The model can help the drivers understand the impact of their driving behaviors and short-term congestions. The macro prediction models estimate average energy consumption across the whole trip considering the features: distance traveled, various road-type features, elevation change, day of the week, time of day, various weather features (temperature, humidity, etc.), and traffic features (speed ratio and jam factor). 3. We have demonstrated that it is possible to transfer the machine learning models we have developed in this project to other teams and cities by using inductive transfer learning. We also showed that the performance of the macro energy prediction models can be improved using a multi-task learning approach where the learning parameters are shared between the models being developed for different vehicle types. The advantage of this approach is improved learning performance as the models can exploit common spatio-temporal and environmental characteristics. 4. Finally, we have developed trip and vehicle assignment and scheduling algorithms that use the energy prediction models and develop a trip to vehicle type (diesel, electric, hybrid) assignment for the whole operation to reduce overall emissions and cost. We have shown through simulations that the proposed algorithms can save $\$$ 48,910 in energy costs and 175 metric tons of CO 2 emission annually for CARTA.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Calculation of Weibull strength parameters and Batdorf flow-density constants for volume- and surface-flaw-induced fracture in ceramics

The calculation of shape and scale parameters of the two-parameter Weibull distribution is described using the least-squares analysis and maximum likelihood methods for volume- and surface-flaw-induced fracture in ceramics with complete and censored samples. Detailed procedures are given for evaluating 90 percent confidence intervals for maximum likelihood estimates of shape and scale parameters, the unbiased estimates of the shape parameters, and the Weibull mean values and corresponding standard deviations. Furthermore, the necessary steps are described for detecting outliers and for calculating the Kolmogorov-Smirnov and the Anderson-Darling goodness-of-fit statistics and 90 percent confidence bands about the Weibull distribution. It also shows how to calculate the Batdorf flaw-density constants by uing the Weibull distribution statistical parameters. The techniques described were verified with several example problems, from the open literature, and were coded. The techniques described were verified with several example problems from the open literature, and were coded in the Structural Ceramics Analysis and Reliability Evaluation (SCARE) design program.

Pai, Shantaram S.↗

Calculation of Weibull strength parameters and Batdorf flow-density constants for volume- and surface-flaw-induced fracture in ceramics

The calculation of shape and scale parametes of the two-parameter Weibull distribution is described using the least-squares analysis and maximum likelihood methods for volume- and surface-flaw-induced fracture in ceramics with complete and censored samples. Detailed procedures are given for evaluating 90 percent confidence intervals for maximum likelihood estimates of shape and scale parameters, the unbiased estimates of the shape parameters, and the Weibull mean values and corresponding standard deviations. Furthermore, the necessary steps are described for detecting outliers and for calculating the Kolmogorov-Smirnov and the Anderson-Darling goodness-of-fit statistics and 90 percent confidence bands about the Weibull distribution. It also shows how to calculate the Batdorf flaw-density constants by using the Weibull distribution statistical parameters. The techniques described were verified with several example problems, from the open literature, and were coded in the Structural Ceramics Analysis and Reliability Evaluation (SCARE) design program.

Shantaram, S. Pai↗

Calculation of Weibull strength parameters, Batdorf flaw density constants and related statistical quantities using PC-CARES

This manual describes the operation and theory of the PC-CARES (Personal Computer-Ceramic Analysis and Reliability Evaluation of Structures) computer program for the IBM PC and compatibles running PC-DOS/MS-DOR OR IBM/MS-OS/2 (version 1.1 or higher) operating systems. The primary purpose of this code is to estimate Weibull material strength parameters, the Batdorf crack density coefficient, and other related statistical quantities. Included in the manual is the description of the calculation of shape and scale parameters of the two-parameter Weibull distribution using the least-squares analysis and maximum likelihood methods for volume- and surface-flaw-induced fracture in ceramics with complete and censored samples. The methods for detecting outliers and for calculating the Kolmogorov-Smirnov and the Anderson-Darling goodness-of-fit statistics and 90 percent confidence bands about the Weibull line, as well as the techniques for calculating the Batdorf flaw-density constants are also described.

Szatmary, Steven A.↗

BKG/DGFI Combination Center Annual Report 2012

This report summarizes the activities of the Federal Agency for Cartography and Geodesy (Bundesamt fuer Kartographie und Geodaesie, BKG) and the German Geodetic Research Institute (Deutsches Geodaetisches Forschungsinstitut, DGFI)BKG/DGFI Combination Center in 2011 and outlines the planned activities for the year 2012. The main focus was to stabilize outlier detection and to update the Web presentation of the combined products.

Bachmann, Sabine↗

Recent Advances in SMAP RFI Processing

The measurements made by the Soil Moisture Active/Passive (SMAP) mission are affected by the presence of Radio Frequency Interference (RFI) in the protected 1400-1427 MHz band. In SMAP data processing, the main protection against RFI is a sophisticated RFI detection algorithm which flags sub-samples in time and frequency that are contaminated by RFI and removes them before estimating the brightness temperature. This contribution presents two additional approaches that have been developed to address the RFI concern in SMAP. The first consists in locating sources of RFI; once located, it becomes possible to report RFI sources to spectrum management authorities, which can lead to less RFI being experienced by SMAP in the future. The second is a new RFI detection method that is based on detecting outliers in the spatial distribution of measured antenna temperatures.

Radio Frequency Interference↗

Feature extraction for structural dynamics model validation

This study focuses on defining and comparing response features that can be used for structural dynamics model validation studies. Features extracted from dynamic responses obtained analytically or experimentally, such as basic signal statistics, frequency spectra, and estimated time-series models, can be used to compare characteristics of structural system dynamics. By comparing those response features extracted from experimental data and numerical outputs, validation and uncertainty quantification of numerical model containing uncertain parameters can be realized. In this study, the applicability of some response features to model validation is first discussed using measured data from a simple test-bed structure and the associated numerical simulations of these experiments. issues that must be considered were sensitivity, dimensionality, type of response, and presence or absence of measurement noise in the response. Furthermore, we illustrate a comparison method of multivariate feature vectors for statistical model validation. Results show that the outlier detection technique using the Mahalanobis distance metric can be used as an effective and quantifiable technique for selecting appropriate model parameters. However, in this process, one must not only consider the sensitivity of the features being used, but also correlation of the parameters being compared.

42 ENGINEERING↗

Revisions to the SRCC Rating Process for Solar Water Heaters: Preprint

In the United States, annual performance ratings for solar water heaters are computed with component-based simulation models driven by typical meteorological year weather and specified water draw. Changes in the process are being implemented to enhance credibility through increased transparency and accuracy. Changes to the process include using a graphical rather than text-based model-buildingtool, performing analytical tests on all components and systems, checking energy balances on every component, loop, and system at every time step, comparing the results to detect outliers and potential errors, and documenting the modeling process in detail. Examples of changes in ratings are shown, along with analytical and comparative testing results.

14 SOLAR ENERGY↗

AUTONOMIE VID

Autonomie Vehicle Information Database (VID) offers a comprehensive list of vehicle specifications since 1990. The database details more than 65,000 vehicles with hundreds of attributes. The database is the result of the development of a general automated data collection framework as well as the development of building blocks for processing, cleaning, integrating and analyzing complex data. The data has undergone several layers of outlier detections processes, machine learning based imputations methods have been used to deal with missing data problems, and new fields have been created according to the rules of feature engineering. Thanks to this streamlined data pipelines, the resulting processed aggregated data should deliver a unique level of information to the user in which the content can be efficiently maintained and updated.

Moswd, Ayman↗

PVAnalytics: A Python Package for Automated Processing of Solar Time Series Data

Multiple publicly available software packages exist that analyze solar time series data, including RdTools and Solar Data Tools, among others. Several of these packages contain their own unique quality assurance (QA) and feature recognition algorithms. The python PVAnalytics package was developed to offer an internally consistent source for these analysis tools, making it easier for the end user to deploy these routines on his or her solar data. The PVAnalytics package currently contains routines for outlier detection, inverter clipping detection, irradiance and temperature checks, orientation checks, and data shift detection, among other functions. These functions have been aggregated from various sources including Solar Forecast Arbiter, RdTools, and the QA process developed by NREL's PV Fleets Initiative. We are continuously adding new functionality to the package, including documentation, examples and algorithms. By bundling QA functionality into a single software package, we hope to make PVAnalytics a comprehensive software library to support analysis of solar metadata and time series data.

data cleaning↗

SU-F-T-97: Outlier Identification in Radiation Therapy Knowledge Modeling

Purpose: To investigate the impact of outliers on knowledge modeling in radiation therapy, and develop a systematic workflow for identifying and analyzing geometric and dosimetric outliers using pelvic cases. Methods: Four groups (G1-G4) of pelvic plans were included: G1 (37 prostate cases), G2 (37 prostate plus lymph node cases), and G3 (37 prostate bed cases) are all clinical IMRT cases. G4 are 10 plans outside G1 re-planned with dynamic-arc to simulate dosimetric outliers. The workflow involves 2 steps: 1. identify geometric outliers, assess impact and clean up; 2. identify dosimetric outliers, assess impact and clean up.1. A baseline model was trained with all G1 cases. G2/G3 cases were then individually added to the baseline model as geometric outliers. The impact on the model was assessed by comparing leverage statistic of inliers (G1) and outliers (G2/G3). Receiver-operating-characteristics (ROC) analysis was performed to determine optimal threshold. 2. A separate baseline model was trained with 32 G1 cases. Each G4 case (dosimetric outliers) was then progressively added to perturb this model. DVH predictions were performed using these perturbed models for remaining 5 G1 cases. Normal tissue complication probability (NTCP) calculated from predicted DVH were used to evaluate dosimetric outliers’ impact. Results: The leverage of inliers and outliers was significantly different. The Area-Under-Curve (AUC) for differentiating G2 from G1 was 0.94 (threshold: 0.22) for bladder; and 0.80 (threshold: 0.10) for rectum. For differentiating G3 from G1, the AUC (threshold) was 0.68 (0.09) for bladder, 0.76 (0.08) for rectum. Significant increase in NTCP started from models with 4 dosimetric outliers for bladder (p<0.05), and with only 1 dosimetric outlier for rectum (p<0.05). Conclusion: We established a systematic workflow for identifying and analyzing geometric and dosimetric outliers, and investigated statistical metrics for detecting. Results validated the necessity for outlier detection and clean-up to enhance model quality in clinical practice. Research Grant: Varian master research grant.

60 APPLIED LIFE SCIENCES↗

Anomaly Detection in Large Sets of High-Dimensional Symbol Sequences

This paper addresses the problem of detecting and describing anomalies in large sets of high-dimensional symbol sequences. The approach taken uses unsupervised clustering of sequences using the normalized longest common subsequence (LCS) as a similarity measure, followed by detailed analysis of outliers to detect anomalies. As the LCS measure is expensive to compute, the first part of the paper discusses existing algorithms, such as the Hunt-Szymanski algorithm, that have low time-complexity. We then discuss why these algorithms often do not work well in practice and present a new hybrid algorithm for computing the LCS that, in our tests, outperforms the Hunt-Szymanski algorithm by a factor of five. The second part of the paper presents new algorithms for outlier analysis that provide comprehensible indicators as to why a particular sequence was deemed to be an outlier. The algorithms provide a coherent description to an analyst of the anomalies in the sequence, compared to more normal sequences. The algorithms we present are general and domain-independent, so we discuss applications in related areas such as anomaly detection.

Budalakoti, Suratna↗

Genetic monitoring of steelhead in the Klickitat River to estimate productivity, straying, and migration timing

Abstract Objective Salmonids with a complex life history variation present challenges for conservation management, but genetic approaches alongside fisheries monitoring can address questions regarding the viability of the natural populations. Methods We genotyped adult (n = 3108) and juvenile (n = 2624) samples of steelhead Oncorhynchus mykiss that were collected in the Klickitat River, Washington, USA at traps in the lower drainage to examine tributary level productivity, straying from outside sources, and variation in adult migration timing. Result Genetic assignment of steelhead from this system indicated that the majority were produced within or near tributaries of the middle Klickitat River (juvenile mean = 72.8%; adult mean = 87.3%). Analyses with parentage-based tagging identified that most hatchery-origin adults were assigned to the Skamania Hatchery (80.8%), as expected, since this has been the release stock for decades within the Klickitat River drainage. Hatchery-origin adults were also identified from programs operating outside the Klickitat River, which were primarily strays from Snake River hatcheries. Most natural-origin steelhead were assigned to the Klickitat River, but there were also natural-origin fish identified as strays from other regions of the Columbia River (22.3% of natural returns). We also examined genes known to be associated with migration timing in adult steelhead observed at the trap and observed a strong relationship between migration date and alleles for early and late migration, but individual outliers were detected across seasons. Conclusion Our results indicate that genetic variation of steelhead in the Klickitat River has been influenced by hatchery programs as well as natural-origin straying from other subbasins, but genetic diversity remains high throughout the subbasin, and both early and late migration alleles are maintained. The genetic diversity present in Klickitat River steelhead may enable this Endangered Species Act listed (threatened) species to better adapt to stochastic environmental conditions compared to less diverse populations.

Collins, Erin E. (ORCID:0000000260978479)↗

1 H NMR metabolomics study of metastatic melanoma in C57BL/6J mouse spleen

Melanoma is a malignant tumor of melanocytes. Although extensive investigations have been done to study metabolic changes in primary melanoma in vivo and in vitro, little effort has been devoted to metabolic profiling of metastatic tumors in organs other than lymph nodes. As such, in this work, NMR-based metabolomics combined with multivariate data analysis is used to study metastatic B16-F10 melanoma in C57BL/6J mouse spleen. Principal Component Analysis (PCA), an unsupervised multivariate data analysis method, is used to detect possible outliers, while Orthogonal Projection to Latent Structure (OPLS), a supervised multivariate data analysis method, is employed to find important metabolites responsible for discriminating the control and the melanoma groups. Two different strategies, i.e., spectral binning and spectral deconvolution, are used to reduce the original spectral data before statistical analysis. Spectral deconvolution is found to be superior for identifying a set of discriminatory metabolites between the control and the melanoma groups, especially when the sample size is small. OPLS results show that the melanoma group can be well separated from its control group. It is found that taurine, glutamate, aspartate, O-Phosphoethanolamine, niacinamide ,ATP, lipids and glycerol derivatives are decreased statistically and significantly while alanine, malate, xanthine, histamine, dCTP, GTP, thymidine, 2'-Deoxyguanosine are statistically and significantly elevated. Furthermore, these significantly changed metabolites are associated with multiple biological pathways and may be potential biomarkers for metastatic melanoma in spleen.

1H NMR metabolomics↗