Search NASA⌕ Search

SEARCH · Search NASA

Results for “bad data detection and identification”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

11 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Robust Distributed State Estimator for Interconnected Transmission and Distribution Networks (Final Report RPPR-1)

This project’s objective is to develop a combined transmission and distribution state estimator which accounts for very large system size and model complexity (by way of distributing the computations) and large number of solar PV units connected to the distribution system on multiple feeders. The project not only provides a robust formulation and solution to this problem but also tests the solution by implementing it on a well-established large utility system. It introduces several improvements with respect to the state of the art in existing state estimation software: (a) The developed state estimator (SE) allows robust and accurate monitoring of bidirectional flows in distribution systems which result due to the distributed energy sources which are not observable and thus not incorporated in generation dispatch; (b) Large utility systems with tens of thousands of transmission buses and hundreds of thousands of distribution nodes are difficult to model as a single integrated system. This shortcoming is addressed by developing a “scalable distributed computational framework” which allows splitting the ultra large system models into several small subsystems and coordinating their solution by a robust and practical state estimation formulation; (c) Measurement errors irrespective of their locations are detected and removed by the developed state estimator. Historically, transmission and distribution systems were analyzed and operated as two independent systems. Given the non-transposed short feeder sections, unevenly loaded phases, strictly radial configuration and unidirectional power flows in the absence of remote generation, distribution system analysis was customized to account for these characteristics. However, some of these assumptions are no longer valid (non-radial configuration, bidirectional power flows) and thus distribution system analysis should be revisited. Furthermore, in the past, the interaction between the transmission and distribution systems was quite passive, where distribution substations were modeled as lumped loads in the transmission system model. With substantial generation injected by renewable generation located in the distribution systems, such modeling will no longer be accurate. The developed state estimator facilitates proper monitoring of the interactions between the transmission and distribution systems and enables smart dispatch of these units which are made observable by the state estimator.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys

As the data volume of astronomical imaging surveys rapidly increases, traditional methods for image anomaly detection, such as visual inspection by human experts, are becoming impractical. We introduce a machine-learning-based approach to detect poor-quality exposures in large imaging surveys, with a focus on the DECam Legacy Survey (DECaLS) in regions of low extinction (i.e., E ( B − V ) < 0.04 ). Our semi-supervised pipeline integrates a vision transformer (ViT), trained via self-supervised learning (SSL), with a k-Nearest Neighbor (kNN) classifier. We train and validate our pipeline using a small set of labeled exposures observed by surveys with the Dark Energy Camera (DECam). A clustering-space analysis of where our pipeline places images labeled in good and bad categories suggests that our approach can efficiently and accurately determine the quality of exposures. Applied to new imaging being reduced for DECaLS Data Release 11, our pipeline identifies 780 problematic exposures, which we subsequently verify through visual inspection. Being highly efficient and adaptable, our method offers a scalable solution for quality control in other large imaging surveys.

Luo, Yufeng (ORCID:0000000246230683)↗
DOE OSTI2020

The good, the bad, and the ugly: Data-driven load profile discord identification in a large building portfolio

Reducing the overall energy consumption and associated greenhouse gas emissions in the building sector is essential for meeting our future sustainability goals. Recently, smart energy metering facilities have been deployed to enable monitoring of energy consumption data with hourly or subhourly temporal resolution. This unprecedented data collection has created various opportunities for advanced data analytics involving load profiles (e.g., building energy benchmarking programs, building-to-grid integration, and calibration of urban-scale energy models). These applications often need preprocessing steps to detect daily load profile discords, such as: 1) outliers due to system malfunctions (the bad) and 2) irregular energy consumption patterns, such as those resulting from holidays (the ugly) compared to normal consumption patterns (the good). However, current preprocessing methods predominantly focus on filtering using statistical threshold values, which fail to capture the contextual discords of daily profiles. In addition, discord detection algorithms in building research are often aimed at finding individual building-level discords, which are not suitable at a large scale. Thus, here, we develop a method for automated load profile discord identification (ALDI) in a large portfolio of buildings (more than 100 buildings). Specifically, ALDI 1) uses the matrix profile (MP) method to quantify the similarities of daily subsequences in time series meter data, 2) compares daily MP values with typical-day MP distributions using the Kolmogorov-Smirnov test, and 3) identifies daily load profile discords in a large building portfolio. We evaluate ALDI using the metering data of both an academic campus and a residential neighborhood. Our results demonstrate that ALDI efficiently discovers measurement errors by system malfunctions and low energy consumption days in the academic campus portfolio, and it detects unique load shape patterns likely driven by occupant behavior and extreme weather conditions in the residential neighborhood.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS2011

Identification of Abnormal System Noise Temperature Patterns in Deep Space Network Antennas Using Neural Network Trained Fuzzy Logic

This paper presents the development of a fuzzy logic function trained by an artificial neural network to classify the system noise temperature (SNT) of antennas in the NASA Deep Space Network (DSN). The SNT data were classified into normal, marginal, and abnormal classes. The irregular SNT pattern was further correlated with link margin and weather data. A reasonably good correlation is detected among high SNT, low link margin and the effect of bad weather; however we also saw some unexpected non-correlations which merit further study in the future.

pattern identification↗
DOE OSTI2021

Phasor-Measurement-Unit-Based Data Analytics Using Digital Twin and PhasorAnalytics Software

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Robust Matrix Completion State Estimation in Distribution Systems

Due to the insufficient measurements in the distribution system state estimation (DSSE), full observability and redundant measurements are difficult to achieve without using the pseudo measurements. The matrix completion state estimation (MCSE) combines the matrix completion and power system model to estimate voltage by exploring the low-rank characteristics of the matrix. This paper proposes a robust matrix completion state estimation (RMCSE) to estimate the voltage in a distribution system under a low-observability condition. Tradition state estimation weighted least squares (WLS) method requires full observability to calculate the states and needs redundant measurements to proceed a bad data detection. The proposed method improves the robustness of the MCSE to bad data by minimizing the rank of the matrix and measurements residual with different weights. It can estimate the system state in a low-observability system and has robust estimates without the bad data detection process in the face of multiple bad data. The method is numerically evaluated on the IEEE 33-node radial distribution system. The estimation performance and robustness of RMCSE are compared with the WLS with the largest normalized residual bad data identification (WLS-LNR), and the MCSE.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Multi-area parameter error identification for large power systems

Power grid model parameters may contain errors due to various reasons. Detecting and correcting parameter errors typically requires significant computational effort due to the size and complexity of the parameter database. While the normalized Lagrange multiplier (NLM) method can effectively detect, identify and correct parameter errors, its computational burden could rapidly grow with increasing system size. This paper addresses this issue by proposing a multi-area parameter error identification method. Each area has its own outlier detection tool for detecting the incorrect parameters and measurements within the area. On the other hand, due to the reduced redundancy at area boundaries, parameter errors on branches incident to boundary buses may not be detected. Such errors are subsequently detected by a coordination level estimator completing the system-wide parameter detection procedure. In conclusion, performance of the developed method is demonstrated using the IEEE 118-bus and 2000-bus Texas synthetic systems.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS2008

Cell-Detection Technique for Automated Patch Clamping

A unique and customizable machinevision and image-data-processing technique has been developed for use in automated identification of cells that are optimal for patch clamping. [Patch clamping (in which patch electrodes are pressed against cell membranes) is an electrophysiological technique widely applied for the study of ion channels, and of membrane proteins that regulate the flow of ions across the membranes. Patch clamping is used in many biological research fields such as neurobiology, pharmacology, and molecular biology.] While there exist several hardware techniques for automated patch clamping of cells, very few of those techniques incorporate machine vision for locating cells that are ideal subjects for patch clamping. In contrast, the present technique is embodied in a machine-vision algorithm that, in practical application, enables the user to identify good and bad cells for patch clamping in an image captured by a charge-coupled-device (CCD) camera attached to a microscope, within a processing time of one second. Hence, the present technique can save time, thereby increasing efficiency and reducing cost. The present technique involves the utilization of cell-feature metrics to accurately make decisions on the degree to which individual cells are "good" or "bad" candidates for patch clamping. These metrics include position coordinates (x,y) in the image plane, major-axis length, minor-axis length, area, elongation, roundness, smoothness, angle of orientation, and degree of inclusion in the field of view. The present technique does not require any special hardware beyond commercially available, off-the-shelf patch-clamping hardware: A standard patchclamping microscope system with an attached CCD camera, a personal computer with an imagedata- processing board, and some experience in utilizing imagedata- processing software are all that are needed. A cell image is first captured by the microscope CCD camera and image-data-processing board, then the image data are analyzed by software that implements the present machine-vision technique. This analysis results in the identification of cells that are "good" candidates for patch clamping (see figure). Once a "good" cell is identified, a patch clamp can be effected by an automated patchclamping apparatus or by a human operator. This technique has been shown to enable reliable identification of "good" and "bad" candidate cells for patch clamping. The ultimate goal in further development of this technique is to combine artificial-intelligence processing with instrumentation and controls in order to produce a complete "turnkey" automated patch-clamping system capable of accurately and reliably patch clamping cells with a minimum intervention by a human operator. Moreover, this technique can be adapted to virtually any cellular-analysis procedure that includes repetitive operation of microscope hardware by a human.

McDowell, Mark↗
DOE OSTI2022

Big Data Synchrophasor Monitoring and Analytics for Resiliency Tracking (BDSMART)

This report contains key findings from a project titled Big Data Synchrophasor Monitoring and Analytics for Resiliency Tracking (BDSMART), which was carried out through a collaborative effort of a team of researchers from Texas A&M Engineering Experiment Station, Temple University, and Quanta Technology, LLC. The in-kind support came from OSIsoft (acquired by AVEVA), which provided their PI Historian software to demonstrate the use case of streaming PMU data. The first section of the report describes the project goals and objectives related to the development of Machine Learning (ML) models capable of detecting and classifying events by processing phasor measurements captured in the field by Phasor Measurement Units (PMUs). The data for this study was contributed by the utilities/ISOs from the Western and Eastern interconnects and ERCOT, further referred to as Interconnect B (IC B), Interconnect A (IC A), and Interconnect C (IC C), respectively. The approach that the BDSMART Research Team proposed and the key research tasks defined by the team are outlined in this section. The next section describes the technical approach. We first discuss the data constraints related to the PMU measurements and data interpretation constraints imposed by the data contributors. They provided neither the topological information of the grid nor PMU placement locations and captured recorded data at very few locations in the system with the reporting rate of either 30 or 60 fps. The recordings are mostly positive sequence voltage, frequency, and ROCOF, and in some limited cases, three-phase voltages and currents. We then reflect on the bad data issues that stem from poor recording practices and vague definitions of the PMU status bits to supposedly be used for bad data identification. Finally, the data discovery points to imprecise time stamps with incomplete event start/end time, as well as inconsistent and incomplete event labeling, which combined make the implementation of the data models using supervising learning quite challenging. Following the data discovery study, we hypothesize that because the IC B data has the most complete labels, we should focus our model development on that data and then test it on data from other interconnects. We also define the common metrics used to evaluate the results from the ML algorithm tests. We concluded this section by summarizing the common ML models we used and explaining how we implemented and tested them. The issues from this section are expanded in the Training Dataset Report from this project. The final section of this report deals with the accomplishments and conclusions. As the accomplishments, we formulate the problem we are solving and what is achieved by solving the problem. We then reflect on each of the analytics tools we developed and point out the performance of each tool when applied to solving the mentioned problems. We reference this work for further details to the papers we published on each tool. In the conclusions, we give recommendations on how to improve future PMU recording practices to facilitate the ML algorithm implementation and guidance for the future standardization work aimed at clarifying the ambiguities associated with the PMU status bits. We finally list future tasks that can bring about further improvements in the proposed algorithms. The issues from this section are expanded in the Training, and Test Dataset Report filed at the project completion date.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Quasars with large proper motions: A selection from the LQAC-5 catalogue combined with Gaia EDR3: Focusing on astrometric and photometric properties

Thanks to the cross-identification between the Gaia EDR3 (E-Data Release 3) catalogue and the quasars’ compiled catalogue LQAC-5 (Large Quasar Astrometric Catalogue-5), accurate astrometric parameters as well as photometric measurements were available for a set of 416 113 quasars. After analysing the astrometric and photometric properties of these quasars, we concentrate our study on objects characterized by significant proper motion, which contradicts the postulate that they are fixed in the celestial sphere. We derived statistical properties of the positions, parallaxes, and proper motions of the quasars cross-matched with Gaia EDR3. We carried out a specific investigation of quasars with a proper motion larger than 10 mas yr -1 , and used colour-colour diagrams in mid-infrared to discriminate these quasars from stellar contamination. We discuss, in detail, a few cases where the abnormal proper motion could be caused by misidentifications. We note that 43% of LQAC-5 quasars were not detected by Gaia EDR3, mainly because of the cut-off magnitude at G = 21. The comparison between r(G) (Gaia) and r (SDSS) shows a normal distribution. The normalized distributions of parallaxes and proper motions are identical to that of the Gaia CRF2, as well as the distribution of the G BP -G RP colour index. We have identified 937 LQAC-5 objects (0.27% of the sample) with Gaia EDR3 proper motion which is significant at the 5σ level. We have clearly shown that below z = 0.35, the integrated magnitude G(r) obtained from SDSS spectra is smaller than G (Gaia), leading to the statement that the quasars are generally resolved under this threshold, which should lead to a bad determination of proper motion. Moreover, to discriminate quasars from stellar contamination, we have plotted them in a mid-infrared colour–colour diagram. Finally, we have made a final selection of 41 objects with μ > 10 mas yr -1 , satisfying drastic constraints, which makes them potential candidates for considering their proper motion as real. In this paper we carried out a complete analysis of the quasars of the LQAC-5 which were cross-identified with Gaia EDR3 ; this serves as an alternative and complementary study with respect to Gaia CRF2, involving a different population of quasars. We selected a set of 41 quasars with a proper motion exceeding 10 mas yr -1 , which can be considered as very high for objects which are a priori fixed in the celestial sphere.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

CHESS 2025: Spectrometer orthorectified at-sensor radiance from NEON AOP imaging spectroscopy surveys

This dataset provides Level 1 (L1) orthorectified at-sensor radiance derived from measurements collected by the Imaging Spectrometer-1 (NIS-1) onboard the NEON (National Ecological Observatory Network) Airborne Observation Platform (AOP) for the 2025 Colorado Headwaters Ecological Spectroscopy Study (CHESS). NIS-1 captures light reflected from the Earth’s surface in 426 discrete wavelength bands as raw digital numbers (DNs; Level 0). These data are then calibrated to physical units (uW/cm²·sr·nm) following the processing steps described in the NEON Imaging Spectrometer Level 1B Calibrated Radiance Algorithm Theoretical Basis Document (ATBD; Gallery 2022). The data delivered here are the primary inputs for the surface reflectance product in “Custom surface reflectance, shade masks, and equivalent water thickness maps for the Colorado Headwaters Ecological Spectroscopy Study” (Carroll et al. 2026). For intertemporal comparison, the radiance data here are most directly relatable to the v2 radiance data in “NEON AOP Imaging Spectroscopy Survey of Upper East River Colorado Watersheds: Raw-Space Radiance and Observational Variable Dataset” (Goulden et al. 2018), to which the same processing methodology was applied. Together, the radiance and reflectance data enable users to exploit the unique reflection signatures of different surface objects for land cover classification, foliar trait mapping, plant vigor assessment, water content estimation, trace-element identification, and other scientific applications. The data were acquired over three study domains in the Upper Gunnison river basin: the upper East River watershed (CRBU); Almont Triangle and Taylor Canyon (ALMO); and Upper Taylor River watershed (UPTA) between 2025-06-13 and 2025-07-15. Within each domain, data are delivered by flightline as orthorectified and calibrated hyperspectral rasters in Hierarchical Data Format version 5 (HDF5) format, with radiance values provided in uW/cm²·sr·nm on a fixed, uniform Universal Transverse Mercator (UTM) grid at 1 meter spatial resolution. The radiance rasters include all 426 NIS-1 spectral bands, along with associated quality-assurance (QA) and diagnostic and ancillary layers needed for atmospheric correction workflows. Orthorectified radiance is produced from pushbroom spectrometer observations by applying NEON’s radiometric calibration (including bad pixel masking, dark subtract, dark pedestal shift correction, electronic panel ghost correction, grating ghost correction, deblur correction and flat-fielding) and spectral calibration (using spectral response function band centers and full-width at half-maximum intensity), followed by geolocation and regridding to the fixed grid. CHESS Project Description: The Colorado Headwaters Ecological Spectroscopy Study (CHESS) comprised a multi-week airborne remote sensing and field observation campaign in the Upper Gunnison Basin, Colorado, conducted in June and July of 2025. Airborne remote sensing was conducted by the National Ecological Observatory Network Airborne Observation Platform (NEON AOP), concurrent with a field campaign run by the Rocky Mountain Biological Laboratory (RMBL), the Lawrence Berkeley National Laboratory (LBNL) and SLAC National Accelerator Laboratory Watershed Function Science Focus Area (SFA), and NASA-JPL (Jet Propulsion Laboratory) Earth Surface Mineral Dust Source Investigation (EMIT) program. Between June 10 and July 18, 2025, the NEON AOP flight team collected high-resolution aerial imaging spectroscopy and Light Detection and Ranging (LiDAR) data over three domains: the Upper East River (CRBU), Almont Triangle (ALMO), and the Upper Taylor Basin (UPTA). In coordination with the flights, a field campaign acquired ground-truth observations, including observations of vegetation composition, foliar traits, forest demography, and subsurface properties in 18 core sampling areas within the domains. Additional surface water observations were taken at over 380 point locations. All CHESS campaign datasets can be found within the CHESS ESS-DIVE data portal: https://data.ess-dive.lbl.gov/portals/chess. Funding Acknowledgement: Field and remote-sensing data acquisition was performed under a grant from the National Aeronautics and Space Administration (80NSSC24K1005). This work was also supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

2018 NEON and 2025 CHESS Campaigns↗