Search NASA⌕ Search

SEARCH · Search NASA

Results for “Imputation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

PVplr-stGNN 0.1.10

PV Performance Loss Rate Estimation using Spatio-temporal Graph Neural Networks PVplr-stGNN is a Python 3 package developed by the SDLE Research Center at Case Western Reserve University in Cleveland OH. This repository contains the full source PVplr-stGNN package. The package contains the PV-stGAE for missingness data detection and imputation and PV-DynGNN for PLR estimation.

Fan, Yangxin [Case Western Reserve Univ., Clevelan↗

Generation and evaluation of synthetic patient data

Background: Machine learning (ML) has made a significant impact in medicine and cancer research; however, its impact in these areas has been undeniably slower and more limited than in other application domains. A major reason for this has been the lack of availability of patient data to the broader ML research community, in large part due to patient privacy protection concerns. High-quality, realistic, synthetic datasets can be leveraged to accelerate methodological developments in medicine. By and large, medical data is high dimensional and often categorical. These characteristics pose multiple modeling challenges. Methods: In this paper, we evaluate three classes of synthetic data generation approaches; probabilistic models, classification-based imputation models, and generative adversarial neural networks. Metrics for evaluating the quality of the generated synthetic datasets are presented and discussed. Results: While the results and discussions are broadly applicable to medical data, for demonstration purposes we generate synthetic datasets for cancer based on the publicly available cancer registry data from the Surveillance Epidemiology and End Results (SEER) program. Specifically, our cohort consists of breast, respiratory, and non-solid cancer cases diagnosed between 2010 and 2015, which includes over 360,000 individual cases. Conclusions: We discuss the trade-offs of the different methods and metrics, providing guidance on considerations for the generation and usage of medical synthetic data.

59 BASIC BIOLOGICAL SCIENCES↗

Genome-Wide Association of Lipid-Lowering Response to Statins in Combined Study Populations

Statins effectively lower total and plasma LDL-cholesterol, but the magnitude of decrease varies among individuals. To identify single nucleotide polymorphisms (SNPs) contributing to this variation, we performed a combined analysis of genome-wide association (GWA) results from three trials of statin efficacy. Bayesian and standard frequentist association analyses were performed on untreated and statin-mediated changes in LDL-cholesterol, total cholesterol, HDL-cholesterol, and triglyceride on a total of 3932 subjects using data from three studies: Cholesterol and Pharmacogenetics (40 mg/day simvastatin, 6 weeks), Pravastatin/ Inflammation CRP Evaluation (40 mg/day pravastatin, 24 weeks), and Treating to New Targets (10 mg/day atorvastatin, 8 weeks). Genotype imputation was used to maximize genomic coverage and to combine information across studies. Phenotypes were normalized within each study to account for systematic differences among studies, and fixed-effects combined analysis of the combined sample were performed to detect consistent effects across studies. Two SNP associations were assessed as having posterior probability greater than 50%, indicating that they were more likely than not to be genuinely associated with statin-mediated lipid response. SNP rs8014194, located within the CLMN gene on chromosome 14, was strongly associated with statin-mediated change in total cholesterol with an 84% probability by Bayesian analysis, and a p-value exceeding conventional levels of genome-wide significance by frequentist analysis (P = 1.8 x 10 -8 ). This SNP was less significantly associated with change in LDL-cholesterol (posterior probability = 0.16, P = 4.0 x 10 -6 ). Bayesian analysis also assigned a 51% probability that rs4420638, located in APOC1 and near APOE, was associated with change in LDL-cholesterol. Using combined GWA analysis from three clinical trials involving nearly 4,000 individuals treated with simvastatin, pravastatin, or atorvastatin, we have identified SNPs that may be associated with variation in the magnitude of statin-mediated reduction in total and LDL-cholesterol, including one in the CLMN gene for which statistical evidence for association exceeds conventional levels of genome-wide significance.

59 BASIC BIOLOGICAL SCIENCES↗

Quality Assurance and Quality Control (QA/QC) of Meteorological Time Series Data for Billy Barr, East River, Colorado USA

A comprehensive Quality Assurance (QA) and Quality Control (QC) statistical framework consists of three major phases: Phase 1—Preliminary raw data sets exploration, including time formatting and combining datasets of different lengths and different time intervals; Phase 2—QA of the datasets, including detecting and flagging of duplicates, outliers, and extreme values; and Phase 3—the development of time series of a desired frequency, imputation of missing values, visualization and a final statistical summary. The time series data collected at the Billy Barr meteorological station (East River Watershed, Colorado) were analyzed. The developed statistical framework is suitable for both real-time and post-data-collection QA/QC analysis of meteorological datasets.The files that are in this data package include one excel file, converted to CSV format (Billy_Barr_raw_qaqc.csv) that contains the raw meteorological data, i.e., input data used for the QA/QC analysis. The second CSV file (Billy_Barr_1hr.csv) is the QA/QC and flagged meteorological data, i.e., output data from the QA/QC analysis. The last file (QAQC_Billy_Barr_2021-03-22.R) is a script written in R that implements the QA/QC and flagging process. The purpose of the CSV data files included in this package is to provide input and output files implemented in the R script.

54 ENVIRONMENTAL SCIENCES↗

QA/QC-ed BCI meteorological drivers

Time series raw datasets of Temperature, Solar Radiation, Relative Humidity, Rainfall, Barometric Pressure, Wind Speed, and Wind Direction for the Barro Colorado (PA-BCI) site for the period from 2003-01-01 to 2016-12-31 were downloaded from the STRI website. The QA/QC protocol of raw datasets included the detection and removal of NAs, outliers and bad data points, imputation of missing data, and creation of the time series with equidistant time stamps. The time series for different meteorological parameters were aligned into a single database to be used for modeling. Statistical QA/QC analysis was performed using a series of R libraries in Rstudio. The changes in the dataset were flagged. The file BCI_met_drivers_2003-2016_QAQC_summary_report.docx provides the details of the QA/QC data analysis.

54 ENVIRONMENTAL SCIENCES↗

Sequential Projection Pursuit Principal Component Analysis – Dealing with Missing Data Associated with New -Omics Technologies

We present a new version of sequential projection pursuit Principal Component Analysis (sppPCA) that has the capability to perform PCA on large multivariate datasets that contain non-random missing values. We demonstrate that sppPCA generates more robust and informative low-dimensional representations of the data than imputation-based approaches and improved downstream statistical analyses, such as clustering or classification. A Java program to run sppPCA is freely available at https://www.biopilot.org/docs/Software/sppPCA.

Webb-Robertson, Bobbie-Jo M.↗

ADEPT Final Report: Dynamic Control of Grid Assets Using Direct AC Converter Cells

Seedling funding was secured through the ARPA-E program to develop a robust, utility-scale Power Converter Augmented Transformer (PCAT). The project designed, fabricated, and tested two 13 kV, 1 MW PCATs operating at an effective switching frequency of up to 50 kHz and at an efficiency of greater than 99%. Two versions of the PCAT were fabricated – one using an imputed DC link (IDCL) topology and the other using a back-to-back (BTB) topology. The IDCL avoids the use of electrolytic capacitors for bulk energy storage, thus improving the lifetime and decreasing the size of the converter, while the BTB topology uses an industry-proven topology that is more commercially viable in the short term. To give the PCAT a lifetime and maintenance profile compatible with typical existing utility assets, a novel Passive Thermal Management System (PTMS) was designed and integrated into both PCATs. The PTMS uses a thermosiphon to remove heat from the power electronic devices without forced cooling and reject it to the ambient environment. In addition, a fail-normal feature was implemented to ensure that in the event of PCAT failure the functionality of the underlying transformer is retained. Attainment of project objectives was verified through testing at the Georgia Tech National Electrical Energy Testing Research and Applications Center (NEETRAC). The IDCL-based PCAT had an efficiency of more than 99.10% and the BTB-based PCAT had an efficiency of more than 99.75%. During testing, the PTMS exceeded project requirements by rejecting the 5 kW output of the converters to a 40 °C environment while keeping devices junction temperatures within their tolerances.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Online mapping and forecasting of epidemics using open-source indicators

Open-source indicators have been proposed as a way of tracking and forecasting disease outbreaks. Some, such are meteorological data, are readily available as reanalysis products. Others, such as those derived from our online behavior (web searches, media article etc.) are gathered easily and are more timely than public health reporting. In this study we investigate how these datastreams may be combined to provide useful epidemiological information. The investigation is performed by building data assimilation systems to track influenza in California and dengue in India. The first does not suffer from incomplete data and was chosen to explore disease modeling needs. The second explores the case when observational data is sparse and disease modeling complexities are beside the point. The two test cases are for opposite ends of the disease tracking spectrum. We find that data assimilation systems that produce disease activity maps can be constructed. Further, being able to combine multiple open-source datastreams is a necessity as any one individually is not very informative. The data assimilation systems have very little in common except that they contain disease models, calibration algorithms and some ability to impute missing data. Thus while the data assimilation systems share the goal for accurate forecasting, they are practically designed to compensate for the shortcomings of the datastreams. Thus we expect them to be disease and location-specific.

42 ENGINEERING↗

Phasor-Measurement-Unit-Based Data Analytics Using Digital Twin and PhasorAnalytics Software

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

24 POWER TRANSMISSION AND DISTRIBUTION↗

FAIRification, Quality Assessment, and Missingness Pattern Discovery for Spatiotemporal Photovoltaic Data

The ongoing growth of the photovoltaic market has pushed the demand for power forecasting and performance evaluation for a huge population of PV power plants. Through access to a large number of time series data sets from different power plants, we have found common issues that impede the modeling process. Namely, the time series data are hard to transfer between groups due to differences in variable nomenclature, and the quality of the data sets can vary. We address the issue of variable nomenclature by FAIRifying spatiotemporal PV time series data. Through the creation of a solar power plant ontology, we propose standards for the naming and structure of metadata used to describe the data from these power plants. Using the structure from this ontology, we have developed both R and Python packages for the automation of the FAIRification process. We have also developed an R package that automates the analysis of the quality of a data set through the designation of letter grades. With access to large time series data sets across many power plants, we can utilize spatiotemporal coherence between the sites in order to improve the quality of our data. To solve the issue of data missingness, we propose the use of Spatiotemporal-GNN autoencoders to detect and impute missing values from a data set by utilizing data from power plants nearby.

14 SOLAR ENERGY↗

Jobs, jobs, jobs: what’s an analyst to do?

Analysts and economists often face the task of using employment metrics to characterize industries of interest. Some key challenges can be understanding where to find employment metrics, the differences in various employment metrics, and when each metric should be used. This article analyzes a variety of publicly available employment data for the United States and compares these data. A detailed description of the intricacies of each data source is provided, which covers factors such as regionality, industry breakout, periodicity, and the types of jobs included. This article provides several case study examples, using the oil and gas extraction, coal mining, and chemical manufacturing sectors to portray challenges data users may face when developing employment estimates that suit their needs. Data users should be aware of a variety of data sources to understand alternative analysis options when data limitations are present and to determine which data source best meets their needs. Instances may occur in which information from one dataset may be used to help impute missing values.

99 GENERAL AND MISCELLANEOUS↗

Who Has the Upper HAND in a Hurricane? Flood Risk, Weather Shocks, and Property Prices

Flood risk is increasing in the United States, but how property markets respond to this hazard is unclear. Focusing on Houston, Texas, we analyze changes in flood risk capitalization in property prices after three major Gulf Coast hurricanes. Here, we compare two measures of flood risk–floodplain designations from the Federal Emergency Management Agency (FEMA), and a hydrologically-imputed, continuous measure, Height Above the Nearest Drainage (HAND). HAND affects property prices after a storm in intuitive ways, while we observe counterintuitive post-storm price premiums within FEMA floodplains. Plausibly exogenous measures like HAND may be useful tools for characterizing flood risk in economic analyses.

Plough, Julian A. [University of North Carolina, C↗

Physics-Informed Deep Learning for Reconstruction of Spatial Missing Climate Information in the Antarctic

Understanding the influence of the Antarctic on the global climate is crucial for the prediction of global warming. However, due to very few observation sites, it is difficult to reconstruct the rational spatial pattern by filling in the missing values from the limited site observations. To tackle this challenge, regional spatial gap-filling methods, such as Kriging and inverse distance weighted (IDW), are regularly used in geoscience. Nevertheless, the reconstructing credibility of these methods is undesirable when the spatial structure has massive missing pieces. Inspired by image inpainting, we propose a novel deep learning method that demonstrates a good effect by embedding the physics-aware initialization of deep learning methods for rapid learning and capturing the spatial dependence for the high-fidelity imputation of missing areas. We create the benchmark dataset that artificially masks the Antarctic region with ratios of 30%, 50% and 70%. The reconstructing monthly mean surface temperature using the deep learning image inpainting method RFR (Recurrent Feature Reasoning) exhibits an average of 63% and 71% improvement of accuracy over Kriging and IDW under different missing rates. With regard to wind speed, there are still 36% and 50% improvements. In particular, the achieved improvement is even better for the larger missing ratio, such as under the 70% missing rate, where the accuracy of RFR is 68% and 74% higher than Kriging and IDW for temperature and also 38% and 46% higher for wind speed. In addition, the PI-RFR (Physics-Informed Recurrent Feature Reasoning) method we proposed is initialized using the spatial pattern data simulated by the numerical climate model instead of the unified average. Compared with RFR, PI-RFR has an average accuracy improvement of 10% for temperature and 9% for wind speed. When applied to reconstruct the spatial pattern based on the Antarctic site observations, where the missing rate is over 90%, the proposed method exhibits more spatial characteristics than Kriging and IDW.

54 ENVIRONMENTAL SCIENCES↗

STAIR 2.0: A Generic and Automatic Algorithm to Fuse Modis, Landsat, and Sentinel-2 to Generate 10 m, Daily, and Cloud-/Gap-Free Surface Reflectance Product

Remote sensing datasets with both high spatial and high temporal resolution are critical for monitoring and modeling the dynamics of land surfaces. However, no current satellite sensor could simultaneously achieve both high spatial resolution and high revisiting frequency. Therefore, the integration of different sources of satellite data to produce a fusion product has become a popular solution to address this challenge. Many methods have been proposed to generate synthetic images with rich spatial details and high temporal frequency by combining two types of satellite datasets—usually frequent coarse-resolution images (e.g., MODIS) and sparse fine-resolution images (e.g., Landsat). In this paper, we introduce STAIR 2.0, a new fusion method that extends the previous STAIR fusion framework, to fuse three types of satellite datasets, including MODIS, Landsat, and Sentinel-2. In STAIR 2.0, input images are first processed to impute missing-value pixels that are due to clouds or sensor mechanical issues using a gap-filling algorithm. The multiple refined time series are then integrated stepwisely, from coarse- to fine- and high-resolution, ultimately providing a synthetic daily, high-resolution surface reflectance observations. We applied STAIR 2.0 to generate a 10-m, daily, cloud-/gap-free time series that covers the 2017 growing season of Saunders County, Nebraska. Moreover, the framework is generic and can be extended to integrate more types of satellite data sources, further improving the quality of the fusion product. View Full-Text

47 OTHER INSTRUMENTATION↗

A COMPREHENSIVE STATISTICAL DESCRIPTION OF RADIO-THROUGH- γ -RAY SPECTRAL ENERGY DISTRIBUTIONS OF ALL KNOWN BLAZARS

We combined multi-wavelength data for blazars from the Roma-BZCAT catalog and analyzed hundreds of X-ray spectra. We present the fluxes and spectral energy distributions (SEDs), in 12 frequency bands from radio to γ -rays, for a final sample of 2214 blazars. Using a model-independent statistical approach, we looked for systematic trends in the SEDs; the most significant trends involved the radio luminosities and X-ray spectral indices of the blazars. We used a principal component analysis (PCA) to determine the basis vectors of the blazar SEDs and, in order to maximize the size of the sample, imputed missing fluxes using the K -nearest neighbors method. Using more than an order of magnitude more data than was available when Fossati et al. first reported trends of SED shape with blazar luminosity, we confirmed the anti-correlation between radio luminosity and synchrotron peak frequency, although with greater scatter than was seen in the smaller sample. The same trend can be seen between bolometric luminosity and synchrotron peak frequency. Finally, we used all of the available blazar data to determine an empirical SED description that depends only on the radio luminosity at 1.4 GHz and the redshift. We verified that this statistically significant relation was not a result of the luminosity–luminosity correlations that are natural in flux-limited samples (i.e., where the correlation is actually caused by the redshift rather than the luminosity).

79 ASTRONOMY AND ASTROPHYSICS↗

A Random Forest Approach to Identifying Young Stellar Object Candidates in the Lupus Star-forming Region

The identification and characterization of stellar members within a star-forming region are critical to many aspects of star formation, including formalization of the initial mass function, circumstellar disk evolution, and star formation history. Previous surveys of the Lupus star-forming region have identified members through infrared excess and accretion signatures. We use machine learning to identify new candidate members of Lupus based on surveys from two space-based observatories: ESA’s Gaia and NASA’s Spitzer. Astrometric measurements from Gaia's Data Release 2 and astrometric and photometric data from the Infrared Array Camera on the Spitzer Space Telescope, as well as from other surveys, are compiled into a catalog for the random forest (RF) classifier. The RF classifiers are tested to find the best features, membership list, non-membership identification scheme, imputation method, training set class weighting, and method of dealing with class imbalance within the data. We list 27 candidate members of the Lupus star-forming region for spectroscopic follow-up. Most of the candidates lie in Clouds V and VI, where only one confirmed member of Lupus was previously known. These clouds likely represent a slightly older population of star formation.

79 ASTRONOMY AND ASTROPHYSICS↗

Functional Data Analysis for Extracting the Intrinsic Dimensionality of Spectra: Application to Chemical Homogeneity in the Open Cluster M67

High-resolution spectroscopic surveys of the Milky Way have entered the Big Data regime and have opened avenues for solving outstanding questions in Galactic archeology. However, exploiting their full potential is limited by complex systematics, whose characterization has not received much attention in modern spectroscopic analyses. In this work, we present a novel method to disentangle the component of spectral data space intrinsic to the stars from that due to systematics. Using functional principal component analysis on a sample of 18,933 giant spectra from APOGEE, we find that the intrinsic structure above the level of observational uncertainties requires ≈10 functional principal components (FPCs). Our FPCs can reduce the dimensionality of spectra, remove systematics, and impute masked wavelengths, thereby enabling accurate studies of stellar populations. To demonstrate the applicability of our FPCs, we use them to infer stellar parameters and abundances of 28 giants in the open cluster M67. We employ Sequential Neural Likelihood, a simulation-based Bayesian inference method that learns likelihood functions using neural density estimators, to incorporate non-Gaussian effects in spectral likelihoods. By hierarchically combining the inferred abundances, we limit the spread of the following elements in M67: Fe ≲ 0.02 dex; C ≲ 0.03 dex; O, Mg, Si, Ni ≲ 0.04 dex; Ca ≲ 0.05 dex; N, Al ≲ 0.07 dex (at 68% confidence). Our constraints suggest a lack of self-pollution by core-collapse supernovae in M67, which has promising implications for the future of chemical tagging to understand the star formation history and dynamical evolution of the Milky Way.

79 ASTRONOMY AND ASTROPHYSICS↗

Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams

Abstract We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.

79 ASTRONOMY AND ASTROPHYSICS↗