Search NASA⌕ Search

SEARCH · Search NASA

Results for “missing values”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Land Surface Phenology from MODIS: Characterization of the Collection 5 Global Land Cover Dynamics Product

Information related to land surface phenology is important for a variety of applications. For example, phenology is widely used as a diagnostic of ecosystem response to global change. In addition, phenology influences seasonal scale fluxes of water, energy, and carbon between the land surface and atmosphere. Increasingly, the importance of phenology for studies of habitat and biodiversity is also being recognized. While many data sets related to plant phenology have been collected at specific sites or in networks focused on individual plants or plant species, remote sensing provides the only way to observe and monitor phenology over large scales and at regular intervals. The MODIS Global Land Cover Dynamics Product was developed to support investigations that require regional to global scale information related to spatiotemporal dynamics in land surface phenology. Here we describe the Collection 5 version of this product, which represents a substantial refinement relative to the Collection 4 product. This new version provides information related to land surface phenology at higher spatial resolution than Collection 4 (500-m vs. 1-km), and is based on 8-day instead of 16-day input data. The paper presents a brief overview of the algorithm, followed by an assessment of the product. To this end, we present (1) a comparison of results from Collection 5 versus Collection 4 for selected MODIS tiles that span a range of climate and ecological conditions, (2) a characterization of interannual variation in Collections 4 and 5 data for North America from 2001 to 2006, and (3) a comparison of Collection 5 results against ground observations for two forest sites in the northeastern United States. Results show that the Collection 5 product is qualitatively similar to Collection 4. However, Collection 5 has fewer missing values outside of regions with persistent cloud cover and atmospheric aerosols. Interannual variability in Collection 5 is consistent with expected ranges of variance suggesting that the algorithm is reliable and robust, except in the tropics where some systematic differences are observed. Finally, comparisons with ground data suggest that the algorithm is performing well, but that end of season metrics associated with vegetation senescence and dormancy have higher uncertainties than start of season metrics.

Land cover dynamics↗

MODIS Collection 6 MAIAC Algorithm

This paper describes the latest version of the algorithm MAIAC (Multi-Angle Implementation of Atmospheric Correction) used for processing the MODIS (Moderate-resolution Imaging Spectroradiometer) Collection6 data record. Since initial publication in 2011-2012, MAIAC has changed considerably to adapt to global processing and improve cloud/snow detection, aerosol retrievals and atmospheric correction of MODIS data. The main changes include (1) transition from a 25 to 1 km scale for retrieval of the spectral regression coefficient (SRC) which helped to remove occasional blockiness at 25 km scale in the aerosol optical depth (AOD) and in the surface reflectance, (2) continuous improvements of cloud detection, (3) introduction of smoke and dust tests to discriminate absorbing fine- and coarse mode aerosols, (4) adding over-water processing, (5) general optimization of the LUT (LookUp-Table)-based radiative transfer for the global processing, and others. MAIAC provides an interdisciplinary suite of atmospheric and land products, including cloud mask (CM), column water vapor (CWV), AOD at 0.47 and 0.55 m, aerosol type (background, smoke or dust) and fine-mode fraction over water; spectral bidirectional reflectance factors (BRF), parameters of Ross-thick Lisparse (RTLS) bidirectional reflectance distribution function (BRDF) model and instantaneous albedo. For snow-covered surfaces, we provide subpixel snow fraction and snow grain size. All products come in standard HDF4 (software library) format at 1 km resolution, except for BRF, which is also provided at 500 m resolution on a sinusoidal grid adopted by the MODIS Land team. All products are provided on per-observation basis in daily files except for the BRDF/Albedo product, which is reported every 8 days. Because MAIAC uses a time series approach, BRDF/Albedo is naturally gap-filled over land where missing values are filled-in with results from the previous retrieval. While the BRDF model is reported for MODIS Land bands 1-7 and ocean band 8, BRF is reported for both land and ocean bands 1-12. This paper focuses on MAIAC cloud detection, aerosol retrievals and atmospheric correction and describes MCD19 data products and quality assurance (QA) flags.

MAIAC Algorithm↗

MLtool: Universal Supervised Machine Learning Tool to Model Tabulated Data

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine learning↗

MLtool Python Code

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine Learning↗

Utility-scale Building Type Assignment Using Smart Meter Data

United States building energy use accounted for 40% of total energy use, 74% of peak demand, and $412 billion in 2019. Building energy modeling allows researchers to simulate building physics, gain insights into possible energy/demand saving opportunities, and assess cost-effective resilience amidst climate change. Many building features needed to create building energy models are readily available such as 2D footprints and LiDAR (height). A critical feature that is not generally obtainable is the building type. In partnership with a utility, a years worth of real-world, 15-minute electrical use data has been examined. The smart meter data is compared to 97 different prototype building energy models to assign building type. Real-world considerations including data preparation, quality assurance, and handling of missing values for advanced metering infrastructure data are addressed. Euclidean distance for pattern-matching of energy use, dynamic time warping, and time-window statistics with machine learning are compared for determining building type from measured electricity use.

Bass, Brett↗

A Data Quality-Aware Framework to Reliably Forecast Photovoltaic Generation and Consumer Load for an Improved Resilience of Microgrids

Photovoltaic (PV) power and consumer load forecasting plays a critical role to ensure operational resilience of the electric grid. Most data-driven forecasting algorithms rely heavily on the continuous availability of good quality data for periodic training and validation. When deployed at the grid’s edge, prolonged disruptions to communications during extreme events degrade data quality. Factors such as missing observations, epistemic uncertainties, data drift, and concept drift are manifestations of data quality that impact the generalization of such field-deployed forecasting models. Currently, there exists no mechanism in the literature to dynamically switch between models under varying degrees of data quality as quantified by certain metrics for each factor highlighted above. This paper addresses this shortcoming by conceptually introducing a data qualityaware framework for reliable PV generation and consumer load forecasting. The framework’s design incorporates components of missing values, divergence tests, and continuous monitoring of generalization performance to detect changes in data quality caused by communications disruptions and trigger specific classes of forecasting models grouped under three use cases (UC1- UC3). As a first step towards validating this framework, real data collected from an actual field microgrid system is used to demonstrate the viability of the three use cases. Results show that the performance is the best in UC1 with an unadjusted R-square value of 0.954, followed by 0.939 for UC2 and 0.757 for UC3.

Sundararajan, Aditya↗

Merging of OMI and AIRS Ozone Data

The OMI Instrument measures ozone using the backscattered light in the UV part of the spectrum. In polar night there are no OMI measurements so we hope to incorporate the AIRS ozone data to fill in these missing regions. AIRS is on the Aqua platform and has been operating since May 2002. AIRS is a multi-detector array grating spectrometer containing 2378 IR channels between 650 per centimeter and 2760 per centimeter which measures atmospheric temperature, precipitable water, water vapor, CO, CH4, CO2 and ozone profiles and column amount. It can also measure effective cloud fraction and cloud top pressure for up to two cloud layers and sea-land skin temperature. Since 2008, OMI has had part of its aperture occulted with a piece of the thermal blanket resulting in several scan positions being unusable. We hope to use the AIRS data to fill in the missing ozone values for those missing scan positions.

OMI↗

Combined Statistical Analyses of Peptide Intensities and Peptide Occurrences Improves Identification of Significant Peptides from MS-based Proteomics Data

Liquid chromatography-mass spectrometry-based (LC-MS) proteomics uses peak intensities of proteolytic peptides to infer the differential abundance of peptides/proteins. However, substantial run-to-run variability in peptide intensities and observations (presence/absence) of peptides makes data analysis quite challenging. The missing abundance values in LC-MS proteomics data are difficult to address with traditional imputation-based approaches because the mechanisms by which data are missing are unknown a priori. Data can be missing due to random mechanisms such as experimental error, or non-random mechanisms such as a true biological effect. We present a statistical approach that uses a test of independence known as a G-test to test the null hypothesis of independence between the number of missing values and the experimental groups. We pair the G-test results evaluating independence of missing data (IMD) with a standard analysis of variance (ANOVA) that uses only means and variances computed from the observed data. Each peptide is therefore represented by two statistical confidence metrics, one for qualitative differential observation and one for quantitative differential intensity. We use two simulated and two real LC-MS datasets to demonstrate the robustness and sensitivity of the ANOVA-IMD approach for assigning confidence to peptides with significant differential abundance among experimental groups.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Scalable tensor factorizations with incomplete data.

The problem of incomplete data - i.e., data with missing or unknown values - in multi-way arrays is ubiquitous in biomedical signal processing, network traffic analysis, bibliometrics, social network analysis, chemometrics, computer vision, communication networks, etc. We consider the problem of how to factorize data sets with missing values with the goal of capturing the underlying latent structure of the data and possibly reconstructing missing values (i.e., tensor completion). We focus on one of the most well-known tensor factorizations that captures multi-linear structure, CANDECOMP/PARAFAC (CP). In the presence of missing data, CP can be formulated as a weighted least squares problem that models only the known entries. We develop an algorithm called CP-WOPT (CP Weighted OPTimization) that uses a first-order optimization approach to solve the weighted least squares problem. Based on extensive numerical experiments, our algorithm is shown to successfully factorize tensors with noise and up to 99% missing data. A unique aspect of our approach is that it scales to sparse large-scale data, e.g., 1000 x 1000 x 1000 with five million known entries (0.5% dense). We further demonstrate the usefulness of CP-WOPT on two real-world applications: a novel EEG (electroencephalogram) application where missing data is frequently encountered due to disconnections of electrodes and the problem of modeling computer network traffic where data may be absent due to the expense of the data collection process.

99 GENERAL AND MISCELLANEOUS↗

Observed and Imputed Volumetric Soil Water Content Timeseries for the New Mexico Elevation Gradient

Reliable soil water content (SWC) data are essential for understanding dryland ecosystem dynamics, but high-frequency SWC sensors often fail, creating gaps in critical datasets. To address this, we developed a Bayesian mixture model that imputes missing SWC using both linear interpolation and an ecosystem water balance model (SOILWAT2), tested across six AmeriFlux eddy covariance tower sites in the New Mexico Elevation Gradient, demonstrating its effectiveness in reconstructing SWC patterns while providing insights into the factors driving SWC variability. Daily volumetric soil water content (SWC) data are provided as csv-formatted spreadsheets for the six AmeriFlux sites (US-Seg, US-Ses, US-Wjs, US-Mpi, US-Vcp, and US-Vcs). For each site there is an observed SWC file (site_SWC_gapfill.csv) and a file that contains imputed SWC (imputed_SWC_site.csv). The observed SWC files contain temperature corrected sensor values, tower precipitation data, as well as outputs from SOILWAT2 simulations that were used to impute SWC. The imputed files contain the original observed SWC values and the imputed missing SWC values. When SWC was missing from the original data, the missing value was imputed based on the Bayesian imputation mixture model. The posterior mean of all imputed values is reported as "mean_X". When the observed SWC was NOT missing, mean_X = observed SWC value (original data). The standard deviation, 2.5th percentile and the 97.5th percentile for the imputed values are also reported in the imputed files. There are readme text files for each file type explaining the contents of each column.

54 ENVIRONMENTAL SCIENCES↗

Search for New Physics in a Final State with Same-Sign Dileptons, Jets, and Missing Transverse Energy at 7 TeV Center of Mass Energy

We report on a search for new physics in a final state with two same sign leptons, missing transverse energy, and significant hadronic activity at a center of mass energy $\sqrt{s}$ = 7 TeV. The data were collected with the CMS detector at the CERN LHC and correspond to an integrated luminosity of 0.98 $\mathrm{fb}^{-1}$. Data--driven methods are developed to estimate the dominant Standard Model backgrounds. No evidence for new physics is observed. The dominant background to the analysis comes from failures of lepton identification in Standard Model $t\bar{t}$ events. The $t\bar{t}$ production cross section in the dilepton final state is measured using 3.1 $\mathrm{fb}^{-1}$ of data. The cross section is measured to be 194 $\pm$ 72 (stat) $\pm$ 24 (syst) $\pm$ 21 (lumi) pb. An algorithm is developed that uses tracking information to improve the reconstruction of missing transverse energy. The reconstruction of missing transverse energy is commissioned using the first collisio ns recorded at 0.9, 2.36 and 7 TeV data. Events with abnormally large values of missing transverse energy are identified as arising from anomalous signals in the calorimeters. Tools are developed to identify and remove these anomalous signals.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Radiation Doses to Members of the U.S. Population from Ubiquitous Radionuclides in the Body: Part 2, Methods and Dose Calculations

This paper is part two of a three-part series investigating annual effective doses to residents of the United States from intakes of ubiquitous radionuclides, including radionuclides occurring naturally, radionuclides whose concentrations are technologically enhanced, and anthropogenic radionuclides. This series of papers explicitly excludes intakes from inhaling 222Rn, 220Rn, and their short-lived decay products; it also excludes intakes of radionuclides in occupational and medical settings. Part one reviewed, summarized, characterized, and grouped all published and some unpublished data for U.S. residents on ubiquitous radionuclide concentrations in tissues and organs. Assumptions about equilibrium with long-lived parents are made for the 28 other radionuclides in these series lacking data. This paper describes the methods developed to group the collected data into source regions described in the Radiation Dose Assessment Resource (RADAR) dosimetric methodology. Methods for converting the various units of data published over 50 years into a standard form are developed and described. Often, meaningful values of uncertainty of measurements were not published so that variability in data sets is confounded with measurement uncertainty. A description of the methods developed to estimate variability is included in this paper. The data described in part one are grouped by gender and age to match the RADAR dosimetric phantoms. Within these phantoms, concentration values are grouped into source tissue regions by radionuclide, and they are imputed for source regions lacking tissue data. Radionuclide concentrations are then imputed for other phantoms’ source regions with missing concentration values, and the uncertainties of the imputed values are increased. The content concentrations of hollow organs are calculated, and activities are apportioned to the bone source regions using assumptions about each radionuclide’s bone-seeking behavior. The data sets are then ready to be used to estimate equivalent doses to target tissues from these source regions. The target tissues are then mapped to lists of tissues with International Commission on Radiation Protection (ICRP) tissue weighting factors, or they are mapped to surrogate tissue regions when there is no direct match. Effective doses, using ICRP tissue weighting factors recommended in 1977, 1990, and 2007, are calculated from tissue and organ equivalent doses.

59 BASIC BIOLOGICAL SCIENCES↗

Automated Demand Response: The Missing Link in the Electricity Value Chain

In 2006, the Public Interest Energy Research Program (PIER) Demand Response Research Center (DRRC) at Lawrence Berkeley National Laboratory initiated research into Automated Demand Response (OpenADR) applications in California industry. The goal is to improve electric grid reliability and lower electricity use during periods of peak demand. The purpose of this research is to begin to define the relationship among a portfolio of actions that industrial facilities can undertake relative to their electricity use. This ?electricity value chain? defines energy management and demand response (DR) at six levels of service, distinguished by the magnitude, type, and rapidity of response. One element in the electricity supply chain is OpenADR, an open-standards based communications system to send signals to customers to allow them to manage their electric demand in response to supply conditions, such as prices or reliability, through a set of standard, open communications. Initial DRRC research suggests that industrial facilities that have undertaken energy efficiency measures are probably more, not less, likely to initiate other actions within this value chain such as daily load management and demand response. Moreover, OpenADR appears to afford some facilities the opportunity to develop the supporting control structure and to"demo" potential reductions in energy use that can later be applied to either more effective load management or a permanent reduction in use via energy efficiency. Under the right conditions, some types of industrial facilities can shift or shed loads, without any, or minimal disruption to operations, to protect their energy supply reliability and to take advantage of financial incentives.1 In 2007 and 2008, 35 industrial facilities agreed to implement OpenADR, representing a total capacity of nearly 40 MW. This paper describes how integrated or centralized demand management and system-level network controls are linked to OpenADR systems. Case studies of refrigerated warehouses and wastewater treatment facilities are used to illustrate OpenADR load reduction potential. Typical shed and shift strategies include: turning off or operating compressors, aerator blowers and pumps at reduced capacity, increasing temperature set-points or pre-cooling cold storage areas and over-oxygenating stored wastewater prior to a DR event. This study concludes that understanding industrial end-use processes and control capabilities is a key to support reduced service during DR events and these capabilities, if DR enabled, hold significant promise in reducing the electricity demand of the industrial sector during utility peak periods.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Overview of the FAF3 Freight Flow Matrix Construction Process

PROJECT DESCRIPTION The FAF3 is a FHWA freight data product which provides a national O-D matrix of commodity flows to, from, and within the United States. FAF3 freight flows are reported in terms of both annual tons and annual dollars of freight moved by mode of transportation. Based largely on the 2007 CFS, FAF3 utilizes domestic freight flow characteristics, geographic regions, and the SCTG commodity coding system from CFS. However, many freight flows were not captured by the 2007 CFS due to scope and sample size limitations. Approximately 100,000 establishments were sampled out of some 754,000 freight moving establishments in 2007 and imports are out of scope entirely. To estimate missing data values, the approach taken in FAF3 was to use a combination of a novel Log-linear modeling approach (LLM) with an iterative proportional fitting (IPF) routine that also uses additional data inputs to fill in the missing pieces. The complete FAF3 O-D Commodity Mode database is made up of 131 Origins x 131 Destinations x 43 Commodity Classes x 8 Modal categories, for annual tons and dollars. This poster illustrates how the 2007 CFS data were integrated with several additional data sources using LLM and IPF to create a comprehensive FAF3 national freight flow matrix. More detailed documentation on the sources and methods utilized in the development of FAF3 are available from the FHWA website at the following website: http://www.ops.fhwa.dot. gov/freight/freight_analysis/faf/index.htm.

99 GENERAL AND MISCELLANEOUS↗

Expectation-maximization algorithms for learning a finite mixture of univariate survival time distributions from partially specified class values

Heterogeneity exists on a data set when samples from di erent classes are merged into the data set. Finite mixture models can be used to represent a survival time distribution on heterogeneous patient group by the proportions of each class and by the survival time distribution within each class as well. The heterogeneous data set cannot be explicitly decomposed to homogeneous subgroups unless all the samples are precisely labeled by their origin classes; such impossibility of decomposition is a barrier to overcome for estimating nite mixture models. The expectation-maximization (EM) algorithm has been used to obtain maximum likelihood estimates of nite mixture models by soft-decomposition of heterogeneous samples without labels for a subset or the entire set of data. In medical surveillance databases we can find partially labeled data, that is, while not completely unlabeled there is only imprecise information about class values. In this study we propose new EM algorithms that take advantages of using such partial labels, and thus incorporate more information than traditional EM algorithms. We particularly propose four variants of the EM algorithm named EM-OCML, EM-PCML, EM-HCML and EM-CPCML, each of which assumes a specific mechanism of missing class values. We conducted a simulation study on exponential survival trees with five classes and showed that the advantages of incorporating substantial amount of partially labeled data can be highly signi cant. We also showed model selection based on AIC values fairly works to select the best proposed algorithm on each specific data set. A case study on a real-world data set of gastric cancer provided by Surveillance, Epidemiology and End Results (SEER) program showed a superiority of EM-CPCML to not only the other proposed EM algorithms but also conventional supervised, unsupervised and semi-supervised learning algorithms.

97 MATHEMATICS AND COMPUTING↗

Status and prospects of investigations into the collinear cluster decay of heavy nuclei

Basic experimental results confirming the existence a new cluster-decay type called collinear cluster tripartition (CCT) are presented. Decays of this type manifest themselves, in particular, as a two-dimensional region of a locally enhanced yield of fragments (bump) that corresponds to specific missing-mass values in the mass-mass distribution of fission fragments. One of the decay modes that contribute to the bump can be treated as a cluster-decay type that is new in relation to the well-known heavy-ion or lead radioactivity. The conclusions drawn from an analysis of correlation mass distributions are confirmed by the results obtained from neutron-gated data, measurements of the nuclear charge for CCT events, and the direct detection of new-decay products.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Bayesian Gaussian process inference for neutron spin echo measurement

Neutron spin echo (NSE) spectroscopy provides unique access to microscopic dynamics, but its application is often constrained by low neutron flux, long acquisition times, and significant noise. Here, we present a Bayesian inference approach based on Gaussian process regression (GPR) to reconstruct high-quality spin echo signals from sparse and noisy data by exploiting correlations in reciprocal space. Benchmarks on synthetic datasets and validation with experimental NSE measurements of dendrimers show that GPR suppresses noise, interpolates missing intensity values, and accommodates irregular observations. The method improves accuracy, shortens acquisition times, and enables high-throughput and real-time studies. Beyond NSE, the framework is broadly applicable to other low signal-to-noise ratio scattering techniques, thereby extending the scope of neutron spectroscopy.

Tung, Chi-Huan [Oak Ridge National Laboratory (ORN↗

CryoTEN: efficiently enhancing cryo-EM density maps using transformers

Abstract Motivation Cryogenic electron microscopy (cryo-EM) is a core experimental technique used to determine the structure of macromolecules such as proteins. However, the effectiveness of cryo-EM is often hindered by the noise and missing density values in cryo-EM density maps caused by experimental conditions such as low contrast and conformational heterogeneity. Although various global and local map-sharpening techniques are widely employed to improve cryo-EM density maps, it is still challenging to efficiently improve their quality for building better protein structures from them. Results In this study, we introduce CryoTEN—a 3D UNETR++ style transformer to improve cryo-EM maps effectively. CryoTEN is trained using a diverse set of 1295 cryo-EM maps as inputs and their corresponding simulated maps generated from known protein structures as targets. An independent test set containing 150 maps is used to evaluate CryoTEN, and the results demonstrate that it can robustly enhance the quality of cryo-EM density maps. In addition, automatic de novo protein structure modeling shows that protein structures built from the density maps processed by CryoTEN have substantially better quality than those built from the original maps. Compared to the existing state-of-the-art deep learning methods for enhancing cryo-EM density maps, CryoTEN ranks second in improving the quality of density maps, while running >10 times faster and requiring much less GPU memory than them. Availability and implementation The source code and data are freely available at https://github.com/jianlin-cheng/cryoten.

Biochemistry & Molecular Biology↗