Search NASA⌕ Search

SEARCH · Search NASA

Results for “missing data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

LANL Meteorological Program: 2020 Data Completeness/Quality Report

Los Alamos National Laboratory (LANL) operates four mesa-top meteorology towers: Technical Area (TA) 06, TA-49, TA-53, and TA-54. An additional tower is located in Mortandad Canyon (TA-5 MDCN), and a rain gauge at North Community (NCOM). A description of the meteorology monitoring network is found in Dewart and Boggs (2014). Mesa-top towers are instrumented at 1.2 meters (m), 11.5 m, 23 m, and 46 m. In addition, TA-06 is instrumented at 92 m. The TA-5 MDCN tower is 10 m in height and is instrumented at 1.2 m and 10 m. Data are collected every 15 minutes. Range checking is done on each measurement every 15 minutes; data that are beyond normal ranges are eliminated from the data set and replaced by a code for missing data. In addition, data are reviewed weekly by meteorologists to identify bad data not identified by range checking. The data steward eliminates these data from the data set and replaces them with a code for missing data. The instrument technicians also review that data and schedule instrument replacement as required. Data completeness is determined by the number of total 15-minute records available versus the total number of possible measurements for the entire year. As a rule, the meteorologists do not attempt to estimate data that are eliminated as bad data. Original datalogger records, containing bad data, can be recalled from program archival storage.

54 ENVIRONMENTAL SCIENCES↗

LANL Meteorological Program: 2022 Data Completeness/Quality Report

Los Alamos National Laboratory (LANL) operates seven mesa-top instrumented meteorology towers: Technical Area (TA) 6, TA-49, TA-53, TA-54, TA-63, TA-54B, and TA-16B. An additional instrumented tower is located in Mortandad Canyon (TA-5 MDCN), and there is a rain gauge at North Community (NCOM), located within the town of Los Alamos. The 10 meter (m) towers at TA-63, TA- 54B, and TA-16B were installed in 2021, and will be included in the 2023 data completeness/quality report. A description of the meteorology monitoring network, prior to the installation of the TA-63, TA-54B, and TA-16B is found in Dewart and Boggs (2014). Four of the mesa-top towers (e.g., TA-6, TA-49, TA-53, and TA-54) are instrumented at the 1.2 m, 11.5 m, 23 m, and 46 m levels. In addition, the TA-6 tower is instrumented at the 92 m level. The TA-5 MDCN tower is 10 m in height and is instrumented at 1.2 m and 10 m. Data are collected and averaged every 15 minutes. Range checking is performed on each measurement every 15 minutes; data that are beyond normal ranges are eliminated from the data set and replaced by a code for missing data. In addition, data are reviewed weekly by qualified meteorologists to identify bad data not identified by the range checking technique. The data steward eliminates these data from the data set and replaces them with a code for missing data. The instrument technicians also review that data and schedule instrument replacement, as required. All instruments are calibrated at a frequency that meets the criteria identified in ANSI/ANS-3.11-2015. Data completeness is determined by the number of total 15-minute records available versus the number of possible measurements for the entire year. As a rule, the meteorologists do not attempt to estimate data that are eliminated as bad data. Original datalogger records, including bad data, can be recalled from program archival storage.

54 ENVIRONMENTAL SCIENCES↗

LANL Meteorological Program: 2023 Data Completeness/Quality Report

Los Alamos National Laboratory (LANL) operates seven mesa-top instrumented meteorology towers: Technical Area (TA) 6, TA-49, TA-53, TA-54, TA-63, TA-54B, and TA-16B. An additional instrumented tower is located in Mortandad Canyon (TA-5 MDCN), and there is a rain gauge at North Community (NCOM), located within the town of Los Alamos. The 10 meter (m) towers at TA-63, TA-54B, and TA-16B have been in testing since they were installed in 2021, and will be included in a future data completeness report. A description of the meteorology monitoring network, prior to the installation of the TA-63, TA-54B, and TA-16B is found in Dewart and Boggs (2014). Four of the mesa-top towers (e.g., TA-6, TA-49, TA-53, and TA-54) are instrumented at the 1.2 m, 11.5 m, 23 m, and 46 m levels. In addition, the TA-6 tower is instrumented at the 92 m level. The TA-5 MDCN tower is 10 m in height and is instrumented at 1.2 m and 10 m. Data are collected and averaged every 15 minutes. Range checking is performed on each measurement every 15 minutes; data that are beyond normal ranges are eliminated from the data set and replaced by a code for missing data. In addition, data are reviewed weekly by qualified meteorologists to identify bad data not identified by the range checking technique. The data steward eliminates these data from the data set and replaces them with a code for missing data. The instrument technicians also review that data and schedule instrument replacement, as required. All instruments are calibrated at a frequency that meets the criteria identified in ANSI/ANS-3.11-2015. Data completeness is determined by the number of total 15-minute records available versus the number of possible measurements for the entire year. As a rule, the meteorologists do not attempt to estimate data that are eliminated as bad data. Original datalogger records, including bad data, can be recalled from program archival storage.

54 ENVIRONMENTAL SCIENCES↗

Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning

Data I/O poses a significant bottleneck in large-scale CFD simulations; thus, practitioners would like to significantly reduce the number of times the solution is saved to disk, yet retain the ability to recover any field quantity (at any time instance) a posteriori . The objective of this work is therefore to accurately recover missing CFD data a posteriori at any time instance, given that the solution has been written to disk at only a relatively small number of time instances. We consider in particular high-order discretizations (e.g., discontinuous Galerkin), as such techniques are becoming increasingly popular for the simulation of highly separated flows. To satisfy this objective, this work proposes a methodology consisting of two stages: 1) dimensionality reduction and 2) dynamics learning. For dimensionality reduction, we propose a novel hierarchical approach. First, the method reduces the number of degrees of freedom within each element of the high-order discretization by applying autoencoders from deep learning. Second, the methodology applies principal component analysis to compress the global vector of encodings. This leads to a low-dimensional state, which associates with a nonlinear embedding of the original CFD data. For dynamics learning, we propose to apply regression techniques (e.g., kernel methods) to learn the discrete-time velocity characterizing the time evolution of this low-dimensional state. In conclusion, a numerical example on a large-scale CFD example characterized by nearly 13 million degrees of freedom illustrates the suitability of the proposed method in an industrial setting.

97 MATHEMATICS AND COMPUTING↗

WE-G-18A-03: Cone Artifacts Correction in Iterative Cone Beam CT Reconstruction

Purpose: For iterative reconstruction (IR) in cone-beam CT (CBCT) imaging, data truncation along the superior-inferior (SI) direction causes severe cone artifacts in the reconstructed CBCT volume images. Not only does it reduce the effective SI coverage of the reconstructed volume, it also hinders the IR algorithm convergence. This is particular a problem for regularization based IR, where smoothing type regularization operations tend to propagate the artifacts to a large area. It is our purpose to develop a practical cone artifacts correction solution. Methods: We found it is the missing data residing in the truncated cone area that leads to inconsistency between the calculated forward projections and measured projections. We overcome this problem by using FDK type reconstruction to estimate the missing data and design weighting factors to compensate the inconsistency caused by the missing data. We validate the proposed methods in our multi-GPU low-dose CBCT reconstruction system on multiple patients' datasets. Results: Compared to the FDK reconstruction with full datasets, while IR is able to reconstruct CBCT images using a subset of projection data, the severe cone artifacts degrade overall image quality. For head-neck case under a full-fan mode, 13 out of 80 slices are contaminated. It is even more severe in pelvis case under half-fan mode, where 36 out of 80 slices are affected, leading to inferior soft-tissue delineation. By applying the proposed method, the cone artifacts are effectively corrected, with a mean intensity difference decreased from ∼497 HU to ∼39HU for those contaminated slices. Conclusion: A practical and effective solution for cone artifacts correction is proposed and validated in CBCT IR algorithm. This study is supported in part by NIH (1R01CA154747-01)

60 APPLIED LIFE SCIENCES↗

A review of imputation strategies for isobaric labeling-based shotgun proteomics

The throughput efficiency and increased depth of coverage provided by isobaric-labeled proteomics measurements have led to increased usage of these techniques. However, the structure of missing data is uniquely different than unlabeled studies. In this review, we compare the efficacy of nine imputation methods on a CPTAC proteomics iTRAQ dataset. Imputation methods were evaluated with regard to accuracy, variability, statistical hypothesis test inference and run time over datasets consisting of varying number of iTRAQ plexes and percentages of missing data. In general, expectation maximization and random forest imputation methods yielded the best performances, and constant-based methods performed poorly consistently across all dataset sizes and percentages of missing values. For datasets with small sample sizes and higher percentages of missing data, results indicate that statistical inference with no imputation may be preferable. Based on the findings in this review, there are core imputation methods that perform higher for isobaric-labeled proteomics data, but great care and consideration as to whether imputation should be used should be given for datasets comprised of a small number of samples, as well as to factors such as computational time and reproducibility of imputation values.

Bramer, Lisa M.↗

The CACAO Method for Smoothing, Gap Filling, and Characterizing Seasonal Anomalies in Satellite Time Series

Consistent, continuous, and long time series of global biophysical variables derived from satellite data are required for global change research. A novel climatology fitting approach called CACAO (Consistent Adjustment of the Climatology to Actual Observations) is proposed to reduce noise and fill gaps in time series by scaling and shifting the seasonal climatological patterns to the actual observations. The shift and scale CACAO parameters adjusted for each season allow quantifying shifts in the timing of seasonal phenology and inter-annual variations in magnitude as compared to the average climatology. CACAO was assessed first over simulated daily Leaf Area Index (LAI) time series with varying fractions of missing data and noise. Then, performances were analyzed over actual satellite LAI products derived from AVHRR Long-Term Data Record for the 1981-2000 period over the BELMANIP2 globally representative sample of sites. Comparison with two widely used temporal filtering methods-the asymmetric Gaussian (AG) model and the Savitzky-Golay (SG) filter as implemented in TIMESAT-revealed that CACAO achieved better performances for smoothing AVHRR time series characterized by high level of noise and frequent missing observations. The resulting smoothed time series captures well the vegetation dynamics and shows no gaps as compared to the 50-60% of still missing data after AG or SG reconstructions. Results of simulation experiments as well as confrontation with actual AVHRR time series indicate that the proposed CACAO method is more robust to noise and missing data than AG and SG methods for phenology extraction.

LAI↗

A Pattern-Recognition-Based Ensemble Data Imputation Framework for Sensors from Building Energy Systems

Building operation data are important for monitoring, analysis, modeling, and control of building energy systems. However, missing data is one of the major data quality issues, making data imputation techniques become increasingly important. There are two key research gaps for missing sensor data imputation in buildings: the lack of customized and automated imputation methodology, and the difficulty of the validation of data imputation methods. In this paper, a framework is developed to address these two gaps. First, a validation data generation module is developed based on pattern recognition to create a validation dataset to quantify the performance of data imputation methods. Second, a pool of data imputation methods is tested under the validation dataset to find an optimal single imputation method for each sensor, which is termed as an ensemble method. The method can reflect the specific mechanism and randomness of missing data from each sensor. The effectiveness of the framework is demonstrated by 18 sensors from a real campus building. The overall accuracy of data imputation for those sensors improves by 18.2% on average compared with the best single data imputation method.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Report of the subpanel on resampling functions

The state of the art and anticipated requirements for resampling in applications of remote sensing are examined. It is noted that the wide range of potential applications and expanded needs for resampling versatility in new imaging sensors leads to a concern for creating standardized test data sets and ground training sites. Several research areas are defined including the development of (1) a theory for defining an input psf of an instrument such that upon resampling of the processed data into the desired output grid the radiometric and geometric properties are a best estimate of the true upwelling radiances; (2) algorithms for estimating missing data or for ignoring missing data; (3) optimal resampling functions; (4) standard data sets; and (5) merit functions for assessing resampling techniques.

Source record↗

Integration of ultra-low coverage whole-genome sequences for reconstructing the evolutionary history of Galapagos giant tortoises

Genomic data from contemporary and historical samples often need to be coupled for evolutionary reconstructions of multitaxon complexes. However, the genetic data recovered from historical samples may result only in ultra-low coverage whole-genome sequences (ulcWGS; <0.15× depth), leading to inaccurate evolutionary inferences given a preponderance of missing data. Using the Galapagos giant tortoise radiation as a study system (Chelonoidis spp., composed of 13 extant and four extinct lineages), we assembled a novel methodological pipeline that removes potential noise introduced by the missing data and enhances the evolutionary signal from ulcWGS samples. We leveraged existing tools for phylogenomic placement (EPA-ng), population genomic structure (smartsnp) and admixture (Admixfrog, NGSadmix) to demonstrate that the evolutionary history of samples can be uncovered with sequencing depths as low as 0.008–0.139×. Importantly, these approaches do not use genotype imputation of the ulcWGS samples, which would require extensive reference datasets. Our application to two cases of extinct lineages of Galapagos giant tortoises, with and without references from the same lineage, demonstrates the general value of the approach. We confirm where the extinct lineages from San Cristóbal and Santa Fe islands fit into the Galapagos giant tortoise radiation, and that these lineages were evolutionarily distinct entities.

ancient DNA↗

Electric Field Data from the North Slope of Alaska

File contains raw as well as calibrated electric field values (V/m) from two CS110 electric field meters. The file also contains raw 2-m U and V winds at the same site. The sampling rate is 1Hz. All files contain raw electric field data, as well as calibrated values to a one week ground-based calibration setup. During this week, simultaneous upward-facing ground based electric field measurements were taken alongside the two downward facing instruments on the pole (CS110-3 at 2m and CS110-2 at 5m). Slopes and intercepts were calculated and applied to the values influenced by the pole to convert the raw data to calibrated values. These calibrated values are absolute electric field measurements. There are no missing data codes. The missong data is not included in the files, and thus any files with missing data will be shorter than the normal 86,400 values. In July 2023, the CS110-3 was replaced with an identical instrument named CS110-8, and the CS110-2 was replaced with an identical instrument called CS110-9. All the variable names remained consistent and the same.

54 ENVIRONMENTAL SCIENCES↗

Confidence-Based Feature Acquisition

Confidence-based Feature Acquisition (CFA) is a novel, supervised learning method for acquiring missing feature values when there is missing data at both training (learning) and test (deployment) time. To train a machine learning classifier, data is encoded with a series of input features describing each item. In some applications, the training data may have missing values for some of the features, which can be acquired at a given cost. A relevant JPL example is that of the Mars rover exploration in which the features are obtained from a variety of different instruments, with different power consumption and integration time costs. The challenge is to decide which features will lead to increased classification performance and are therefore worth acquiring (paying the cost). To solve this problem, CFA, which is made up of two algorithms (CFA-train and CFA-predict), has been designed to greedily minimize total acquisition cost (during training and testing) while aiming for a specific accuracy level (specified as a confidence threshold). With this method, it is assumed that there is a nonempty subset of features that are free; that is, every instance in the data set includes these features initially for zero cost. It is also assumed that the feature acquisition (FA) cost associated with each feature is known in advance, and that the FA cost for a given feature is the same for all instances. Finally, CFA requires that the base-level classifiers produce not only a classification, but also a confidence (or posterior probability).

Wagstaff, Kiri L.↗

A cross-sectional study of the temporal evolution of electricity consumption of six commercial buildings

Current approaches to building efficiency diagnoses include conventional energy audit techniques that can be expensive and time consuming. In contrast, virtual energy audits of readily available 15-minute-interval building electricity consumption are being explored to provide quick, inexpensive, and useful insights into building operation characteristics. A cross sectional analysis of six buildings in two different climate zones provides methods for data cleaning, population-based building comparisons, and relationships (correlations) of weather and electricity consumption. Data cleaning methods have been developed to categorize and appropriately filter or correct anomalous data including outliers, missing data, and erroneous values (resulting in < 0.5% anomalies). The utility of a cross-sectional analysis of a sample set of building's electricity consumption is found through comparisons of baseload, daily consumption variance, and energy use intensity. Correlations of weather and electricity consumption 15-minute interval datasets show important relationships for the heating and cooling seasons using computed correlations of a Time-Specific-Averaged- Ordered Variable (exterior temperature) and corresponding averaged variables (electricity consumption)(TSAOV method). The TSAOV method is unique as it introduces time of day as a third variable while also minimizing randomness in both correlated variables through averaging. This study found that many of the pair-wise linear correlation analyses lacked strong relationships, prompting the development of the new TSAOV method to uncover the causal relationship between electricity and weather. We conclude that a combination of varied HVAC system operations, building thermal mass, plug load use, and building set point temperatures are likely responsible for the poor correlations in the prior studies, while the correlation of time-specific-averaged-ordered temperature and corresponding averaged variables method developed herein adequately accounts for these issues and enables discovery of strong linear pair-wise correlation R values. TSAOV correlations lay the foundation for a new approach to building studies, that mitigates plug load interferences and identifies more accurate insights into weather-energy relationship for all building types. Over all six buildings analyzed the TSAOV method reported very significant average correlations per building of 0.94 to 0.82 in magnitude. Our rigorous statistics-based methods applied to 15- minute-interval electricity data further enables virtual energy audits of buildings to quickly and inexpensively inform energy savings measures.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Ozone data and mission sampling analysis

A methodology was developed to analyze discrete data obtained from the global distribution of ozone. Statistical analysis techniques were applied to describe the distribution of data variance in terms of empirical orthogonal functions and components of spherical harmonic models. The effects of uneven data distribution and missing data were considered. Data fill based on the autocorrelation structure of the data is described. Computer coding of the analysis techniques is included.

Robbins, J. L.↗

Dig Hazard Assessment Using a Stereo Pair of Cameras

This software evaluates the terrain within reach of a lander s robotic arm for dig hazards using a stereo pair of cameras that are part of the lander s sensor system. A relative level of risk is calculated for a set of dig sectors. There are two versions of this software; one is designed to run onboard a lander as part of the flight software, and the other runs on a PC under Linux as a ground tool that produces the same results generated on the lander, given stereo images acquired by the lander and downlinked to Earth. Onboard dig hazard assessment is accomplished by executing a workspace panorama command sequence. This sequence acquires a set of stereo pairs of images of the terrain the arm can reach, generates a set of candidate dig sectors, and assesses the dig hazard of each candidate dig sector. The 3D perimeter points of candidate dig sectors are generated using configurable parameters. A 3D reconstruction of the terrain in front of the lander is generated using a set of stereo images acquired from the mast cameras. The 3D reconstruction is used to evaluate the dig goodness of each candidate dig sector based on a set of eight metrics. The eight metrics are: 1. The maximum change in elevation in each sector, 2. The elevation standard deviation in each sector, 3. The forward tilt of each sector with respect to the payload frame, 4. The side tilt of each sector with respect to the payload frame, 5. The maximum size of missing data regions in each sector, 6. The percentage of a sector that has missing data, 7. The roughness of each sector, and 8. Monochrome intensity standard deviation of each sector. Each of the eight metrics forms a goodness image layer where the goodness value of each sector ranges from 0 to 1. Goodness values of 0 and 1 correspond to high and low risk, respectively. For each dig sector, the eight goodness values are merged by selecting the lowest one. Including the merged goodness image layer, there are nine goodness image layers for each stereo pair of mast images.

Rankin, Arturo L.↗

Statistical theory and methodology for remote sensing data analysis with special emphasis on LACIE

Crop proportion estimators for determining crop acreage through the use of remote sensing were evaluated. Several studies of these estimators were conducted, including an empirical comparison of the different estimators (using actual data) and an empirical study of the sensitivity (robustness) of the class of mixture estimators. The effect of missing data upon crop classification procedures is discussed in detail including a simulation of the missing data effect. The final problem addressed is that of taking yield data (bushels per acre) gathered at several yield stations and extrapolating these values over some specified large region. Computer programs developed in support of some of these activities are described.

Odell, P. L.↗

Equations for Scoring Rules When Data Are Missing

A document presents equations for scoring rules in a diagnostic and/or prognostic artificial-intelligence software system of the rule-based inference-engine type. The equations define a set of metrics that characterize the evaluation of a rule when data required for the antecedence clause(s) of the rule are missing. The metrics include a primary measure denoted the rule completeness metric (RCM) plus a number of subsidiary measures that contribute to the RCM. The RCM is derived from an analysis of a rule with respect to its truth and a measure of the completeness of its input data. The derivation is such that the truth value of an antecedent is independent of the measure of its completeness. The RCM can be used to compare the degree of completeness of two or more rules with respect to a given set of data. Hence, the RCM can be used as a guide to choosing among rules during the rule-selection phase of operation of the artificial-intelligence system..

James, Mark↗

PHOTOMETRIC REDSHIFTS AND QUASAR PROBABILITIES FROM A SINGLE, DATA-DRIVEN GENERATIVE MODEL

We describe a technique for simultaneously classifying and estimating the redshift of quasars. It can separate quasars from stars in arbitrary redshift ranges, estimate full posterior distribution functions for the redshift, and naturally incorporate flux uncertainties, missing data, and multi-wavelength photometry. We build models of quasars in flux-redshift space by applying the extreme deconvolution technique to estimate the underlying density. By integrating this density over redshift, one can obtain quasar flux densities in different redshift ranges. This approach allows for efficient, consistent, and fast classification and photometric redshift estimation. This is achieved by combining the speed obtained by choosing simple analytical forms as the basis of our density model with the flexibility of non-parametric models through the use of many simple components with many parameters. We show that this technique is competitive with the best photometric quasar classification techniques-which are limited to fixed, broad redshift ranges and high signal-to-noise ratio data-and with the best photometric redshift techniques when applied to broadband optical data. We demonstrate that the inclusion of UV and NIR data significantly improves photometric quasar-star separation and essentially resolves all of the redshift degeneracies for quasars inherent to the ugriz filter system, even when included data have a low signal-to-noise ratio. For quasars spectroscopically confirmed by the SDSS 84% and 97% of the objects with Galaxy Evolution Explorer UV and UKIDSS NIR data have photometric redshifts within 0.1 and 0.3, respectively, of the spectroscopic redshift; this amounts to about a factor of three improvement over ugriz-only photometric redshifts. Our code to calculate quasar probabilities and redshift probability distributions is publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗