Search NASA⌕ Search

SEARCH · Search NASA

Results for “data imputation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2021

Salvaging Data Records with Missing Data: Data Imputation using the Multivariate t Distribution

When doing multivariate data analysis, one commonobstacle is the presence of incomplete observations, i.e., observationsfor which one or more key fields are blank. Missing datais often countered by deleting entire observations that containmissing data. The negative effects of deleting entire observationsare multiple: deleting observations reduces sample size andcan also result in biased inferences even if data is missing atrandom. In addition, knowledge contained within incompleteobservations is knowledge lost when they are deleted– and theeffort spent collecting that knowledge is effort wasted. Data imputationmethods, or methods of statistically “filling-in” missingdata, can help combat small sample sizes by using the existinginformation in partially complete observations with the end goalof producing less biased and higher confidence inferences. Whena sample from a multivariate normal population is only partiallycomplete, and the missing data meets appropriate assumptions(missing at random), robust data imputation of the missing datacan be implemented with monotone data augmentation (MDA)using the multivariate t distribution.Missing data imputation is applied to data from the NASA InstrumentCost Model (NICM) using the MDA algorithm underthe assumption of having a multivariate t distribution with fixeddegrees of freedom. A sensitivity analysis to the degrees offreedom parameter is presented to demonstrate robustness ofthe multivariate t distribution when dealing with small samplesas compared to the multivariate normal distribution.

DiNicola, Michael↗
NASA NTRS2015

Astronaut Preflight Cardiovascular Variables Associated with Vascular Compliance are Highly Correlated with Post-Flight Eye Outcome Measures in the Visual Impairment Intracranial Pressure (VIIP) Syndrome Following Long Duration Spaceflight

The detection of the first VIIP case occurred in 2005, and adequate eye outcome measures were available for 31 (67.4%) of the 46 long duration US crewmembers who had flown on the ISS since its first crewed mission in 2000. Therefore, this analysis is limited to a subgroup (22 males and 9 females). A "cardiovascular profile" for each astronaut was compiled by examining twelve individual parameters; eleven of these were preflight variables: systolic blood pressure, pulse pressure, body mass index, percentage body fat, LDL, HDL, triglycerides, use of anti‐lipid medication, fasting serum glucose, and maximal oxygen uptake in ml/kg. Each of these variables was averaged across three preflight annual physical exams. Astronaut age prior to the long duration mission, and inflight salt intake was also included in the analysis. The group of cardiovascular variables for each crew member was compared with seven VIIP eye outcome variables collected during the immediate post‐flight period: anterior-posterior axial length of the globe measured by ultrasound and optical biometry; optic nerve sheath diameter, optic nerve diameter, and optic nerve to sheath ratio‐ each measured by ultrasound and magnetic resonance imaging (MRI), intraocular pressure (IOP), change in manifest refraction, mean retinal nerve fiber layer (RNFL) on optical coherence tomography (OCT), and RNFL of the inferior and superior retinal quadrants. Since most of the VIIP eye outcome measures were added sequentially beginning in 2005, as knowledge of the syndrome improved, data were unavailable for 22.0% of the outcome measurements. To address the missing data, we employed multivariate multiple imputation techniques with predictive mean matching methods to accumulate 200 separate imputed datasets for analysis. We were able to impute data for the 22.0% of missing VIIP eye outcomes. We then applied Rubin's rules for collapsing the statistical results across our 200 multiply imputed data sets to assess the canonical correlation between the eye outcomes and the twelve astronaut cardiovascular variables available for all 31 subjects. Results: A highly significant canonical correlation was observed among the canonical solutions (p<.00001), with an average best canonical correlation of.97. The results suggest a strong association between astronauts' measures of cardiovascular health and the seven eye outcomes of the VIIP syndrome used in this analysis. Furthermore, the "joint test" revealed a significant difference in cardiovascular profile between male and female astronauts (Prob > F = 0.00001). Overall, female astronauts demonstrated a significantly healthier cardiovascular status. Individually, the female astronauts had significantly healthier profiles on seven of twelve cardiovascular variables than the men (p values ranging from <0.0001 to <0.05). Male astronauts did not demonstrate significantly healthier values on any of the twelve cardiovascular variables measured

Otto, Christian↗
NASA NTRS2014

Missing Data and Multiple Imputation: An Unbiased Approach

The default method of dealing with missing data in statistical analyses is to only use the complete observations (complete case analysis), which can lead to unexpected bias when data do not meet the assumption of missing completely at random (MCAR). For the assumption of MCAR to be met, missingness cannot be related to either the observed or unobserved variables. A less stringent assumption, missing at random (MAR), requires that missingness not be associated with the value of the missing variable itself, but can be associated with the other observed variables. When data are truly MAR as opposed to MCAR, the default complete case analysis method can lead to biased results. There are statistical options available to adjust for data that are MAR, including multiple imputation (MI) which is consistent and efficient at estimating effects. Multiple imputation uses informing variables to determine statistical distributions for each piece of missing data. Then multiple datasets are created by randomly drawing on the distributions for each piece of missing data. Since MI is efficient, only a limited number, usually less than 20, of imputed datasets are required to get stable estimates. Each imputed dataset is analyzed using standard statistical techniques, and then results are combined to get overall estimates of effect. A simulation study will be demonstrated to show the results of using the default complete case analysis, and MI in a linear regression of MCAR and MAR simulated data. Further, MI was successfully applied to the association study of CO2 levels and headaches when initial analysis showed there may be an underlying association between missing CO2 levels and reported headaches. Through MI, we were able to show that there is a strong association between average CO2 levels and the risk of headaches. Each unit increase in CO2 (mmHg) resulted in a doubling in the odds of reported headaches.

Foy, M.↗
NASA NTRS2019

Predictive Modeling for Differential Diagnosis and Mortality Risk Assessment

The prevalence of electronic health record (EHR) systems has brought prodigious biomedical informatics opportunity. Automated machine learning methods can effectively utilize such data and have become common tools for healthcare predictive modeling. Researches in medical informatics have explored the potential of deep learning and classical models in emergent care scenarios. In particular, predicting differential diagnoses for admissions have proven useful in decreasing unnecessary lab tests and improving inpatient triage decision-making. Moreover, identification of high-risk patients for in-hospital mortality is vitally important to maximize allocation of medical resources.The Medical Information Mart for Intensive Care (MIMIC-III) database, containing de-identified critical care inpatient was used in our study. This data set captures hospital patient laboratory measurements, pharmacologic prescriptions, diagnostic data and procedure event recordings. When considering adult patients and discounting admissions with ICU length of stay less than 24 hours, there were 37,787 unique admissions and 30,414 total patients. We examined the top 25 most prevalent ICD-9 group-level disease specificities in MIMIC-III using a multi-label classification model. In-hospital mortality was modeled as binary classification with 4,155 (13%) adult patients that expired, of which 3,138 (75.5%) were in the ICU setting. The metrics AUC, F1 score, sensitivity and specificity values calculated for each disease label measured prediction performance.The usage of ICD-9 group codes reduced feature dimension from 14,567 to 942 and greatly improved distribution of patient diagnostic categories. Disease temporal patterns were captured by considering the most frequently sampled 6 vital signs and 13 laboratory values. Missing data were imputed at each time-stamp. Time-series raw hourly average values were converted into 5 summary features (mean, standard deviation, number of observations, min & max values). Patient demographic variables such as age, gender, marital status and ethnicity were also factored into the modeling. Choi et al showed that contextual embedding of medical data, diagnostic and procedural codes alone can predict future diagnoses with sensitivity as high as 0.79. We utilized an embedding technique called word2vec which allowed sparse representations of medical history to be transformed into dense word vectors. The mappings captured contextual information by treating each admission as a sentence and learning the most likely neighboring words in a sliding window fashion. Binary and multi-label classification was achieved via collapse models, which do not consider temporal information, as well as recurrent neural networks with regularization, Softmax output layer activation together with categorical cross-entropy as the loss function.

US Army collaboration↗
NASA NTRS1978

Urine monitoring system failure analysis and operational verification test report

Failure analysis and testing of a prototype urine monitoring system (UMS) are reported. System performance was characterized by a regression formula developed from volume measurement test data. When the volume measurement test data. When the volume measurement data was imputted to the formula, the standard error of the estimate calculated using the regression formula was found to be within 1.524% of the mean of the mass of the input. System repeatability was found to be somewhat dependent upon the residual volume of the system and the evaporation of fluid from the separator. The evaporation rate was determined to be approximately 1cc/minute. The residual volume in the UMS was determined by measuring the concentration of LiCl in the flush water. Observed results indicated residual levels in the range of 9-10ml, however, results obtained during the flushing efficiency test indicated a residual level of approximately 20ml. It is recommended that the phase separator pumpout time be extended or the design modified to minimize the residual level.

Glanfield, E. J.↗
NASA NTRS2004

Clustering with Missing Values: No Imputation Required

Clustering algorithms can identify groups in large data sets, such as star catalogs and hyperspectral images. In general, clustering methods cannot analyze items that have missing data values. Common solutions either fill in the missing values (imputation) or ignore the missing data (marginalization). Imputed values are treated as just as reliable as the truly observed data, but they are only as good as the assumptions used to create them. In contrast, we present a method for encoding partially observed features as a set of supplemental soft constraints and introduce the KSC algorithm, which incorporates constraints into the clustering process. In experiments on artificial data and data from the Sloan Digital Sky Survey, we show that soft constraints are an effective way to enable clustering with missing values.

constraints↗
NASA NTRS2017

Relationship Between Carbon Dioxide Levels and Reported Congestion and Headaches on the International Space Station

Congestion is commonly reported during spaceflight, and most crewmembers have reported using medications for congestion during International Space Station (ISS) missions. Although congestion has been attributed to fluid shifts during spaceflight, fluid status reaches equilibrium during the first week after launch while congestion continues to be reported throughout long duration missions. Congestion complaints have anecdotally been reported in relation to ISS CO2 levels; this evaluation was undertaken to determine whether or not an association exists. METHODS: Reported headaches, congestion symptoms, and CO2 levels were obtained for ISS expeditions 2-31, and time-weighted means and single-point maxima were determined for 24-hour (24hr) and 7-day (7d) periods prior to each weekly private medical conference. Multiple imputation addressed missing data, and logistic regression modeled the relationship between probability of reported event of congestion or headache and CO2 levels, adjusted for possible confounding covariates. The first seven days of spaceflight were not included to control for fluid shifts. Data were evaluated to determine the concentration of CO2 required to maintain the risk of congestion below 1% to allow for direct comparison with a previously published evaluation of CO2 concentrations and headache. RESULTS: This study confirmed a previously identified significant association between CO2 and headache and also found a significant association between CO2 and congestion. For each 1-mm Hg increase in CO2, the odds of a crew member reporting congestion doubled. The average 7-day CO2 would need to be maintained below 1.5 mmHg to keep the risk of congestion below 1%. The predicted probability curves of ISS headache and congestion curves appear parallel when plotted against ppCO2 levels with congestion occurring at approximately 1mmHg lower than a headache would be reported. DISCUSSION: While the cause of congestion is multifactorial, this study showed congestion is associated with CO2 levels on ISS. Data from additional expeditions could be incorporated to further assess this finding. CO2 levels are also associated with reports of headaches on ISS. While it may be expected for astronauts with congestion to also complain of headaches, these two symptoms are commonly mutually exclusive. Furthermore, it is unknown if a temporal CO2 relationship exists between congestion and headache on ISS. CO2 levels were time-weighted for 24hr and 7d, and thus the time course of congestion leading to headache was not assessed; however, congestion could be an early CO2-related symptom when compared to headache. Future studies evaluating the association of CO2-related congestion leading to headache would be difficult due to the relatively stable daily CO2 levels on ISS currently, but a systematic study could be implemented on-orbit if desired.

Cole, Robert↗
NASA NTRSDate not supplied

Characterizing Spatiotemporal Uncertainty in Interpolated Meteorological Data

Interpolated meteorological data invariably contain errors. These errors have structure in time and space, particularly autocorrelation, which can cause the effects of errors to compound when model outputs are aggregated temporally or spatially. One way to account for this uncertainty is with a probabilistic model from which samples can be drawn that are coherent with respect to underlying spatial and temporal covariance structure. This work describes a probabilistic method for spatial interpolation of point-wise meteorological time series. Observational data from weather stations are generally sparse in space and dense in time (but sometimes missing). The method works by projecting time series onto orthogonal basis vectors and spatially interpolating each resulting component independently. Under suitable assumptions, and data transformations to better satisfy those assumptions, Gaussian process regression provides a complete description of the joint predictive distribution over a Gaussian random field. Spatiotemporally coherent realizations are generated as the sum of conditional (spatial) simulations of each orthogonal (temporal) component. Data-derived and generic orthogonal bases are considered. In addition to spatial interpolation, imputation of missing observational data is examined. The method is applied using near-surface air temperature over the Western United States and validated by comparing theoretical versus actual coverage of predictive distributions and analyzing the degree to which spatial and temporal covariance structure is reproduced. Computational considerations, relating to conditional simulation of random fields, are also addressed.

Conor T Doherty↗
NASA NTRS1982

Imputing historical statistics, soils information, and other land-use data to crop area

In foreign crop condition monitoring, satellite acquired imagery is routinely used. To facilitate interpretation of this imagery, it is advantageous to have estimates of the crop types and their extent for small area units, i.e., grid cells on a map represent, at 60 deg latitude, an area nominally 25 by 25 nautical miles in size. The feasibility of imputing historical crop statistics, soils information, and other ancillary data to crop area for a province in Argentina is studied.

Perry, C. R., Jr.↗
NASA NTRS2006

Diurnal Variability of Vertical Structure from a TRMM Passive Microwave "Virtual Radar" Retrieval

Robust description of the diurnal cycle from TRMM observations is complicated by the limitations of Low Earth Orbit (LEO) sampling; from a 'climatological' perspective, sufficient sampling must exist to control for both spatial and seasonal variability, before tackling an additional diurnal component (e.g., with 8 additional 3-hourly or 24 1-hourly bins). For documentation of vertical structure, the narrow sample swath of the TRMM Precipitation Radar limits the resolution of any of these components. A neural-network based 'virtual radar" retrieval has been trained and internally validated, using multifrequency / multipolarization passive microwave(TM1) brightness temperatures and textures parameters and lightning (LIS) observations, as inputs, and PR volumetric reflectivity as targets (outputs). By training the algorithms (essentially highly multivariate, nonlinear regressions) on a very large sample of high-quality co-located data from the center of the TRMM swath, 3D radar reflectivity and derived parameters (VIL, IWC, Echo Tops, etc.) can be retrieved across the entire TMI swath, good to 8-9% over the dynamic range of parameters. As a step in the retrieval (and as an output of the process), each TMI multifrequency pixel (at 85 GHz resolution) is classified into one of the 25 archetypal radar profile vertical structure "types", previously identified using cluster analysis. The dynamic range of retrieved vertical structure appears to have higher fidelity than the current (Version 6) experimental GPROF hydrometeor vertical structure retrievals. This is attributable to correct representation of the prior probabilities of vertical structure variability in the neural network training data, unlike the GPROF cloud-resolving model training dataset used in the V6 algorithms. The LIS lightning inputs are supplementary inputs, and a separate offline neural network has been trained to impute (predict) LIS lightning from passive-microwave-only data. The virtual radar retrieval is thus, in principle, extensible to Aqua/AMSR-E and NPOESS/CMIS passive microwave instruments. The virtual radar approach yields a threefold increase in effective sampling from the mission, albeit of lower-quality "retrieved" data, reducing the variance of local estimates by one third (or the standard deviation by-0.57). In this talk, the variance reduction is leveraged to more finely resolve global diurnal variability in both space and time (local hour).

Boccippio, Dennis J.↗
NASA NTRS2022

MLtool: Universal Supervised Machine Learning Tool to Model Tabulated Data

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine learning↗
NASA NTRS2022

MLtool Python Code

Machine Learning (ML) is a subfield of Artificial Intelligence that gives computers the ability to learn from past data without being explicitly programmed. The predictive capabilities of ML models have already been used to facilitate several scientific breakthroughs. However, the practical application of ML is often limited due to the gaps in technical knowledge of its users. The common issue faced by many scientific researchers is the inability to choose the appropriate ML pipelines that are needed to treat real-world data, which is often sparse and noisy. To solve this problem, we have developed an automated Machine Learning tool (MLtool) that includes a set of ML algorithms and approaches to aid scientific researchers. The current version of MLtool is implemented as an object-oriented Python code that is easily extensible. It includes 44 different regression algorithms used to model data. MLtool helps users select the best model for their data, based on the scoring metrics used. Besides regression algorithms, MLtool also includes a suite of pre- and post-processing techniques such as missing value imputation, categorical variable encoding, input feature normalization, uncertainty quantification, exploratory data analysis (EDA), etc. MLtool was tested on several publicly available multi-dimensional data sets and was found capable of making accurate predictions.

Machine Learning↗
NASA NTRS2022

Bayesian Rules of Thumb: Robust Uncertainty Quantification in Early Project Cost Estimation

Systems engineers often make use of cost Rules ofThumb in order to estimate cost during early phases of projectformulation. These Rules of Thumb typically take the form ofa sequence of percentages over which a total cost is allocatedacross NASA WBS elements. Rules of Thumb can then be usedto extrapolate cost from one or more known WBS elements tothe remaining unknown WBS elements, assisting early projectformulation architecture studies (such as those in JPL’s Team Xand A Team).A number of issues can arise when generating and using costRules of Thumb. For example, many records of project costsconsist of incomplete data. Typical methods of dealing withincomplete cost allocation data include (a) ignoring missionswith incomplete data, or (b) taking averages of the non-zero percentagesacross missions, but both of these methods can result inbiased estimates if the existence of incomplete data correlateswith total mission cost or any particular WBS element. Anothercommon example is cost reported in one or more incorrect WBSelements. This is especially prevalent in smaller missions whereit is more common for engineers to perform tasks that fall underthe purview of multiple WBS elements.Furthermore, a Rule of Thumb estimate is typically reported asa point estimate; there is no reported uncertainty around thepercentages used to generate an allocation. Even in the rarecase in which confidence intervals around mean percentages areprovided, there may be positive or negative correlations betweenWBS elements which can skew estimates.Here we attempt to address these problems by formulatingprobabilistic Rules of Thumb in which a distribution of allocationschemes, rather than a single allocation scheme, is generated.We use a bootstrap imputation method to simultaneouslyaccount for uncertainty in the missing data while using allavailable information contained in the dataset. The imputeddatasets are then input into a multivariate Bayesian modelwhich accounts for correlations between WBS elements andproperly accounts for uncertainty in the final Rule of Thumbpercentages and predictions. We describe the mathematicalmodel and provides snippets of R code utilizing the brms(Bayesian Regression Models using Stan) package. To illustratethis model, we generate a Bayesian Level 2 WBS Cost Rule ofThumb for MIDEX (Medium-Class Explorers) missions withdata extracted from NASA’s CADRe. We then compare thismethod’s performance with the classical Rule of Thumb method.

Hooke, Melissa A↗
NASA NTRS2016

Exploration Analysis of Carbon Dioxide Levels and Ultrasound Measures of the Eye During ISS Missions

Enhanced screening for the Visual Impairment/Intracranial Pressure (VIIP) Syndrome, including in-flight ultrasound, was implemented in 2010 to better characterize the changes in vision observed in some long-duration crewmembers. Suggested possible risk factors for VIIP include cardiovascular changes, diet, anatomical and genetic factors, and environmental conditions. As a potent vasodilator, carbon dioxide (CO (sub 2)), which is chronically elevated on the International Space Station (ISS) relative to typical indoor and outdoor ambient levels on Earth, seems a plausible contributor to VIIP. In an effort to understand the possible associations between CO (sub 2) and VIIP, this study analyzes the relationship between ambient CO (sub 2) levels on ISS and ultrasound measures of the eye obtained from ISS fliers. CO (sub 2) measurements will be pulled directly from Operational Data Reduction Complex for the Lab and Node 3 major constituent analyzers (MCAs) on ISS or from sensors located in the European Columbus module, as available. CO (sub 2) measures between ultrasound sessions will be summarized using standard time series class metrics in MATLAB including time-weighted means and variances. Cumulative CO (sub 2) exposure metrics will also be developed. Regression analyses will be used to quantify the relationships between the CO (sub 2) metrics and specific ultrasound measures. Generalized estimating equations will adjust for the repeated measures within individuals. Multiple imputation techniques will be used to adjust for any possible biases in missing data for either CO (sub 2) or ultrasound measures. These analyses will elucidate the possible relationship between CO (sub 2) and changes in vision and also inform future analysis of inflight VIIP data.

Young, M.↗