Search NASA⌕ Search

SEARCH · Search NASA

Results for “missing data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Scalable Hyper-parameter Estimation for Gaussian Process Based Time Series Analysis

Gaussian process (GP) is increasingly becoming popular as a kernel machine learning tool for non-parametric data analysis. Recently, GP has been applied to model non-linear dependencies in time series data. GP based analysis can be used to solve problems of time series prediction, forecasting, missing data imputation, change point detection, anomaly detection, etc. But the use of GP to handle massive scientific time series data sets has been limited, owing to its expensive computational complexity. The primary bottleneck is the handling of the covariance matrix whose size is quadratic in the length of the time series. In this paper we propose a scalable method that exploit the special structure of the covariance matrix for hyper-parameter estimation in GP based learning. The proposed method allows estimation of hyper parameters associated with GP in quadratic time, which is an order of magnitude improvement over standard methods with cubic complexity. Moreover, the proposed method does not require explicit computation of the covariance matrix and hence has memory requirement linear to the length of the time series as opposed to the quadratic memory requirement of standard methods. To further improve the computational complexity of the proposed method, we provide a parallel version to concurrently estimate the log likelihood for a set of time series which is the key step in the hyper-parameter estimation. Performance results on a multi-core system show that our proposed method provides significant speedups as high as 1000, even when running in serial mode, while maintaining a small memory footprint. The parallel version exploits the natural parallelization potential of the serial algorithm and is shown to perform significantly better than the serial faster algorithm, with speedups as high as 10.

Chandola, Varun↗

Data mining the missing ordered phases of Li/Na metal oxides

Data-driven discovery of Li-ion and Na-ion battery materials has been pioneered by generic materials data platforms such as the Materials Project. After decades of progress, it is timely to ask whether there remain underexplored compositional spaces. Here, in this work, we present a systematic data-mining effort to uncover missing ordered binary, ternary and quaternary Li/Na-containing metal oxides using high-throughput density functional theory (DFT). Building on 19,120 stable and metastable oxides entries from the Materials Project, we performed 13,245 additional calculations through isovalent substitutions of known ground states, experimentally reported compounds, and specific prototype structures. Our study identifies 36 new ground states within the GGA/GGA + U convex hull and 45 within the r 2 SCAN convex hull. Additionally, we identified 840 metastable compounds from GGA/GGA + U and 979 from r 2 SCAN that are absent in the present Materials Project databases. Moreover, we have tripled the metastable materials in compositional spaces with a molar ratio of cation/anion >1, highlighting the overlooked opportunities in this compositional space.

25 ENERGY STORAGE↗

Provenance infrastructure for multimodal x-ray experiments and reproducible analysis

Exploring relationships between material structures and desired properties for material discovery and synthesis requires mining machine-readable databases and linking experimental data with sufficiently rich annotations. There are vast gaps both in these data sources and in the connections between them. We present a provenance-based data management and analysis framework that enables capturing, persisting, and reanalyzing experimental data from the BNL National Synchrotron Light Source II (NSLS-II). Our system leverages NSLS-II Bluesky for experimental data acquisition, captures analysis parameters, and enriches the search space with results from the scientific literature. We describe in detail the challenges and benefits of data-driven approaches when comparing computational and experimental data, such as the multiple possible scenarios of interest in often ill-posed inverse problems and missing data. Our infrastructure utilizes the NSLS-II event model and includes discovery mechanisms across multiple beamlines. A text and data mining module using natural language processing classification techniques extracts relevant information from the literature that is added to data sources. We describe a provenance acquisition and analysis rerun workflow based on a directed acyclic graph that keeps track of process and data provenance and allows to replay analysis.

97 MATHEMATICS AND COMPUTING↗

2016 Floodplain Geochemistry from the East River Watershed, Colorado

This dataset includes groundwater geochemistry data from floodplain piezometers collected as a part of the Watershed Function Scientific Focus Area (SFA) located in the Upper Colorado River Basin. The data were collected in order to investigate the role of hyporheic exchange and other river corridor processes on riverine export of solutes. Floodplain piezometers installed in several meanders (Meander C, Meander D, and Meander O) in the vicinity of the Pumphouse were sampled at several dates in 2016. Data includes in-field measurements (pH, electrical conductivity, oxidation reduction potential, dissolved oxygen, dissolved Fe(II), dissolved sulfide, and groundwater level) along with laboratory measurements (dissolved inorganic carbon, dissolved organic carbon, metals and major cations). Five files are included in this dataset, including: sample locations and depths in a csv file (metadata.csv), aqueous geochemistry data in a csv file, analytical detection limits in a csv file, a file-level metadata (flmd.csv) file, and a data dictionary (dd.csv) file. Data that was below the detection limit is reported as one half of the detection limit and missing data is listed as "-9999".

54 ENVIRONMENTAL SCIENCES↗

Overview of the FAF3 Freight Flow Matrix Construction Process

PROJECT DESCRIPTION The FAF3 is a FHWA freight data product which provides a national O-D matrix of commodity flows to, from, and within the United States. FAF3 freight flows are reported in terms of both annual tons and annual dollars of freight moved by mode of transportation. Based largely on the 2007 CFS, FAF3 utilizes domestic freight flow characteristics, geographic regions, and the SCTG commodity coding system from CFS. However, many freight flows were not captured by the 2007 CFS due to scope and sample size limitations. Approximately 100,000 establishments were sampled out of some 754,000 freight moving establishments in 2007 and imports are out of scope entirely. To estimate missing data values, the approach taken in FAF3 was to use a combination of a novel Log-linear modeling approach (LLM) with an iterative proportional fitting (IPF) routine that also uses additional data inputs to fill in the missing pieces. The complete FAF3 O-D Commodity Mode database is made up of 131 Origins x 131 Destinations x 43 Commodity Classes x 8 Modal categories, for annual tons and dollars. This poster illustrates how the 2007 CFS data were integrated with several additional data sources using LLM and IPF to create a comprehensive FAF3 national freight flow matrix. More detailed documentation on the sources and methods utilized in the development of FAF3 are available from the FHWA website at the following website: http://www.ops.fhwa.dot. gov/freight/freight_analysis/faf/index.htm.

99 GENERAL AND MISCELLANEOUS↗

Verification and Validation Tests of Gamma Library of MC2-3 for Coupled Neutron and Gamma Heating Calculation

For the accurate assessment of the heat generation rate in fast reactors, the gamma library of MC2 -3 and the MC2 -3 + GAMSOR procedure employing the coupled neutron and gamma heating calculation has been thoroughly verified against Monte Carlo results and validated using the ZPPR-15D gamma dose measurement data. NJOY outputs are post-processed in a consistent way with the NJOY procedure to avoid any missing data or double counting of data. Prompt heating for 379 out of 391 isotopes in the gamma library was verified against MCNP6.2 to within 1% relative error in total heating for most isotopes. For both a simple one-dimensional slab problem representing a sodium cooled fast reactor and the Experimental Breeder Reactor II (EBR-II) benchmark problem, the root-mean-square values of assembly power error were less than 0.5% for fuel assemblies, ~1% in blankets and ~1 to ~3% in reflectors compared to MCNP6.2 results. The most plausible cause for the 3% error in a reflector assembly is believed to be the error in the multigroup neutron cross sections for the reflector assembly. For validation, gamma doses measured with thermoluminiscent dosimeters (TLDs) in the ZPPR-15D experiment were calculated using GAMSOR. Due to the uncertainty in the TLD measurement with regards to the energy deposition of photons and neutrons, the validation data leads to a 12.7% uncertainty on the experimental measurement. With this uncertainty bound, the calculated doses all fell within one standard deviation of the measured value. Combined with the accurate calculation of reaction rate distributions and neutron spectrum measurements, these results indicate good agreement for the neutron and gamma heating calculations that were performed.

coupled neutron and gamma heating↗

Feasibility of Economic Analysis of Radiation Therapy Oncology Group (RTOG) 91-11 Using Medicare Data

Purpose: The specific aim of this analysis was to evaluate the feasibility of performing a cost-effectiveness analysis using Medicare data from patients treated on a randomized Phase III clinical trial. Methods and Materials: Cost data included Medicare Part A and Part B costs from all providers-inpatient, outpatient, skilled nursing facility, home health, hospice, and physicians-and were obtained from the Centers for Medicare and Medicaid Services for patients eligible for Medicare, treated on Radiation Therapy Oncology Group (RTOG) 9111 between 1992 and 1996. The 47-month expected discounted (annual discount rate of 3%) cost for each arm of the trial was calculated in 1996 dollars, with Kaplan-Meier sampling average estimates of survival probabilities for each month and mean monthly costs. Overall and disease-free survival was also discounted 3%/year. The analysis was performed from a payer's perspective. Incremental cost-effectiveness ratios were calculated comparing the chemotherapy arms to the radiation alone arm. Results: Of the 547 patients entered, Medicare cost data and clinical outcomes were available for 66 patients. Reasons for exclusion included no RTOG follow-up, Medicare HMO enrollment, no Medicare claims since trial entry, and trial entry after 1996. Differences existed between groups in tumor characteristics, toxicity, and survival, all which could affect resource utilization. Conclusions: Although we were able to test the methodology of economic analysis alongside a clinical trial using Medicare data, the results may be difficult to translate to the entire trial population because of non-random missing data. Methods to improve Medicare data capture and matching to clinical trial samples are required.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Random Forest Regressor-Based Approach for Detecting Fault Location and Duration in Power Systems

Power system failures or outages due to short-circuits or “faults” can result in long service interruptions leading to significant socio-economic consequences. It is critical for electrical utilities to quickly ascertain fault characteristics, including location, type, and duration, to reduce the service time of an outage. Existing fault detection mechanisms (relays and digital fault recorders) are slow to communicate the fault characteristics upstream to the substations and control centers for action to be taken quickly. Fortunately, due to availability of high-resolution phasor measurement units (PMUs), more event-driven solutions can be captured in real time. In this paper, we propose a data-driven approach for determining fault characteristics using samples of fault trajectories. A random forest regressor (RFR)-based model is used to detect real-time fault location and its duration simultaneously. This model is based on combining multiple uncorrelated trees with state-of-the-art boosting and aggregating techniques in order to obtain robust generalizations and greater accuracy without overfitting or underfitting. Four cases were studied to evaluate the performance of RFR: 1. Detecting fault location (case 1), 2. Predicting fault duration (case 2), 3. Handling missing data (case 3), and 4. Identifying fault location and length in a real-time streaming environment (case 4). A comparative analysis was conducted between the RFR algorithm and state-of-the-art models, including deep neural network, Hoeffding tree, neural network, support vector machine, decision tree, naive Bayesian, and K-nearest neighborhood. Experiments revealed that RFR consistently outperformed the other models in detection accuracy, prediction error, and processing time.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Systematic Review of Radiation Therapy Toxicity Reporting in Randomized Controlled Trials of Rectal Cancer: A Comparison of Patient-Reported Outcomes and Clinician Toxicity Reporting

The use of multimodal treatments for rectal cancer has improved cancer-related outcomes but makes monitoring toxicity challenging. Optimizing future radiation therapy regimens requires collection and publication of detailed toxicity data. This review evaluated the quality of toxicity information provided in randomized controlled trials (RCTs) of radiation therapy in rectal cancer and focused on the difference between clinician-reported and patient-reported toxicity. Medline, EMBASE, and the Cochrane Library were searched (January 1995-July 2013) for RCTs reporting late toxicity in patients treated with regimens including preoperative (chemo)radiation therapy. Data on toxicity measures and information on toxicity reported were extracted using Quantitative Analyses of Normal Tissue Effects in the Clinic recommendations. International Society for Quality of Life Research standards on patient-reported outcomes (PROs) were used to evaluate the quality of patient-reported toxicity. Twenty-one RCT publications met inclusion criteria out of 4144 articles screened. All PRO studies reported higher rates of toxicity symptoms than clinician-reported studies and reported on a wider range and milder symptoms. No clinician-reported study published data on sexual dysfunction. Of the clinician-reported studies, 55% grouped toxicity data related to an organ system together (eg “Bowel”), and 45% presented data only on more-severe (grade ≥3) toxicity. In comparison, all toxicity grades were reported in 79% of PRO publications, and all studies (100%) presented individual symptom toxicity data (eg bowel urgency). However, PRO reporting quality was variable. Only 43% of PRO studies presented baseline data, 28% did not use any psychometrically validated instruments, and only 29% of studies described statistical methods for managing missing data. Analysis of these trials highlights the lack of reporting standards for adverse events and reveals the differences between clinician and patient reporting of toxicity. Recommendations for improving the quality of adverse event data collection are provided, with the aim of improving critical appraisal of outcomes for future studies.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

QA/QC of the East River, Colorado, discharge and geochemical time series datasets (Almont, BCC, and Pump House) to be used for modeling of hydrogeochemical balance

The following datasets were QA/QC-ed (Quality Assurance/Quality Control): 1. Brush Creek Confluence (BCC) discharge data (from Helen Malenda, USGS, Colorado School of Mines), which were calculated using the pressure transducer data and rating curves. The original 15 min time series data were presented as mean daily discharge. 2. Almont discharge data from United States Geological Survey (USGS). The original data were in 15 min time intervals, and were averaged to mean daily discharge time series. 3. Pump House discharge data as mean daily discharge (downloaded from the SFA portal). 4. BCC and Pump House chemistry data from SFA data portal and/or original spreadsheets provided by Roelof Versteeg. The following challenging QA/QC problems of the datasets were resolved: Missing data with the duration of gaps up to >1 month; Duplicated dates; Anomalies and outliers of discharge and concentrations; Time stamps of measurements of the discharge and concentrations are not aligned (hydrogeochemical balance calculations require the timestamps to be aligned). All QA/QC-ed datasets are given as csv files. The csv files were prepared using the xts files with multiple worksheets, which are also included in the data packages. Figures of the QA/QC-ed datasets are given in the jpeg and pdf formats. The QA/QC-ed datasets have been used to quantify discharge and chemical concentrations in river water in order to understand riverine exports of water and dissolved constituents in the East River watershed. These datasets served as a basis in the presentation given by P. Fox et al. at the 2021 Goldschmidt Conference.

54 ENVIRONMENTAL SCIENCES↗

Fundamental Science-Based Simulation of Nuclear Waste Forms

This report presents a hierarchical multiscale modeling scheme based on two-way information exchange. To account for all essential phenomena in waste forms over geological time scales, the models have to span length scales from nanometer to kilometer and time scales from picoseconds to millenia. A single model cannot cover this wide range and a multi-scale approach that integrates a number of different at-scale models is called for. The approach outlined here involves integration of quantum mechanical calculations, classical molecular dynamics simulations, kinetic Monte Carlo and phase field methods at the mesoscale, and continuum models. The ultimate aim is to provide science-based input in the form of constitutive equations to integrated codes. The atomistic component of this scheme is demonstrated in the promising waste form xenotime. Density functional theory calculations have yielded valuable information about defect formation energies. This data can be used to develop interatomic potentials for molecular dynamics simulations of radiation damage. Potentials developed in the present work show a good match for the equilibrium lattice constants, elastic constants and thermal expansion of xenotime. In novel waste forms, such as xenotime, a considerable amount of data needed to validate the models is not available. Integration of multiscale modeling with experimental work is essential to generate missing data needed to validate the modeling scheme and the individual models. Density functional theory can also be used to fill knowledge gaps. Key challenges lie in the areas of uncertainty quantification, verification and validation, which must be performed at each level of the multiscale model and across scales. The approach used to exchange information between different levels must also be rigorously validated. The outlook for multiscale modeling of wasteforms is quite promising.

Continuum Model↗

AUTONOMIE VID

Autonomie Vehicle Information Database (VID) offers a comprehensive list of vehicle specifications since 1990. The database details more than 65,000 vehicles with hundreds of attributes. The database is the result of the development of a general automated data collection framework as well as the development of building blocks for processing, cleaning, integrating and analyzing complex data. The data has undergone several layers of outlier detections processes, machine learning based imputations methods have been used to deal with missing data problems, and new fields have been created according to the rules of feature engineering. Thanks to this streamlined data pipelines, the resulting processed aggregated data should deliver a unique level of information to the user in which the content can be efficiently maintained and updated.

Moswd, Ayman↗

Online mapping and forecasting of epidemics using open-source indicators

Open-source indicators have been proposed as a way of tracking and forecasting disease outbreaks. Some, such are meteorological data, are readily available as reanalysis products. Others, such as those derived from our online behavior (web searches, media article etc.) are gathered easily and are more timely than public health reporting. In this study we investigate how these datastreams may be combined to provide useful epidemiological information. The investigation is performed by building data assimilation systems to track influenza in California and dengue in India. The first does not suffer from incomplete data and was chosen to explore disease modeling needs. The second explores the case when observational data is sparse and disease modeling complexities are beside the point. The two test cases are for opposite ends of the disease tracking spectrum. We find that data assimilation systems that produce disease activity maps can be constructed. Further, being able to combine multiple open-source datastreams is a necessity as any one individually is not very informative. The data assimilation systems have very little in common except that they contain disease models, calibration algorithms and some ability to impute missing data. Thus while the data assimilation systems share the goal for accurate forecasting, they are practically designed to compensate for the shortcomings of the datastreams. Thus we expect them to be disease and location-specific.

42 ENGINEERING↗

Optimizing Geothermal Drilling: Oil and Gas Technology Transfer

There is a significant amount of financial risk associated with geothermal drilling; however, there are opportunities to improve upon current practices and technologies used. The scope of this drilling operational study included 21 geothermal wells and 21 oil and gas wells. The goal was to determine a 'perfect well' using historical data to compare the best oil and gas well to the best geothermal well. Unfortunately, limitations encountered in the study included missing data (bit records, mud information, etc.), poor data collection, and difficult to ascertain handwriting. An online software database was used to format drilling data to IADC coded daily drilling reports and generate analysis figures. Six major issues have been found in geothermal drilling operations. These problems include lost circulation, rig/equipment selection, cementing, penetration rate, drilling program, and time management. As a result of these issues, geothermal drilling averages 56.4 days longer than drilling comparable oil and gas wells in the wells in this study. Roughly $13.9 million would be lost due to non-productive time in the 21 geothermal wells and only $1.3 million in the oil and gas wells, assuming a cost of $50,000 per day. Comparable events such as drilling the same sized hole, tripping in/out, cementing, and running the same size casing took substantially less time in the oil and gas wells. Geothermal wells were drilled using older and/or less advanced technology to depths less than 10,000 feet, while oil and gas wells reached 12,500 feet faster with purpose built rigs. A new approach is now underway that will optimize drilling programs throughout the drilling industry. It is the use of Mechanical Specific Energy (MSE) as a tool to realize efficient drilling processes. However, a work-flow must also be established in order for there to be an efficient drilling program. Potential improvements for current geothermal operations are: the use of electronic records, real time services, and official glossary terms to describe rig operations, and advanced drilling rigs/technology.

15 GEOTHERMAL ENERGY↗

Optimizing Geothermal Drilling: Oil and Gas Technology Transfer

There is a significant amount of financial risk associated with geothermal drilling. This study of drilling operations seeks opportunities to improve upon current practices and technologies. The scope of this study included analyzing 21 geothermal wells and 21 oil and gas wells. The goal was to determine a 'Perfect Well' using historical data to compare the best oil and gas well to the best geothermal well. Unfortunately, limitations encountered in the study included missing data (bit records, mud information, etc.) and poor data collection practices An online software database was used to format drilling data to IADC coded daily drilling reports and generate figures for analysis. Six major issues have been found in geothermal drilling operations. These problems include lost circulation, rig/ equipment selection, cementing, penetration rate, drilling program, and time management. As a result of these issues, geothermal drilling averaged 56.4 days longer than drilling comparable oil and gas wells in the wells in this study. Roughly $13.9 million was spent on non-productive time in the 21 geothermal wells, compared with only $1.3 million in the oil and gas wells, assuming a cost of $50,000 per day. Comparable events such as drilling the same sized hole, tripping in/out, cementing, and running the same size casing took substantially less time in the oil and gas wells. Geothermal wells were drilled using older and/or less advanced technology to depths less than 10,000 feet, while oil and gas wells reached 12,500 feet faster with purpose built rigs. A new approach is now underway that will optimize drilling programs throughout the drilling industry using Mechanical Specific Energy (MSE) as a tool to realize efficient drilling processes. Potential improvements for current geothermal operations are: the use of electronic records, real time services, and official glossary terms to describe rig operations, and advanced drilling rigs/technology.

15 GEOTHERMAL ENERGY↗

An iterative bidirectional gradient boosting approach for CVR baseline estimation

Here this paper presents a novel Iterative Bidirectional Gradient Boosting Model (IBi-GBM) for estimating the baseline of Conservation Voltage Reduction (CVR) programs. In contrast to many existing methods, we treat CVR baseline estimation as a missing data retrieval problem. The approach involves dividing the load and its corresponding temperature profiles into three periods: pre-CVR, CVR, and post-CVR. To restore the missing load profile during the CVR period, the method employs a three-step process. First, a forward-pass GBM is executed using data from the pre-CVR period as inputs. Subsequently, a backward-pass GBM is applied using data from the post-CVR period. The two restored load profiles are reconciled, considering pre-calculated weights derived from forecasting accuracy, and only the leftmost and rightmost points are retained. The newly restored points are then included as inputs for the subsequent iteration. This iterative procedure continues until the original load data in the CVR period is fully restored. We develop IBi-GBM using actual smart meter and Supervisory Control and Data Acquisition (SCADA) data. Our results demonstrate that IBi-GBM exhibits robust performance across various data resolutions and in different seasons and outperforms existing methods by achieving a 1-2% reduction in normalized Root Mean Square Error (nRMSE).

42 ENGINEERING↗

Imputation of urban environmental sensor data using gated attention bidirectional long short-term memory (GA-BiLSTM): methods, performance, and implications

Urban environmental monitoring networks frequently encounter significant data gaps due to sensor malfunctions, environmental disturbances, and communication failures. Reliable approaches to address these gaps are essential for ensuring the continuity and quality of environmental data streams. In this study, we developed a gated attention bidirectional long short-term memory (GA-BiLSTM) model to impute missing data in a dense urban monitoring network. Using observations from the CROCUS network in Chicago, we evaluated GA-BiLSTM against widely used approaches (XGBoost and K-nearest neighbors) under scenarios of both short-term intermittent gaps and prolonged outages. GA-BiLSTM consistently outperformed comparative methods, particularly during extended outages of up to ten days, demonstrating its ability to capture spatiotemporal dependencies across sensor nodes. Beyond performance metrics, feature importance and spatial network analyses highlighted the unexpected but critical predictive role of peripheral rural nodes, underlining their strategic value for maintaining robust urban monitoring systems. These results emphasize that advanced imputation methods can substantially improve the reliability of environmental monitoring networks and support more resilient data infrastructures for urban sustainability.

Data imputation↗

Soil Water Percolation Flux, April 2017 to March 2019, BR-Ma2, Manaus

Soil water percolation flux measurements derived from six passive wick flux meters across three topographic positions (valley, slope and plateau). Data has been processed to compute the mean for each topographic position where two flux meters were installed at each topographic position. Missing data represented a few percent of the total data set and were filled using two approaches. In the event one of the flux meters malfunctioned, we used data from the other sensor from the same topographic position without averaging. When both sensors were not working, such as during the period from September to October 2017 for the slope and valley and July 2018 for the plateau, data was filled using a nonlinear regression model that was based on local precipitation data and coefficients that were empirically derived for each topographic position. The metadata tab included in the .csv has additional information on locations where the sensors were installed, and other installation/maintenance details. Contact ksolander@lanl.gov if you need to use this dataset for additional information.

54 ENVIRONMENTAL SCIENCES↗