SEARCH · Search NASA
Results for “missing values”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
AGR 5/6/7 Data Qualification Report for ATR Cycles 162B through 168A
This report provides the qualification status of experimental data for the Advanced Gas Reactor (AGR) 5/6/7 fuel irradiation. AGR-5/6/7 was conducted in the Advanced Test Reactor (ATR) at Idaho National Laboratory (INL) in support of development and qualification of tri-structural isotropic (TRISO) low-enriched fuel for use in high temperature gas-cooled reactors. The objectives of the AGR-5/6/7 experiments are to: (i) irradiate reference-design fuel particles to support fuel qualification, (ii) establish operating margins for the fuel beyond normal operating conditions, and (iii) provide irradiated-fuel performance data and irradiated-fuel samples for post-irradiation examination (PIE) and safety testing. The test train contains five separate capsules that were independently controlled and monitored. Each capsule contains multiple 12.51-mm-long compacts filled with low enriched uranium carbide/oxide (UCO) TRISO fuel particles. The primary objective of the AGR-5/6 test (Capsules 1, 2, 4, and 5) is to verify successful performance of the reference-design fuel under normal operating conditions. The AGR-7 test (Capsule 3) was designed to explore fuel performance at higher temperatures to demonstrate the capability of the fuel to withstand conditions beyond normal operating conditions in support of plant design and licensing. AGR 5/6/7 will also provide irradiated-fuel performance data on fission-gas release from failed particles during irradiation. The AGR-5/6/7 capsules were irradiated in the ATR northeast flux trap location. The experiment began on February 16, 2018 and ended on July 22, 2020, spanning nine ATR cycles over two and a half years. Thus, the AGR-5/6/7 fuel compacts were irradiated for a total of 360.9 effective full power days. The AGR 5/6/7 experiment was able to remain in the reactor core during all three Powered Axial Locator Mechanism (PALM) cycles (163A, 165A, and 167A) without overheating its fuel compacts. This report includes irradiation monitoring data from nine ATR Cycles: 162B, 163A, 164A, 164B, 165A, 166A, 166B, 167A, and 168A, as stored in the Nuclear Data Management and Analysis System (NDMAS). During irradiation, data records consisted of instantaneous measurements recorded every minute and provided by text files automatically every 2 hours. The AGR 5/6/7 data streams addressed in this report include thermocouple (TC) temperatures, sweep gas data (flow rates [capsule inlet, outlet, and downstream at detector], pressure, and moisture content), and Fission Product Monitoring System (FPMS) data (release rates and release to birth rate ratios [R/Bs]) for each of the five capsules. A total of 94,989,908 TC temperature and sweep gas data records were received and processed by NDMAS for AGR 5/6/7 irradiation. Of these records, 41,593,387 (or 43.7% of the total) met data collection and accuracy requirements and are labeled as Qualified. A total of 57,746,693 TC temperature readings were captured from 54 installed TCs. Among them, 10,034,676 TC temperature records (only 17.4%) were Qualified and 47,701,371 TC temperatures (or 82.6%) are Failed due to 48 TC failures (63.5%) and due to missing values (19.1%). To assess performance of the operational TCs, analysis of daily correlations between TCs found no evidence of virtual junction failure for any TCs. Analyses on control charts of TC temperature differences revealed trending in TC readings for TC2, 4, 5, and 13 in Capsule 3, but there is no conclusive indication of TC drift failure that caused those trends. Therefore, TC control charts are not used to disqualify TC data, but only for users’ consideration. For sweep gas flow rates, a total of 31,519,747 gas flow records (84.4%) are Qualified for use for AGR-5/6/7 experiment; 5,723,468 gas flow records (15.4%) are Failed due mostly to missing values; and 74,641 high sweep gas flow rates (0.2 %) are Trend. A large number of Failed missing TC temperature and gas flow values were caused by an error in the data output script that outputted a ‘NULL’ value when values were unchanged. This problem was fixed during the outage of Cycle 166B, which led to a substantially decreased number of missing values during the last three cycles. Nonetheless, a large amount of non-missing data remained because of the high data acquisition frequency (1-minute) and still provided sufficient data to effectively monitor the experiment as designed. For FPMS data, NDMAS received and processed fission product release and R/B data for nine ATR cycles, when ATR core reached full power during AGR 5/6/7 irradiation. These data consist of 110,388 release rate records and 110,388 R/B records for the twelve radionuclides (Kr 85m, Kr 87, Kr 88, Kr 89, Kr 90, Xe 131m, Xe 133, Xe 135, Xe 135m, Xe 137, Xe 138, and Xe 139) for each of the five capsules. Equivalent numbers of uncertainty records associated the release rates and R/B values were provided. To date, qualification status of the FPMS data stored in the NDMAS dat
Packaging and distributing ecological data from multisite studies
Studies of global change and other regional issues depend on ecological data collected at multiple study areas or sites. An information system model is proposed for compiling diverse data from dispersed sources so that the data are consistent, complete, and readily available. The model includes investigators who collect and analyze field measurements, science teams that synthesize data, a project information system that collates data, a data archive center that distributes data to secondary users, and a master data directory that provides broader searching opportunities. Special attention to format consistency is required, such as units of measure, spatial coordinates, dates, and notation for missing values. Often data may need to be enhanced by estimating missing values, aggregating to common temporal units, or adding other related data such as climatic and soils data. Full documentation, an efficient data distribution mechanism, and an equitable way to acknowledge the original source of data are also required.
Explaining Missing Data in Graphs: A Constraint-based Approach
Abstract: This paper introduces a constraint-based approach to clarify missing values in graphs. Our method capitalizes on a set S of graph data constraints. An explanation is a sequence of operational enforcement of S towards the recovery of interested yet missing data (e.g., attribute values, edges). We show that constraint-based approach helps us to understand not only why a value is missing, but also how to recover the missing value. We study S-explanation problem, which is to compute the optimal explanations with guarantees on the informativeness and conciseness. We show the problem is in ?P^2 for established graph data constraints such as graph keys and graph association rules. We develop an efficient bidirectional algorithm to compute optimal explanations, without enforcing S on the entire graph. We also show our algorithm can be easily extended to support graph refinement within limited time, and to explain missing answers. Using real-world graphs, we experimentally verify the effectiveness and efficiency of our algorithms.
Meteorological and Soil Data from Ecohydrology Sensor Towers at Pump House and Snodgrass Mountain in East River Watershed, Colorado, 2019-2025
This data package includes hourly meteorological and soil sensor data at eight ecohydrology monitoring sites in East River Watershed, Colorado as part of the Watershed Function Scientific Focus Area (WFSFA) research led by Lawrence Berkeley National Lab (LBNL). Four field sites were located on the hillslope of East River (ER) near Pump House (PH) at Mount Crested Butte (ER-PHS1 to 4), and the other four are in the Snodgrass Mountain (SG) area (SG-EHS5 to 8). In terms of vegetation cover, three sites are in montane grasslands (ER-PHS1, ER-PHS2, and SG-EHS5), three are below evergreen conifer canopy (ER-PHS3, SG-EHS6, and SG-EHS7), and two are below deciduous aspen canopy (ER-PHS4 and SG-EHS8). The monitoring period began in October 2019 at the East River sites, in October 2020 at SG-EHS5 and SG-EHS6, and in October 2021 at SG-EHS7 and SG-EHS8. In September 2024, all four East River sites were fully retired. The four Snodgrass Mountain sites remain active. Each site is equipped with a comprehensive suite of meteorological sensors on a tripod and soil sensors that measure weather, energy fluxes, and soil variables. This data package includes measurements from ten different types of sensors and up to thirteen individual sensors per site, including (1) a weather station (measurement height ranges from 2.8~3.8 meters (m) above ground), (2) a quantum sensor for photosynthetic active radiation (PAR) (2.4~3.3m), (3) a net radiometer (1.7~2.1m), (4) an infrared radiometer (1.6~2.2m), (5) a sonic distance sensor (1.5~1.9m), (6) a soil carbon dioxide (CO2) flux chamber (0m), (7) a soil heat flux plate (-0.05m below ground), (8) a soil oxygen sensor (-0.3m), (9) a soil water potential sensor (-0.3m), and (10) soil water content sensors at 3~4 depths (-1.15 ~ -0.1m). A total of twenty-three variables is reported in this data package, including (1) atmospheric variables: air temperature (TA), atmospheric pressure (PA), vapor pressure (VP), and vapor pressure deficit (VPD), (2) precipitation variables: rain precipitation (P) and snow depth (D_SNOW), (3) energy fluxes variables: four-component net radiation (NETRAD) (shortwave/longwave incoming/outgoing radiation, SW_IN, SW_OUT, LW_IN, LW_OUT), photosynthetic photon flux density (PPFD), and soil heat flux (G), (4) soil variables: soil water content (SWC), soil water potential (SWP), soil temperature (TS), soil bulk electrical conductivity (COND_SOIL), and soil gaseous oxygen concentration (O2_SOIL), (5) wind variables: two-dimensional wind speed (WS), gust speed (WS_MAX), and wind direction (WD), and (6) surface variables: surface infrared temperature (T_CANOPY) and soil CO2 flux (CO2_SOIL). Please see the Methods section for data processing and QA/QC steps taken to generate the hourly datasets. The following files are included in this data package (notes on version: v{x}-{y}, where x is the metadata version, and y is the data version, when applicable): (1) “metadata_site_v{x}-{y}.csv” - a site metadata file that summarizes location information of all sites, including site ID, description, coordinates, timeframe, elevation, and vegetation cover, (2) “metadata_instrument_v{x}-{y}.csv” - an instrument metadata file that summarizes sensor information of all sites, including sensor manufacturer and model, measurement height, and sampling and averaging interval of all variables, (3) "data_{SITE_ID}_v{x}-{y}.csv" - eight data files that contain hourly data of each site indicated by {SITE_ID} in the filename, (4) “/figure/data_{SITE_ID}_v{x}-{y}.png" - eight figures that help visualize data of each site indicated by {SITE_ID} in the filename, (5) “/photo/*” - photos of each site indicated by {SITE_ID} in the filename, and (6) four file level metadata (flmd.csv) and data dictionary (*_dd.csv) files that summarize file, header, column, and variable information of all files. Notes: (1) Measurement height: Each variable name is followed by conventional positional qualifiers “H_V_R”, where H indicates the relative horizontal positions of that specific variable, V the vertical positions, and R the replicates. In this data package, only the vertical qualifier V varies, and V increases from the highest vertical position (V=1) to the lowest. Variables with the same qualifier are not necessarily measured by the same sensor, and the same variable with the same qualifier across different sites are not necessarily measured at the same height. Please refer to “metadata_instrument.csv” for the sensor information and measurement heights, and whether a variable is measured below the canopy. (2) Variable availability: Snow depth is not available at ER-PHS3 and SG-EHS7. SWC, soil temperature, and soil bulk EC at the deepest depth (<-1m) are not available at SG-EHS6 and SG-EHS7. The missing value code for numeric variables is -9999, except for SWP. For SWP, the missing value code is +9999, because SWP values are negative. (3) Sampling frequency: Please refer to “metadata_instrument.csv” for the increase of sampling frequency of some variables from 30-min to 1-min at ER-PHS1 to 4 in July 2020. (4) Sensors: While the methods of each sensor are not detailed, all sensors are commercially available, and their methods can be found in their manuals. Please refer to “metadata_instrument.csv” for the sensor manufacturer and model information. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.
Incomplete Data in Smart Grid: Treatment of Values in Electric Vehicle Charging Data
In this paper, five imputation methods namely Constant (zero), Mean, Median, Maximum Likelihood, and Multiple Imputation methods have been applied to compensate for missing values in Electric Vehicle (EV) charging data. The outcome of each of these methods have been used as the input to a prediction algorithm to forecast the EV load in the next 24 hours at each individual outlet. The data is real world data at the outlet level from the UCLA campus parking lots. Given the sparsity of the data, both Median and Constant (=zero) imputations improved the prediction results. Since in most missing value cases in our database, all values of that instance are missing, the multivariate imputation methods did not improve the results significantly compared to univariate approaches.
AUTOMATIC CLASSIFICATION OF VARIABLE STARS IN CATALOGS WITH MISSING DATA
We present an automatic classification method for astronomical catalogs with missing data. We use Bayesian networks and a probabilistic graphical model that allows us to perform inference to predict missing values given observed data and dependency relationships between variables. To learn a Bayesian network from incomplete data, we use an iterative algorithm that utilizes sampling methods and expectation maximization to estimate the distributions and probabilistic dependencies of variables from data with missing values. To test our model, we use three catalogs with missing data (SAGE, Two Micron All Sky Survey, and UBVI) and one complete catalog (MACHO). We examine how classification accuracy changes when information from missing data catalogs is included, how our method compares to traditional missing data approaches, and at what computational cost. Integrating these catalogs with missing data, we find that classification of variable objects improves by a few percent and by 15% for quasar detection while keeping the computational cost the same.
Cara Status and Upcoming Enhancements
RIC Miss Values in Summary TableTabular presentation of miss vector in Summary Section RIC Uncertainty Values in Details SectionNumerical presentation of miss component uncertainty values in Details SectionGreen Events with Potentially Maneuverable Secondary ObjectsAll potentially maneuverable secondary objects will be reported out to 7-days prior to TCA for LEO events and 10-days for NONLEO events, regardless of risk (relates to MOWG Action Item 1309-11) All green events with potentially active secondary objects included in Summary ReportsAllows more time for contacting other OOBlack Box FixSometimes a black square appeared in the summary report where the ASW RIC time history plot should beAppendix Orbit RegimeMission Name MismatchPc 0 Plotting BugAll Pc points less than 1e-10 (zero) are now plotted as 1e-10 (instead of not at all)Maneuver Indication FixManeuver indicator now present even if maneuver was in the past.
Search for gluino mediated bottom- and top-squark production in multijet final states in pp collisions at 8 TeV
A search for supersymmetry is presented based on events with large missing transverse energy, no isolated electron or muon, and at least three jets with one or more identified as a bottom-quark jet. A simultaneous examination is performed of the numbers of events in exclusive bins of the scalar sum of jet transverse momentum values, missing transverse energy, and bottom-quark jet multiplicity. The sample, corresponding to an integrated luminosity of 19.4 inverse femtobarns, consists of proton-proton collision data recorded at a center-of-mass energy of 8 TeV with the CMS detector at the LHC in 2012. The observed numbers of events are found to be consistent with the standard model expectation, which is evaluated with control samples in data. The results are interpreted in the context of two simplified supersymmetric scenarios in which gluino pair production is followed by the decay of each gluino to an undetected lightest supersymmetric particle and either a bottom or top quark-antiquark pair, characteristic of gluino mediated bottom- or top-squark production. Using the production cross section calculated to next-to-leading-order plus next-to-leading-logarithm accuracy, and in the limit of a massless lightest supersymmetric particle, we exclude gluinos with masses below 1170 GeV and 1020 GeV for the two scenarios, respectively.
Detecting Anomalies from End-to-End Internet Performance Measurements (PingER) Using Cluster Based Local Outlier Factor
PingER (Ping End-to-End Reporting) is a worldwide end-to-end Internet performance measurement framework. It was developed by the SLAC National Accelerator Laboratory, Stanford, USA and running from the last 20 years. It has more than 700 monitoring agents and remote sites which monitor the performance of Internet links around 170 countries of the world. At present, the size of the compressed PingER data set is about 60 GB comprising of 100,000 flat files. The data is publicly available for valuable Internet performance analyses. However, the data sets suffer from missing values and anomalies due to congestion, bottleneck links, queuing overflow, network software misconfiguration, hardware failure, cable cuts, and social upheavals. Therefore, the objective of this paper is to detect such performance drops or spikes labeled as anomalies or outliers for the PingER data set. In the proposed approach, the raw text files of the data set are transformed into a PingER dimensional model. The missing values are imputed using the k-NN algorithm. The data is partitioned into similar instances using the k-means clustering algorithm. Afterward, clustering is integrated with the Local Outlier Factor (LOF) using the Cluster Based Local Outlier Factor (CBLOF) algorithm to detect the anomalies or outliers from the PingER data. Lastly, anomalies are further analyzed to identify the time frame and location of the hosts generating the major percentage of the anomalies in the PingER data set ranging from 1998 to 2016.
Soil Water Retention and Hydraulic Conductivity Data and Model at Pump House in East River Watershed, Colorado 2019-2024
This data package includes soil water retention and hydraulic conductivity data and model fitting results from measurements of ex-situ soil samples and in-situ soil sensors near Pump House at Mount Crested Butte in the East River Watershed. Soil water retention curves (SWRC) characterize soil water content as a function of soil water potential. SWRC depends on soil texture and pore structure and can be used to describe the constraints on biogeochemical processes in terms of soil water availability. In this data package, the sample identification follows the format ER-X-Y, where ER refers to East River, X is the location identifier, and Y is the depth identifier at the same X (shallow Y=1). Specifically, ER-PHS, ER-LMC, ER-LMF, and ER-SMN are associated with ecohydrology sites under the East-Taylor Watershed Community Observatory Sites directory, and ER-RBTn (upslope n=1) are sampling transects during the 2019 Rootball Campaign. The sample and location information can be found in metadata.csv. Sampling and Measurements Each sample falls into one of the three sampling methods – (1) intact cores, (2) repacked samples, or (3) soil sensors – and one of the two measurement methods – (a) laboratory or (b) in-situ. Both intact cores and repacked samples were measured using the laboratory methods, which include measurements of soil water potential (HYPROP & WP4C, METER), saturated (KSAT, METER) and unsaturated hydraulic conductivity (HYPROP). The in-situ method uses a pair of co-located soil sensors to measure volumetric water content (TEROS12, METER) and soil water potential (TEROS21, METER), and the hydraulic conductivity was not measured. In comparison, the laboratory methods progress from full saturation to dry conditions, and the in-situ method includes both dry-to-wet and wet-to-dry cycles. The sampling and measurement methods for each sample can be found in metadata.csv, and more information about the measurements is detailed in the Methods section below. Models Retention and hydraulic conductivity data were fitted with four van-Genuchten-type models (specified by “model_name” column in the files): (1) traditional constrained van Genuchten model (“vG_constrained”), (2) traditional unconstrained van Genuchten model (“vG_unconstrained”), (3) PDI-variant of the constrained van Genuchten model (“vG_constrained_PDI”), and (4) PDI-variant of the unconstrained van Genuchten model (“vG_unconstrained_PDI”). The difference between the constrained (1: n) and the unconstrained (2: n, m) van Genuchten models is the number of pore-size distribution parameters in the model equations, giving the unconstrained model more degrees of freedom when fitting the data. Between the traditional and the PDI-variant models, model fitting differs the most at the dry end of the measurements. The traditional models allow infinite suction at the residual water content (water content does not drop below residual water content), and the PDI-variant models enforce a soil water potential value of pF=6.8 (~ -630 MPa) at oven-dryness (water content reaches 0). The inclusion of the van-Genuchten-type models is due to their common application. If other retention models are required, users can access the data in data.csv for further data fitting. More information about the models can be found in the Methods section below. Fitting Tasks The model fitting can be categorized into three levels of tasks (specified by “fitting_task” column in the files). Level 1 (“fit_retention”) only includes retention data fitting (the only level available for the in-situ method). Level 2 (“fit_retention_conductivity”) includes both retention and hydraulic conductivity data fitting, and the saturated hydraulic conductivity (Ks, a parameter of the hydraulic conductivity functions) is fixed by the measurements from KSAT. Level 3 (“fit_retention_conductivity_Ks”) also includes both retention and hydraulic conductivity data fitting, but Ks is a fitted parameter without the constraints from KSAT measurements. Among the same retention models (e.g. vG_constrained models of the same sample), level 1 should produce the best retention data fitting. Level 2 should have the highest misfit of the retention and hydraulic conductivity data, because the retention and hydraulic conductivity functions share common model parameters, and the unsaturated hydraulic conductivity (HYPROP) data fitting is subject to Ks measured independently by KSAT. Level 3 should have mid-level misfits of the retention and hydraulic conductivity data. While level 3 fits the hydraulic conductivity data better than level 2, the fitted Ks value might be unreasonable due to the lack of constraints at the wet end of the measurements. General recommendation when using this data package: (1) Choice of sampling methods: Intact cores and in-situ soil sensors could be prioritized because these sampling methods are less destructive. While the repacked samples were packed to the target bulk density (estimated post-sampling, when sample volume was known), these samples had altered pore structures. Nevertheless, intact cores might suffer from sample gaps that would lead to overestimation of Ks (sample gaps can be inferred from the “soil_sample_volume” column in metadata.csv when the value is < 249). In-situ method also has higher uncertainty in characterizing the wet end of the SWRC because of sensor limitations and the difficulty in reaching full saturation under natural conditions. (2) Choice of fitting tasks: When only retention data is needed, level 1 (“fit_retention”) should be prioritized. When both retention and hydraulic conductivity data are needed, level 2 (“fit_retention_conductivity”) could be prioritized. (3) Choice of models: This could depend on what the downstream models call for. If no specific model is required, model misfit could be used as a ranking criterion. Model misfit values in terms of RMSE can be found in model_parameters.csv. The following files are included in this data package: (1) metadata.csv – This file includes the general information of each sample, including location (description, geocoordinates, elevation), sampling and measurements details (method, depth, time or period, volume, instruments), and soil physical properties (bulk density, saturated hydraulic conductivity, only applicable to physical soil samples). (2) data.csv – This file includes soil water potential, volumetric water content, and unsaturated hydraulic conductivity data of each sample. Column “instrument” specifies the instrument (HYPROP, WP4C, or TEROS) used to perform the measurements. (3) model_fit.csv – This file includes soil water potential, volumetric water content, and unsaturated hydraulic conductivity fitted from the four models and three fitting tasks. Column “model_name” specifies the retention model used, and “fitting_task” specifies the level of data fitting. Missing values indicate that the variable does not apply to that fitting task. (4) model_parameters.csv – This file includes the fitted model parameters, model misfits, and conventional water content thresholds (field capacity and wilting point) from the four models and three fitting tasks. Column “model_name” specifies the retention model used, and “fitting_task” specifies the level of data fitting. Missing values indicate that the parameter does not apply to that model and/or that fitting task. (5) data_Ks.csv – This file includes the saturated hydraulic conductivity measurements from KSAT. (6) /figure/*.png – This folder includes three quick visualizations of the data, retention model fitting results and misfits, and hydraulic conductivity model fitting results, misfits, and parameters. The model fitting results are separated by samples and fitting tasks and colored by models. Zoom-in required. (7) /hyprop/*.bdhx – This folder includes proprietary hyprop files that require the free Labros SoilView-Analysis (METER) to open. Users can explore data fitting using other retention models (i.e. Brooks-Corey, Fredlund-Xing, Kosugi, bimodal models). Be aware that Ks value is pre-entered under “Fitting tab, Conductivity functions parameters” for level 2 fitting. If the value is lost, please refer to metadata.csv under “Ks” column. (8) Six file-level metadata that summarize file, header, column, and variable information of all files. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.
Soil Water Retention and Hydraulic Conductivity Data and Model at Trail Creek in Taylor River Watershed, Colorado 2024-2025
This data package includes soil water retention and hydraulic conductivity data and model fitting results from measurements of ex-situ soil samples and in-situ soil sensors near Trail Creek. Soil water retention curves (SWRC) characterize soil water content as a function of soil water potential. SWRC depends on soil texture and pore structure and can be used to describe the constraints on biogeochemical processes in terms of soil water availability. In this data package, the sample identification follows the format TR-X-Y, where TR refers to Trail Creek, X is the treatment block identifier, and Y is the location identifier. Specifically, TR-ASCC1 is the control treatment block under the Adaptive Silviculture for Climate Change (ASCC) project, and TR-ASCC2 is the clear-cut treatment block. TR-ASCC-EHSn is associated with ecohydrology sites under the East-Taylor Watershed Community Observatory Sites directory, and TR-ASCC-ERTn (upslope n=1) are ecohydrology sites along the electrical resistivity tomography transects. The sample and location information can be found in metadata.csv, and the data from the soil sensors will be included in a future data version when the observation period becomes sufficiently long for data analysis. Sampling and Measurements Each sample falls into one of the two sampling methods – (1) intact cores or (2) soil sensors – and one of the two measurement methods – (a) laboratory or (b) in-situ. The intact cores were measured using the laboratory methods, which include measurements of soil water potential (HYPROP & WP4C, METER), saturated (KSAT, METER) and unsaturated hydraulic conductivity (HYPROP). The in-situ method uses a pair of co-located soil sensors to measure volumetric water content (TEROS12, METER) and soil water potential (TEROS21, METER), and the hydraulic conductivity was not measured. In comparison, the laboratory methods progress from full saturation to dry conditions, and the in-situ method includes both dry-to-wet and wet-to-dry cycles. The sampling and measurement methods for each sample can be found in metadata.csv, and more information about the measurements is detailed in the Methods section below. Models Retention and hydraulic conductivity data were fitted with four van-Genuchten-type models (specified by “model_name” column in the files): (1) traditional constrained van Genuchten model (“vG_constrained”), (2) traditional unconstrained van Genuchten model (“vG_unconstrained”), (3) PDI-variant of the constrained van Genuchten model (“vG_constrained_PDI”), and (4) PDI-variant of the unconstrained van Genuchten model (“vG_unconstrained_PDI”). The difference between the constrained (1: n) and the unconstrained (2: n, m) van Genuchten models is the number of pore-size distribution parameters in the model equations, giving the unconstrained model more degrees of freedom when fitting the data. Between the traditional and the PDI-variant models, model fitting differs the most at the dry end of the measurements. The traditional models allow infinite suction at the residual water content (water content does not drop below residual water content), and the PDI-variant models enforce a soil water potential value of pF=6.8 (~ -630 MPa) at oven-dryness (water content reaches 0). The inclusion of the van-Genuchten-type models is due to their common application. If other retention models are required, users can access the data in data.csv for further data fitting. More information about the models can be found in the Methods section below. Fitting Tasks The model fitting can be categorized into three levels of tasks (specified by “fitting_task” column in the files). Level 1 (“fit_retention”) only includes retention data fitting (the only level available for the in-situ method). Level 2 (“fit_retention_conductivity”) includes both retention and hydraulic conductivity data fitting, and the saturated hydraulic conductivity (Ks, a parameter of the hydraulic conductivity functions) is fixed by the measurements from KSAT. Level 3 (“fit_retention_conductivity_Ks”) also includes both retention and hydraulic conductivity data fitting, but Ks is a fitted parameter without the constraints from KSAT measurements. Among the same retention models (e.g. vG_constrained models of the same sample), level 1 should produce the best retention data fitting. Level 2 should have the highest misfit of the retention and hydraulic conductivity data, because the retention and hydraulic conductivity functions share common model parameters, and the unsaturated hydraulic conductivity (HYPROP) data fitting is subject to Ks measured independently by KSAT. Level 3 should have mid-level misfits of the retention and hydraulic conductivity data. While level 3 fits the hydraulic conductivity data better than level 2, the fitted Ks value might be unreasonable due to the lack of constraints at the wet end of the measurements. General recommendation when using this data package: (1) Choice of sampling methods: Intact cores might suffer from sample gaps that would lead to overestimation of Ks (sample gaps can be inferred from the “soil_sample_volume” column in metadata.csv when the value is < 249). In-situ method has higher uncertainty in characterizing the wet end of the SWRC because of sensor limitations and the difficulty in reaching full saturation under natural conditions. (2) Choice of fitting tasks: When only retention data is needed, level 1 (“fit_retention”) should be prioritized. When both retention and hydraulic conductivity data are needed, level 2 (“fit_retention_conductivity”) could be prioritized. (3) Choice of models: This could depend on what the downstream models call for. If no specific model is required, model misfit could be used as a ranking criterion. Model misfit values in terms of RMSE can be found in model_parameters.csv. The following files are included in this data package: (1) metadata.csv – This file includes the general information of each sample, including location (description, geocoordinates, elevation), sampling and measurements details (method, depth, time or period, volume, instruments), and soil physical properties (bulk density, saturated hydraulic conductivity, only applicable to physical soil samples). (2) data.csv – This file includes soil water potential, volumetric water content, and unsaturated hydraulic conductivity data of each sample. Column “instrument” specifies the instrument (HYPROP, WP4C, or TEROS) used to perform the measurements. (3) model_fit.csv – This file includes soil water potential, volumetric water content, and unsaturated hydraulic conductivity fitted from the four models and three fitting tasks. Column “model_name” specifies the retention model used, and “fitting_task” specifies the level of data fitting. Missing values indicate that the variable does not apply to that fitting task. (4) model_parameters.csv – This file includes the fitted model parameters, model misfits, and conventional water content thresholds (field capacity and wilting point) from the four models and three fitting tasks. Column “model_name” specifies the retention model used, and “fitting_task” specifies the level of data fitting. Missing values indicate that the parameter does not apply to that model and/or that fitting task. (5) data_Ks.csv – This file includes the saturated hydraulic conductivity measurements from KSAT. (6) /figure/*.png – This folder includes three quick visualizations of the data, retention model fitting results and misfits, and hydraulic conductivity model fitting results, misfits, and parameters. The model fitting results are separated by samples and fitting tasks and colored by models. Zoom-in required. (7) /hyprop/*.bdhx – This folder includes proprietary hyprop files that require the free Labros SoilView-Analysis (METER) to open. Users can explore data fitting using other retention models (i.e. Brooks-Corey, Fredlund-Xing, Kosugi, bimodal models). Be aware that Ks value is pre-entered under “Fitting tab, Conductivity functions parameters” for level 2 fitting. If the value is lost, please refer to metadata.csv under “Ks” column. (8) Six file-level metadata that summarize file, header, column, and variable information of all files. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.
Soil Water Retention and Hydraulic Conductivity Data and Model at Snodgrass Mountain in East River Watershed, Colorado 2020-2025
This data package includes soil water retention and hydraulic conductivity data and model fitting results from measurements of ex-situ soil samples and in-situ soil sensors at Snodgrass Mountain. Soil water retention curves (SWRC) characterize soil water content as a function of soil water potential. SWRC depends on soil texture and pore structure and can be used to describe the constraints on biogeochemical processes in terms of soil water availability. In this data package, the sample identification follows the format SG-X-Y, where SG refers to Snodgrass Mountain, X is the location identifier, and Y is the depth identifier at the same X (shallow Y=1). Specifically, SG-EHS is associated with ecohydrology sites under the East-Taylor Watershed Community Observatory Sites directory, and SG-ERTn (upslope n=1) are points along the Snodgrass electrical resistivity tomography transect not associated with the existing site names in the directory. The sample and location information can be found in metadata.csv. Sampling and Measurements Each sample falls into one of the three sampling methods – (1) intact cores, (2) repacked samples, or (3) soil sensors – and one of the two measurement methods – (a) laboratory or (b) in-situ. Both intact cores and repacked samples were measured using the laboratory methods, which include measurements of soil water potential (HYPROP & WP4C, METER), saturated (KSAT, METER) and unsaturated hydraulic conductivity (HYPROP). The in-situ method uses a pair of co-located soil sensors to measure volumetric water content (TEROS12, METER) and soil water potential (TEROS21, METER), and the hydraulic conductivity was not measured. In comparison, the laboratory methods progress from full saturation to dry conditions, and the in-situ method includes both dry-to-wet and wet-to-dry cycles. The sampling and measurement methods for each sample can be found in metadata.csv, and more information about the measurements is detailed in the Methods section below. Models Retention and hydraulic conductivity data were fitted with four van-Genuchten-type models (specified by “model_name” column in the files): (1) traditional constrained van Genuchten model (“vG_constrained”), (2) traditional unconstrained van Genuchten model (“vG_unconstrained”), (3) PDI-variant of the constrained van Genuchten model (“vG_constrained_PDI”), and (4) PDI-variant of the unconstrained van Genuchten model (“vG_unconstrained_PDI”). The difference between the constrained (1: n) and the unconstrained (2: n, m) van Genuchten models is the number of pore-size distribution parameters in the model equations, giving the unconstrained model more degrees of freedom when fitting the data. Between the traditional and the PDI-variant models, model fitting differs the most at the dry end of the measurements. The traditional models allow infinite suction at the residual water content (water content does not drop below residual water content), and the PDI-variant models enforce a soil water potential value of pF=6.8 (~ -630 MPa) at oven-dryness (water content reaches 0). The inclusion of the van-Genuchten-type models is due to their common application. If other retention models are required, users can access the data in data.csv for further data fitting. More information about the models can be found in the Methods section below. Fitting Tasks The model fitting can be categorized into three levels of tasks (specified by “fitting_task” column in the files). Level 1 (“fit_retention”) only includes retention data fitting (the only level available for the in-situ method). Level 2 (“fit_retention_conductivity”) includes both retention and hydraulic conductivity data fitting, and the saturated hydraulic conductivity (Ks, a parameter of the hydraulic conductivity functions) is fixed by the measurements from KSAT. Level 3 (“fit_retention_conductivity_Ks”) also includes both retention and hydraulic conductivity data fitting, but Ks is a fitted parameter without the constraints from KSAT measurements. Among the same retention models (e.g. vG_constrained models of the same sample), level 1 should produce the best retention data fitting. Level 2 should have the highest misfit of the retention and hydraulic conductivity data, because the retention and hydraulic conductivity functions share common model parameters, and the unsaturated hydraulic conductivity (HYPROP) data fitting is subject to Ks measured independently by KSAT. Level 3 should have mid-level misfits of the retention and hydraulic conductivity data. While level 3 fits the hydraulic conductivity data better than level 2, the fitted Ks value might be unreasonable due to the lack of constraints at the wet end of the measurements. General recommendation when using this data package: (1) Choice of sampling methods: Intact cores and in-situ soil sensors could be prioritized because these sampling methods are less destructive. While the repacked samples were packed to the target bulk density (estimated post-sampling, when sample volume was known), these samples had altered pore structures. Nevertheless, intact cores might suffer from sample gaps that would lead to overestimation of Ks (sample gaps can be inferred from the “soil_sample_volume” column in metadata.csv when the value is < 249). In-situ method also has higher uncertainty in characterizing the wet end of the SWRC because of sensor limitations and the difficulty in reaching full saturation under natural conditions. (2) Choice of fitting tasks: When only retention data is needed, level 1 (“fit_retention”) should be prioritized. When both retention and hydraulic conductivity data are needed, level 2 (“fit_retention_conductivity”) could be prioritized. (3) Choice of models: This could depend on what the downstream models call for. If no specific model is required, model misfit could be used as a ranking criterion. Model misfit values in terms of RMSE can be found in model_parameters.csv. The following files are included in this data package: (1) metadata.csv – This file includes the general information of each sample, including location (description, geocoordinates, elevation), sampling and measurements details (method, depth, time or period, volume, instruments), and soil physical properties (bulk density, saturated hydraulic conductivity, only applicable to physical soil samples). (2) data.csv – This file includes soil water potential, volumetric water content, and unsaturated hydraulic conductivity data of each sample. Column “instrument” specifies the instrument (HYPROP, WP4C, or TEROS) used to perform the measurements. (3) model_fit.csv – This file includes soil water potential, volumetric water content, and unsaturated hydraulic conductivity fitted from the four models and three fitting tasks. Column “model_name” specifies the retention model used, and “fitting_task” specifies the level of data fitting. Missing values indicate that the variable does not apply to that fitting task. (4) model_parameters.csv – This file includes the fitted model parameters, model misfits, and conventional water content thresholds (field capacity and wilting point) from the four models and three fitting tasks. Column “model_name” specifies the retention model used, and “fitting_task” specifies the level of data fitting. Missing values indicate that the parameter does not apply to that model and/or that fitting task. (5) data_Ks.csv – This file includes the saturated hydraulic conductivity measurements from KSAT. (6) /figure/*.png – This folder includes three quick visualizations of the data, retention model fitting results and misfits, and hydraulic conductivity model fitting results, misfits, and parameters. The model fitting results are separated by samples and fitting tasks and colored by models. Zoom-in required. (7) /hyprop/*.bdhx – This folder includes proprietary hyprop files that require the free Labros SoilView-Analysis (METER) to open. Users can explore data fitting using other retention models (i.e. Brooks-Corey, Fredlund-Xing, Kosugi, bimodal models). Be aware that Ks value is pre-entered under “Fitting tab, Conductivity functions parameters” for level 2 fitting. If the value is lost, please refer to metadata.csv under “Ks” column. (8) Six file-level metadata that summarize file, header, column, and variable information of all files. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.
Development of quantitative structure-activity relationship (QSAR) models to predict the carcinogenic potency of chemicals
Determining the carcinogenicity and carcinogenic potency of new chemicals is both a labor-intensive and time-consuming process. In order to expedite the screening process, there is a need to identify alternative toxicity measures that may be used as surrogates for carcinogenic potency. Alternative toxicity measures for carcinogenic potency currently being used in the literature include lethal dose (dose that kills 50% of a study population [LD{sub 50}]), lowest-observed-adverse-effect-level (LOAEL) and maximum tolerated dose (MTD). The purpose of this study was to investigate the correlation between tumor dose (TD{sub 50}) and three alternative toxicity measures as an estimator of carcinogenic potency. A second aim of this study was to develop a Classification and Regression Tree (CART) between TD{sub 50} and estimated/experimental predictor variables to predict the carcinogenic potency of new chemicals. Rat TD{sub 50}s of 590 structurally diverse chemicals were obtained from the Cancer Potency Database, and the three alternative toxicity measures considered in this study were estimated using TOPKAT, a toxicity estimation software. Though poor correlations were obtained between carcinogenic potency and the three alternative toxicity (both experimental and TOPKAT) measures for the CPDB chemicals, a CART developed using experimental data with no missing values as predictor variables provided reasonable estimates of TD{sub 50} for nine chemicals that were part of an external validation set. However, if experimental values for the three alternative measures, mutagenicity and logP are not available in the literature, then either the CART developed using missing experimental values or estimated values may be used for making a prediction.
Physics-Informed Deep Learning for Reconstruction of Spatial Missing Climate Information in the Antarctic
Understanding the influence of the Antarctic on the global climate is crucial for the prediction of global warming. However, due to very few observation sites, it is difficult to reconstruct the rational spatial pattern by filling in the missing values from the limited site observations. To tackle this challenge, regional spatial gap-filling methods, such as Kriging and inverse distance weighted (IDW), are regularly used in geoscience. Nevertheless, the reconstructing credibility of these methods is undesirable when the spatial structure has massive missing pieces. Inspired by image inpainting, we propose a novel deep learning method that demonstrates a good effect by embedding the physics-aware initialization of deep learning methods for rapid learning and capturing the spatial dependence for the high-fidelity imputation of missing areas. We create the benchmark dataset that artificially masks the Antarctic region with ratios of 30%, 50% and 70%. The reconstructing monthly mean surface temperature using the deep learning image inpainting method RFR (Recurrent Feature Reasoning) exhibits an average of 63% and 71% improvement of accuracy over Kriging and IDW under different missing rates. With regard to wind speed, there are still 36% and 50% improvements. In particular, the achieved improvement is even better for the larger missing ratio, such as under the 70% missing rate, where the accuracy of RFR is 68% and 74% higher than Kriging and IDW for temperature and also 38% and 46% higher for wind speed. In addition, the PI-RFR (Physics-Informed Recurrent Feature Reasoning) method we proposed is initialized using the spatial pattern data simulated by the numerical climate model instead of the unified average. Compared with RFR, PI-RFR has an average accuracy improvement of 10% for temperature and 9% for wind speed. When applied to reconstruct the spatial pattern based on the Antarctic site observations, where the missing rate is over 90%, the proposed method exhibits more spatial characteristics than Kriging and IDW.
Detecting differential protein expression in large-scale population proteomics
Mass spectrometry-based high-throughput quantitative proteomics shows great potential in clinical biomarker studies, identifying and quantifying thousands of proteins in biological samples. However, methods are needed to appropriately handle issues/challenges unique to mass spectrometry data in order to detect as many biomarker proteins as possible. One issue is that different mass spectrometry experiments generate quite different total numbers of quantified peptides, which can result in more missing peptide abundances in an experiment with a smaller total number of quantified peptides. Another issue is that the quantification of peptides is sometimes absent, especially for less abundant peptides and such missing values contain the information about the peptide abundance. Here, we propose a Significance Analysis for Large-scale Proteomics Studies (SALPS) that handles missing peptide intensity values caused by the two mechanisms mentioned above. Our model has a robust performance in both simulated data and proteomics data from a large clinical study. Because varying patients’ sample qualities and deviating instrument performances are not avoidable for clinical studies performed over the course of several years, we believe that our approach will be useful to analyze large-scale clinical proteomics data.
Cleaned 5-Minute Resolution Air Quality and Meteorological Data from Nine TCEQ CAMS Sites in Houston, Texas (Nov 2021 – Oct 2022)
These data encompass 5-minute air monitoring and meteorological observations collected in the greater Houston, Texas metropolitan region, at nine (9) Continuous Ambient Monitoring Stations (CAMS) operated by the Texas Commission on Environmental Quality (TCEQ) between November 1, 2021 and October 31, 2022. The CAMS sites (CAMS 1, 8, 35, 45, 148, 403, 405, 410, and 1052) were chosen because their instrumentation includes measurements of PM2.5. These sites also provide continuous multi-parameter air-quality and meteorological measurements. Particulate matter (PM2.5, PM10) was sampled along with several trace gases, including ozone (O3), nitrogen oxides (NO, NO2, NOx), sulfur dioxide (SO2), and carbon monoxide (CO). The data set also contains standard surface meteorological parameters (temperature, humidity, pressure, wind speed, and wind direction). Several sites also include AutoGC-based measurements of volatile organic compounds (VOCs). Air monitoring instruments deployed at the selected sites comprise the following systems: BAM-1020 or TEOM (PM2.5), Thermo Scientific TEI 49i (O3), TEI 42i (NOx), and AutoGCs (VOCs). This data set is similar to the data included within the houairq5mX1.00 datastream, except for a few additional quality control steps. A systematic data cleaning and verification process was performed on the data set to ensure its quality and preparation for analysis. Removal of non-numeric status flags (e.g., [LIM], [QAS], [SPZ], [CAL], [PMA], [AQI], [SPN], [MAL]) was accomplished by employing rule-based string parsing to extract valid numerical values. Missing entries were set to -9999; however, invalid or anomalous values (e.g., 99999) were retained as originally reported by the TCEQ to preserve data provenance. The time sequence was verified for completeness, removal of duplicates, and uniformity at 5-minute intervals. Column labeling was standardized, and corresponding values were assessed for physical plausibility. All timestamps in the data set were reported in Coordinated Universal Time (UTC) as provided by the TCEQ. Further, the latitude and longitude coordinates were added for each CAMS site. A subset of the data (June 1–September 30, 2022) has been used in the following publication: Subba et al. 2025. “Implications of sea breeze circulations on boundary layer aerosols in the southern coastal Texas region.” EGUsphere 2025: 1–49, https://doi.org/10.5194/egusphere-2025-2659.
A review of imputation strategies for isobaric labeling-based shotgun proteomics
The throughput efficiency and increased depth of coverage provided by isobaric-labeled proteomics measurements have led to increased usage of these techniques. However, the structure of missing data is uniquely different than unlabeled studies. In this review, we compare the efficacy of nine imputation methods on a CPTAC proteomics iTRAQ dataset. Imputation methods were evaluated with regard to accuracy, variability, statistical hypothesis test inference and run time over datasets consisting of varying number of iTRAQ plexes and percentages of missing data. In general, expectation maximization and random forest imputation methods yielded the best performances, and constant-based methods performed poorly consistently across all dataset sizes and percentages of missing values. For datasets with small sample sizes and higher percentages of missing data, results indicate that statistical inference with no imputation may be preferable. Based on the findings in this review, there are core imputation methods that perform higher for isobaric-labeled proteomics data, but great care and consideration as to whether imputation should be used should be given for datasets comprised of a small number of samples, as well as to factors such as computational time and reproducibility of imputation values.