Search NASA⌕ Search

SEARCH · Search NASA

Results for “data skew”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Probabilistic assessment of cloud fraction using Bayesian blending of independent datasets: Feasibility study of a new method

We describe and evaluate a novel method to blend two observed cloud fraction (CF) datasets through Bayesian posterior estimation. The research reported here is a feasibility study designed to explore the method. In this proof‐of‐concept study, we illustrate the approach using specific observational datasets from the U. S. Department of Energy Atmospheric Radiation Measurement Program's Southern Great Plains site in the central United States, but the method is quite general and is readily applicable to other datasets. The total sky image (TSI) camera observations are used to determine the prior distribution. A regression model and the active remote sensing of clouds (ARSCL) radar/lidar observations are used to determine the likelihood function. The posterior estimate is a probability density function (pdf) of the CF whose mean is taken to be the optimal blend of the two observations. The data at hourly, daily, 5‐day, monthly, and annual time scales are considered. Some physical and probabilistic properties of the CFs are explored from radar/lidar, camera, and satellite observations and from simulations using the Community Atmosphere Model (CAM5). Our results imply that (a) the Beta distribution is a reasonable model for CF for both short‐ and long‐time means, the 5‐day data are skewed right, and the annual data are almost normally distributed, and (b) the Bayesian method developed successfully yields a pdf of CF, rather than a deterministic CF value, and it is feasible to blend the TSI and ARSCL data with a capability for bias correction.

58 GEOSCIENCES↗

Clustering High-dimensional Toxicogenomics Data with Rare Signals

Toxicogenomics studies the gene and protein activities to drug treatments or toxic exposures. As the drugs and genes are numerous, toxicogenomics data are naturally high dimensional, with dimension sizes up to millions. In addition, the distribution of toxicogenomics data is oftentimes skewed, and they contain rare but important signals representing a cell or organism’s response to toxicity. The combination of high dimension and extremely skewed distribution of toxicogenomics data makes clustering analysis extremely challenging.We present our study of clustering toxicogenomics data using classical approaches such as principal component analysis as well as deep learning approaches such as auto-encoders. Our experiments show that these approaches fail to preserve rare signals and produce high-quality clusters. We then explore augmenting matrix factorization with deep learning techniques such as attention mechanism to produce latent representations for clustering. Our technique is able to better preserve rare signals after dimensionality reduction than prior approaches. Furthermore, we combine our augmented matrix factorization with a mechanism similar to autoencoder to balance separable clusters and low regeneration errors. Our experiments demonstrate better clustering with our proposed approach.

Cong, Guojing↗

Design study for multi-channel tape recorder system, volume 2

Skew test data are presented on a tape recorder transport with a double capstan drive for a 100 KHz tone recorded on five tracks simultaneously. Phase detectors were used to measure the skew when the center channel was the 100 KHz reference.

Source record↗

Remark on Algorithm 1012: Computing Projections with Large Datasets

In ACM TOMS Algorithm 1012, the DELAUNAYSPARSE software is given for performing Delaunay interpolation in medium to high dimensions. When extrapolating outside the convex hull of the training set, DELAUNAYSPARSE calls the nonnegative least squares solver DWNNLS to compute projections onto the convex hull. However, DWNNLS and many other available sum-of-squares optimization solvers were not intended for usage with many variable problems, which result from the large training sets that are typical in machine learning applications. Thus, a new PROJECT subroutine is given, based on the highly customizable quadratic program solver BQPD. This solution is shown to be as robust as DELAUNAYSPARSE for projection onto both synthetic and real-world datasets, where other available solvers frequently fail. Although it is intended as an update for DELAUNAYSPARSE, due to the difficulty and prevalence of the problem, this solution is likely to be of external interest as well.

97 MATHEMATICS AND COMPUTING↗

Measurement of Iodine, Iodide, and Iodate in Hanford Tank Waste: Technology Transfer from PNNL to Hanford 222-S Laboratory

This report summarizes the measurement of the inorganic chemical forms of iodine in Hanford tank waste. The Hanford tank wastes have roughly a part per million iodine. Part of the iodine is the long-lived beta emitter 129I, and the rest is stable 127I. Because tank waste iodine is radioactive and radiotoxic, its chemical form must be known so that its behavior in the vitrification process can be reliably known and anticipated in various waste streams (glass, secondary liquid and solid waste, and offgas emissions) at Hanford. Iodine in tank wastes exists in the inorganic chemical forms iodide and iodate (and possibly also periodate) and also as organic forms such as alkyl iodides. The measurement of organic iodides is very different from the measurement of inorganic forms and is outside the scope of this report. The inorganic chemical forms are chemically separated, in sequence, from the raw tank waste using solvent extractions and several redox reactions, then measured by ICP-MS. The inorganic chemical forms of iodine are chemically reactive, and so is the tank waste. The chemistry must be carefully designed to avoid unintended reactions that could convert one form of iodine to another during the analysis, which would skew the data and report iodine in the wrong chemical form. This iodine analysis, developed at PNNL, is being transferred to the DOE 222-S Laboratory on the Hanford Site, in Washington State, for implementation during waste processing operations.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Applied Meteorology Unit (AMU)

This report summarizes the Applied Meteorology Unit (AMU) activities for the second quarter of Fiscal Year 2007 (January - March 2007). Tasks reported on are: Obiective Lightning Probability Tool, Peak Wind Tool for General Forecasting, Situational Lightning Climatologies for Central Florida, Anvil Threat Corridor Forecast Tool in AWIPS, Volume Averaqed Heiqht lnteq rated Radar Reflectivity (VAHIRR), Tower Data Skew-t Tool, and Weather Research and Forecastini (WRF) Model Sensitivity Study

Bauman, William↗

Algae Tile Data: 2004-2007, BPA-51; Preliminary Report, October 28, 2008.

Multiple files containing 2004 through 2007 Tile Chlorophyll data for the Kootenai River sites designated as: KR1, KR2, KR3, KR4 (Downriver) and KR6, KR7, KR9, KR9.1, KR10, KR11, KR12, KR13, KR14 (Upriver) were received by SCS. For a complete description of the sites covered, please refer to http://ktoi.scsnetw.com. To maintain consistency with the previous SCS algae reports, all analyses were carried out separately for the Upriver and Downriver categories, as defined in the aforementioned paragraph. The Upriver designation, however, now includes three additional sites, KR11, KR12, and the nutrient addition site, KR9.1. Summary statistics and information on the four responses, chlorophyll a, chlorophyll a Accrual Rate, Total Chlorophyll, and Total Chlorophyll Accrual Rate are presented in Print Out 2. Computations were carried out separately for each river position (Upriver and Downriver) and year. For example, the Downriver position in 2004 showed an average Chlorophyll a level of 25.5 mg with a standard deviation of 21.4 and minimum and maximum values of 3.1 and 196 mg, respectively. The Upriver data in 2004 showed a lower overall average chlorophyll a level at 2.23 mg with a lower standard deviation (3.6) and minimum and maximum values of (0.13 and 28.7, respectively). A more comprehensive summary of each variable and position is given in Print Out 3. This lists the information above as well as other summary information such as the variance, standard error, various percentiles and extreme values. Using the 2004 Downriver Chlorophyll a as an example again, the variance of this data was 459.3 and the standard error of the mean was 1.55. The median value or 50th percentile was 21.3, meaning 50% of the data fell above and below this value. It should be noted that this value is somewhat different than the mean of 25.5. This is an indication that the frequency distribution of the data is not symmetrical (skewed). The skewness statistic, listed as part of the first section of each analysis, quantifies this. In a symmetric distribution, such as a Normal distribution, the skewness value would be 0. The tile chlorophyll data, however, shows larger values. Chlorophyll a, in the 2004 Downriver example, has a skewness statistic of 3.54, which is quite high. In the last section of the summary analysis, the stem and leaf plot graphically demonstrates the asymmetry, showing most of the data centered around 25 with a large value at 196. The final plot is referred to as a normal probability plot and graphically compares the data to a theoretical normal distribution. For chlorophyll a, the data (asterisks) deviate substantially from the theoretical normal distribution (diagonal reference line of pluses), indicating that the data is non-normal. Other response variables in both the Downriver and Upriver categories also indicated skewed distributions. Because the sample size and mean comparison procedures below require symmetrical, normally distributed data, each response in the data set was logarithmically transformed. The logarithmic transformation, in this case, can help mitigate skewness problems. The summary statistics for the four transformed responses (log-ChlorA, log-TotChlor, and log-accrual ) are given in Print Out 4. For the 2004 Downriver Chlorophyll a data, the logarithmic transformation reduced the skewness value to -0.36 and produced a more bell-shaped symmetric frequency distribution. Similar improvements are shown for the remaining variables and river categories. Hence, all subsequent analyses given below are based on logarithmic transformations of the original responses.

13 HYDRO ENERGY↗

Unbalanced Parallel I/O: An Often-Neglected Side Effect of Lossy Scientific Data Compression

Lossy compression techniques have demonstrated promising results in significantly reducing the scientific data size while guaranteeing the compression error bounds. However, one important yet often neglected side effect of lossy scientific data compression is its impact on the performance of parallel I/O. Our key observation is that the compressed data size is often highly skewed across processes in lossy scientific compression. To understand this behavior, we conduct extensive experiments where we apply three lossy compressors MGARD, ZFP, and SZ, which are specifically designed and optimized for scientific data, to three real-world scientific applications Gray-Scott simulation, WarpX, and XGC. Our analysis result demonstrates that the size of the compressed data is always skewed even if the original data is evenly decomposed among processes. Such skewness widely exists in different scientific applications using different compressors as long as the information density of the data varies across processes. We then systematically study how this side effect of lossy scientific data compression impacts the performance of parallel I/O. We observe that the skewness in the sizes of the compressed data often leads to I/O imbalance, which can significantly reduce the efficiency of I/O bandwidth utilization if not properly handled. In addition, writing data concurrently to a single shared file through MPI-IO library is more sensitive to the unbalanced I/O loads. Therefore, we believe our research community should pay more attention to the unbalanced parallel I/O caused by lossy scientific data compression.

Wang, Xinying↗

Cost of Fish Exclusion and Passage Technologies for Hydropower

Hydropower represents a reliable source of renewable energy and accounts for approximately 7% of the total electrical generation in the United States. Future expansion of hydropower is likely to be in the form of either smaller new stream development projects or powering existing non-powered dams. For these new projects to be successful, careful analysis of risks, costs, and uncertainty to offset reduced power production as well as ensuring the protection and safe passage of migratory fish to gain public support, will be required. Exclusion and passage are two common approaches to protect fish from entrainment and impingement at hydropower facilities. The thresholds for entrainment risk and requirements for exclusion and passage often differ depending on the species involved, the characteristics of the facility, and the goals of stakeholders. While the costs associated with environmental mitigations represent a large proportion of the total costs required for the licensing of hydropower facilities, little quantitative information is present within the literature regarding the specific costs of fish exclusion and passage. Working with FOA awardee Natel Energy, scientists at Oak Ridge National Laboratory were tasked with assessing the capital construction costs for downstream fish exclusion and passage infrastructure. This report used keyword searches of an existing environmental mitigation cost data set and manual extraction of additional cost data associated with protection, mitigation, and enhancement (PM&E) measures related to positive barrier screening and passage from regulatory licensing documents available in the Federal Energy Regulatory Commission (FERC) eLibrary. This approach yielded a total of 50 PM&E mitigation measures with estimated capital construction costs pertaining to positive barrier screens, 142 pertaining to passage studies, and 26 pertaining to passage-related studies. PM&E measures associated with positive barrier screens represented <10% of the 171 total FERC project dockets available in the data set. These data were highly skewed toward conventional relicensing projects, as <7% were associated with new stream development (NSD) projects. Results from these data indicate highly variable costs associated with fish screening, with flow-normalized costs one to two orders of magnitude higher for screening with the highest exclusion capability (≤0.09 in. spacing) compared with coarser screening (1 to 2 in.). Furthermore, estimated capital costs of passage infrastructure were positively related to the scale of the project based on installed capacity for some, but not all, types of passage. These data provide an initial baseline for estimating exclusion and passage costs for hydropower development and may help developers consider options for more fish-friendly generation technologies, though gaps remain relating to a lack of data, particularly for NSD projects. More data may still be available within the FERC eLibrary, but significant effort will be required to manually identify and extract the data for future analyses.

13 HYDRO ENERGY↗

Fifth Status Report: Testing of Aged Softwood Fiberboard Material for the 9975 Shipping Package

Samples have been prepared from a 9975 lower fiberboard subassembly fabricated from softwood fiberboard. Physical, mechanical and thermal properties have been measured following varying periods of conditioning in each of several environments. These tests have been conducted in the same manner as previous testing on cane fiberboard samples. Overall, similar aging trends are observed for softwood and cane fiberboard samples, with a few differences. Some softwood fiberboard properties tend to degrade faster in elevated humidity environments, while some cane fiberboard properties degrade faster in the hotter dry environments. As a result, it is premature to assume both materials will age at the same rates, and the preliminary aging models developed for cane fiberboard might not apply to softwood fiberboard. However, it is expected that both cane and softwood fiberboard assemblies will perform satisfactorily in conforming packages stored in a typical KAC storage environment for up to 15 years. Aging and testing of softwood fiberboard will continue and additional data will be collected. Additional samples will be added to each aging environment, to support development of an aging model specific to softwood fiberboard. Post-conditioning data have been measured on samples from a single softwood fiberboard assembly, and baseline data are also available from a limited number of vendor-provided samples. This provides minimal information on the possible sample-to-sample variation exhibited by softwood fiberboard. Data to date are generally consistent with the range seen in cane fiberboard, but some portions of the data trends are skewed toward the lower end of that range. Two additional softwood fiberboard source packages have been obtained and will begin to provide data on the range of variability of this material.

36 MATERIALS SCIENCE↗

Is the assumption of normality or log-normality for continuous response data critical for benchmark dose estimation?

Continuous responses (e.g. body weight) are widely used in risk assessment for determining the benchmark dose (BMD) which is used to derive a U.S. EPA reference dose. One critical question that is not often addressed in dose–response assessments is whether to model the continuous data as normally or log-normally distributed. Additionally, if lognormality is assumed, and only summarized response data (i.e., mean ± standard deviation) are available as is usual in the peer-reviewed literature, the BMD can only be approximated. In this study, using the “hybrid” method and relative deviation approach, we first evaluate six representative continuous dose–response datasets reporting individual animal responses to investigate the impact on BMD/BMDL estimates of (1) the distribution assumption and (2) the use of summarized versus individual animal data when a log-normal distribution is assumed. We also conduct simulation studies evaluating model fits to various known distributions to investigate whether the distribution assumption has influence on BMD/BMDL estimates. Our results indicate that BMDs estimated using the hybrid method are more sensitive to the distribution assumption than counterpart BMDs estimated using the relative deviation approach. The choice of distribution assumption has limited impact on the BMD/BMDL estimates when the within dose-group variance is small, while the lognormality assumption is a better choice for relative deviation method when data are more skewed because of its appropriateness in describing the relationship between mean and standard deviation. Additionally, the results suggest that the use of summarized data versus individual response data to characterize log-normal distributions has minimal impact on BMD estimates. - Highlights: • We investigate to what extent the distribution assumption can affect BMD estimates. • Both real data analysis and simulation study are conducted. • BMDs estimated using hybrid method are more sensitive to distribution assumption. • Summarized continuous data are adequate for BMD estimation.

60 APPLIED LIFE SCIENCES↗

Digital tape unit test facility software

Two computer programs are described which are used for the collection and analysis of data from the digital tape unit test facility (DTUTF). The data are the recorded results of skew tests made on magnetic digital tapes which are used on computers as input/output media. The results of each tape test are keypunched onto an 80 column computer card. The format of the card is checked and the card image is stored on a master summary tape via the DTUTF card checking and tape updating system. The master summary tape containing the results of all the tape tests is then used for analysis as input to the DTUTF histogram generating system which produces a histogram of skew vs. date for selected data, followed by some statistical analysis of the data.

Jackson, J. T.↗

Regression for Assessing Contaminant Plume Trends

The Department of Energy’s (DOE) Hanford site has significant dissolved-phase groundwater contaminant plumes, for which remediation activities are ongoing. Assessing concentration trends is critical for making decisions regarding remedy implementation or performance. The SOCRATES single-page web application provides a quality-assured framework for data access and consistent data analytics using standard methods. To provide trend information for remedial decisions, statistical analyses based on ordinary least squares (OLS) are already supported by SOCRATES. The presence of censored data, however, can skew the result of an OLS regression. Other methods, such as the Akritas-Theil-Sen (ATS) or Tobit regressions can be more powerful. These two regression methods are used by Hanford site personnel to analyze groundwater contaminant concentration data, which often include nondetects (censored data). The ATS method uses comparisons of observation-pair ranks and the Tobit method uses a maximum likelihood estimation method. The Tobit method can also accommodate covariates, such as river stage or groundwater level fluctuations. Both ATS and Tobit regressions can be performed with the statistics software R, but to incorporate these methods into SOCRATES, their R code was ported into JavaScript. For quality assurance verification, the new code was tested with hexavalent chromium concentration data from the Hanford 100-K West Area and compared to regression analyses that were previously conducted on this data using R.

100-K Area↗

Estimated capital costs of fish exclusion technologies for hydropower facilities

Hydropower is a reliable source of renewable energy, and its future expansion is likely to be in the form of either smaller new stream development (NSD) projects or powering existing non-powered dams. Thresholds for entrainment risk to fish and the requirements for fish exclusion at hydropower facilities often differ depending on the species involved, the characteristics of the facility, and the goals of stakeholders, but little quantitative information is present within the literature regarding the specific costs of fish exclusion measures. Cost data associated with protection, mitigation, and enhancement (PM&E) measures related to positive barrier screening were identified using keyword searches of an existing environmental mitigation cost data set and manual extraction from regulatory licensing documents available in the Federal Energy Regulatory Commission (FERC) eLibrary. This approach yielded a total of 50 p.m.&E mitigation measures with estimated capital construction costs pertaining to positive barrier screens and represented <10% of the 171 total FERC project dockets available in the data set. These data were highly skewed toward conventional relicensing projects, as <7% were associated with NSD projects. Results indicate highly variable costs are associated with fish screening, with flow-normalized costs one to two orders of magnitude higher for screening with the highest exclusion capability (≤0.09 in. spacing) compared with coarser screening (1–2 in.). These data provide an initial baseline for estimating exclusion costs for hydropower development and may help developers consider options for more fish-friendly generation technologies, though gaps remain relating to a lack of data, particularly for NSD projects.

13 HYDRO ENERGY↗

Planck 2013 results. XXIV. Constraints on primordial non-Gaussianity

The Planck nominal mission cosmic microwave background (CMB) maps yield unprecedented constraints on primordial non-Gaussianity (NG). Using three optimal bispectrum estimators, separable template-fitting (KSW), binned, and modal, we obtain consistent values for the primordial local, equilateral, and orthogonal bispectrum amplitudes, quoting as our final result f NL local = 2.7 ± 5.8, f NL equil = -42 ± 75, and f NL orth = -25 ± 39 (68% CL statistical). Non-Gaussianity is detected in the data; using skew-C &ell; statistics we find a nonzero bispectrum from residual point sources, and the integrated-Sachs-Wolfe-lensing bispectrum at a level expected in the ΛCDM scenario. The results are based on comprehensive cross-validation of these estimators on Gaussian and non-Gaussian simulations, are stable across component separation techniques, pass an extensive suite of tests, and are confirmed by skew-C &ell; , wavelet bispectrum and Minkowski functional estimators. Beyond estimates of individual shape amplitudes, we present model-independent, three-dimensional reconstructions of the Planck CMB bispectrum and thus derive constraints on early-Universe scenarios that generate primordial NG, including general single-field models of inflation, excited initial states (non-Bunch-Davies vacua), and directionally-dependent vector models. We provide an initial survey of scale-dependent feature and resonance models. These results bound both general single-field and multi-field model parameter ranges, such as the speed of sound, c s ≥ 0.02 (95% CL), in an effective field theory parametrization, and the curvaton decay fraction r D ≥ 0.15 (95% CL). The Planck data significantly limit the viable parameter space of the ekpyrotic/cyclic scenarios. The amplitude of the four-point function in the local model τ NL < 2800 (95% CL). Taken together, these constraints represent the highest precision tests to date of physical mechanisms for the origin of cosmic structure.

79 ASTRONOMY AND ASTROPHYSICS↗

SECOND STATUS REPORT: TESTING OF AGED SOFTWOOD FIBERBOARD MATERIAL FOR THE 9975 SHIPPING PACKAGE

Samples have been prepared from a softwood fiberboard lower subassembly. Physical, mechanical and thermal properties have been measured following varying periods of conditioning in each of several environments. These tests have been conducted in the same manner as previous testing on cane fiberboard samples. Overall, similar aging trends are observed for softwood and cane fiberboard samples, with a few differences. On the positive side, the softwood fiberboard data to date shows less sample-to-sample variation in physical properties than cane fiberboard, and the thermal conductivity decreases at a slower rate at 250F for softwood fiberboard than for cane fiberboard. On the other hand, the softwood fiberboard physical property samples generally show degradation rates greater than cane fiberboard samples in the 185F 30%RH environment. Testing following additional conditioning will continue and the addition of samples in other elevated humidity environment(s) will be pursued to identify the extent of these trends. Post-conditioning data have been measured on samples from a single softwood fiberboard assembly, and baseline data are also available from a limited number of vendor-provided samples. This provides minimal information on the possible sample-to-sample variation exhibited by softwood fiberboard. Data to date are generally consistent with the range seen in cane fiberboard, but some portions of the data trends are skewed toward the lower end of that range. Further understanding of the variability of softwood fiberboard properties will require testing of additional material.

36 MATERIALS SCIENCE↗