Search NASA⌕ Search

SEARCH · Search NASA

Results for “data skew”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Toward equitable environmental exposure modeling through convergence of data, open, and citizen sciences: an example of air pollution exposure modeling amidst increasing wildfire smoke

Exposure modeling is critical in environmental epidemiology and human health but may face challenges (e.g., skewed data, unequal error, context-insensitive validation, and computational demands). Modeling decisions reflect the intended use of the models and the values that modelers prioritize. We aimed to provide a conceptual framework and machine learning (ML) modeling protocols that address these issues. With 500m-gridded hourly PM 2.5 and O 3 levels in Illinois before, during, and after the 2023 Canadian wildfire season as a motivating example, we conducted modeling experiments to evaluate modeling methods, guided by three domains we propose based on theories of science: 1) Data Diversity, leveraging open and citizen science data to enhance inclusivity, parsimony, and representativeness; 2) Equitable Accuracy, ensuring fairly distributed uncertainties across subpopulations; and 3) Sustainable Modeling, balancing accuracy with reducing computational demands to promote accessibility for under-resourced researchers. Here, we found that ML with publicly available data can achieve high accuracy. Depending on methods, performance may vary substantially, even with identical input data. Large but skewed data may reduce performance. Misuse of cross-validation protocols can underestimate prediction error; although we observed R 2 s of ∼98 %, the modeled estimates varied significantly, indicating the need for careful model validation. By using new modeling protocols including representativeness-considered training and validation data and a new loss function, we achieved high agreement between estimates and ground-based measurements (e.g., R 2 = ∼90 % for PM 2.5 ; ∼80 % for O 3 ), equally distributed errors across sociodemographic strata and urban–rural divides, and reduction in computation time—from several weeks or months to a few days.

Exposure assessment↗

Distributions of the Mean for Lognormally Distributed Data - 19427

In environmental sciences, a distribution of the mean is often needed for right-skewed data. For example, risk assessments are often based on distributions of the mean to obtain upper confidence limits (UCLs) for analyte concentrations, and, in probabilistic performance assessment (PPA) models, parameters are given distributions rather than deterministic values, and they are usually given distributions of their mean rather than the population distributions to account for the large spatial and temporal domains to which they are applied. How this distribution of the mean is developed depends on the characteristics of the parameter's full distribution and how the parameter relates to its use. In this presentation, I will discuss the implications of a right skew in distribution development. Parameters may vary over orders of magnitude and/or may be right-skewed. The material property of saturated hydraulic conductivity, Ks, is a prime example of such a parameter. Right-skewed parameters are often assumed to follow lognormal distributions. If X follows a lognormal distribution, then log(X) follows a normal distribution. For some right-skewed parameters, practitioners go back and forth between the two forms of the data depending on their needs. For example, it may be preferred to use the untransformed version of X in a model or equation, but it is easier to visualize the parameter values on a logarithmic scale. However, distribution development for the average value of the parameter will differ depending on whether X or log(X) is modeled. This difference is due to Jensen's Inequality, a theorem which states that the expected value of a concave function of a variable is less than or equal to the same function of the expected value of the variable; a logarithm is a concave function. In other words, the mean of log(X) is always less than or equal to the log of the mean of X and therefore the mean estimated from a distribution based on log(X) is less than or equal to the mean estimated from a distribution based on X. If a parameter is naturally considered on a log scale by subject matter experts, then distributions developed for the mean of log(X) may be more consistent with the use of the parameters in the literature than distributions developed for X. The statistical decisions of distribution development will be discussed within the context of right-skewed, lognormal data. These issues include method-of-moments versus maximum likelihood estimation of the distribution parameters and the use of bootstrapping to aid in the visualization of a distribution of the mean. The details surrounding Jensen's Inequality will be explained and visualized within the context of distribution development of a right-skewed parameter such as saturated hydraulic conductivity. (authors)

54 ENVIRONMENTAL SCIENCES↗

DeStager: feature guided in-situ data management in distributed deep memory hierarchies

In-situ analytics have been increasingly adopted by leadership scientific applications to gain fast insights into massive output data of simulations. With the current practice, systems buffer the output data in DRAM for analytics processing, constraining it to DRAM capacity un-used by the simulation. The rapid growth of data size requires alternative approaches to accommodating data-rich analytics, such as using solid-state disks to increase effective memory capacity. For this purpose, this paper explores software solutions for exploring the deep memory hierarchies expected on future high-end machines. Leveraging the fact that many analytics are sensitive to data features (regions-of-interest) hidden in the data being processed, the approach incorporates the knowledge of the data features into in-situ data management. It uses adaptive index creation/refinement to reduce the overhead of index management. In addition, it uses data features to predict data skew and improve load balance through controlling data distribution and placement on distributed staging servers. The experimental results show that such feature-guided optimizations achieve substantial improvements over state-of-the-art approaches for managing output data in-situ.

Computer Science↗

Resource-Adaptive Federated Text Generation with Differential Privacy

In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client’s data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.

Wang, Jiayi [ORNL]↗

On the measurement of shape: With applications to lunar regolith

With the renewed commitment from NASA and other commercial entities for a presence on the Moon, the importance of understanding the characteristics of lunar regolith and how to utilize it have become the target of increasing scrutiny. Much of what is known about lunar regolith was collected during and immediately after the Apollo program, however, analytical techniques and instrumentation have advanced in leaps and bounds in the subsequent decades. Specifically, dynamic image analysis systems have advanced to the point that millions of particles can have morphological characteristics automatically determined in relatively short time frames. Particle morphological data was collected on several lunar samples and a pair of widely used regolith simulants to ascertain the accuracy of these simulants and to explore statistical analysis methods of these large datasets. It is found that these morphology datasets can vary widely depending on the particle size of the particles, and simple averaging of the data skews the results heavily towards the numerically abundant size ranges, the fines. Different reporting methods are suggested to ameliorate these problems. When applied to the lunar regolith, the particles are noted to be less morphologically complex than initially suspected. Compared to the lunar material, the simulants are found to contain some more morphological variability and angular grains. Such difference is likely due to the wildly different comminution processes that these different powder systems are subjected to.

2D shape↗

Effective Quantification of Wind-Speed Variability at Different Temporal Scales

RCoV, a statistically robust and resistant metric, effectively characterizes and correlates the spreads of the distributions of wind resources and wind-energy productions. Using RCoV, a wind farm with high wind-speed fluctuations possesses high variations in wind-energy generation and vice versa. Because the long-term correlations between the wind-speed and energy-production interannual variabilities are weak and decrease with the length of data, we do not recommend calculating variabilities with yearly mean data. Skewness and kurtosis drastically change with averaging time frames. Nonzero skewness and kurtosis illustrate that the Gaussian assumption is principally inadequate in most of the United States for all averaging time frames of wind speeds.

17 WIND ENERGY↗

Analysis of BOSS galaxy data with weighted skew-spectra

We present the first application of the weighted skew-spectra to analyze non-Gaussian information in galaxy survey data. Using the tree-level galaxy skew-spectra together with the one-loop power spectrum multipoles, we analyze the Sloan Digital Sky Survey (SDSS)-III Baryon Oscillation Spectroscopic Survey (BOSS) galaxy clustering data, and target our search towards the equilateral bispectrum shape of primordial non-Gaussianity. We use the Effective Field Theory model for the galaxy power spectrum and bispectrum, and account for systematic effects, such as the survey geometry. From our likelihood analysis, we find f NL equil = -34 +296 -334 at 68% CL, consistent with previous works, while systematic errors from our treatment of the survey geometry lead to an unreliable estimation of f NL ortho . As a result, we further constrain the bias and counterterm parameters, while keeping the cosmology fixed to Planck 2018 values. As a check, we also validate our analysis pipeline using theNseries simulation suite.

79 ASTRONOMY AND ASTROPHYSICS↗

Clustering High-dimensional Toxicogenomics Data with Rare Signals

Toxicogenomics studies the gene and protein activities to drug treatments or toxic exposures. As the drugs and genes are numerous, toxicogenomics data are naturally high dimensional, with dimension sizes up to millions. In addition, the distribution of toxicogenomics data is oftentimes skewed, and they contain rare but important signals representing a cell or organism’s response to toxicity. The combination of high dimension and extremely skewed distribution of toxicogenomics data makes clustering analysis extremely challenging.We present our study of clustering toxicogenomics data using classical approaches such as principal component analysis as well as deep learning approaches such as auto-encoders. Our experiments show that these approaches fail to preserve rare signals and produce high-quality clusters. We then explore augmenting matrix factorization with deep learning techniques such as attention mechanism to produce latent representations for clustering. Our technique is able to better preserve rare signals after dimensionality reduction than prior approaches. Furthermore, we combine our augmented matrix factorization with a mechanism similar to autoencoder to balance separable clusters and low regeneration errors. Our experiments demonstrate better clustering with our proposed approach.

Cong, Guojing↗

Remark on Algorithm 1012: Computing Projections with Large Datasets

In ACM TOMS Algorithm 1012, the DELAUNAYSPARSE software is given for performing Delaunay interpolation in medium to high dimensions. When extrapolating outside the convex hull of the training set, DELAUNAYSPARSE calls the nonnegative least squares solver DWNNLS to compute projections onto the convex hull. However, DWNNLS and many other available sum-of-squares optimization solvers were not intended for usage with many variable problems, which result from the large training sets that are typical in machine learning applications. Thus, a new PROJECT subroutine is given, based on the highly customizable quadratic program solver BQPD. This solution is shown to be as robust as DELAUNAYSPARSE for projection onto both synthetic and real-world datasets, where other available solvers frequently fail. Although it is intended as an update for DELAUNAYSPARSE, due to the difficulty and prevalence of the problem, this solution is likely to be of external interest as well.

97 MATHEMATICS AND COMPUTING↗

Measurement of Iodine, Iodide, and Iodate in Hanford Tank Waste: Technology Transfer from PNNL to Hanford 222-S Laboratory

This report summarizes the measurement of the inorganic chemical forms of iodine in Hanford tank waste. The Hanford tank wastes have roughly a part per million iodine. Part of the iodine is the long-lived beta emitter 129I, and the rest is stable 127I. Because tank waste iodine is radioactive and radiotoxic, its chemical form must be known so that its behavior in the vitrification process can be reliably known and anticipated in various waste streams (glass, secondary liquid and solid waste, and offgas emissions) at Hanford. Iodine in tank wastes exists in the inorganic chemical forms iodide and iodate (and possibly also periodate) and also as organic forms such as alkyl iodides. The measurement of organic iodides is very different from the measurement of inorganic forms and is outside the scope of this report. The inorganic chemical forms are chemically separated, in sequence, from the raw tank waste using solvent extractions and several redox reactions, then measured by ICP-MS. The inorganic chemical forms of iodine are chemically reactive, and so is the tank waste. The chemistry must be carefully designed to avoid unintended reactions that could convert one form of iodine to another during the analysis, which would skew the data and report iodine in the wrong chemical form. This iodine analysis, developed at PNNL, is being transferred to the DOE 222-S Laboratory on the Hanford Site, in Washington State, for implementation during waste processing operations.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Unbalanced Parallel I/O: An Often-Neglected Side Effect of Lossy Scientific Data Compression

Lossy compression techniques have demonstrated promising results in significantly reducing the scientific data size while guaranteeing the compression error bounds. However, one important yet often neglected side effect of lossy scientific data compression is its impact on the performance of parallel I/O. Our key observation is that the compressed data size is often highly skewed across processes in lossy scientific compression. To understand this behavior, we conduct extensive experiments where we apply three lossy compressors MGARD, ZFP, and SZ, which are specifically designed and optimized for scientific data, to three real-world scientific applications Gray-Scott simulation, WarpX, and XGC. Our analysis result demonstrates that the size of the compressed data is always skewed even if the original data is evenly decomposed among processes. Such skewness widely exists in different scientific applications using different compressors as long as the information density of the data varies across processes. We then systematically study how this side effect of lossy scientific data compression impacts the performance of parallel I/O. We observe that the skewness in the sizes of the compressed data often leads to I/O imbalance, which can significantly reduce the efficiency of I/O bandwidth utilization if not properly handled. In addition, writing data concurrently to a single shared file through MPI-IO library is more sensitive to the unbalanced I/O loads. Therefore, we believe our research community should pay more attention to the unbalanced parallel I/O caused by lossy scientific data compression.

Wang, Xinying↗

Cost of Fish Exclusion and Passage Technologies for Hydropower

Hydropower represents a reliable source of renewable energy and accounts for approximately 7% of the total electrical generation in the United States. Future expansion of hydropower is likely to be in the form of either smaller new stream development projects or powering existing non-powered dams. For these new projects to be successful, careful analysis of risks, costs, and uncertainty to offset reduced power production as well as ensuring the protection and safe passage of migratory fish to gain public support, will be required. Exclusion and passage are two common approaches to protect fish from entrainment and impingement at hydropower facilities. The thresholds for entrainment risk and requirements for exclusion and passage often differ depending on the species involved, the characteristics of the facility, and the goals of stakeholders. While the costs associated with environmental mitigations represent a large proportion of the total costs required for the licensing of hydropower facilities, little quantitative information is present within the literature regarding the specific costs of fish exclusion and passage. Working with FOA awardee Natel Energy, scientists at Oak Ridge National Laboratory were tasked with assessing the capital construction costs for downstream fish exclusion and passage infrastructure. This report used keyword searches of an existing environmental mitigation cost data set and manual extraction of additional cost data associated with protection, mitigation, and enhancement (PM&E) measures related to positive barrier screening and passage from regulatory licensing documents available in the Federal Energy Regulatory Commission (FERC) eLibrary. This approach yielded a total of 50 PM&E mitigation measures with estimated capital construction costs pertaining to positive barrier screens, 142 pertaining to passage studies, and 26 pertaining to passage-related studies. PM&E measures associated with positive barrier screens represented <10% of the 171 total FERC project dockets available in the data set. These data were highly skewed toward conventional relicensing projects, as <7% were associated with new stream development (NSD) projects. Results from these data indicate highly variable costs associated with fish screening, with flow-normalized costs one to two orders of magnitude higher for screening with the highest exclusion capability (≤0.09 in. spacing) compared with coarser screening (1 to 2 in.). Furthermore, estimated capital costs of passage infrastructure were positively related to the scale of the project based on installed capacity for some, but not all, types of passage. These data provide an initial baseline for estimating exclusion and passage costs for hydropower development and may help developers consider options for more fish-friendly generation technologies, though gaps remain relating to a lack of data, particularly for NSD projects. More data may still be available within the FERC eLibrary, but significant effort will be required to manually identify and extract the data for future analyses.

13 HYDRO ENERGY↗

Regression for Assessing Contaminant Plume Trends

The Department of Energy’s (DOE) Hanford site has significant dissolved-phase groundwater contaminant plumes, for which remediation activities are ongoing. Assessing concentration trends is critical for making decisions regarding remedy implementation or performance. The SOCRATES single-page web application provides a quality-assured framework for data access and consistent data analytics using standard methods. To provide trend information for remedial decisions, statistical analyses based on ordinary least squares (OLS) are already supported by SOCRATES. The presence of censored data, however, can skew the result of an OLS regression. Other methods, such as the Akritas-Theil-Sen (ATS) or Tobit regressions can be more powerful. These two regression methods are used by Hanford site personnel to analyze groundwater contaminant concentration data, which often include nondetects (censored data). The ATS method uses comparisons of observation-pair ranks and the Tobit method uses a maximum likelihood estimation method. The Tobit method can also accommodate covariates, such as river stage or groundwater level fluctuations. Both ATS and Tobit regressions can be performed with the statistics software R, but to incorporate these methods into SOCRATES, their R code was ported into JavaScript. For quality assurance verification, the new code was tested with hexavalent chromium concentration data from the Hanford 100-K West Area and compared to regression analyses that were previously conducted on this data using R.

100-K Area↗

Estimated capital costs of fish exclusion technologies for hydropower facilities

Hydropower is a reliable source of renewable energy, and its future expansion is likely to be in the form of either smaller new stream development (NSD) projects or powering existing non-powered dams. Thresholds for entrainment risk to fish and the requirements for fish exclusion at hydropower facilities often differ depending on the species involved, the characteristics of the facility, and the goals of stakeholders, but little quantitative information is present within the literature regarding the specific costs of fish exclusion measures. Cost data associated with protection, mitigation, and enhancement (PM&E) measures related to positive barrier screening were identified using keyword searches of an existing environmental mitigation cost data set and manual extraction from regulatory licensing documents available in the Federal Energy Regulatory Commission (FERC) eLibrary. This approach yielded a total of 50 p.m.&E mitigation measures with estimated capital construction costs pertaining to positive barrier screens and represented <10% of the 171 total FERC project dockets available in the data set. These data were highly skewed toward conventional relicensing projects, as <7% were associated with NSD projects. Results indicate highly variable costs are associated with fish screening, with flow-normalized costs one to two orders of magnitude higher for screening with the highest exclusion capability (≤0.09 in. spacing) compared with coarser screening (1–2 in.). These data provide an initial baseline for estimating exclusion costs for hydropower development and may help developers consider options for more fish-friendly generation technologies, though gaps remain relating to a lack of data, particularly for NSD projects.

13 HYDRO ENERGY↗

Subfield crop yields and temporal stability in thousands of US Midwest fields

Understanding subfield crop yields and temporal stability is critical to better manage crops. Several algorithms have proposed to study within-field temporal variability but they were mostly limited to few fields. In this study, a large dataset composed of 5520 yield maps from 768 fields provided by farmers was used to investigate the influence of subfield yield distribution skewness on temporal variability. The data are used to test two intuitive algorithms for mapping stability: one based on standard deviation and the second based on pixel ranking and percentiles. The analysis of yield monitor data indicates that yield distribution is asymmetric, and it tends to be negatively skewed (p < 0.05) for all of the four crops analyzed, meaning that low yielding areas are lower in frequency but cover a larger range of low values. The mean yield difference between the pixels classified as high-and-stable and the pixels classified as low-and-stable was 1.04 Mg ha –1 for maize, 0.39 Mg ha –1 for cotton, 0.34 Mg ha –1 for soybean, and 0.59 Mg ha —1 for wheat. The yield of the unstable zones was similar to the pixels classified as low-and-stable by the standard deviation algorithm, whereas the two-way outlier algorithm did not exhibit this bias. Furthermore, the increase in the number years of yield maps available induced a modest but significant increase in the certainty of stability classifications, and the proportion of unstable pixels increased with the precipitation heterogeneity between the years comprising the yield maps.

59 BASIC BIOLOGICAL SCIENCES↗

Comparison of Machine Learning Approaches for Prediction of the Equivalent Alkane Carbon Number for Microemulsions Based on Molecular Properties

The chemical properties of oils are vital in the design of microemulsion systems. The hydrophilic–lipophilic difference equation used to predict microemulsions’ phase behavior expresses the oils’ physiochemical properties as the equivalent alkane carbon number (EACN). The experimental determination of EACN requires knowledge of the temperature dependence of the microemulsion system and the effects of different surfactant concentrations. Thus, the experimental determination is time-intensive and tedious, requiring days to months for proper separations. Furthermore, the experiments require high purity of chemicals because microemulsions are sensitive to impurities. Our work focuses on the quick and reliable predictions of the EACN with machine learning (ML) models. Due to the immaturity of ML chemical predictions, we compare three graph neural networks (GNNs) and a gradient-boosted tree algorithm, known as XGBoost. The GNNs use the molecular structures represented as simplified molecular-input line-entry system (SMILES) codes for the initial input, which allows us to assess whether geometry optimization is necessary for reliable results. The XGBoost model also begins with the SMILES representations of the molecules but uses molecular descriptors instead of geometry optimizations. As a result, the best model tested (crystal graph convolutional neural network with Merck molecular force field-94) has an error of 1.15 EACN units of the true EACN for unknown data with the errors skewed toward zero and an R² score of 0.9

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Nonparametric, data-based kernel interpolation for particle-tracking simulations and kernel density estimation

Traditional interpolation techniques for particle tracking include binning and convolutional formulas that use pre-determined (i.e., closed-form, parameteric) kernels. In many instances, the particles are introduced as point sources in time and space, so the cloud of particles (either in space or time) is a discrete representation of the Green’s function of an underlying PDE. As such, each particle is a sample from the Green’s function; therefore, each particle should be distributed according to the Green’s function. In short, the kernel of a convolutional interpolation of the particle sample “cloud” should be a replica of the cloud itself. This idea gives rise to an iterative method by which the form of the kernel may be discerned in the process of interpolating the Green’s function. When the Green’s function is a density, this method is broadly applicable to interpolating a kernel density estimate based on random data drawn from a single distribution. We formulate and construct the algorithm and demonstrate its ability to perform kernel density estimation of skewed and/or heavy-tailed data including breakthrough curves.

42 ENGINEERING↗

Outlier analyses of the Protein Data Bank archive using a probability-density-ranking approach

Outlier analyses are central to scientific data assessments. Conventional outlier identification methods do not work effectively for Protein Data Bank (PDB) data, which are characterized by heavy skewness and the presence of bounds and/or long tails. We have developed a data-driven nonparametric method to identify outliers in PDB data based on kernel probability density estimation. Unlike conventional outlier analyses based on location and scale, Probability Density Ranking can be used for robust assessments of distance from other observations. Analyzing PDB data from the vantage points of probability and frequency enables proper outlier identification, which is important for quality control during deposition-validation-biocuration of new three-dimensional structure data. Ranking of Probability Density also permits use of Most Probable Range as a robust measure of data dispersion that is more compact than Interquartile Range. The Probability-Density-Ranking approach can be employed to analyze outliers and data-spread on any large data set with continuous distribution.

59 BASIC BIOLOGICAL SCIENCES↗