Search NASA⌕ Search

SEARCH · Search NASA

Results for “data distribution”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Mind the gap: the power of combining photometric surveys with intensity mapping

The long wavelength modes lost to bright foregrounds in the interferometric 21-cm surveys can partially be recovered using a forward modeling approach that exploits the non-linear coupling between small and large scales induced by gravitational evolution. In this work, we build upon this approach by considering how adding external galaxy distribution data can help to fill in these modes. We consider supplementing the 21-cm data at two different redshifts with a spectroscopic sample (good radial resolution but low number density) loosely modeled on DESI-ELG at z=1 and a photometric sample (high number density but poor radial resolution) similar to LSST sample at z=1 and z=4 respectively. We find that both the galaxy samples are able to reconstruct the largest modes better than only using 21-cm data, with the spectroscopic sample performing significantly better than the photometric sample despite much lower number density. Here, we demonstrate the synergies between surveys by showing that the primordial initial density field is reconstructed better with the combination of surveys than using either of them individually. Methodologically, we also explore the importance of smoothing the density field when using bias models to forward model these tracers for reconstruction.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Measurement of the small-scale 3D Lyman- α forest power spectrum

Small-scale correlations measured in the Lyman-α (Lyα) forest encode information about the intergalactic medium and the primordial matter power spectrum. In this article, we present and implement a simple method to measure the 3-dimensional power spectrum, P 3D , of the Lyα forest at wavenumbers k corresponding to small, ~ Mpc scales. In order to estimate P 3D from sparsely and unevenly distributed data samples, we rely on averaging 1-dimensional Fourier Transforms, as previously carried out to estimate the 1-dimensional power spectrum of the Lyα forest, P 1D . Further, this methodology exhibits a very low computational cost. We confirm the validity of this approach through its application to Nyx cosmological hydrodynamical simulations. Subsequently, we apply our method to the eBOSS DR16 Lyα forest sample, providing as a proof of principle, a first P 3D measurement averaged over two redshift bins z = 2.2 and z = 2.4. This work highlights the potential for forthcoming P 3D measurements, from upcoming large spectroscopic surveys, to untangle degeneracies in the cosmological interpretation of P 1D .

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Urbanization drives convergence in soil profile texture and carbon content

Urban development has driven extensive modification of the global landscape. This shift in land use and land cover alters ecological functioning, and thereby affects sustainable management agendas. Urbanization fundamentally reshapes the soils that underlay landscapes, and throughout the soil profile, extends impacts of urbanization far below the landscape surface. The impacts of urbanization on deeper soils that are beyond the reach of regular land management are largely unknown, and validation of general theories of convergent ecosystem properties are thwarted by a dearth of both level of measurement effort and the substantial heterogeneity in soils and urban landscapes. Here, we examined two soil properties with strong links to ecological functioning—carbon and mineral-fraction particle size—measured in urban soils, and compared them to their pre-urbanization conditions across a continental gradient encompassing global soil diversity. We hypothesized that urbanization drove convergence of soils properties from heterogeneous pre-urban conditions towards homogeneous urban conditions. Based on our observations, we confirm the hypothesis. Both soil carbon and particle size converged toward an intermediate value in the full data distribution, from pre-urban to urban conditions. These outcomes in urban soils were observed to uniformly be fine textured soils with overall lower carbon content. Although these properties are desirable for supporting urban infrastructure (e.g. buildings, pipes), they constrain the potential to render ecosystem services. Since soil profile texture and carbon content were convergent and observed across 11 cities, we suggest that these property profiles can be used as a universal urban soil profile to: 1) provide a clear prediction for how urbanization will shift soil properties from pre-urban conditions, 2) facilitate the adoption of commonly-accepted soil profiles for process models, and 3) offer a reference point to test against urban management strategies and how they impact soil resources.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Deep kernel methods learn better: from cards to process optimization

Abstract The ability of deep learning methods to perform classification and regression tasks relies heavily on their capacity to uncover manifolds in high-dimensional data spaces and project them into low-dimensional representation spaces. In this study, we investigate the structure and character of the manifolds generated by classical variational autoencoder (VAE) approaches and deep kernel learning (DKL). In the former case, the structure of the latent space is determined by the properties of the input data alone, while in the latter, the latent manifold forms as a result of an active learning process that balances the data distribution and target functionalities. We show that DKL with active learning can produce a more compact and smooth latent space which is more conducive to optimization compared to previously reported methods, such as the VAE. We demonstrate this behavior using a simple cards dataset and extend it to the optimization of domain-generated trajectories in physical systems. Our findings suggest that latent manifolds constructed through active learning have a more beneficial structure for optimization problems, especially in feature-rich target-poor scenarios that are common in domain sciences, such as materials synthesis, energy storage, and molecular discovery. The Jupyter Notebooks that encapsulate the complete analysis accompany the article.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Accurate field-level weak lensing inference for precision cosmology

We present miko, a catalog-to-cosmology pipeline for general flat-sky field-level inference, which provides access to cosmological information beyond the two-point statistics. In the context of weak lensing, we identify several new field-level analysis systematics (such as aliasing, Fourier mode-coupling, and density-induced shape noise), quantify their impact on cosmological constraints, and correct the biases to a percent level. Next, we find that model misspecification can lead to both absolute bias and incorrect uncertainty quantification for the inferred cosmological parameters in realistic simulations. The Gaussian map prior infers unbiased cosmological parameters, regardless of the true data distribution, but it yields overconfident uncertainties. The log-normal map prior quantifies the uncertainties accurately, although it requires careful calibration of the shift parameters for unbiased cosmological parameters. Here, we demonstrate systematics control down to the 2% level for both models, making them suitable for ongoing weak lensing surveys.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Contrastive learning for robust representations of neutrino data

In neutrino physics, analyses often depend on large simulated datasets, making it essential for models to generalize effectively to real-world detector data. Contrastive learning, a well-established technique in deep learning, offers a promising solution to this challenge. By applying controlled data augmentations to simulated data, contrastive learning enables the extraction of robust and transferable features. This improves the ability of models trained on simulations to adapt to real experimental data distributions. In this paper, we investigate the application of contrastive learning methods in the context of neutrino physics. Through a combination of empirical evaluations and theoretical insights, we demonstrate how contrastive learning enhances model performance and adaptability. Additionally, we compare it to other domain adaptation techniques, highlighting the unique advantages of contrastive learning for this field. Published by the American Physical Society 2025

Wilkinson, Alex (ORCID:0000000253404506)↗
DOE OSTI2022

A Scalable Pipeline for Gigapixel Whole Slide Imaging Analysis on Leadership Class HPC Systems

Whole Slide Imaging (WSI) captures microscopic details of a patient's histopathological features at multiple res-olutions organized across different levels. Images produced by WSI are gigapixel-sized, and saving a single image in memory requires a few gigabytes which is scarce since a complicated model occupies tens of gigabytes. Performing a simple met-ric operation on these large images is also expensive. High-performance computing (HPC) can help us quickly analyze such large images using distributed training of complex deep learning models. One popular approach in analyzing these images is to divide a WSI image into smaller tiles (patches) and then train a simpler model with these reduced-sized but large numbers of patches. However, we need to solve three pre-processing challenges efficiently for pursuing this patch-based approach. 1) Creating small patches from a high-resolution image can result in a high number (hundreds of thousands per image) of patches. Storing and processing these images can be challenging due to a large number of I/O and arithmetic operations. To reduce I/Oand memory accesses, an optimal balance between the size and number of patches must exist to reduce I/O and memory accesses. 2) WSI images may have tiny annotated regions for cancer tissue and a significant portion with normal and fatty tissues; correct patch sampling should avoid dataset imbalance. 3) storing and retrieving many patches to and from disk storage might incur I/O latency while training a deep learning model. An efficient distributed data loader should reduce I/O latency during the training and inference steps. This paper explores these three challenges and provides empirical and algorithmic solutions deployed on the Summit supercomputer hosted at the Oak Ridge Leadership Computing Facility.

Dash, Sajal↗
DOE OSTI2021

Spectral Effects in Albedo and Rearside Irradiance Measurement for Bifacial Performance Estimation

We investigate the impact of spectral dependence of ground surface reflectivity on albedo and rearside irradiance measurements necessary for bifacial photovoltaic (PV) module performance estimation and monitoring. Because PV modules are spectrally selective, albedo and irradiance measurements performed with common irradiance sensors may require spectral mismatch corrections when used for performance prediction. We investigate via simulation the differences in spectrally responsive albedo measured with thermopile pyranometers and crystalline silicon PV reference cells in comparison to a typical crystalline-silicon bifacial PV module. Simulations are performed for nine different representative ground surface materials using simulated solar spectra together with spectral reflectivity data distributed with the SMARTS simulation software. For the materials considered, the results show that albedo spectral mismatch relative to the bifacial module is distributed over a range of ±9.2% for thermopile pyranometers versus only ±3.7% for a typical PV reference cell. We consider the impact of this spectrally-responsive albedo mismatch on bifacial PV module rearside irradiance measurements. Using synthesized rearside spectral irradiance distributions, we find that for the nine different ground surface materials the predicted rearside irradiance measurement deviates from the effective irradiance observed by the PV module by on the order of 16.5 W/m 2 for the pyranometer and 3.6 W/m 2 for the PV reference cell. We discuss the implications for bifacial albedo and irradiance measurement.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2025

FedEFsz: Fair Cross-Silo Federated Learning System With Error-Bounded Lossy Compression

Cross-Silo federated learning systems have been identified as an efficient approach to scaling DNN training across geographically-distributed data silos to preserve the privacy of the training data. Communication efficiency and fairness are two major issues that need to be both satisfied when federated learning systems are deployed in practice. Simultaneously guaranteeing both of them, however, is exceptionally difficult because simply combining communication reduction and fairness optimization approaches often causes non-converged training or drastic accuracy degradation. Here, to bridge this gap, we propose FedEFsz. On the one hand, it integrates the state-of-the-art error-bounded lossy compressor SZ3 into cross-silo federated learning systems to significantly reduce communication traffic during the training. On the other hand, it achieves a high fairness (i.e., rather consistent model accuracy and performance across different clients) through a carefully designed heuristic algorithm that can tune the error-bound of SZ3 for different clients during the training. Extensive experimental results based on a GPU cluster with 65 GPU cards show that FedEFsz improves the fairness across different benchmarks by up to 60.88% and meanwhile reduces the communication traffic by up to 315×.

Cross-Silo Federated Learning Systems↗
DOE OSTI2026

DP-TwoLevel: two-stage gradient subspace learning for differentially private federated learning

Federated learning (FL) enables collaborative model training across distributed data sources without sharing raw data, but faces fundamental challenges in communication efficiency and privacy. Differentially private (DP) training mitigates information leakage but introduces noise that degrades model performance, especially in high-dimensional settings. We propose DP-TwoLevel, a hierarchical gradient projection method that improves utility under fixed DP constraints by exploiting low-dimensional structure in model updates. Our approach learns a two-level PCA-based representation of gradients and applies DP noise in a reduced-dimensional subspace, thereby lowering the effective noise magnitude while preserving dominant signal components. We evaluate the method across three datasets (MNIST, Fashion-MNIST, CIFAR-10) and three privacy regimes (ϵ∈0.5, 1.0, 2.0). Across nine experimental settings, DP-TwoLevel consistently outperforms DP-FedAvg, achieving an average accuracy improvement of 9.44%, with larger gains observed in lower ϵ(higher-noise) regimes (up to +22.31%). We further analyze scalability across models ranging from 100K to 1.49M parameters and identify a variance-based success criterion: performance remains strong when the projection preserves more than 75% of gradient variance, degrades in a marginal regime (65–75%), and fails below this threshold. Our results demonstrate that structure-aware dimensionality reduction can significantly improve the privacy–utility tradeoff in FL without modifying formal privacy guarantees. We also provide empirical evidence of scaling limitations for global projections and motivate per-layer extensions for larger models.

Kotevska, Olivera [ORNL] (ORCID:0000000316772243)↗
DOE OSTI2021

Measurements of multiplicity fluctuations of identified hadrons in inelastic proton–proton interactions at the CERN Super Proton Synchrotron

Measurements of multiplicity fluctuations of identified hadrons produced in inelastic p+p interactions at 31, 40, 80, and 158 $\text {Ge}\text {V}/c$ beam momentum are presented. Three different measures of multiplicity fluctuations are used: the scaled variance $\omega $ and strongly intensive measures $\Sigma $ and $\Delta $. These fluctuation measures involve second and first moments of joint multiplicity distributions. Data analysis is preformed using the Identity method which corrects for incomplete particle identification. Strongly intensive quantities are calculated in order to allow for a direct comparison to corresponding results on nucleus–nucleus collisions. The results for different hadron types are shown as a function of collision energy. A comparison with predictions of string-resonance Monte-Carlo models: Epos, Smash and Venus, is also presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Software and computing for Run 3 of the ATLAS experiment at the LHC

The ATLAS experiment has developed extensive software and distributed computing systems for Run 3 of the LHC. These systems are described in detail, including software infrastructure and workflows, distributed data and workload management, database infrastructure, and validation. The use of these systems to prepare the data for physics analysis and assess its quality are described, along with the software tools used for data analysis itself. An outlook for the development of these projects towards Run 4 is also provided.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

ARIC 2019 workshop report: The 2nd ACM SIGSPATIAL International Workshop on Advances in Resilient and Intelligent Cities: Chicago, IL, USA November 5, 2019

The advancements in sensor technology and ubiquity of connected devices has enabled the generation of large volume of disparate, dynamic and geographically distributed data both by scientific communities and citizens. With astonishing technological innovations and convergence, there have been major changes in peoples daily activities and social interaction. The socio-technological innovations motivate the concept of smart and connected cities. A smart city, however, is subjected to the same challenges as a conventional city, such as environmental damages, hazard impacts, access to services and resources, due to continuous population and economic growth. Therefore, it is imperative to improve our understanding of Resilient and Intelligent Cities in order to leverage technologies and artificial intelligence to tackle the challenges cities face, which range from climate change, public health, traffic congestion, economic growth, to digital divide, social equity, political movements, and cultural conflicts, among others.

Kar, Bandana↗
DOE OSTI2022

Autodataman

The autodataman tool is intended to be a very lightweight climate data database and data distribution engine.

Ordonez, AnaC↗
DOE OSTI2024

MLUQ (Uncertainty quantification for ML closure models) [SWR-24-36]

Data-based closure models are increasingly being used to replace physic-based closure models because of their flexibility and the growing availability of data. However, closure models are subject to uncertainty because of lack of data in parts of the input space (epistemic uncertainty) or because of noise in the data (aleatoric uncertainty). This software contains a toolbox for training bayesian neural nets that can estimate both uncertainties. A specific treatment is provided to ensure accuracy outside of the data distribution. A set of tools are provided to reduce the dimensionality of the uncertain parameter space, thereby enabling fast uncertainty propagation.

Hassanaly, Malik↗
DOE OSTI2020

GridOPTICS/GridPACK

GridPACK is a software framework consisting of a set of modules designed to simplify the development of programs that model the power grid and run on parallel, high performance computing platforms. It also contains several fully developed applications, including powerflow, dynamic simulation, state estimation, Kalman filter analysis (dynamic state estimation), contingency analysis and real time path rating. These applications can be used either standalone or as components in more complicated workflows that combine several different types of application together. The framework modules are available as a combination of libraries and software templates and consist of components for setting up and distributing power grid networks, support for modeling the behavior of individual buses and branches in the network, converting the network models to the corresponding algebraic equations, and parallel routines for manipulating and solving large algebraic systems. The framework also contains a module for distributing tasks evenly amongst computing resources, even if individual tasks vary widely in their execution times. Additional modules support input and output, basic statistical analysis of contingency based calculations, distributed data structures, as well as basic profiling and error management.

Palmer, Bruce↗
DOE OSTI2026

DDCP framework

DDCP protocol software 1.0 This repository contains the C++ implementation of version 1.x of the Distributed Data Communications Protocol (DDCP). DDCP provides request/reply, feature discovery, data transfer, control, interrupt, and transaction support for communicating with accelerator instrumentation over UDP. The standard server port is 65000. The framework is a source dependency for services that communicate directly with DDCP hardware. It is not a deployable service by itself.

Joshi, Shreya [Fermi National Accelerator Laborato↗
DOE OSTI2021

The Atmospheric Carbon and Transport (ACT)-America Mission

The Atmospheric Carbon and Transport (ACT)-America NASA Earth Venture Suborbital Mission set out to improve regional atmospheric greenhouse gas (GHG) inversions by exploring the intersection of the strong GHG fluxes and vigorous atmospheric transport that occurs within the midlatitudes. In this study, two research aircraft instrumented with remote and in situ sensors to measure GHG mole fractions, associated trace gases, and atmospheric state variables collected 1,140.7 flight hours of research data, distributed across 305 individual aircraft sorties, coordinated within 121 research flight days, and spanning five 6-week seasonal flight campaigns in the central and eastern United States. Flights sampled 31 synoptic sequences, including fair-weather and frontal conditions, at altitudes ranging from the atmospheric boundary layer to the upper free troposphere. The observations were complemented with global and regional GHG flux and transport model ensembles. We found that midlatitude weather systems contain large spatial gradients in GHG mole fractions, in patterns that were consistent as a function of season and altitude. We attribute these patterns to a combination of regional terrestrial fluxes and inflow from the continental boundaries. These observations, when segregated according to altitude and air mass, provide a variety of quantitative insights into the realism of regional CO 2 and CH 4 fluxes and atmospheric GHG transport realizations. The ACT-America dataset and ensemble modeling methods provide benchmarks for the development of atmospheric inversion systems. As global and regional atmospheric inversions incorporate ACT-America’s findings and methods, we anticipate these systems will produce increasingly accurate and precise subcontinental GHG flux estimates.

54 ENVIRONMENTAL SCIENCES↗