Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ordering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Polarization and variability of compact sources measured in Planck time-ordered data

This paper introduces a new Planck Catalog of Polarized and Variable Compact Sources (PCCS-PV) comprising 153 sources, the majority of which are extragalactic. The data include both the total flux density and linear polarization measured by Planck with frequency coverage from 30 to 353 GHz, and temporal spacing ranging from days to years. We classify most sources as beamed, extragalactic radio sources; the catalog also includes several radio galaxies, Seyfert galaxies, and Galactic and Magellanic Cloud sources, including H II regions and planetary nebulae. An advanced extraction method applied directly to the multifrequency Planck time-ordered data, rather than the mission sky maps, was developed to allow an assessment of the variability of polarized sources. Our analysis of the time-ordered data from the Planck mission, tod2flux, allowed us to catalog the time-varying emission and polarization properties for these sources at the full range of polarized frequencies employed by Planck, 30-353 GHz. PCCS-PV provides the time-and frequency-dependent, polarized flux densities for all 153 sources. To illustrate some potential applications of the PCCS-PV, we conducted preliminary comparisons of our measurements of selected sources with published data from other astronomical instruments. In summary, we find general agreement between the Planck and the Institut de Radioastronomie Millimétrique (IRAM) polarization measurements as well as with the Metsähovi 37 GHz values at closely similar epochs. Here, these combined measurements also show the value of PCCS-PV results and the PCCS2 catalog for filling in missing spectral (or temporal) coverage and helping to define the spectral energy distributions of extragalactic sources. In turn, these results provide useful clues as to the physical properties of the sources.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Conservative high-order data transfer method on generalized polygonal meshes

A conservative data transfer (remap) between two meshes is an important step of arbitrary Lagrangian-Eulerian (ALE) hydrodynamics simulations. High-order numerical methods for ALE simulations require both high-order (curvilinear) meshes and high-order remap algorithms. Here we develop a conservative and bounds-preserving method for accurate remapping of discrete fields on generalized polygonal meshes with curvilinear edges. The properties of the proposed method are studied theoretically and numerically for various (smooth and non-smooth) mesh deformations and discrete fields that represent smooth and discontinuous functions.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Virtual Log-Structured Storage for High-Performance Streaming

Over the past decade, given the higher number of data sources (e.g., Cloud applications, Internet of things) and critical business demands, Big Data transitioned from batch-oriented to real-time analytics. Stream storage systems, such as Apache Kafka, are well known for their increasing role in real-time Big Data analytics. For scalable stream data ingestion and processing, they logically split a data stream topic into multiple partitions. Stream storage systems keep multiple data stream copies to protect against data loss while implementing a stream partition as a replicated log. This architectural choice enables simplified development while trading cluster size with performance and the number of streams optimally managed. This paper introduces a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and consume in parallel data streams. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability). We implement the virtual log technique in the KerA stream storage system. When comparing with Apache Kafka, KerA improves the cluster ingestion throughput by up to 4x when multiple producers write over hundreds of data streams.

consistent stream ordering↗
DOE OSTI2020

Pilot Study for Determining HVAC Fault Prevalence from Fault Monitoring Data

HVAC faults in U.S. commercial buildings are estimated to waste 0.7 quads of energy annually and many companies have deployed software at scale for fault detection and diagnosis (FDD). Yet the lack of comprehensive published data on fault prevalence makes it difficult for researchers, software developers, and building operators to target the most important types of faults and accelerate the widespread deployment of this technology. A multi-lab effort is underway to quantify the prevalence of commercial building HVAC faults. This paper describes initial analysis conducted on over 2-years of FDD data from 12 campus buildings to inform the methodology and data collection strategy of the full study. The analysis found a large variability in fault prevalence across different buildings, pieces of equipment, and fault types, many duplicate fault alarms, and challenges in validating faults with secondary data sources such as manually entered work order data. This paper also discusses the requirements for a successful full study, which necessitates the acquisition of a large, diverse dataset, from buildings that vary in end-use, climate region, and installed FDD software platform, as well as robust validation data easily linked to the FDD data. It is especially important to perform this analysis on data from multiple FDD providers and building owners to ensure that the results are agnostic to the software methodology used and building-specific settings. Future work will present the results of the full study, which will contain data from several different partner organizations, spanning a large proportion of U.S. climate zones.

Newman, Sarah F.↗
DOE OSTI2023

BEYONDPLANCK II. CMB mapmaking through Gibbs sampling

We present a Gibbs sampling solution to the mapmaking problem for cosmic microwave background (CMB) measurements that builds on existing destriping methodology. Gibbs sampling breaks the computationally heavy destriping problem into two separate steps: noise filtering and map binning. Considered as two separate steps, both are computationally much cheaper than solving the combined problem. This provides a huge performance benefit as compared to traditional methods and it allows us, for the first time, to bring the destriping baseline length to a single sample. Here, we applied the Gibbs procedure to simulated Planck 30 GHz data. We find that gaps in the time-ordered data are handled efficiently by filling them in with simulated noise as part of the Gibbs process. The Gibbs procedure yields a chain of map samples, from which we are able to compute the posterior mean as a best-estimate map. The variation in the chain provides information on the correlated residual noise, without the need to construct a full noise covariance matrix. However, if only a single maximum-likelihood frequency map estimate is required, we find that traditional conjugate gradient solvers converge much faster than a Gibbs sampler in terms of the total number of iterations. The conceptual advantages of the Gibbs sampling approach lies in statistically well-defined error propagation and systematic error correction. This methodology thus forms the conceptual basis for the mapmaking algorithm employed in the BEYONDPLANCK framework, which implements the first end-to-end Bayesian analysis pipeline for CMB observations.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

BeyondPlanck: XII. Cosmological parameter constraints with end-to-end error propagation

We present cosmological parameter constraints estimated using the Bayesian BEYONDPLANCK analysis framework. This method supports seamless end-to-end error propagation from raw time-ordered data onto final cosmological parameters. As a first demonstration of the method, we analyzed time-ordered Planck LFI observations, combined with selected external data (WMAP 33–61 GHz, Planck HFI DR4 353 and 857 GHz, and Haslam 408 MHz) in the form of pixelized maps that are used to break critical astrophysical degeneracies. Overall, all the results are generally in good agreement with previously reported values from Planck 2018 and WMAP, with the largest relative difference for any parameter amounting about 1σ when considering only temperature multipoles between 30 ≤ ℓ ≤ 600. In cases where there are differences, we note that the BEYONDPLANCK results are generally slightly closer to the high-ℓ HFI-dominated Planck 2018 results than previous analyses, suggesting slightly less tension between low and high multipoles. Using low-ℓ polarization information from LFI and WMAP, we find a best-fit value of τ = 0.066 ± 0.013, which is higher than the low value of τ = 0.052 ± 0.008 derived from Planck 2018 and slightly lower than the value of 0.069 ± 0.011 derived from the joint analysis of official LFI and WMAP products. Most importantly, however, we find that the uncertainty derived in the BEYONDPLANCK processing is about 30 % greater than when analyzing the official products, after taking into account the different sky coverage. We argue that this uncertainty is due to a marginalization over a more complete model of instrumental and astrophysical parameters, which results in more reliable and more rigorously defined uncertainties. We find that about 2000 Monte Carlo samples are required to achieve a robust convergence for a low-resolution cosmic microwave background (CMB) covariance matrix with 225 independent modes, and producing these samples takes about eight weeks on a modest computing cluster with 256 cores.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Method and apparatus for maintaining data coherence in a non-uniform compute device

A data processing apparatus includes one or more host processors with first processing units, one or more caches with second processing unit, a non-cache memory having a third processing unit and a reorder buffer operable to maintain data order during execution of a program of instructions. An instruction scheduler routes instructions to the processing units. Data coherence is maintained by control logic that blocks access to data locations in use by a selected processing unit other than the selected processing unit until data associated with the data locations are released from the reorder buffer. Data stored in the cache is written to the memory if it is already in a modified state, otherwise the state is set to the modified state. A memory controller may be used to restrict access to memory locations to be operated on.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Navigation and Meteorological Data from Multiple Sensors on Airborne Platform (NAVMET-AIR) Value-Added Product Report

The Navigation and Meteorological Data from Multiple Sensors on Airborne Platform (NAVMET-AIR) value-added product encompasses the aafnaviwg data set in the ARM Data Center (ADC), known as the IWG file, named for the Inter-Agency Working Group for Airborne Data and Telemetry Systems (IWGADTS). Its purpose is to produce a suite of tools to promote standardization of instrument interface, data format, and data processing. This data set’s contents are navigational and meteorological state variables at 1 Hz. It also contains higher-order data flags created by ARM Aerial Facility (AAF) scientists to aid analysis such as periods when the aircraft is flying level, operating in maneuvers, or flying through cloud. Due to the nature of research flights, the payload of the aircraft consists of many duplicate and overlapping instruments to ensure, in case of instrument failure, there is a backup to record the data. Measurements from multiple instruments are consolidated into a single file. Each variable of this data set is carefully chosen from the onboard instrumentation and quality checked by AAF scientists to create this wholistic and most accurate data set for airborne research. This document is intended to be a hub towards the individual “read me” files produced for each campaign. For reference about what instrument was used for calculations on specific days, the read me files are located in the ARM IOP archive at the addresses listed, or access the IOP (intensive operational period) database at https://www.archive.arm.gov/.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

BEYONDPLANCK XIV. Polarized foreground emission between 30 and 70 GHz

We constrain polarized foreground emission between 30 and 70 GHz with the Planck Low Frequency Instrument (LFI) and WMAP data within the global Bayesian BeyondPlanck framework. We combine for the first time full-resolution Planck LFI time-ordered data with low-resolution WMAP sky maps at 33, 40 and 61 GHz. Spectral parameters are fit with a likelihood defined at the native resolution of each frequency channel. This analysis represents the first implementation of true multi-resolution component separation applied to CMB observations for both amplitude and spectral energy distribution (SED) parameters. For synchrotron emission, we approximate the SED as a power-law in frequency and find that the low signal-to-noise ratio of the current data strongly limits the number of free parameters that may be robustly constrained. We partition the sky into four large disjoint regions (High Latitude; Galactic Spur; Galactic Plane; and Galactic Center), each associated with its own power-law index. We find that the High Latitude region is prior-dominated, while the Galactic Center region is contaminated by residual instrumental systematics. The two remaining regions appear to be signal-dominated, and for these we derive spectral indices of $β$ S Spur = -3.17 ± 0.06 and $β$ S Plane = -3.03 ± 0.07, in good agreement with previous results. For thermal dust emission we assume a modified blackbody model and we fit a single power-law index across the full sky. We find $β$ d = 1.64 ± 0.03, which is slightly steeper than reported from Planck HFI data, but still statistically consistent at the 2σ confidence level.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

pyDRESCALk

Modern data scientists are tasked to analyze ever-growing data sets with increasingly complex relationships. Tensor decompositions have come to play a central role in identifying underlying latent structures in higher-order data. The problem of fitting tensor models to different distributions is complicated by the combinations of size, dimensionality, and sparsity present in real world data. The situation demands efficient algorithms designed for shared-memory and distributed systems. This work will present new research that tackles these challenges on several different fronts, leveraging optimizations in numerical algorithms and sparse tensor representations in heterogeneous high performance computing environments.

Bhattarai, Manish↗
DOE OSTI2025

The three-point form factor of Tr ϕ 3 to six loops

We study the three-point form factor of the length-three half-BPS operator (Tr ϕ 3 ) in planar $\mathcal{N}$ = 4 Super-Yang-Mills theory, using analyticity and integrability methods. We find that the functions describing the form factor in perturbation theory live in the same restrictive space of multiple polylogarithms as the one describing the form factor of the stress-tensor operator (Tr ϕ 2 ). Furthermore, we find that the leading-order data in the collinear limit provided by the form factor operator product expansion (FFOPE) is enough to fix the form factor uniquely, at least through six loops. We perform various tests of our results using the subleading FFOPE corrections. We also analyze the form factor in the Regge limit where two Mandelstam invariants are large; we obtain a compact representation for the form factor in this limit which is valid to all orders in the coupling.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

The combination of ever-growing scientific datasets and distributed workflow complexity creates I/O performance bottlenecks due to data volume, velocity, and variety. Although the increasing use of descriptive data formats (e.g., HDF5, netCDF) helps organize these datasets, it also creates obscure bottlenecks due to the need to translate high level operations into file addresses and then into low-level I/O operations. To address this challenge, we introduce DaYu, a method and toolset for analyzing (a) semantic relationships between logical datasets and file addresses, (b) how dataset operations translate into I/O, and (c) the combination across entire workflows. DaYu's analysis and visualization enables identification of critical bottlenecks and reasoning about remediation. We describe our methodology and propose optimization guidelines. Evaluation on scientific workflows demonstrates up to 3.7x performance improvements in I/O time for obscure bottlenecks. The time and storage overhead for DaYu's time-ordered data is typically under 0.2% of runtime and 0.25% of data volume, respectively.

Tang, Meng↗
DOE OSTI2022

Improved uranium isotopic ratio determinations for the liquid sampling-atmospheric pressure glow discharge orbitrap mass spectrometer by use of moving average processing

The liquid sampling-atmospheric pressure glow discharge (LS-APGD) is a versatile combined atomic and molecular (CAM) ionization source capable of ionizing elemental species, small polar compounds, low-polarity polycyclic aromatic hydrocarbons, and proteins. While the LS-APGD has been proven capable of determining the 235U/238U isotope ratio in enriched and natural uranium, recent efforts have strived to reduce the analysis time of individual measurements by employing higher-order data processing techniques, specifically, moving average methods. Additionally, the use of a moving average (of data windows of various widths) improves the precision of the ratio measurements and reduces the number of scans needed to be collected to generate high-precision results. Additionally, use of the moving average minimizes the quantity of sample that needs to be analyzed while reducing the measurement time. These reductions have allowed isotope ratio (IR) determinations to be made using injections instead of direct infusion of the analyte. For example, employing a 40-point window width in the moving average to direct infusion data reduced the percent relative standard deviation (% RSD) of the 235U/238U values from 3.4% to 0.5%. For the injection of a 100 μL aliquot, the % RSD was reduced from 5.3% to 0.7%.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2023

BEYONDPLANCK III. Commander3

We describe the computational infrastructure for end-to-end Bayesian cosmic microwave background (CMB) analysis implemented by the BeyondPlanck Collaboration. The code is called Commander3. It provides a statistically consistent framework for global analysis of CMB and microwave observations and may be useful for a wide range of legacy, current, and future experiments. The paper has three main goals. Firstly, we provide a high-level overview of the existing code base, aiming to guide readers who wish to extend and adapt the code according to their own needs or re-implement it from scratch in a different programming language. Secondly, we discuss some critical computational challenges that arise within any global CMB analysis framework, for instance in-memory compression of time-ordered data, fast Fourier transform optimization, and parallelization and load-balancing. Thirdly, we quantify the CPU and RAM requirements for the current BEYONDPLANCK analysis, finding that a total of 1.5 TB of RAM is required for efficient analysis and that the total cost of a full Gibbs sample for LFI is 170 CPU-hrs, including both low-level processing and high-level component separation, which is well within the capabilities of current low-cost computing facilities. The existing code base is made publicly available under a GNU General Public Library (GPL) license.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

BEYONDPLANCK VI. Noise characterization and modeling

We present a Bayesian method for estimating instrumental noise parameters and propagating noise uncertainties within the global BeyondPlanck Gibbs sampling framework, and apply this to Planck Low Frequency Instrument (LFI) time-ordered data. Following previous literature, we initially adopt a 1/ f model for the noise power spectral density (PSD), but find the need for an additional lognormal component in the noise model for the 30 and 44 GHz bands. We implement an optimal Wiener-filter (or constrained realization) gap-filling procedure to account for masked data. We then use this procedure to both estimate the gapless correlated noise in the time-domain, n corr , and to sample the noise PSD parameters, ξ n = {σ 0 , f knee , α, A p }. In contrast to previous Planck analyses, we assume piecewise stationary noise only within each pointing period (PID), not throughout the full mission, but we adopt the LFI Data Processing Center (DPC) results as priors on α and f knee . On average, we find best-fit correlated noise parameters that are mostly consistent with previous results, with a few notable exceptions. However, a detailed inspection of the time-dependent results reveals many important findings. First and foremost, we find strong evidence for statistically significant temporal variations in all noise PSD parameters, many of which are directly correlated with satellite housekeeping data. Second, while the simple 1/ f model appears to be an excellent fit for the LFI 70 GHz channel, there is evidence for additional correlated noise not described by a 1/ f model in the 30 and 44 GHz channels, including within the primary science frequency range of 0.1–1 Hz. In general, most 30 and 44 GHz channels exhibit deviations from 1/ f at the 2–3σ level in each one hour pointing period, motivating the addition of the lognormal noise component for these bands. For some periods of time, we also find evidence of strong common mode noise fluctuations across the entire focal plane. Overall, we conclude that a simple 1/ f profile is not adequate to fully characterize the Planck LFI noise, even when fitted hour-by-hour, and a more general model is required. These findings have important implications for large-scale CMB polarization reconstruction with the Planck LFI data, and the current work is a first attempt at understanding and mitigating these issues.

79 ASTRONOMY AND ASTROPHYSICS↗