How to Design a Quantum Streaming Algorithm Without Knowing Anything About Quantum Computing
Proceedings article for the SIAM Symposium on Simplicity in Algorithms (SOSA25).
Kallaugher, John Michael Goddard [Sandia National ↗
SEARCH · Search NASA
Results for “streaming algorithms”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Fast shared-memory streaming multilevel graph partitioning
In this report we show that a fast parallel graph partitioner can benefit many applications by reducing data transfers. The online methods for partitioning graphs have to be fast and they often rely on simple one-pass streaming algorithms, while the offline methods for partitioning graphs contain more involved algorithms and the most successful methods in this category belong to the multilevel approaches. In this work, we assess the feasibility of using streaming graph partitioning algorithms within the multilevel framework. Our end goal is to come up with a fast parallel offline multilevel partitioner that can produce competitive cutsize quality. We rely on a simple but fast and flexible streaming algorithm throughout the entire multilevel framework. This streaming algorithm serves multiple purposes in the partitioning process: a clustering algorithm in the coarsening, an effective algorithm for the initial partitioning, and a fast refinement algorithm in the uncoarsening. Its simple nature also lends itself easily for parallelization. The experiments on various graphs show that our approach is on the average up to 5.1x faster than the multi-threaded MeTiS, which comes at the expense of only 2x worse cutsize.
97 MATHEMATICS AND COMPUTING↗
Majorize-Minimize Algorithms for Streaming Generalized Canonical Polyadic Tensor Decompositions.
Abstract not provided.
Gilman, Kyle↗
Discovering hydrothermalism from Afar: In Situ methane instrumentation and change-point detection for decision-making
Seafloor hydrothermalism plays a critical role in fundamental interactions between geochemical and biological processes in the deep ocean. A significant number of hydrothermal vents are hypothesized to exist, but many of these remain undiscovered due in part to the difficulty of detecting hydrothermalism using standard sensors on rosettes towed in the water column or robotic platforms performing surveys. Here, we use in situ methane sensors to complement standard sensing technology for hydrothermalism discovery and compare sensors on a towed rosette and an autonomous underwater vehicle (AUV) during a 17 km long transect in the Northern Guaymas Basin in the Gulf of California. This transect spatially intersected with a known hydrothermally active venting site. These data show that methane signalled possible hydrothermal-activity 1.5–3 km laterally (100–150 m vertically) from a known vent. Methane as a signal for hydrothermalism performed similarly to standard turbidity sensors (plume detection 2.2–3.3 km from reference source), and more sensitively and clearly than temperature, salinity, and oxygen instruments which readily respond to physical mixing in background seawater. We additionally introduce change-point detection algorithms—streaming cross-correlation and regime identification—as a means of real-time hydrothermalism discovery and discuss related data supervision technologies that could be used in planning, executing, and monitoring explorative surveys for hydrothermalism.
58 GEOSCIENCES↗
Evaluation & Development of Algorithms & Techniques for Streaming Detector Readout
The advancement in microelectronics capabilities, computing, and data science in the last decade has been remarkable. With the start of the 12 GeV Science Program at Jefferson Lab (JLab) and the conceptual detector design for the upcoming Ion Collider (EIC) in mind, we have evaluated and developed the possibility of evolving and improving the existing nuclear science research workflow based on these advances. Specifically, we have prototyped components of streaming readout.
43 PARTICLE ACCELERATORS↗
Streaming Matching and Edge Cover in Practice
Graph algorithms with polynomial space and time requirements often become infeasible for massive graphs with billions of edges or more. State-of-the-art approaches therefore employ approximate serial, parallel, and distributed algorithms to tackle these challenges. However, such approaches require storing the entire graph in memory and thus need access to costly computing resources such as clusters and supercomputers. In this paper, we present practical streaming approaches for solving massive graph problems using limited memory for two prototypical graph problems: maximum weighted matching and minimum weighted edge cover. For matching, we conduct a thorough computational study on two of the semi-streaming algorithms including a recent breakthrough result that achieves a $1/(2+\varepsilon)$-approximation of the weight while using $O( n \log W /\epsilon)$ memory (here $n$ is the number of vertices and $W$ is the maximum edge weight), designed by Paz and Schwartzman [SODA, 2017]. Empirically, we show that the semi-streaming algorithms produce matchings whose weight is close to the best $1/2$-approximate offline algorithm while requiring less time and an order-of-magnitude less memory. For minimum weighted edge cover, we develop three novel semi-streaming algorithms. Two of these algorithms require a single pass through the input graph, require $O(n \log n)$ memory, and provide a 2-approximation guarantee on the objective. We also leverage a relationship between approximate maximum weighted matching and approximate minimum weighted edge cover to develop a two-pass $3/2+\epsilon$-approximate algorithm with the memory requirement of Paz and Schwartzman's semi-streaming matching algorithm. These streaming approaches are compared against the state-of-the-art 3/2-approximate offline algorithm. The semi-streaming matching and the novel edge cover algorithms proposed in this paper can process graphs with several billions of edges in under 30 minutes using 6 GB of memory, which is at least an order of magnitude improvement from the offline (non-streaming) algorithms. For the largest graph, the best alternative offline parallel approximation algorithm (GPA+ROMA) could not finish in three hours even while employing hundreds of processors and 1 TB of memory. We also demonstrate an application of the semi-streaming algorithm by computing a matching using linearly bounded memory on item intersection graphs derived from three machine learning datasets, whereas the existing offline algorithms could not complete on one of these datasets since their memory requirements exceeded 1TB.
Ferdous, S M.↗
Streaming Compression of Scientific Data via Weak-SINDy
Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.
97 MATHEMATICS AND COMPUTING↗
Via Machinae 2.0 : Full-sky, model-agnostic search for stellar streams in Gaia DR2
ABSTRACT We present an update to Via Machinae, an automated stellar stream-finding algorithm based on the deep learning anomaly detector ANODE. Via Machinae identifies stellar streams within Gaia, using only angular positions, proper motions, and photometry, without reference to a model of the Milky Way potential for orbit integration or stellar distances. This new version, Via Machinae 2.0, includes many improvements and refinements to nearly every step of the algorithm, that altogether result in more robust and visually distinct stream candidates than our original formulation. In this work, we also provide a quantitative estimate of the false positive rate of Via Machinae 2.0 by applying it to a simulated Gaia-mock catalogue based on galaxia, a smooth model of the Milky Way that does not contain substructure or stellar streams. Finally, we perform the first full-sky search for stellar streams with Via Machinae 2.0, identifying 102 streams at high significance within the Gaia Data Release 2, of which only 10 have been previously identified. While follow-up observations for further confirmation are required, taking into account the false positive rate presented in this work, we expect approximately 90 of these stream candidates to correspond to real stellar structures.
Astronomy & Astrophysics↗
Evidence of a Dwarf Galaxy Stream Populating the Inner Milky Way Halo
Stellar streams produced from dwarf galaxies provide direct evidence of the hierarchical formation of the Milky Way. Here, we present the first comprehensive study of the LMS-1 stellar stream, that we detect by searching for wide streams in the Gaia EDR3 data set using the STREAMFINDER algorithm. This stream was recently discovered by Yuan et al. We detect LMS-1 as a 60° long stream to the north of the galactic bulge, at a distance of ~20 kpc from the Sun, together with additional components that suggest that the overall stream is completely wrapped around the inner Galaxy. Using spectroscopic measurements from the Large Sky Area Multi-Object Fiber Spectroscopic Telescope, the Sloan Digital Sky Survey, and the Apache Point Observatory Galactic Evolution Experiment, we infer that the stream is very metal-poor (<[Fe/H]> = –2.1) with a significant metallicity dispersion (σ [Fe/H] = 0.4), and it possesses a large radial velocity dispersion (σ v = 20 ± 4 km s–1 ). These estimates together imply that LMS-1 is a dwarf galaxy stream. The orbit of LMS-1 is close to polar, with an inclination of 75° to the galactic plane. Both the orbit and metallicity of LMS-1 are remarkably similar to the globular clusters NGC 5053, NGC 5024, and the stellar stream Indus. As a result, these findings make LMS-1 an important contributor to the stellar population of the inner Milky Way halo.
79 ASTRONOMY AND ASTROPHYSICS↗
Real Time implementation of Artificial Intelligence compression algorithm for High-Speed Streaming Readout signals
The new generation of high-energy physics experiments plans to acquire data in streaming mode. With this approach, it is possible to access the information of the whole detector (organized in time slices) for optimal and lossless triggering of data acquisitions. With this approach, data rates, especially in large detectors, are often very high, and the network is likely to be the bottleneck for the entire Streaming Read Out system. The aim of this work is to study the implementation of a lossy compression algorithm based on Artificial Intelligence: an Autoencoder. With Machine Learning it is possible to achieve a high compression ratio and fast inference time with only a small degradation of the signals, almost negligible for the specific application. This work explores different configurations of the Autoencoder and the implementation on different hardware. Different Autoencoder configurations are explored to find the best trade-off between compression ratio and reconstruction loss, both for signals and energy spectrum. Different hardware implementations are also explored to find the best platform to achieve real-time performance for the specific application.
Rossi, Fabio (ORCID:0009000385713885)↗
GreediRIS: Scalable influence maximization using distributed streaming maximum cover
Not Available
97 MATHEMATICS AND COMPUTING↗
Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN
Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.
Anomaly detection↗
Machine learning on FPGA for event selection
Real-time data processing is a frontier field in experimental particle physics. The application of FPGAs at the trigger level is used by many current and planned experiments (CMS, LHCb, Belle2, PANDA). Usually they use conventional processing algorithms. LHCb has implemented Machine Learning (ML) elements for real-time data processing with a triggered readout system that runs most of the ML algorithms on a computer farm. The work described in this article aims to test the ML-FPGA algorithms for streaming data acquisition. Herein, there are many experiments working in this area and they have a lot in common, but there are many specific solutions for detector and accelerator parameters that are worth exploring further. This report describes the purpose of the work and progress in evaluating the ML-FPGA application.
47 OTHER INSTRUMENTATION↗
Refining HPCToolkit for application performance analysis at exascale
As part of the US Department of Energy’s Exascale Computing Project (ECP), Rice University has been refining its HPCToolkit performance tools to better support measurement and analysis of applications executing on exascale supercomputers. To efficiently collect performance measurements of GPU-accelerated applications, HPCToolkit employs novel non-blocking data structures to communicate performance measurements between tool threads and application threads. To attribute performance information in detail to source lines, loop nests, and inlined call chains, HPCToolkit performs parallel analysis of large CPU and GPU binaries involved in the execution of an exascale application to rapidly recover mappings between machine instructions and source code. To analyze terabytes of performance measurements gathered during executions at exascale, HPCToolkit employs distributed-memory parallelism, multithreading, sparse data structures, and out-of-core streaming analysis algorithms. To support interactive exploration of profiles up to terabytes in size, HPCToolkit’s hpcviewer graphical user interface uses out-of-core methods to visualize performance data. The result of these efforts is that HPCToolkit now supports collection, analysis, and presentation of profiles and traces of GPU-accelerated applications at exascale. These improvements have enabled HPCToolkit to efficiently measure, analyze and explore terabytes of performance data for executions using as many as 64K MPI ranks and 64K GPU tiles on ORNL’s Frontier supercomputer. HPCToolkit’s support for measurement and analysis of GPU-accelerated applications has been employed to study a collection of open-science applications developed as part of ECP. This paper reports on these experiences, which provided insight into opportunities for tuning applications, strengths and weaknesses of HPCToolkit itself, as well as unexpected behaviors in executions at exascale.
Adhianto, Laksono↗
Scheduling and Control of Flexible Building Loads for Grid Services Based on a Virtual Battery Model
This paper presents a framework for modeling, scheduling, and controlling residential thermostatically controlled loads (TCLs) to provide multiple grid services, such as energy shifting, peak load reduction, and ancillary services. A modeling method is proposed to characterize the aggregate flexibility from heterogeneous TCLs using a battery-equivalent model. Based on the flexibility model, a multi-period optimal scheduling formulation is developed to best utilize the flexibility from building loads and maximize total benefits from stacked value streams. An algorithm is proposed to control individual TCLs to follow the desired power consumption in real-time. The proposed methods are illustrated and validated through simulations.
Wu, Di↗
A dictionary learning algorithm for compression and reconstruction of streaming data in preset order
There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.
97 MATHEMATICS AND COMPUTING↗
Anomaly Detection, Localization and Classification using Drifting Synchrophasor Data Streams
With ongoing automation and digitization of the electric power system, several Phasor Measurement Units(PMUs) have been deployed for monitoring and control. PMU data can have multiple anomalies, and many of the researchers in the past have concentrated on training machine/deep learning algorithms offline for anomaly detection over PMU data (i.e., not in real time). These machine/deep learning algorithms, when trained offline on a sample rather than a population of the dataset, fail to consider the dynamic behavior of the power grid in real-time, resulting in low accuracy. Considering the dynamic behavior of the power grid (e.g., change in load, generation, distributed energy resources (DERs) switching, network, controls), the definition of data anomalies varies in time and requires online training. A fundamental challenge is to enable online (i.e., real-time) training of machine/deep learning algorithms for anomaly detection over streaming PMU data. While machine/deep learning is often desirable to manage data streams, training a deep learning algorithm over streaming PMU data is nontrivial due to changes in data statistics caused by dynamic streaming data. This paper proposes PMUNET: a novel device-level deep learning-based data-driven approach for anomaly detection, localization, and classification over streaming PMU data, using online learning and multivariate data-drift detection algorithm .Two variants of PMUNET, Dynamic data Change Driven Learning (DCDL) and Continuity Driven Learning (CDL), are proposed and compared. DCDL aims to train the deep learning algorithm whenever the definition of anomaly changes due to the power grid dynamics. On the other hand, CDL continuously trains the deep learning algorithm over the PMU data-stream. The experimental results verify that DCDL outperforms CDL and other efficient anomaly detection methods over multiple events such as faults and load/ generator/capacitor/DERs variations/switching for IEEE 14 and 39 Bus test system as well as real PMU industrial data. The result verifies that DCDL variant of PMUNET improves over existing approach with a gain of 2% - 10% in terms of accuracy, false-positive rate, and false-negative rate.
adversarial deep learning↗