Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2016

Implementation of a Multichannel Serial Data Streaming Algorithm using the Xilinx Serial RapidIO Solution

In the current world of applications that use reconfigurable technology implemented on field programmable gate arrays (FPGAs), there is a need for flexible architectures that can grow as the systems evolve. A project has limited resources and a fixed set of requirements that development efforts are tasked to meet. Designers must develop robust solutions that practically meet the current customer demands and also have the ability to grow for future performance. This paper describes the development of a high speed serial data streaming algorithm that allows for transmission of multiple data channels over a single serial link. The technique has the ability to change to meet new applications developed for future design considerations. This approach uses the Xilinx Serial RapidIO LOGICORE Solution to implement a flexible infrastructure to meet the current project requirements with the ability to adapt future system designs.

RapidIO↗
DOE OSTI2020

Fast shared-memory streaming multilevel graph partitioning

In this report we show that a fast parallel graph partitioner can benefit many applications by reducing data transfers. The online methods for partitioning graphs have to be fast and they often rely on simple one-pass streaming algorithms, while the offline methods for partitioning graphs contain more involved algorithms and the most successful methods in this category belong to the multilevel approaches. In this work, we assess the feasibility of using streaming graph partitioning algorithms within the multilevel framework. Our end goal is to come up with a fast parallel offline multilevel partitioner that can produce competitive cutsize quality. We rely on a simple but fast and flexible streaming algorithm throughout the entire multilevel framework. This streaming algorithm serves multiple purposes in the partitioning process: a clustering algorithm in the coarsening, an effective algorithm for the initial partitioning, and a fast refinement algorithm in the uncoarsening. Its simple nature also lends itself easily for parallelization. The experiments on various graphs show that our approach is on the average up to 5.1x faster than the multi-threaded MeTiS, which comes at the expense of only 2x worse cutsize.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Discovering hydrothermalism from Afar: In Situ methane instrumentation and change-point detection for decision-making

Seafloor hydrothermalism plays a critical role in fundamental interactions between geochemical and biological processes in the deep ocean. A significant number of hydrothermal vents are hypothesized to exist, but many of these remain undiscovered due in part to the difficulty of detecting hydrothermalism using standard sensors on rosettes towed in the water column or robotic platforms performing surveys. Here, we use in situ methane sensors to complement standard sensing technology for hydrothermalism discovery and compare sensors on a towed rosette and an autonomous underwater vehicle (AUV) during a 17 km long transect in the Northern Guaymas Basin in the Gulf of California. This transect spatially intersected with a known hydrothermally active venting site. These data show that methane signalled possible hydrothermal-activity 1.5–3 km laterally (100–150 m vertically) from a known vent. Methane as a signal for hydrothermalism performed similarly to standard turbidity sensors (plume detection 2.2–3.3 km from reference source), and more sensitively and clearly than temperature, salinity, and oxygen instruments which readily respond to physical mixing in background seawater. We additionally introduce change-point detection algorithms—streaming cross-correlation and regime identification—as a means of real-time hydrothermalism discovery and discuss related data supervision technologies that could be used in planning, executing, and monitoring explorative surveys for hydrothermalism.

58 GEOSCIENCES↗
NASA NTRS1986

Heating rates in tropical anvils

An IR radiative transfer model for cirrus clouds was developed on the basis of data acquired with a U-2 aircraft. The emission and scattering of radiation was expressed with a two-stream algorithm that considered the cloud in 10 layers. Exponential sums were employed to quantify gaseous absorption by H2O, CO2 and O3 over the IR spectrum of 4.5-250 microns. Ice water content had the greatest impact on radiative heating of the high altitude anvils, although the vertical extent of the anvil and the cloudiness of the underlying atmosphere were also important.

Ackerman, T. P.↗
NASA NTRS1987

New cellular automaton model for magnetohydrodynamics

A new type of two-dimensional cellular automation method is introduced for computation of magnetohydrodynamic fluid systems. Particle population is described by a 36-component tensor referred to a hexagonal lattice. By appropriate choice of the coefficients that control the modified streaming algorithm and the definition of the macroscopic fields, it is possible to compute both Lorentz-force and magnetic-induction effects. The method is local in the microscopic space and therefore suited to massively parallel computations.

Chen, Hudong↗
NASA NTRS1993

Three dimensional modeling of cirrus during the 1991 FIRE IFO 2: Detailed process study

A three-dimensional model of cirrus cloud formation and evolution, including microphysical, dynamical, and radiative processes, was used to simulate cirrus observed in the FIRE Phase 2 Cirrus field program (13 Nov. - 7 Dec. 1991). Sulfate aerosols, solution drops, ice crystals, and water vapor are all treated as interactive elements in the model. Ice crystal size distributions are fully resolved based on calculations of homogeneous freezing of solution drops, growth by water vapor deposition, evaporation, aggregation, and vertical transport. Visible and infrared radiative fluxes, and radiative heating rates are calculated using the two-stream algorithm described by Toon et al. Wind velocities, diffusion coefficients, and temperatures were taken from the MAPS analyses and the MM4 mesoscale model simulations. Within the model, moisture is transported and converted to liquid or vapor by the microphysical processes. The simulated cloud bulk and microphysical properties are shown in detail for the Nov. 26 and Dec. 5 case studies. Comparisons with lidar, radar, and in situ data are used to determine how well the simulations reproduced the observed cirrus. The roles played by various processes in the model are described in detail. The potential modes of nucleation are evaluated, and the importance of small-scale variations in temperature and humidity are discussed. The importance of competing ice crystal growth mechanisms (water vapor deposition and aggregation) are evaluated based on model simulations. Finally, the importance of ice crystal shape for crystal growth and vertical transport of ice are discussed.

Jensen, Eric J.↗
NASA NTRS1985

Study of one- and two-dimensional filtering and deconvolution algorithms for a streaming array computer

The development of vector processing computers of the streaming array architectures has made possible a dramatic decrease in the time required for the solution of problems, i.e., for those algorithms which readily lend themselves to sequential operations on long vectors. There has concurrently been rapid growth in the applications of the techniques generally known as mathematical digital filtering, also called signal analysis, digital signal processing, time series analysis, and digital image processing. The best known applications of these techniques is to seismic data and two-dimensional images, data types consisting of very large collections of numbers. A major limitation for these cases is the size and speed of the computer available. Therefore, a move to the streaming array architecture can result in marked improvement in the data analysis techniques which may be employed.

Ioup, G. E.↗
DOE OSTI2020

Evaluation & Development of Algorithms & Techniques for Streaming Detector Readout

The advancement in microelectronics capabilities, computing, and data science in the last decade has been remarkable. With the start of the 12 GeV Science Program at Jefferson Lab (JLab) and the conceptual detector design for the upcoming Ion Collider (EIC) in mind, we have evaluated and developed the possibility of evolving and improving the existing nuclear science research workflow based on these advances. Specifically, we have prototyped components of streaming readout.

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

Streaming Matching and Edge Cover in Practice

Graph algorithms with polynomial space and time requirements often become infeasible for massive graphs with billions of edges or more. State-of-the-art approaches therefore employ approximate serial, parallel, and distributed algorithms to tackle these challenges. However, such approaches require storing the entire graph in memory and thus need access to costly computing resources such as clusters and supercomputers. In this paper, we present practical streaming approaches for solving massive graph problems using limited memory for two prototypical graph problems: maximum weighted matching and minimum weighted edge cover. For matching, we conduct a thorough computational study on two of the semi-streaming algorithms including a recent breakthrough result that achieves a $1/(2+\varepsilon)$-approximation of the weight while using $O( n \log W /\epsilon)$ memory (here $n$ is the number of vertices and $W$ is the maximum edge weight), designed by Paz and Schwartzman [SODA, 2017]. Empirically, we show that the semi-streaming algorithms produce matchings whose weight is close to the best $1/2$-approximate offline algorithm while requiring less time and an order-of-magnitude less memory. For minimum weighted edge cover, we develop three novel semi-streaming algorithms. Two of these algorithms require a single pass through the input graph, require $O(n \log n)$ memory, and provide a 2-approximation guarantee on the objective. We also leverage a relationship between approximate maximum weighted matching and approximate minimum weighted edge cover to develop a two-pass $3/2+\epsilon$-approximate algorithm with the memory requirement of Paz and Schwartzman's semi-streaming matching algorithm. These streaming approaches are compared against the state-of-the-art 3/2-approximate offline algorithm. The semi-streaming matching and the novel edge cover algorithms proposed in this paper can process graphs with several billions of edges in under 30 minutes using 6 GB of memory, which is at least an order of magnitude improvement from the offline (non-streaming) algorithms. For the largest graph, the best alternative offline parallel approximation algorithm (GPA+ROMA) could not finish in three hours even while employing hundreds of processors and 1 TB of memory. We also demonstrate an application of the semi-streaming algorithm by computing a matching using linearly bounded memory on item intersection graphs derived from three machine learning datasets, whereas the existing offline algorithms could not complete on one of these datasets since their memory requirements exceeded 1TB.

Ferdous, S M.↗
DOE OSTI2025

Streaming Compression of Scientific Data via Weak-SINDy

Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Via Machinae 2.0 : Full-sky, model-agnostic search for stellar streams in Gaia DR2

ABSTRACT We present an update to Via Machinae, an automated stellar stream-finding algorithm based on the deep learning anomaly detector ANODE. Via Machinae identifies stellar streams within Gaia, using only angular positions, proper motions, and photometry, without reference to a model of the Milky Way potential for orbit integration or stellar distances. This new version, Via Machinae 2.0, includes many improvements and refinements to nearly every step of the algorithm, that altogether result in more robust and visually distinct stream candidates than our original formulation. In this work, we also provide a quantitative estimate of the false positive rate of Via Machinae 2.0 by applying it to a simulated Gaia-mock catalogue based on galaxia, a smooth model of the Milky Way that does not contain substructure or stellar streams. Finally, we perform the first full-sky search for stellar streams with Via Machinae 2.0, identifying 102 streams at high significance within the Gaia Data Release 2, of which only 10 have been previously identified. While follow-up observations for further confirmation are required, taking into account the false positive rate presented in this work, we expect approximately 90 of these stream candidates to correspond to real stellar structures.

Astronomy & Astrophysics↗
DOE OSTI2021

Evidence of a Dwarf Galaxy Stream Populating the Inner Milky Way Halo

Stellar streams produced from dwarf galaxies provide direct evidence of the hierarchical formation of the Milky Way. Here, we present the first comprehensive study of the LMS-1 stellar stream, that we detect by searching for wide streams in the Gaia EDR3 data set using the STREAMFINDER algorithm. This stream was recently discovered by Yuan et al. We detect LMS-1 as a 60° long stream to the north of the galactic bulge, at a distance of ~20 kpc from the Sun, together with additional components that suggest that the overall stream is completely wrapped around the inner Galaxy. Using spectroscopic measurements from the Large Sky Area Multi-Object Fiber Spectroscopic Telescope, the Sloan Digital Sky Survey, and the Apache Point Observatory Galactic Evolution Experiment, we infer that the stream is very metal-poor (<[Fe/H]> = –2.1) with a significant metallicity dispersion (σ [Fe/H] = 0.4), and it possesses a large radial velocity dispersion (σ v = 20 ± 4 km s–1 ). These estimates together imply that LMS-1 is a dwarf galaxy stream. The orbit of LMS-1 is close to polar, with an inclination of 75° to the galactic plane. Both the orbit and metallicity of LMS-1 are remarkably similar to the globular clusters NGC 5053, NGC 5024, and the stellar stream Indus. As a result, these findings make LMS-1 an important contributor to the stellar population of the inner Milky Way halo.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Real Time implementation of Artificial Intelligence compression algorithm for High-Speed Streaming Readout signals

The new generation of high-energy physics experiments plans to acquire data in streaming mode. With this approach, it is possible to access the information of the whole detector (organized in time slices) for optimal and lossless triggering of data acquisitions. With this approach, data rates, especially in large detectors, are often very high, and the network is likely to be the bottleneck for the entire Streaming Read Out system. The aim of this work is to study the implementation of a lossy compression algorithm based on Artificial Intelligence: an Autoencoder. With Machine Learning it is possible to achieve a high compression ratio and fast inference time with only a small degradation of the signals, almost negligible for the specific application. This work explores different configurations of the Autoencoder and the implementation on different hardware. Different Autoencoder configurations are explored to find the best trade-off between compression ratio and reconstruction loss, both for signals and energy spectrum. Different hardware implementations are also explored to find the best platform to achieve real-time performance for the specific application.

Rossi, Fabio (ORCID:0009000385713885)↗
NASA NTRS1990

Edge detection applied to SST fields

An algorithm designed to detect fronts automatically in satellite-derived sea-surface temperature (SST) fields is presented. The algorithm is operated at different levels to detect and differentiate between false and true edges. For purposes of comparison, the algorithm is applied to a test set of 98 SST images to detect the northern edge of the Gulf Stream. The algorithm successfully detected valid temperature fronts and ignored false edges, and also produced statistics about the temperature fronts that are useful in the subsequent analysis of these fronts. It is assumed that the algorithm performs equally well on other SST fronts such as those associated with rings, the subtropical convergence, or the shelf/slope fronts.

Cayula, Jean-Francois↗