Advanced data structures for monitoring national security cyber streams.
Abstract not provided.
Phillips, Cynthia A.↗
SEARCH · Search NASA
Results for “Streaming Data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Towards an Intelligent Data Delivery Service
The ATLAS Event Streaming Service (ESS) at the LHC is an approach to preprocess and deliver data for Event Service (ES) that has implemented a fine-grained approach for ATLAS event processing. The ESS allows one to asynchronously deliver only the input events required by ES processing, with the aim to decrease data traffic over WAN and improve overall data processing throughput. A prototype of ESS was developed to deliver streaming events to fine-grained ES jobs. Based on it, an intelligent Data Delivery Service (iDDS) is under development to decouple the "cold format" and the processing format of the data, which also opens the opportunity to include the production systems of other HEP experiments. Here we will at first present the ESS model view and its motivations for iDDS system. Then we will also present the iDDS schema, architecture and the applications of iDDS.
97 MATHEMATICS AND COMPUTING↗
Complex Event Recognition Architecture
Complex Event Recognition Architecture (CERA) is the name of a computational architecture, and software that implements the architecture, for recognizing complex event patterns that may be spread across multiple streams of input data. One of the main components of CERA is an intuitive event pattern language that simplifies what would otherwise be the complex, difficult tasks of creating logical descriptions of combinations of temporal events and defining rules for combining information from different sources over time. In this language, recognition patterns are defined in simple, declarative statements that combine point events from given input streams with those from other streams, using conjunction, disjunction, and negation. Patterns can be built on one another recursively to describe very rich, temporally extended combinations of events. Thereafter, a run-time matching algorithm in CERA efficiently matches these patterns against input data and signals when patterns are recognized. CERA can be used to monitor complex systems and to signal operators or initiate corrective actions when anomalous conditions are recognized. CERA can be run as a stand-alone monitoring system, or it can be integrated into a larger system to automatically trigger responses to changing environments or problematic situations.
Fitzgerald, William A.↗
Applying Transfer Learning for Street-Scale Nuisance Flood Forecasting in Coastal-Urban Cities
An important challenge with Machine Learning (ML) is its transferability; that is, whether a ML model trained on one set of data can be applied to a second set of data without requiring a full re-training of the model. Transfer Learning (TL) addresses this challenge by transferring knowledge learned in the source domain (the data it was trained on) to the target domain (a second set of data that is statistically different but related, which the model was not trained on). This study investigates the use of TL for street-scale nuisance flood forecasting by exploring whether a ML model trained on data collected for one set of streets can effectively forecast flooding for another set of streets in the same city using TL. The envisioned use case is a city deploying a new flood depth monitoring sensor on a street and using TL to apply a ML model, trained on sensor data from an existing flood depth sensor network, to this new street. Eventually, the new flood depth sensor will have a sufficient dataset for training its own ML model, but TL can be used to fill the gap in time while this new dataset is being generated. This method is explored using a Long Short-Term Memory (LSTM) model trained on data for the flood-prone streets of Norfolk City, Virginia. The data used for training includes environmental time series (rainfall, tide), topographic features (Digital Elevation Model (DEM), Topographic Wetness Index (TWI), Depth To Water (DTW)), and street-scale flood depth time series obtained from a high-fidelity physics-based model, acting as a synthetic street-scale stream depth sensor dataset since actual stream depth sensor data is generally unavailable for most cities. A set of 180 flood-prone streets was used to train a base model, while another set of 180 flood-prone streets was used to re-train that model using different TL strategies. The results show that full-weight re-training proved most effective and minimal re-training of only the output layer was insufficient. The advantage of TL was most pronounced when target data was limited, meaning data collected at the new water depth sensor location included generally less than 18 flood events. As target data increased beyond 18 flood events, the benefit of TL diminished relative to training a ML model directly on the local flood events. These findings can assist cities as they implement street-scale flood sensing systems to create accurate forecasts for new sensing locations that do not yet have sufficient data records to train a local ML model.
Roy, Binata [Univ. of Virginia, Charlottesville, V↗
Experimental Investigation of Transition to Turbulence as Affected by Passing Wakes
Experimental results from a study of the effects of passing wakes upon laminar-to-turbulent transition in a low-pressure turbine passage are presented. The test section geometry is designed to simulate the effects of unsteady wakes resulting from rotor-stator interaction upon laminar-to-turbulent transition in turbine blade boundary layers and separated flow regions over suction surfaces. Single-wire, thermal anemometry techniques were used to measure time-resolved and phase-averaged, wall-normal profiles of velocity, turbulence intensity, and intermittency at multiple streamwise locations over the turbine airfoil suction surface. These data are compared to steady state, wake-free data collected in the same geometry to identify the effects of wakes upon laminar-to-turbulent transition. Results are presented for flows with a Reynolds number based on suction surface length and exit velocity of 50,000 and an approach flow turbulence intensity of 2.5 percent. From these data, the effects of passing wakes and associated increased turbulence levels and varying pressure gradients on transition and separation in the near-wall flow are presented. The results show that the wakes affect transition both by virtue of their difference in turbulence level from that of the free-stream but also by virtue of their velocity deficit relative to the freestream velocity, and the concomitant change in angle of attack and temporal pressure gradients. The results of this study seem to support the theory that bypass transition is a response of the near-wall viscous layer to pressure fluctuations imposed upon it from the free-stream flow. The data also show a significant lag between when the wake is present over the surface and when transition begins. The accompanying CD-ROM includes tabulated data, animations, higher resolution plots, and an electronic copy of this report.
Kaszeta, Richard W.↗
Experimental Investigation of Transition to Turbulence as Affected by Passing Wakes
Experimental results from a study of the effects of passing wakes upon laminar-to-turbulent transition in a low-pressure turbine passage are presented. The test section geometry is designed to simulate the effects of unsteady wakes resulting from rotor-stator interaction upon laminar-to-turbulent transition in turbine blade boundary layers and separated flow regions over suction surfaces. Single-wire, thermal anemometry techniques were used to measure time- resolved and phase-averaged, wall-normal profiles of velocity, turbulence intensity, and intermittency at multiple streamwise locations over the turbine airfoil suction surface. These data are compared to steady state, wake-free data collected in the same geometry to identify the effects of wakes upon laminar-to-turbulent transition. Results are presented for flows with a Reynolds number based on suction surface length and exit velocity of 50,000 and an approach flow turbulence intensity of 2.5 percent. From these data, the effects of passing wakes and associated increased turbulence levels and varying pressure gradients on transition and separation in the near-wall flow are presented. The results show that the wakes affect transition both by virtue of their difference in turbulence level from that of the free-stream but also by virtue of their velocity deficit relative to the freestream velocity, and the concomitant change in angle of attack and temporal pressure gradients. The results of this study seem to support the theory that bypass transition is a response of the near-wall viscous layer to pressure fluctuations imposed upon it from the free-stream flow. The data also show a significant lag between when the wake is present over the surface and when transition begins. The accompanying CD-ROM includes tabulated data, animations, higher resolution plots, and an electronic copy of this report.
Kaszeta, Richard W.↗
X-ray bright points and high-speed wind streams: A preliminary analysis from Yohkoh and Ulysses data
The following aspect of the solar wind mass flux, and of its variation, is examined: whether coronal plumes might be responsible for the long-term variability of the mass flux in high-speed streams emanating from coronal holes. The assumption that plumes are rooted in coronal bright points (BP's) is made. The behavior of X-ray BP's, imaged by the Yohkoh soft X-ray telescope (SXT), during a seven month period when Ulysses experiments observed a series of recurrent high-speed streams, is analyzed. If plumes/BP's are sources of the wind mass flux, changes in the coronal hole BP density to mimic changes of the mass flux in high-speed streams are expected. SOHO will have the capability of measuring the solar wind speed/density at small heliocentric distances while simultaneously observing coronal BP's and coronal plumes.
Poletto, Giannina↗
Exploring DAOS as a Burst Buffer for a 100 Gbps DAQ Real-Time Streaming System
We present an experimental evaluation of a burst buffer for a real-time DAQ streaming system designed to transmit instrument data to remote data centers. The system is based on EJ-FAT, a load balancing system capable of Nx 100Gbps streams, distributing data from event sources to processing nodes. We explore applying the DAOS system as a burst buffer to serve a number of purposes: improve resiliency, elasticity and add new functions into the processing pipeline. In the evaluation a sender transmits events over a 100Gbps network to a receiver integrated with DAOS to store the reassembled events using DAOS APIs. We evaluate the system for possible bottlenecks and provide end-to-end evaluation with a burst buffer using DAOS storage abstractions. We show that a receiver node can support 38.1 Gbps. This proves the viability of our approach and allows us to extend this work to investigate scale-out properties and new streaming optimizations.
Mei, Xinxin↗
Portable Radiation Package (PRP) Instrument Handbook
The Portable Radiation Package (PRP) was developed to provide basic radiation information in locations such as ships at sea where proper exposure is remote and difficult, the platform is in motion, and azimuth alignment is not fixed. Development of the PRP began at Brookhaven National Laboratory (BNL) in the mid-1990s and versions of it were deployed on ships in the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Climate Research Facility’s Nauru-99 project. The PRP was deployed on ships in support of the National Aeronautics and Space Administration (NASA) Sensor Intercomparison for Marine Biological and Interdisciplinary Ocean Studies (SIMBIOS) program. Over the years the measurements have remained the same while the post-processing data analysis, especially for the FRSR, has evolved. This document describes the next-generation Portable Radiation Package (PRP2) that was developed for the DOE ARM Facility, under contract no. 9F-31462 from Argonne National Laboratory (ANL). The PRP2 has the same scientific principles that were well validated in prior studies, but has upgraded electronic hardware. The PRP2 approach is completely modular, both in hardware and software. Each sensor input is treated as a separate serial stream into the data collection computer. In this way the operator has complete access to each component of the system for purposes of error checking, calibration, and maintenance. The resulting system is more reliable, easier to install in complex situations, and more amenable to upgrade.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Mapping National Forest Aboveground Biomass in Mexico By Integrating GEDI and Landsat Times Series Data
Mexico is one of the countries with great potential for the UN's Reducing Emissions from Deforestation and Forest Degradation (REDD+) program, a key nature-based solution for the forest sector. To monitor carbon stock changes, there is a growing demand for unbiased Monitoring Reporting Verification (MRV) systems to facilitate effective forest management and climate change mitigation strategies. Remote sensing-based national aboveground biomass density (AGBD) estimation over Mexico is scarce and often limited to one-time static mapping, leading to spatiotemporal inconsistency in inputs. As an effort under NASA's Carbon Monitoring System (CMS) program, we have developed a remote sensing-based approach to create consistent historical AGBD maps of Mexico using multi-stream remote sensing data, including spaceborne lidar GEDI and long-term Landsat time series, as well as topographic information. We employ the continuous change detection and classification (CCDC) algorithm for temporal modeling of Landsat surface reflectance, followed by the inference of forest AGBD using a random forest machine learning algorithm with the temporal information of land surface dynamics extracted by the CCDC as input. GEDI provides unprecedented forest structure and AGBD sampling datasets for model training and validation practices. In this presentation, we share the progress made in developing a spatially explicit mapping of historical AGBD changes associated with land surface changes and post-disturbance landscapes.
Taejin Park↗
Distributed out-of-memory NMF on CPU/GPU architectures
We propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10 -6 .
97 MATHEMATICS AND COMPUTING↗
A Data Analysis Center for Electromagnetic and Hadronic Interaction. Products of the DAC members
The Data Analysis Center (DAC) of the Center for Nuclear Studies (CNS) at the George Washington University (GW) has made significant progress in its program to enhance and expand the partial-wave (and multipole) analyses of fundamental two- and three-body reactions (such as pion-nucleon, photon-nucleon, and nucleon-nucleon scattering) by maintaining and augmenting the analysis codes and databases associated with these reactions. These efforts provide guidance to experimental groups at the international level, forming an important link between theory and experiment. A renaissance in light hadron spectroscopy is underway as a continuous stream of polarization data issues from existing precision electromagnetic facilities and the coming Jefferson Lab 12 GeV Upgrade. Our principal goals have been focused on supporting the national N* resonance physics program. We have also continued to study topics more generally related to the problems associated with partial-wave analysis. On the Experimental side of the CNS DAC. Its primary goal is the enhancement of the body of data necessary for our analyses of fundamental γ - N reactions. We perform experiments that study the dynamics responsible for the internal structure of the nucleon and its excitations. Our principal focus is on the N* programs at JLab and MAMI. At JLab we study spin-polarization observables using polarized photons, protons and neutrons and yielding charged final states. Similarly at MAMI we study neutral meson photoproduction off polarized protons and neutrons. We use the Crystal Ball and TAPS spectrometers (CBT) to detect photons and neutrons to measure the photoproduction of π0, η, 2π0, π0η, and K0 off the neutron. The CBT program complements our program at JLab, which studies reactions resulting in charged final states. We are also involved in a renewed effort to make neutral pion photoproduction measurements close to threshold at Mainz. In addition to the programs underway, we are contributing to the future by participation in preparations for the coming JLab 12 GeV Upgrade. GW students are involved in tests of the detectors proposed to be used with CLAS12, i.e., for the CentralTime-of-Flight Barrel (CTOF). WJB is heavily involved in the MUSE quest at PSI to solve the Proton Radius Puzzle.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Evidence of a Dwarf Galaxy Stream Populating the Inner Milky Way Halo
Stellar streams produced from dwarf galaxies provide direct evidence of the hierarchical formation of the Milky Way. Here, we present the first comprehensive study of the LMS-1 stellar stream, that we detect by searching for wide streams in the Gaia EDR3 data set using the STREAMFINDER algorithm. This stream was recently discovered by Yuan et al. We detect LMS-1 as a 60° long stream to the north of the galactic bulge, at a distance of ~20 kpc from the Sun, together with additional components that suggest that the overall stream is completely wrapped around the inner Galaxy. Using spectroscopic measurements from the Large Sky Area Multi-Object Fiber Spectroscopic Telescope, the Sloan Digital Sky Survey, and the Apache Point Observatory Galactic Evolution Experiment, we infer that the stream is very metal-poor (<[Fe/H]> = –2.1) with a significant metallicity dispersion (σ [Fe/H] = 0.4), and it possesses a large radial velocity dispersion (σ v = 20 ± 4 km s–1 ). These estimates together imply that LMS-1 is a dwarf galaxy stream. The orbit of LMS-1 is close to polar, with an inclination of 75° to the galactic plane. Both the orbit and metallicity of LMS-1 are remarkably similar to the globular clusters NGC 5053, NGC 5024, and the stellar stream Indus. As a result, these findings make LMS-1 an important contributor to the stellar population of the inner Milky Way halo.
79 ASTRONOMY AND ASTROPHYSICS↗
The Artificial Scientist: in-Transit Machine Learning of Plasma Simulations
Large-scale simulations or scientific experiments produce petabytes of data per run. This poses massive challenges for I/O and storage when scientific analysis workflows are run manually offline. Unsupervised deep learning-based techniques to extract patterns and non-linear relations from these large amounts of data provide a way to build scientific understanding from raw data, reducing the need for manual pre-selection of analysis steps, but require exascale compute and memory to process the full dataset available. In this paper, we demonstrate a heterogeneous streaming workflow in which plasma simulation data is streamed directly to a Machine Learning (ML) application training a model on the simulation data in-transit, completely circumventing the capacity-constrained filesystem bottleneck. This workflow employs openPMD to provide a high level interface to describe scientific data and also uses ADIOS2, to transfer volumes of data that exceed the capabilities of the filesystem. We employ experience replay to avoid catastrophic forgetting in learning from this non-steady state process in a continual manner and adapt it to improve model convergence while learning in-transit. As a proof-of-concept, we approach the ill-posed inverse problem of predicting particle dynamics from radiation in a particle-incell (PIConGPU) simulation of the Kelvin-Helmholtz instability (KHI). We detail hardware-software co-design challenges as we scale PIConGPU to full Frontier, the Top-1 system as of June 2024 Top500 list.
Kelling, Jeffrey [Helmholtz-Zentrum Dresden Rossen↗
COVID-19 Data Curation Effort: An Initial Analysis of the Data
During the COVID-19 pandemic of 2020, major case reporting outlets quickly coalesced around two or three primary vendors. Johns Hopkins University and The New York Times were among the more prominent, and all were of great value to the nation, particularly during the uncertain early stages of the pandemic. They primarily focused on three major attributes: number of new cases, deaths, and recovery. Recognizing that many states were reporting very detailed data sets (e.g., hospital beds) at a county level or finer, the ORNL Pandemic Modeling team embarked on a major data curation effort from March to June 2020 for the purpose of capturing this wealth of detailed data. The challenge of curating this data was daunting. The number of attributes reported by the states grew on almost on a weekly basis. States were routinely shifting their web tool strategies away from easily parsable HTML-based formatting to new Tableau and ArcGIS content. This growth in the sheer number of attributes, combined with the unpredictable shifts in data format, meant an aggressive and agile combination of automated scripting and manual scraping was required to capture new daily streams. Further, the team had to scale up staff and widen its approach for capture and storage. As a result, the team collected more than 11 million data points. Following the close of this data collection effort on June 30 th , 2020, the team embarked on a major effort to appraise what had been collected, including an inventory list, spatial completeness, temporal completeness, scale and geographic characteristics, and a determination. A report on this matter was submitted on September 15 th , 2020, titled “DOE COVID-19 Data Curation Effort: Overview of Data Collection Coverage”. Over 2000 unique attributes had been netted over a wide range of spatial scales, including state, county, zip codes, health regions, and census blocks. Over 11 million individual data points were collected across these attributes, and spatial coverage (in total) included all 50 states and multiple territories. What became apparent in the process is that in the absence of any data standards, many states reported a wide variety of unique attributes that were not always compatible with attributes reported in other states. As time continued, states began adding new attributes and offering finer grain detail in some older attributes. This meant that not all data streams existed for the entire time period; in fact, the number tended to increase dramatically towards the end. Often, states would begin an attribute series and then stop altogether. These highly variable and uncertain conditions illuminated the need for harmonization approaches that would reconcile and conflate changing attribute names and detail over time. For example, grouping racial data reported as either Black or African American, depending on the state, into a single harmonized attribute. These choices would make a within-state analysis possible during the time period and lead to potential between-state analytics later on. This was almost entirely a manual decision process, requiring some subjective decision-making at times, to prevent a fragmented, short-lived collection of time series fragments that would offer few insights into trends, patterns, and correlates. This report imports harmonized data for state and county into the World Spatio-Temporal Analytics and Mapping Project (WSTAMP). WSTAMP is a major space-time analysis and visualization tool developed at ORNL for the National Geospatial-Intelligence Agency specifically for this kind of exploratory analysis. WSTAMP offers a rich analytical and graphical environment consisting of a wide range of analytics. These include time series plots, statistical summaries, data mining techniques, trend and pattern detection, and hypothesis generation.
59 BASIC BIOLOGICAL SCIENCES↗
Effects of Nose Radius and Extreme Cooling on Boundary-Layer Transition for Three Smooth 15°-Cone-Cylinders in Free Flight at Mach Numbers to 8.50
Three highly polished 15°- included- angle cone- cylinders with hemispherical tips of several diameters ( 2, 3, and 4 in.) have been flown in order to obtain boundary- layer transition data at very low wall to local stream temperature ratios, and heat- transfer data. All surfaces had a 2-microinch average roughness height. Laminar flow existed over the entire hemispherical nose of the 2- and 3-inch-tip- diameter models throughout the complete flight history. Extreme cooling to wall to local stream temperature ratios at the sonic point as low as 0.20 did not cause transition on the nose for diameters as large as 3 inches. However, extreme cooling did cause early transition on the 4-inch model where it appears probable that transition occurred forward of the 45° station at a wall to local stream temperature ratio of about 0.26. Variations in tip diameter influenced transition downstream of the nose under conditions of extreme cooling. The 2-inch- tip model was laminar at all cone- cylinder stations at temperature ratios as low as 0.32 whereas the 3- and 4-inch-tip models were turbulent at the same local flow conditions but at higher wall to local temperature ratios. Transition on the cone and cylinder of the 3- and 4-inch- tip bodies appeared to be sensitive to local Mach number, and occurred at higher local temperature ratios when values of local Mach number were higher. Increasing the nose diameter from 2 to 3 inches significantly changed the local flow conditions for which laminar flow existed on the cone- cylinder afterbody. However, a further increase in tip size t o a 4-inch diameter had no discernable effect on the local flow conditions at transition. The transition results of the 3- and 4-inch-nose-diameter smooth bodies are similar to those observed on a 7/8-inch-nose-diameter body with roughened surfaces. Turbulent boundary layers resulted in both cases at very low wall to local stream temperature ratios. Both laminar and turbulent heat-transfer data were in good agreement with theoretical Stanton numbers when heat-transfer reduction due to tip blunting was considered.
CYLINDRICAL AFTERBODY↗
Towards Interactive, Reproducible Analytics at Scale on HPC Systems
The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.
containers↗
Science Goal Monitor: Science Goal Driven Automation for NASA Missions
Infusion of automation technologies into NASA s future missions will be essential because of the need to: (1) effectively handle an exponentially increasing volume of scientific data, (2) successfully meet dynamic, opportunistic scientific goals and objectives, and (3) substantially reduce mission operations staff and costs. While much effort has gone into automating routine spacecraft operations to reduce human workload and hence costs, applying intelligent automation to the science side, i.e., science data acquisition, data analysis and reactions to that data analysis in a timely and still scientifically valid manner, has been relatively under-emphasized. In order to introduce science driven automation in missions, we must be able to: capture and interpret the science goals of observing programs, represent those goals in machine interpretable language; and allow spacecrafts onboard systems to autonomously react to the scientist's goals. In short, we must teach our platforms to dynamically understand, recognize, and react to the scientists goals. The Science Goal Monitor (SGM) project at NASA Goddard Space Flight Center is a prototype software tool being developed to determine the best strategies for implementing science goal driven automation in missions. The tools being developed in SGM improve the ability to monitor and react to the changing status of scientific events. The SGM system enables scientists to specify what to look for and how to react in descriptive rather than technical terms. The system monitors streams of science data to identify occurrences of key events previously specified by the scientist. When an event occurs, the system autonomously coordinates the execution of the scientist s desired reactions. Through SGM, we will improve om understanding about the capabilities needed onboard for success, develop metrics to understand the potential increase in science returns, and develop an operational prototype so that the perceived risks associated with increased use of automation can be reduced.
Koratkar, Anuradha↗