Search NASA⌕ Search

SEARCH · Search NASA

Results for “exercise”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

Guideline 36 Conformance Test v2.0

ASHRAE Guideline 36 provides guidance for how typical commercial building HVAC systems should be controlled to reduce energy costs and improve the indoor environmental quality for occupants. An industry-led effort is developing test scripts for how control manufacturers can test that the control logic in their control products follows Guideline 36. This software intends to support that effort in two ways. First, by providing a testbed for control manufacturers to run the test scripts against their own control products and compare their generated results to the expected results provided by the test scripts. Second, by providing a testbed that can be used during the development of test scripts to rapidly and repeatably verify they exercise Guideline 36 logic as desired and generate accurate expected results that control products can be compared to. An initial version of the software has already been developed and released open-source by a previous CEC-funded project that implemented the first use case (https://github.com/LBNL-ETA/guideline36_conformance_test/blob/master/LICENSE.txt), while this software disclosure will add the functionality of the second use case, as well as update the software where needed, all open-source. We intend to extend and contribute to the existing open-source software repository.

Blum, David [Lawrence Berkeley National Laboratory↗

Comprehensive Approach to Measure the Mobility Energy Productivity of Freight Transport

Freight travel accounts for a major share of the energy consumed in the transportation sector in any country, and the United States is no exception. Understanding and modeling freight movement are critical, particularly in the context of capturing the impact of emerging technologies on freight travel and its externalities. The domain of freight modeling and forecasting has been gaining pace in recent years, but advancement in comprehensive freight performance metrics is still lagging. Conventional freight performance metrics such as truck-miles, ton-miles, or value-miles are unidimensional and aggregate in nature, making them unsuitable to accurately capture the impact of emerging transportation trends on the performance or productivity of freight systems. Addressing the research need, this paper presents the “Freight Mobility Energy Productivity” metric to quantify freight productivity of current as well as future freight systems, accounting for various costs associated with freight transport. The proposed metric was implemented using data from the Freight Analysis Framework along with other published sources, and shows intuitive results in quantifying freight productivity. Further, a scenario analysis exercise was conducted to test the capability of the metric in tracking improvements in system-level freight productivity as a result of vehicle electrification. The relative differences in Freight Mobility Energy Productivity scores help identify which zones benefit from the vehicle powertrain technology improvement. The results of the scenario analysis reinforce confidence that the proposed metric can be used as a decision support tool in assessing the productivity of existing as well as future freight trends and technologies.

47 OTHER INSTRUMENTATION↗

ZFP: A compressed array representation for numerical computations

HPC trends favor algorithms and implementations that reduce data motion relative to FLOPS. We investigate the use of lossy compressed data arrays in place of traditional IEEE floating point arrays to store the primary data of calculations. Simulation is fundamentally an exercise in controlled approximation, and error introduced by finite-precision arithmetic (or lossy compression) is just one of several sources of error that need to be managed to ensure sufficient accuracy in a computed result. We describe ZFP, a compressed numerical format designed for in-memory storage of multidimensional arrays, and summarize theoretical results that demonstrate that the error of repeated lossy compression can be bounded and controlled. Furthermore, we establish a relationship between grid resolution and compression-induced errors and show that, contrary to conventional floating point, ZFP reduces finite-difference errors with finer grids. We present example calculations that demonstrate data reduction by 4x or more with negligible impact on solution accuracy. Our results further demonstrate several orders-of-magnitude increase in accuracy using ZFP over IEEE floating point and Posits for the same storage budget.

Lindstrom, Peter↗

Development of a two-zone zero-dimensional mixing model for actively fueled prechamber engines

In this numerical work, a zero-dimensional, two-zone mixing model was developed and validated to capture the mixture preparation process of an actively fueled prechamber engine. The model was developed for ethanol-gasoline fuel blends but is applicable to various fuels of interest relevant to prechamber applications. The model was exercised to demonstrate the effects of boundary conditions, fuel composition, and residual content on the mixture stoichiometry predictions. The 0D model showed reasonable agreement in prechamber equivalence ratio predictions with higher fidelity computational fluid dynamics modeling, but the accuracy of model was constrained to the governing assumptions of the injection process. The modeling assumptions surrounding the non-dimensional injection of mass into the prechamber were scrutinized to identify key areas of model improvement. The results indicate heat transfer effects play a crucial role in the evaporation of the fuel within the confined auxiliary chamber, especially when considering fuels with high latent heat of vaporization. Additionally, the presence of fuel films that collect on the prechamber surfaces complicated the modeled evaporative cooling loses as the modeling suggests the films evaporate primarily by means of conduction from the boundary. Furthermore, the developed model serves as a simplified framework to generate calibration maps of prechamber fueling or run real time as an open loop control strategy such that desired prechamber mixture proportions are effectively targeted.

Zero-dimensional↗

The public health exposome and pregnancy-related mortality in the United States: a high-dimensional computational analysis

Racial inequities in maternal mortality in the U.S. continue to be stark. The 2015–2018, 4-year total population, county-level, pregnancy-related mortality ratio (PRM; deaths per 100,000 live births; National Center for Health Statistics (NCHS), restricted use mortality file) was linked with the Public Health Exposome (PHE). Using data reduction techniques, 1591 variables were extracted from over 62,000 variables for use in this analysis, providing information on the relationships between PRM and the social, health and health care, natural, and built environments. Graph theoretical algorithms and Bayesian analysis were applied to PHE/PRM linked data to identify latent networks. PHE variables most strongly correlated with total population PRM were years of potential life lost and overall life expectancy. Population-level indicators of PRM were overall poverty, smoking, lack of exercise, heat, and lack of adequate access to food. In this high-dimensional analysis, overall life expectancy, poverty indicators, and health behaviors were found to be the strongest predictors of pregnancy-related mortality. This provides strong evidence that maternal death is part of a broader constellation of both similar and unique health behaviors, social determinants and environmental exposures as other causes of death.

60 APPLIED LIFE SCIENCES↗

Male Runners With Lower Energy Availability Have Impaired Skeletal Integrity Compared to Nonathletes

Abstract Context Female athletes, particularly runners, with insufficient caloric intake for their energy expenditure [low energy availability (EA) or relative energy deficiency] are at risk for impaired skeletal integrity. Data are lacking in male runners. Objective To determine whether male runners at risk for energy deficit have impaired bone mineral density (BMD), microarchitecture, and estimated strength. Design Cross-sectional. Setting Clinical research center. Participants 39 men (20 runners, 19 controls), ages 16–30 years. Main Outcome Measures Areal BMD (dual-energy x-ray absorptiometry); tibia and radius volumetric BMD and microarchitecture (high-resolution peripheral quantitative computed tomography); failure load (microfinite element analysis); serum testosterone, estradiol, leptin; energy availability. Results Mean age (24.5 ± 3.8 y), lean mass, testosterone, and estradiol levels were similar; body mass index, percent fat mass, leptin, and lumbar spine BMD Z-score (−1.4 ± 0.8 vs −0.8 ± 0.8) lower (P < .05); and calcium intake and running mileage higher (P ≤ .01) in runners vs controls. Runners with EA <median had lower lumbar spine (−1.5 ± 0.7, P = .028), while runners with EA ≥median had higher hip (0.3 ± 0.7 vs −0.4 ± 0.5, P = .002), BMD Z-scores vs controls. After adjusting for calcium intake and running mileage, runners with EA <median had lower mean tibial total and trabecular volumetric BMD, trabecular bone volume fraction, cortical porosity, and apparent modulus vs controls (P < .05). Appendicular lean mass and serum estradiol (R ≥ 0.45, P ≤ .046), but not testosterone, were positively associated with tibial failure load among runners. Conclusions Despite weight-bearing activity, skeletal integrity is impaired in male runners with lower caloric intake relative to exercise energy expenditure, which may increase bone stress injury risk. Lower estradiol and lean mass are associated with lower tibial strength in runners.

Haines, Melanie S. (ORCID:0000000310024754)↗

MCS solutions for the TVA Watts Bar Unit 1 multi-physics depletion benchmark

The high-fidelity solution for exercises 1-3 of the recently developed TVA Watts Bar Unit 1 multi-physics and multi-cycle depletion benchmark using MCS was coupled with two different thermal-hydraulics (TH) codes: TH1D and COBRA-TF (CTF), and their solutions were compared. The solutions from MCS/CTF and MCS/TH1D for typical Pressurized Water Reactor (PWR) problems are quite close. And to evaluate MCS solutions for the benchmark, their results were also compared against measured data as well as publicly available high-fidelity solutions. The MCS criticality solutions at Hot Zero Power (HZP) deviate within 50 pcm against the reference solutions. While for the critical boron concentration (CBC) search calculations, the difference on the MCS calculated CBC is about 7 ppm, and the root-mean-squared (RMS) errors for the assembly power distribution and outlet coolant temperature are less than 0.5% and 1 Celsius degree respectively. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Benchmarking of ENDF/B-VIII.0 cross sections for LWR ex-core transport applications

ENDF/B-VIII.0 cross section data is benchmarked on a simple Monte-Carlo model representative of the ex-core geometry of a PWR used for reactor vessel neutron fluence calculations. The benchmarking indicates that the ENDF/B-VIII.0 results show large differences as compared to previous evaluations, which have been extensively compared to measured data. This exercise suggests that ENDF/B-VIII.0 nuclear data is not suitable for ex-core LWR radiation transport applications. Evaluators are aware of the problems with the ENDF/B-VIII.0 evaluation of iron and work is underway to improve the evaluations for an upcoming release.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

A refined assessment of nuclear data target accuracy requirements for ALFRED

The interest in target accuracy assessment has been recently renewed within the OECD/NEA Working Party on International Nuclear Data Evaluation Cooperation with an update exercise, thanks to the invaluable initiative of Massimo Salvatores. This work, moving from evaluations on the main core integral parameters of the Advanced Lead-cooled Fast Reactor European Demonstrator (ALFRED), and the identification of the associated target accuracies needed for improving its design, approaches the retrieval of target accuracy requirements for nuclear data, as the inverse problem of uncertainty quantification. The preliminary results obtained in a previous work are here refined by explicitly taking into account the total contribution due to correlations among variables in the optimization constraint, while its various components are evaluated individually. Moreover, three different sets of cost parameters are considered to introduce a bias in the solution in order to take in due account the relative difficulties in the execution of new differential experiments as needed for achieving the proposed target accuracies. By this, the obtained results are deemed usefully informative, and proposed for the update of a High Priority List which identifies the isotopes-reaction couples of most interest for future refinement experiments, especially when integrated with further analogous information collected on other advanced nuclear systems. Among the cross-sections whose refinement contributes the most to the reduction of the multiplication factor, the fission of Pu{sup 239} was found, which needs to be fixed below 1% in the range of interest for ALFRED (2 keV-4 MeV). Another contribution for the amelioration of the multiplication factor calculation accuracy, is due to the capture channel of Pu{sup 239}, for which the initial high uncertainty (around 15%) can be significantly reduced resorting to easily practicable experiments. The same occurs for the inelastic of Pb{sup 207}, for which the initial uncertainty value of around 50% can be drastically reduced up to target values between 4% and 7%.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Historical review and proof-of-concept future method demonstration of adaptive mesh refinement in nuclear engineering for increased fidelity and computational efficiency

As the nuclear industry's use of computational tool increases, the need for increased fidelity and computational efficiency is well known. While most approaches to increased fidelity rely on applying a fine mesh over the problem domain, a more efficient method is to apply an adaptive mesh refinement (AMR) algorithm to the mesh definition. In the field of nuclear engineering, AMR has previously been used in conjunction with deterministic methods, including: S{sub N} transport methods, Lattice Boltzmann Methods, and COMSOL. The future of AMR in nuclear engineering is to couple it to a Monte Carlo code with the goal of reducing calculation time. A proof-of-concept example yielded positive results for using the gradient of the flux as a refinement criteria. The refinement criteria was varied from 0.01 to 0.10, which yielded a recommended range of 0.01 to 0.04, and the number of refinement iterations was varied from 0 to 7, with diminishing returns seen after 5 iterations. After the success of the proof-of-concept exercise, work began on creating a full program coupling MCNP6.2 and the AMR algorithm in the deal.II library. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Estimating the Contributions to Human Error Probability from the Convolution of the Distribution of Time Available and Time Required

As part of their duties, Human Reliability Analysis must often evaluate if crews in nuclear power plants (NPPs) can complete tasks associated with a human-failure event within time limits. For example, the time required in NPP scenarios is determined by systematic and structured walkthroughs, feasibility studies, recorded times from training exercises, and interviews with experienced operators and experts. Typically, a point estimate is derived for the estimate (mean, maximum, or 95th percentile of time required). Using point-estimate values can mask the risk associated with variability among crews, plant conditions and set-up, environmental conditions, and other impact factors under which these actions are executed. While point estimates for time required and time available have served the industry well, without considering the uncertainty they could lead to biased understanding about the risk. The Integrated Human Event Analysis System - General Methodology (IDHEAS-G) model (developed by the US Nuclear Regulatory Commission, NRC) for human error probability calculates human error probability by summing two probabilities: insufficient time and cognitive error. As such, the model takes a more holistic approach by considering the full distributions for time required and time available to calculate the human error probability because the time available to complete the task is insufficient. In this study, we expand on the work of the NRC and discuss methods for estimating these time considerations. For example, for the time required, the impact of Performance Influencing Factors (PIFs) on the distribution was divided into impacts that are aleatory in nature, such as crew-to-crew variability, and those that are epistemic (i.e., the PIFs). Starting with the factors that introduce aleatory uncertainty, a first-order distribution was developed from a large set of time required (i.e., NPP task completion times) data for the range of operator actions that occur in the NPP control room under simulated accident conditions. The first-order distribution can then be adjusted to account for epistemic uncertainty using research associated with the impact of applicable PIFs on the time required. We also develop guidance for analysts to address the probability distributions for the time available. The guidance we developed on how to estimate time required and time available distributions is based on the identification of pertinent research and data, data analyses, and expert knowledge elicitation.

human error probability, human performance, time e↗

Evaluation of Oak Ridge National Laboratory Health Physics Research Reactor Operation Data for Critical Benchmark Creation

The Oak Ridge National Laboratory (ORNL) Health Physics Research Reactor (HPRR) was a research reactor designed and built at ORNL in 1961. The critical assembly used a highly enriched uranium and molybdenum alloy as the fuel and could be operated in steady-state or burst modes. The HPRR has recently been the object of an investigation to create a criticality benchmark. Such benchmarks are very important, as they are used primarily to show the accuracy of newly developed modeling codes and to help experimental validation and reactor licensing. The evaluated experiments considered in this paper were carried out between 1974 and 1986 from various HPRR activities such as steady-state subcritical, steady-state critical, and burst prompt super-critical operations of the reactor for dosimetry, irradiation, or training purposes. By using the HPRR experimental logbook information and the as-built drawings of the critical assembly, a highly detailed model of the HPRR was created with SCALE 6.2.4/KENO-VI, and a first version of a critical benchmark of the HPRR was developed following the International Criticality Safety Benchmark Evaluation Project (ICSBEP) guidelines for thorough description and uncertainty/sensitivity quantification. Unfortunately, in most of the evaluated experiments, the obtained difference between calculated and experimental k eff is around 1,000 pcm, corresponding to a relative error of approximately 1%, beyond the quality standards of the ICSBEP recommending a relative error below 0.1%. Moreover, the derived experimental uncertainty is high, around 4% relative, mainly due to the U-Mo fuel density uncertainty, but also from numerous other factors. For these reasons, the creation of a valuable critical benchmark from HPRR operation data is thus far compromised. In this paper, the different steps of the experiments’ evaluation are summarized, and the reasons for the experimental/calculation discrepancies and potential ways to solve them are explored. This paper also aims to remind us always to exercise considerable care when performing experimental work, and to record all the data possible for potential future uses.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Current Status of the DOE/NNSA Nuclear Criticality Safety Program Hands-on Criticality Safety Training Courses

In 2011, the US Department of Energy/National Nuclear Security Administration (DOE/NNSA) Nuclear Criticality Safety Program (NCSP) developed and piloted a 2-week nuclear criticality safety (NCS) practitioner course to support training and qualification of new NCS staff. The course was developed in accordance with the American National Standard Institute/American Nuclear Society (ANSI/ANS) standard for NCS training and qualifications (ANSI/ANS-8.26-2007). In 2013, an NCS manager’s course was developed for process supervisors, managers, regulators, and other professionals with NCS-related responsibilities. These courses consist of classroom education, facility training, and hands-on subcritical and critical experiments training. Each course is currently offered twice per year. The 2-week practitioner course offers a week of classroom training, with practical workshops and exercises focused on teaching students how to perform an NCS evaluation. The second week of training involves hands-on critical and subcritical experiments and measurements. The first week is offered in Las Vegas, Nevada, at the DOE Nevada Field Office or the National Atomic Testing Museum. Depending on the student’s clearance level, the second week is offered at Sandia National Laboratory (SNL) (uncleared and L-cleared students) or at the National Criticality Experiments Research Center (NCERC) (Q-cleared students). The 1-week manager’s course is offered at SNL or NCERC, depending on clearance or interest, and includes classroom and hands-on critical and subcritical experiments and measurements. This paper provides an overview and status report for the DOE/NNSA NCSP training courses in NCS and provides information about future course offerings.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Core Design Optimization of the Westinghouse Lead Fast Reactor

Westinghouse is pursuing an advanced Nuclear Power Plant design based on Lead Fast Reactor (LFR) technology for global commercialization. To achieve an optimal combination of key attributes, such as safety, sustainability, and economic competitiveness, Westinghouse and ANL partnered in developing and applying a formalized core design optimization strategy. An LFR analysis workflow was developed to automate a suite of reactor physics, fuels performance, safety, and economics simulations on a selected LFR concept. The workflow streamlines analysis of a wide range of LFR designs with different dimensions and fuel types to assess their viability and economic performance, significantly reducing human processing time and risks of processing errors. The LFR optimization exercise was defined, resulting in selection of the design constraints (geometric, neutronics, thermo-mechanical, safety, thermal-hydraulics, and economics) and performance metrics researched (minimization of both the fuels LCOE and the first core inventory cost). A total of 14 varied design parameters were considered, including assembly dimensions, coolant temperature, and enrichment distribution throughout the core. The LFR analysis workflow was connected to DAKOTA for sensitivity and optimization analyses. Due to the extremely large size of the potential LFR optimization solution space relative to the computing time required to characterize one LFR solution, a multi-stage optimization approach was proposed to breakdown the problem into several stages with more reasonable sizes. This optimization approach enabled finding various viable core solutions with different cost tradeoffs that were considered by Westinghouse and justify selection of a smaller core with multi-batch 2-year cycle length.

Stauff, Nicolas E.↗

A proposed framework for the development and qualitative evaluation of West Nile virus models and their application to local public health decision-making

West Nile virus (WNV) is a globally distributed mosquito-borne virus of great public health concern. The number of WNV human cases and mosquito infection patterns vary in space and time. Many statistical models have been developed to understand and predict WNV geographic and temporal dynamics. However, these modeling efforts have been disjointed with little model comparison and inconsistent validation. In this paper, we describe a framework to unify and standardize WNV modeling efforts nationwide. WNV risk, detection, or warning models for this review were solicited from active research groups working in different regions of the United States. A total of 13 models were selected and described. The spatial and temporal scales of each model were compared to guide the timing and the locations for mosquito and virus surveillance, to support mosquito vector control decisions, and to assist in conducting public health outreach campaigns at multiple scales of decision-making. Our overarching goal is to bridge the existing gap between model development, which is usually conducted as an academic exercise, and practical model applications, which occur at state, tribal, local, or territorial public health and mosquito control agency levels. The proposed model assessment and comparison framework helps clarify the value of individual models for decision-making and identifies the appropriate temporal and spatial scope of each model. This qualitative evaluation clearly identifies gaps in linking models to applied decisions and sets the stage for a quantitative comparison of models. Specifically, whereas many coarse-grained models (county resolution or greater) have been developed, the greatest need is for fine-grained, short-term planning models (m–km, days–weeks) that remain scarce. We further recommend quantifying the value of information for each decision to identify decisions that would benefit most from model input.

60 APPLIED LIFE SCIENCES↗

Novel perspective on a conventional technique: Impact of ultra-low temperature on bacterial viability and protein extraction

Ultra-low temperature (ULT) storage of microbial biomass is routinely practiced in biological laboratories. However, there is very little insight regarding the effects of biomass storage at ULT and the structure of the cell envelope, on cell viability. Eventually, these aspects influence bacterial cell lysis which is one of the critical steps for biomolecular extraction, especially protein extraction. Therefore, we studied the effects of ULT-storage (-80°C) on three different bacterial platforms: Escherichia coli , Bacillus subtilis and the cyanobacterium Synechocystis sp. PCC 6803. By using a propidium iodide assay and a modified MTT assay we determined the impact of ULT storage on cellular viability. Subsequently, the protein extraction efficiency was determined by analyzing the amount of protein released following the storage. The results successfully established that longer the ULT-storage time lower is the cell viability and larger is the protein extraction efficiency. Interestingly, E . coli and B . subtilis exhibited significant reduction in cell viability over Synechocystis 6803. This indicates that the cell membrane structure and composition may play a major role on cell viability in ULT storage. Interestingly, E . coli exhibited concomitant increase in cell lysis efficiency resulting in a 4.5-fold increase (from 109 μg/ml of protein on day 0 to 464 μg/ml of protein on day 2) in the extracted protein titer following ULT storage. Furthermore, our investigations confirmed that the protein function, tested through the extraction of fluorescent proteins from cells stored at ULT, remained unaltered. These results established the plausibility of using ULT storage to improve protein extraction efficiency. Towards this, the impact of shorter ULT storage time was investigated to make the strategy more time efficient to be adopted into protocols. Interestingly, E . coli transformants expressing mCherry yielded 2.7-fold increase (93 μg/mL to 254 μg/mL) after 10 mins, while 4-fold increase (380 μg/mL) after 120 mins of ULT storage in the extracted soluble protein. We thereby substantiate that: (1) the storage time of bacterial cells in -80°C affect cell viability and can alter protein extraction efficiency; and (2) exercising a simple ULT-storage prior to bacterial cell lysis can improve the desired protein yield without impacting its function.

59 BASIC BIOLOGICAL SCIENCES↗

Semi-Analytical Hierarchical Bayesian Inference of Nonlinear Model Structure in Stochastic Dynamics: Applied to Compartmental Models of Infectious Diseases

A Bayesian computational framework for parsimonious inference in stochastic nonlinear dynamical systems is presented. This framework enables the concurrent estimation of system states, time-varying parameters, time-invariant parameters, and the optimal sparsity structure of the model parameters. Because differential equation-based models are often simplified mechanistic or phenomenological representations, robust inference from noisy measurement data requires explicit treatment of model error and uncertainty. Model error and time-varying parameters can be represented as random processes, enabling inference while making minimal assumptions about the underlying sources of discrepancy and variability. Adopting stochastic differential equation representations affords the model significant flexibility, but can also render it susceptible to overfitting during statistical inversion, where the inferred model may track noise rather than the underlying signal. To alleviate the effects of overfitting and to enable the discovery of the optimal sparse representation of the time-invariant parameters, a Bayesian sparse learning algorithm is embedded within the framework. This sparse learning framework adopts an approximate hierarchical Bayesian setting defined by a series of semi-analytical expressions. The model structure inference framework is validated using a stochastic compartmental model for tracking and forecasting active cases of an infectious disease. Compartmental models describe population-level infectious disease dynamics through interactions among population fractions grouped by disease state. Mathematically, such models consist of a system of coupled ordinary differential equations. This example adopts an expressive compartmental model that includes multiple possible interactions between disease states, motivated by early uncertainty surrounding COVID-19 reinfection dynamics and their implications for long-term epidemic forecasting. The sparse learning exercise permits the inference of a priori unknown epidemiological dynamics from simulated public health data, discovering the nested compartmental model that optimizes the trade-off between average data-fit and model complexity. It is shown that inducing sparsity among the model parameters eliminates redundant interactions between compartments, equivalently revealing the optimal coupling structure between differential equations.

97 MATHEMATICS AND COMPUTING↗

Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow

Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks, a key tool for reasoning about temporal completeness in infinite streams.First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches:• Computing a single correct answer, as in notifications.• Reasoning about a lack of data, as in dip detection.• Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models.• Safely and punctually garbage collecting obsolete inputs and intermediate state.• Surfacing a reliable signal of overall pipeline health.Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.

Akidau, Tyler↗