Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines

• DOI: 10.1145/3318464.3389723
• Publication Date: 2020-05-29T13:12:33Z
• Authors: Bonaventura Del M...; Steffen Zeuch; Tilmann Rabl; Volker Markl
• Keywords: Stream Processing; Control reconfiguration; State management

Scale-out stream processing engines (SPEs) are powering large big data applications on high velocity streams. Industrial setups require SPEs to sustain outages, varying rates, and low-latency processing. need transparently reconfigure stateful queries during runtime. However, state-of-the-art not ready yet handle on-the-fly reconfigurations of with terabytes state due three problems. These network overhead for migration, consistency, In this paper, we propose Rhino, a library efficient running in the presence very distributed state. Rhino provides handover protocol migration consistently efficiently migrate among servers. Overall, our evaluation shows that scales sizes up TBs, reconfigures query 15 times faster than state-of-the-art, reduces latency by orders magnitude upon reconfiguration.