Evaluation of Data Enrichment Methods for Distributed Stream Processing Systems

• DOI: 10.48550/arxiv.2307.14287
• Publication Date: 2023-01-01
• Authors: Dominik Scheinert; Fabian Casares; Morgan K. Geldenhuys; Kevin Styp-Rekowski; Odej Kao
• Keywords: Stream Processing; Data Processing; Data processing system

Stream processing has become a critical component in the architecture of modern applications. With exponential growth data generation from sources such as Internet Things, business intelligence, and telecommunications, real-time unbounded streams necessity. DSP systems provide solution to this challenge, offering high horizontal scalability, fault-tolerant execution, ability process multiple single job. Often enough though, need be enriched with extra information for correct processing, which introduces additional dependencies potential bottlenecks. In paper, we present an in-depth evaluation enrichment methods identify different use cases stream systems. Using representative system conducting realistic cloud environment, found that outsourcing can improve performance specific cases. However, increased resource consumption highlights solutions specifically designed performance-intensive workloads cloud-based