Stateful processing of clickstream events using Apache Beam

16 Jun

5:30pm UTC

TALKS

Stateful processing of clickstream events using Apache Beam

By: Jainik Vora

Intuit migrated its clickstream enrichment pipeline from batch to stream processing by implementing stateless and stateful enrichments using Apache Beam running on Flink Runner. The pipeline processes over 2 billion events a day with milliseconds latency enabling a variety of fraud and customer behavior use cases in real-time. In this talk, we will share our leanings and benefits of moving from batch to streaming, go in depth on stateful processing which we implemented using State and Timer APIs of Apache Beam, cost management and optimization.