Stateful processing of clickstream events using Apache Beam

Intuit migrated its clickstream enrichment pipeline from batch to stream processing by implementing stateless and stateful enrichments using Apache Beam running on Flink Runner. The pipeline processes over 2 billion events a day with milliseconds latency enabling a variety of fraud and customer behavior use cases in real-time. In this talk, we will share our leanings and benefits of moving from batch to streaming, go in depth on stateful processing which we implemented using State and Timer APIs of Apache Beam, cost management and optimization.