LinkedInBook a call
All articles

Data Engineering

Building Real-Time Analytics with Kafka, Flink, and ClickHouse

28 March 202517 min readBy Bayseian Engineering

Architecture and implementation of streaming data pipelines for real-time analytics, handling millions of events per second with sub-second latency.

Introduction: The Real-Time Analytics Challenge

Modern applications generate millions of events per second: user clicks, sensor data, financial transactions, IoT telemetry. Traditional batch processing (running jobs nightly or hourly) is too slow for use cases that demand sub-second insights: fraud detection, recommendation engines, operational dashboards, and anomaly detection.

Real-time analytics streaming architectures enable processing data as it arrives, providing immediate insights and triggering automated actions. This post covers production-ready streaming architectures using Apache Kafka, Apache Flink, and cloud-native services.

  • •Fraud Detection: Analyze transactions in real-time to block suspicious activity before the money moves, not in a nightly batch review after the fact.
  • •Personalization: Update recommendations instantly based on user behavior: the difference between reacting to what a user did five minutes ago and what they're doing right now.
  • •Operational Monitoring: Dashboard metrics with <1 second latency, so an incident shows up on the dashboard before customers start filing tickets.
  • •IoT Analytics: Process sensor data from thousands of devices as it arrives, since batching telemetry defeats the purpose of real-time alerting on equipment failure.
  • •Log Analytics: Real-time error detection and alerting catches a bad deploy in minutes instead of during the next morning's log review.
  1. 1.Stream Ingestion: Kafka, Kinesis, Pub/Sub
  2. 2.Stream Processing: Flink, Spark Streaming, Kafka Streams
  3. 3.State Management: RocksDB, Redis, DynamoDB
  4. 4.Data Sink: Elasticsearch, ClickHouse, TimescaleDB, S3
  5. 5.Monitoring: Prometheus, Grafana, Datadog

Architecture Overview

End-to-end latency under 100ms at 1M events/sec with 99.9%+ availability.

Stream Processing with Apache Flink

Why Flink for Streaming?

Flink provides true event-time processing, exactly-once semantics, and low-latency state management, which is what makes it a good fit for production real-time analytics.

  1. 1.Event Time Processing: Handle late-arriving data correctly. A mobile event delayed by a flaky connection still lands in the window it actually happened in, not the window it arrived in.
  2. 2.Stateful Processing: Maintain aggregations, windows, joins across events without re-reading the whole stream from the start every time.
  3. 3.Exactly-Once Semantics: No duplicate processing, even across retries and failures. That matters when the output feeds billing or fraud decisions.
  4. 4.High Throughput: Process millions of events/second on a cluster that scales by adding task managers, not by rewriting the job.
  5. 5.Fault Tolerance: Automatic recovery from failures via checkpointing. A task manager crash resumes from the last checkpoint instead of losing state.

Common Streaming Patterns:

1. Windowed Aggregations: Count/sum events over time windows, the basis for any "events per minute" or "rolling average" metric.
2. Stream Enrichment: Join streaming data with reference tables so events carry context (user plan, geography) without a separate lookup service.
3. Pattern Detection: Identify sequences (e.g., user journey analysis). This catches multi-step behaviors that a single-event view can't see.
4. Anomaly Detection: ML models on streaming data flag outliers within the same window they occur in, not after a batch job runs hours later.
5. Stream-to-Stream Joins: Correlate events from multiple sources, like matching a click stream to a purchase stream to measure conversion in real time.

Python
# Flink SQL for Real-Time Analytics
# Example: Real-time user activity dashboard

from pyflink.table import EnvironmentSettings, TableEnvironment
from pyflink.table.window import Tumble

# Setup Flink environment
env_settings = EnvironmentSettings.in_streaming_mode()
table_env = TableEnvironment.create(env_settings)

# Configure Kafka source
table_env.execute_sql("""
    CREATE TABLE user_events (
        user_id STRING,
        event_type STRING,
        event_timestamp BIGINT,
        page_url STRING,
        session_id STRING,
        device_type STRING,
        country STRING,
        event_time AS TO_TIMESTAMP(FROM_UNIXTIME(event_timestamp)),
        WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'user-events',
        'properties.bootstrap.servers' = 'kafka:9092',
        'properties.group.id' = 'flink-analytics',
        'scan.startup.mode' = 'latest-offset',
        'format' = 'json',
        'json.fail-on-missing-field' = 'false',
        'json.ignore-parse-errors' = 'true'
    )
""")

# Real-time aggregation: Events per minute by device type
table_env.execute_sql("""
    CREATE TABLE events_per_minute AS
    SELECT
        window_start,
        window_end,
        device_type,
        COUNT(*) as event_count,
        COUNT(DISTINCT user_id) as unique_users,
        COUNT(DISTINCT session_id) as sessions
    FROM TABLE(
        TUMBLE(TABLE user_events, DESCRIPTOR(event_time), INTERVAL '1' MINUTE)
    )
    GROUP BY window_start, window_end, device_type
""")

# Stream enrichment: Join with user profile data
table_env.execute_sql("""
    CREATE TABLE user_profiles (
        user_id STRING,
        plan_type STRING,
        signup_date STRING,
        PRIMARY KEY (user_id) NOT ENFORCED
    ) WITH (
        'connector' = 'jdbc',
        'url' = 'jdbc:postgresql://postgres:5432/users',
        'table-name' = 'user_profiles',
        'lookup.cache.max-rows' = '10000',
        'lookup.cache.ttl' = '1 hour'
    )
""")

# Enriched event stream
enriched_events = table_env.sql_query("""
    SELECT 
        e.user_id,
        e.event_type,
        e.event_time,
        e.page_url,
        e.device_type,
        p.plan_type,
        p.signup_date
    FROM user_events e
    LEFT JOIN user_profiles FOR SYSTEM_TIME AS OF e.event_time AS p
    ON e.user_id = p.user_id
""")

# Anomaly detection: Users with >100 events in 5 minutes
anomalies = table_env.sql_query("""
    SELECT
        user_id,
        window_start,
        COUNT(*) as event_count
    FROM TABLE(
        TUMBLE(TABLE user_events, DESCRIPTOR(event_time), INTERVAL '5' MINUTE)
    )
    GROUP BY window_start, user_id
    HAVING COUNT(*) > 100
""")

# Write to Elasticsearch for real-time dashboards
table_env.execute_sql("""
    CREATE TABLE es_events_per_minute (
        window_start TIMESTAMP(3),
        device_type STRING,
        event_count BIGINT,
        unique_users BIGINT,
        sessions BIGINT,
        PRIMARY KEY (window_start, device_type) NOT ENFORCED
    ) WITH (
        'connector' = 'elasticsearch-7',
        'hosts' = 'http://elasticsearch:9200',
        'index' = 'events-per-minute',
        'document-id.key-delimiter' = '-',
        'sink.flush-on-checkpoint' = 'true',
        'sink.bulk-flush.max-actions' = '1000',
        'sink.bulk-flush.interval' = '1s',
        'format' = 'json'
    )
""")

# Insert aggregated data into Elasticsearch
table_env.execute_sql("""
    INSERT INTO es_events_per_minute
    SELECT * FROM events_per_minute
""")

# Write anomalies to Kafka for alerting
table_env.execute_sql("""
    CREATE TABLE kafka_anomalies (
        user_id STRING,
        window_start TIMESTAMP(3),
        event_count BIGINT
    ) WITH (
        'connector' = 'kafka',
        'topic' = 'user-anomalies',
        'properties.bootstrap.servers' = 'kafka:9092',
        'format' = 'json'
    )
""")

table_env.execute_sql("""
    INSERT INTO kafka_anomalies
    SELECT * FROM anomalies
""")

# Execute all jobs
table_env.execute("Real-Time Analytics Pipeline")

State Management and Windowing

Stateful Stream Processing:

  • •Running aggregations (count, sum, avg over time windows), computed incrementally as events arrive instead of recomputed from scratch on every query.
  • •Session tracking (user activity within a session). The processor has to remember a user's prior events to know a session is still active.
  • •Pattern detection (sequence of events): matching "A then B then C" requires holding onto A and B until C arrives or the pattern times out.
  • •Stream joins (correlating events from multiple streams). One side of the join may arrive seconds or minutes before the other, so both sides must be buffered until they match.

Windowing Strategies:

  • •Use case: Metrics every 1 minute
  • •Example: "Events per minute by device type"
  • •Use case: Moving averages, trend detection
  • •Example: "Average response time over last 5 minutes, updated every 30 seconds"
  • •Use case: User session analytics
  • •Example: "User activity within a session (15-min inactivity timeout)"
  • •Use case: Custom aggregation logic
  • •Example: "Alert after 10 failed login attempts"

State Backend Options:

  • •Disk-based, scales to terabytes
  • •Supports incremental checkpoints
  • •Slightly higher latency than in-memory
  • •Faster access (<1ms)
  • •Limited by JVM memory
  • •Use for small state (<1GB)
  • •Auto-cleanup old state
  • •Reduces memory footprint
  • •Example: Keep only last 7 days of user activity

State spills from fast memory to RocksDB to durable S3 as it ages.

Production Considerations

Scalability:

  • •Increase Kafka partitions and Flink parallelism together. Flink can't parallelize past the partition count, so scaling one without the other caps throughput.
  • •Rule of thumb: 1 Flink task per Kafka partition, which keeps consumer assignment even and avoids idle parallelism.
  • •Example: 12 partitions = 12 Flink parallel instances, the max useful parallelism for that topic.
  • •Monitor task queue sizes. A growing queue is the earliest signal that processing can't keep up with ingestion, before latency alerts fire.
  • •Add more Flink task managers if backpressure detected, which is usually the fix if the bottleneck is compute, not a downstream sink.
  • •Consider sampling if processing can't keep up, since delivering approximate results for 100% of events sometimes beats exact results for a shrinking fraction.
  • •CPU-bound: Increase parallelism. More task slots split the compute-heavy work across more cores.
  • •Memory-bound: Increase heap size or use RocksDB. RocksDB trades some latency for state that no longer has to fit in JVM heap.
  • •Network-bound: Optimize serialization (Avro, Protobuf). Compact binary formats cut the bytes shuffled between operators, which is often the real bottleneck at high throughput.

Fault Tolerance:

  • •Frequency: Every 1-5 minutes. More frequent checkpoints shrink the replay window after a failure but add overhead to every checkpoint cycle.
  • •Incremental checkpoints for large state (RocksDB): only the delta since the last checkpoint is written, so checkpoint time doesn't grow linearly with state size.
  • •Store checkpoints in S3/HDFS (durable storage), so a full cluster failure doesn't also mean losing the recovery point.
  • •Requires idempotent sinks or transactional writes, since Flink's internal exactly-once guarantee doesn't extend past the sink unless the sink cooperates.
  • •Kafka sink: Use transactional producer, which makes writes atomic with the checkpoint so a replay can't produce duplicate messages downstream.
  • •Database sink: Use upserts with event ID so a replayed event overwrites the same row instead of inserting a duplicate.

Monitoring & Alerting:

  • •Throughput: Events/second processed
  • •Latency: End-to-end processing time (p50, p95, p99)
  • •Backpressure: Task buffer utilization
  • •Checkpoint Duration: Time to complete checkpoint
  • •State Size: Growth over time
  • •Backpressure > 80% for 5 minutes
  • •Checkpoint failure
  • •Job restart
  • •Latency p99 > 5 seconds
  • •Consumer lag > 1 million messages

Cost Optimization:

  • •Use spot instances for non-critical jobs (50-70% savings), acceptable for jobs that checkpoint frequently and can tolerate a task manager being reclaimed.
  • •Right-size task managers (4-8 cores typical). Oversized task managers waste spend, undersized ones create backpressure under load.
  • •Auto-scale based on consumer lag, since lag is a more honest signal of load than CPU utilization alone.
  • •Compress checkpoints (Snappy, ZSTD). This cuts both storage cost and the time spent writing and reading checkpoints during recovery.
  • •Use S3 Intelligent-Tiering for old checkpoints, which are rarely accessed after a job has run stably for a while.
  • •Clean up old savepoints. Savepoints accumulate indefinitely unless pruned, quietly growing the storage bill.
  • •Co-locate Kafka and Flink in same AZ. Cross-AZ traffic carries both a latency and a per-GB cost penalty at this volume.
  • •Use compression for Kafka messages to cut both broker storage and the bytes shuffled across the network per event.
  • •Batch small messages, since per-message overhead dominates cost and latency when messages are tiny.
  • •Traffic: 1M events/second
  • •Infrastructure: 10 Kafka brokers, 20 Flink task managers
  • •Latency: p99 < 500ms
  • •Cost: $15K/month (AWS)
  • •Availability: 99.9%

## Conclusion Real-time analytics streaming lets organizations act on data as it arrives instead of hours later: detecting fraud, updating recommendations, and catching incidents while they still matter. Production pipelines built on Apache Flink and Kafka give you the scalability, fault tolerance, and low latency that mission-critical workloads need. The key principles for streaming success: - Event-time processing with proper watermarking handles late-arriving data correctly - Stateful computations with windowing enable complex analytics like aggregations, pattern detection, and stream joins - Exactly-once semantics ensure data accuracy even with failures and restarts - Horizontal scalability through partitioning allows processing millions of events per second The difference between a proof-of-concept and a production streaming system lies in operational maturity: proper checkpointing for fault tolerance, comprehensive monitoring for visibility, and careful resource allocation for cost efficiency. A well-architected streaming platform can process petabytes of data daily at sub-second latency. At Bayseian, we've built real-time analytics pipelines for clients processing billions of events daily, powering fraud detection systems, recommendation engines, and operational dashboards. Our approach emphasizes proper windowing strategies, efficient state management, and comprehensive observability from day one. Whether you're building your first streaming pipeline or scaling an existing one, the patterns here give you a foundation to build on. Start with simple tumbling window aggregations and add complexity only as your use cases demand it. Ready to build production-grade streaming analytics? Contact us at contact@bayseian.com to discuss your real-time data architecture.

KafkaFlinkReal-timeStreamingAnalytics

Related Articles

Data Engineering

Optimizing Apache Spark on AWS EMR for Petabyte-Scale Data Processing

Deep dive into performance tuning Spark clusters on EMR, memory management, partitioning strategies, and cost reduction techniques for processing massive datasets.

Data Engineering

Building a Serverless Data Lake on AWS: S3, Athena, and Glue

Complete guide to architecting a cost-effective, scalable data lake using AWS services with automated ETL pipelines and real-time analytics capabilities.

Working on something like this?

No pitch, just a practical conversation with the team that builds and operates these systems in production.

Start a conversation