Building Real-Time Data Pipelines with Kafka and Spark Structured Streaming

Why Real-Time Matters

Batch processing served enterprises well for decades, but modern business demands — fraud detection, live dashboards, personalization engines, and IoT analytics — require data freshness measured in seconds, not hours. Apache Kafka combined with Spark Structured Streaming provides a proven, scalable architecture for real-time data pipelines.

Architecture Overview

The reference architecture consists of three layers. The ingestion layer uses Kafka producers to capture events from applications, databases (via CDC with Debezium), and IoT devices. The processing layer uses Spark Structured Streaming to consume from Kafka topics, apply transformations, aggregations, and enrichments. The serving layer writes processed data to low-latency stores like Apache Druid, ClickHouse, or Elasticsearch for real-time querying.

Kafka Design Patterns

Proper Kafka topic design is foundational to a successful real-time pipeline:

• Use a consistent naming convention: domain.entity.version (e.g., payments.transactions.v1)
• Set partition counts based on target throughput — each partition handles roughly 10MB/s
• Use Avro or Protobuf with a Schema Registry to enforce data contracts
• Implement dead-letter queues for messages that fail processing

Spark Structured Streaming Best Practices

Structured Streaming provides exactly-once semantics with proper checkpointing. Configure your streams with trigger intervals that balance latency against throughput — trigger(processingTime="10 seconds") works for most use cases. Use watermarks to handle late-arriving data gracefully, and leverage foreachBatch for complex sink logic that needs transactional guarantees.

Handling Schema Evolution

Real-time systems must handle schema changes without downtime. Use a Schema Registry (Confluent or AWS Glue) to manage schema versions. Design your Spark jobs to handle both old and new schema versions simultaneously during transition periods. Forward-compatible schemas (adding optional fields) are safer than backward-incompatible changes.

Monitoring and Observability

Real-time pipelines require real-time monitoring. Track Kafka consumer lag to detect processing bottlenecks, Spark streaming batch durations to identify slow micro-batches, end-to-end latency from event production to serving layer availability, and data quality metrics including null rates, schema violations, and duplicate counts. Alert on consumer lag exceeding your SLA threshold — this is the single most important metric for real-time pipeline health.

When Not to Go Real-Time

Real-time adds complexity and cost. If your business can tolerate 15-minute delays, consider micro-batch processing instead. If your data volumes are modest, a simple Kafka-to-database CDC pipeline without Spark may suffice. Choose the simplest architecture that meets your latency requirements.