Scaling Kafka Streams on Kubernetes: How We Cut Costs by 80%

Hello, I'm Taewan Kim from the Data Engineering Cell in the Tech Division.

In this post, I'll share how we designed a scaling policy suited to Kafka consumer workloads while migrating Log Transformer, a Kafka Streams application that ran on AWS EC2, to Kubernetes (k8s), and what came of it.

Here's what this post covers:

• Why CPU-based HPA is a poor fit for Kafka consumer workloads
• Designing scaling policies around workload-specific metrics with KEDA and consumer lag
• Getting ahead of the midnight traffic spike with a cron trigger
• Running multiple threads per pod to reduce JVM overhead and rebalance events
• The result: more than 80% lower infrastructure costs compared with EC2

#Background: The Limits of Log Transformer on EC2

Log Transformer is a Kafka Streams application that reads raw logs from multiple Kafka topics, validates and transforms them, and then publishes the transformed logs to game-specific Kafka topics. Because it routes raw logs from multiple source topics into topics split by game and purpose, downstream pipelines can consume only the topics they need.

It ran on 72 c6g.2xlarge instances and had the following limitations.

• Instance counts that didn't follow the traffic pattern: Originally, we attached a cron-based schedule to the ASG (Auto Scaling Group) so that it switched between 36 and 72 instances depending on the time of day. This approach was unstable, though, so we ended up pinning it at 72, the peak capacity, and kept that size even during low-traffic hours.
• Rigid scaling: Keeping up with traffic changes meant someone had to change the ASG settings by hand each time.
• A complex deployment pipeline: Spinnaker-based EC2 deployments had many stages and took a long time to roll out, so even a simple configuration change was a burden.

All of these limits came down to the lack of autoscaling that reacts to traffic in real time and of fast, simple deployments. On k8s, autoscalers such as HPA and KEDA can react to traffic changes automatically, and declarative manifests and GitOps deployments reduce the burden of configuration changes. So we decided to move Log Transformer to k8s.

As the first step of the migration, we split the single application that consumed multiple source topics together into independent Kafka Streams applications, one per source topic. Each topic has a different traffic pattern, so we wanted to apply scaling policies independently and isolate failures and deployments per topic. With that split done, we got down to designing the scaling policy.

#First Attempt: Why CPU-Based HPA Failed

After moving to k8s, the first thing we tried was a basic CPU-based HPA (Horizontal Pod Autoscaler). The replica counts and thresholds in the YAML in this post are example values for illustration.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: log-transformer-hpa
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

The result wasn't what we expected. Within a consumer group, each partition is assigned to only one consumer (here, a pod), so load spreads evenly only when partitions are divided evenly among consumers. In practice, partitions weren't distributed evenly across consumers, so while some pods hit 100% CPU, the average CPU stayed below the target and HPA never reacted.

#Partitions That Don't Divide Evenly, and Concentrated Load

Kafka Streams groups partitions into tasks and distributes them with StreamsPartitionAssignor, handing out tasks in proportion to the number of stream threads each instance has. The problem comes next. If the number of tasks isn't divisible by the total number of threads, the leftover tasks have to go somewhere, so some consumers get one extra task each and take on that much more load.

This isn't a weakness of any particular assignment strategy. As long as the number of tasks isn't divisible by the total number of threads, the imbalance remains no matter which assignment strategy you use.

[Example: 6 partitions, 4 consumers]

6 / 4 = 1 remainder 2 → 2 consumers each take one extra partition.

Consumer A: partition 0, 1  → CPU 80%
Consumer B: partition 2, 3  → CPU 80%
Consumer C: partition 4     → CPU 40%
Consumer D: partition 5     → CPU 40%

Average CPU: 60%
HPA target: 70%
→ Scaling condition not met. Consumers A and B fall behind.

#The Assumption Behind Average CPU

Here, lag means the number of messages in a topic that consumers haven't processed yet. On consumers that were assigned more partitions, processing falls behind and lag builds up, and those pods' CPU shoots up to 100%. But with many pods, the average CPU stays at or below the target, so HPA never reacts.

Average CPU is a valid metric only on the assumption that the workload is spread evenly across all of an application's pods. That assumption doesn't hold for workloads like Kafka consumers, where load is tied to partitions.

#Solution: Scaling on Consumer Lag with KEDA

The core of the problem was that CPU didn't properly represent the real load of this workload. We needed to scale on lag, the metric that directly reflects the load. HPA does support external metrics, but that requires attaching and operating a separate metrics adapter that exposes Kafka lag. So we adopted KEDA (Kubernetes Event-Driven Autoscaling), which can use a wide range of event sources as scaling criteria.

#Adopting KEDA and Setting lagThreshold

KEDA's Kafka scaler queries a consumer group's lag directly and scales based on that value.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: log-transformer
spec:
  scaleTargetRef:
    name: log-transformer
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker:9092
        consumerGroup: log-transformer-group
        topic: log-topic
        lagThreshold: '100'

lagThreshold is the amount of lag one replica is expected to handle. KEDA scales replicas to the total lag divided by this value, and the replica count never exceeds the number of partitions.

#Benefits of Lag-Based Scaling

• Based on workload-specific metrics: Decisions are made on a metric that comes from the Kafka consumer workload itself rather than a general-purpose one.
• Reacts even when partition counts are unbalanced: Because it uses the total lag rather than individual consumers' CPU, scaling decisions still happen when partitions are distributed unevenly and the average CPU looks low. If traffic concentrates on a single partition, however, adding replicas won't increase that partition's throughput, and the partition design itself needs another look.
• Supports scale-to-zero: During hours when no data arrives on a source topic, the pod count can drop to zero, and pods start up to process messages once they begin to pile up again. We applied this to development-environment topics with intermittent traffic to eliminate idle costs, while production topics keep a minimum number of replicas to avoid processing delays.

#Refinement: The Midnight Traffic Spike and the Cron Trigger

After applying KEDA, the application ran reliably most of the time, but there was one window where lag-based scaling alone responded too late.

Many games reset features such as daily missions and daily check-ins at midnight (00:00), and at that moment related batch jobs and daily aggregation logs pour in all at once, causing a traffic spike. Consumer lag-based scaling reacts only after lag has built up, so it inevitably lags behind such a sharp traffic increase. On top of that, new pods only start processing after they become ready and the consumer group rebalance completes, plus JVM warm-up time. Until the scale-out finishes, lag keeps climbing, and log delivery to the downstream pipelines is delayed accordingly.

#Getting Ahead with a Cron Trigger

In addition to the Kafka scaler, KEDA supports a cron trigger. We solved the problem by increasing the pod count in advance, just before midnight.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: log-transformer
spec:
  scaleTargetRef:
    name: log-transformer
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker:9092
        consumerGroup: log-transformer-group
        topic: log-topic
        lagThreshold: '100'
    - type: cron
      metadata:
        timezone: Asia/Seoul
        start: '50 23 * * *' # scale out at 23:50
        end: '30 0 * * *' # end the cron trigger's active window at 00:30
        desiredReplicas: '8'

When the Kafka lag trigger and the cron trigger are used together, KEDA takes the higher replica count of the two. It scales out to 8 in advance at 23:50, absorbs the midnight traffic, and after 00:30 scales in naturally based on lag.

#Optimization: Fewer Rebalance Events Through Multiple Threads

With the scaling policy and the midnight response in place, we finally redesigned the pod configuration itself. A simple setup with one Kafka Streams thread per pod is easy to operate, but JVM startup overhead grows linearly with the number of pods, and pod lifecycle events (restarts, rolling deploys, evictions) directly trigger consumer group rebalances.

To reduce this, we configured each pod to run multiple Kafka Streams threads at once. Kafka Streams supports running multiple stream threads per instance (= pod) through the num.stream.threads setting, and each thread is independently assigned tasks (groups of partitions) to process (Kafka Streams Architecture). From the consumer group's point of view, each thread behaves like an independent consumer, so increasing the number of threads per pod leaves partition distribution and processing consistency unchanged. In other words, this change kept the total number of consumer threads the same and reduced only the number of pods, which let us lower the number of JVM instances, the memory overhead, and the absolute frequency of lifecycle events all at once.

However, as more threads are bundled into a single pod, the blast radius of a single pod failure grows. So we didn't increase threads per pod without limit, but capped it according to the CPU allocation rule below.

To avoid CPU contention between threads, we set the thread count at one CPU core per thread and sized the pod's CPU request/limit to match. For example, a pod running 6 threads is allocated 6 cores. Kafka Streams' capacity planning guide also recommends sizing the number of stream threads to the available CPU cores; we ran somewhat conservatively at one core per thread to make throughput predictable. This keeps per-pod throughput stable, and "a single pod's capacity to process the backlog," which lag-based scaling relies on, also converges to a predictable value.

#Results: Resources That Follow Traffic, and Lower Costs

#Cost Savings

Compared with running fixed EC2 instances, moving to k8s + KEDA cut infrastructure costs by more than 80%.

More important than the 80% figure itself is the change in how we operate. On EC2, we had to keep instances running at peak capacity at all times, but with KEDA the deployment shrinks to 2 pods overnight and scales out only at peaks, allocating just as many pods as the load at that moment requires. The 80% was measured under the current traffic pattern; the key point is that resources follow the traffic even as the pattern changes.

#Less Operational Burden

• Adding a game only requires adding values to the Helm chart.
• Scaling policies are managed in the Helm chart, so they can be reviewed and their history tracked.
• Deployments go through ArgoCD with just an image tag and environment variable change, which greatly reduced deployment preparation and rollout time compared with the previous Spinnaker-based EC2 pipeline.

#Remaining Work: Distributing Partitions Evenly Down to the Thread Level

Lag-based scaling works stably in the current setup, but the underlying problem of partitions not being distributed evenly across consumers remains.

KEDA's ensureEvenDistributionOfPartitions option addresses this by limiting the replica count to divisors of the total number of partitions. However, this option only checks partitions / replicas. In a setup where each pod runs multiple threads, the units that actually divide up the partitions are threads, not pods, so the condition that has to hold is whether partitions / (replicas × threads per pod) is an integer. Because we had already split deployments by source topic, each pod is dedicated to a single topic, so the number of threads per pod here is the number of threads processing that topic.

If 24 partitions are processed with 4 threads per pod, the option allows replica counts of 1, 2, 3, 4, 6, 8, 12, and 24 (the divisors of 24), but only counts of 1, 2, 3, and 6 distribute tasks evenly across threads. The other values show two symptoms.

• 4 replicas: Each pod gets an even 6 tasks, but 24 tasks are split across 16 threads in total, so 8 threads get 2 tasks and the other 8 get 1. Throughput between threads can differ by up to 2x.
• 8 replicas or more: The number of threads exceeds the number of tasks. With 8 replicas, 8 of the 32 threads, and with 24 replicas, 72 of the 96 threads, receive no tasks at all and just spin through empty poll loops.

The same goes for the replica upper bound. KEDA limits the replica count to at most the number of partitions, but this upper bound is also calculated without knowing the number of threads per pod. Ultimately, extending even distribution down to the thread level requires KEDA to know the number of threads per pod when deciding replicas.

To fill this gap, we opened an issue proposing a new option in the KEDA repository, and a PR implementing an option named threadsPerConsumer is now under review. If you set this value to the number of threads per pod together with ensureEvenDistributionOfPartitions, KEDA picks replicas from the divisors of partitions / threadsPerConsumer rather than of partitions. In the earlier example, only the divisors of 6 instead of 24, that is 1, 2, 3, and 6, become candidates, guaranteeing even distribution at both the pod and the thread level.

More fundamentally, we're also writing a component in Rust that will replace Log Transformer, written in Scala, and take on an expanded role. We expect lower JVM startup overhead and GC latency, reduced CPU and memory usage, and higher throughput. We'll share the results in a separate post once they're in, along with the progress of the threadsPerConsumer option.

#Closing: Scaling Metrics That Fit the Workload

What this migration reconfirmed for us comes down to one thing.

Choose scaling metrics to match the load characteristics of your workload.

CPU utilization is a perfectly valid metric for workloads whose load spreads evenly across pods. But for workloads like Kafka consumers, where load is tied to partitions, average CPU alone made it hard to catch real processing delays. By using consumer lag as the criterion, we were able to make our scaling more accurate.

Beyond Kafka, KEDA provides scalers for a wide range of event sources, including RabbitMQ, AWS SQS, and Prometheus metrics. Being able to easily bring in metrics other than CPU as scaling criteria was its biggest advantage for us.

If you're wrestling with similar questions about scaling Kafka consumers, we recommend looking into consumer lag-based scaling.

Thank you.

Devsisters is hiring.

Visit our careers site for details.

View openings