Devsisters Tech Blog

We Fixed the Definitions, Not the Queries: Adopting Databricks Metric Views

Fri, 18 Sep 2026 00:00:00 GMT

Hello, I'm Hyunki Park from the Data Intelligence Cell in the Tech Division. In this post, I'll share how we introduced Databricks Unity Catalog's Metric Views into the data pipeline of a newly launched game: how we organized our data into layers so that dashboards and analytics agents read the same metric definitions, the design rules we settled on along the way, and the results.

#Background: Metrics That Were Calculated Differently in Every Query

The period right after a game launches is when data requests are at their most numerous and most urgent. There were many sources — snapshots of the game server DB, analytics logs emitted by the client and server, metadata tables — and the conditions were complicated too: which users to exclude from metrics, how to handle games that included bots, which point-in-time values to use. Meanwhile, as many dashboards were built in a short period and analysis requests poured in, each dashboard, and each ad hoc analysis query, was defining these conditions on its own. Each dashboard's dataset computed the aggregations it needed inside its own query. This was a good way to build quickly, but after a few weeks three problems surfaced.

The same metric was defined with a different formula in each dataset. Take win rate alone: the condition for identifying games with a decided outcome differed from dataset to dataset, and the criteria for cleansing abnormal data (excluding operator accounts, excluding abnormal users) were broad in some places and narrow in others. Changing a definition meant editing many places, and the moment we missed one, the numbers across charts stopped matching.
The cleansing cost before aggregation was paid again every time a dashboard was opened. Game logs often record multiple participants and items for a single event together in nested fields. To calculate a metric, we had to explode these fields into rows, join data from different sources, and filter out duplicates. Because this ran again with every dashboard dataset query, the cleansing query in front of the metric often cost more money and time than the metric query itself.
The analytics agent defined metrics on its own every time. When we asked our in-house analytics agent for "the top 10 cookies by win rate last week," it answered by reading uncleansed log data. In the process, the agent decided on a variety of criteria by itself — whether to exclude churned users, whether to set a minimum sample size, which criteria to cleanse by, and so on. We did write these conditions down in the guide documents (skills) the agent reads. But as the conditions attached to each metric grew, the skills got longer, every new condition meant editing a skill, and even then we couldn't fully prevent the tables and conditions from varying slightly from request to request.

The first and second problems could be solved to some extent with human time and effort, but the third could not be solved with our existing data environment alone.

#What Is a Metric View?

A Metric View is a semantic layer feature provided by Databricks Unity Catalog. Its purpose is to store metric definitions in the catalog as view objects, so that dashboards, SQL, and agents alike query that view and read the same values.

A semantic layer is a layer where you give names to metrics such as "win rate" or "matchmaking success rate" and define in one place what data each metric aggregates and how. Consumers don't write the aggregation expressions themselves; they simply select the defined metric fields. It's a long-established concept, seen in forms like BI tool data models, dbt's Semantic Layer, and LookML, and its purpose is to have everyone read the same value from a single definition instead of recalculating the same metric in each place it's consumed.

Databricks' Metric View is an implementation of this concept inside Unity Catalog. Once you define measures and dimensions in YAML on top of a table, consumers call metrics with the measure() function and can read the same definition at any granularity just by changing the group by. With a regular view, the aggregation and grouping granularity are fixed when the view is created; with a Metric View, only the definition is fixed, and the granularity is decided at query time.

Let's look at the difference between the two using win rate as an example. With a regular view, you have to decide the aggregation granularity when you create the view.

create view win_rate_daily_by_mode as
select base_date, mode_id,
  count(*) filter (where is_win) * 100.0 / count(*) as win_rate
from battle_play
group by base_date, mode_id

This view can only answer daily, per-mode win rates. If you need win rate per cookie, you have to create another view with a different group by, and if you take the avg() of this view's daily win rates to get a monthly win rate, days with many plays and days with few plays are weighted equally, giving you the wrong value.

A Metric View, on the other hand, writes the same definition split into dimensions and measures.

# Metric view definition
version: 1.1
source: battle_play
dimensions:
  - name: base_date
    expr: base_date
  - name: mode_id
    expr: mode_id
  - name: cookie_id
    expr: cookie_id
measures:
  - name: win_rate
    expr: count(*) filter (where is_win) * 100.0 / count(*)

-- Example metric view queries
-- Daily, per mode
select base_date, mode_id, measure(win_rate) as win_rate
from win_rate_metric_view
group by base_date, mode_id

-- Monthly
select date_trunc('month', base_date) as base_month, measure(win_rate) as win_rate
from win_rate_metric_view
group by date_trunc('month', base_date)

The win rate definition lives in one place in YAML, and as the example above shows, you can run queries with different GROUP BY clauses. With the earlier daily view, you couldn't use that view for a monthly query, but with a Metric View, even a monthly query doesn't average the daily win rates; it recalculates the win rate from all rows in that month, so the value is accurate.

As mentioned earlier, the semantic layer concept existed before, but it usually lived inside BI tools. As a result, a definition only worked within that tool, and to see the same metric in another tool, in SQL, or in a notebook, you had to define it again. A Metric View, by contrast, lives not in a BI tool but in the data catalog (Databricks' Unity Catalog), so any tool can query the same definition via SQL, and who can see the metric is managed in the catalog the same way as for any other table. That's why dashboards, SQL, notebooks, external BI tools, and agents can all read the same definition. In fact, in its Metric View launch announcement, Databricks puts particular emphasis on that last consumer, as shown below.

Analysts, engineers, executives and now AI agents often interpret the same data differently, resulting in metric drift, conflicting reports and declining trust. In the era of agentic AI, where agents reason over data and act autonomously, fragmented definitions do not just create confusion, they scale it.

Source: Databricks, "Announcing General Availability and Open Sourcing of Unity Catalog Business Semantics" (2026)

That last consumer is also the reason we wanted to use Metric Views. If dashboard performance had been the only goal, a data mart would have been enough. But when the analytics agent needed to analyze data along a wider variety of dimensions than the groupings used in dashboards, it had to query data at a lower layer than the data mart, and when it did, it read uncleansed log data and defined metrics on its own, producing values that diverged from those shown on the dashboards. The line from Databricks' launch announcement — fragmented definitions "scale" confusion — was playing out exactly in our environment. So we needed a layer from which the agent could read definitions, and that layer had to share the same definitions as the dashboards we were using.

#Design: Definitions in One Layer, Cleansing Beneath It

At Devsisters, we had been managing our data warehouse in layers split into Bronze, Silver, and Gold, following Databricks' Medallion Architecture. So where should Metric Views sit?

In terms of role, a Metric View is a consumption layer like Gold, but it differs in that it holds only definitions without fixing the aggregation granularity in advance. The source data for a Metric View could also be the Bronze layer, but as I mentioned earlier, we didn't want to pay processing costs such as handling nested fields over and over at query time. So we decided to put fully processed data in Silver and use Metric Views in a restricted way: like the Gold layer, they query data in Silver.

So we made Metric Views views that read Silver, not tables, classified them as a separate layer named Semantic, and placed it on top of Silver, alongside Gold. In the end, we defined each layer's role to fit our data structure as follows.

Layer	Role
Bronze	Raw, as is. Never modified.
Silver	Data integrity. Explodes nested fields, joins records left in different sources, removes duplicates, and attaches cleansing flags. Every value that measure logic reads is produced here. No metrics are calculated here.
Gold	Consumption optimization. Physical tables precomputed at a fixed aggregation granularity. Built only when metrics that can't be served from a pre-aggregated cache, such as unique user counts or quantiles, need to be viewed quickly at a fixed granularity; all other metrics go in Metric Views.
Semantic (Metric View)	Business meaning. Defines measure formulas and dimensions in one place.

#Reducing Query Cost by Reading Pre-Aggregated Results

A regular view re-aggregates from the source table every time it's queried. When Metric Views first launched, they also stored only the definition and not the results, so their default behavior was the same. Since every query read the definition and re-aggregated, we expected that dashboards repeatedly opening the same results would need separate, precomputed Gold tables. However, the materialization feature added later resolved most of this concern. If you specify frequently used dimension combinations, Databricks precomputes and stores the aggregated results for those combinations. I'll call what is stored this way an aggregated materialization. When a consumer queries with that combination, it reads the stored results instead of recomputing the definition. By registering the combinations our dashboards use here, most dashboards rendered fast enough.

That said, this cache doesn't work for every measure. Just as summing daily play counts gives you monthly play counts, it can only be used for measures whose pre-aggregated values remain correct when combined again. Unique user counts and quantiles don't yield monthly values when you sum daily values, so such measures are recomputed at query time from an unaggregated materialization, which stores all the individual, unaggregated rows. As a result, they get slower the longer the query period, so we built Gold tables only for datasets where such metrics need to be viewed quickly and frequently at a fixed granularity. We'll come back to this distinction under the name additivity in the Rules section, and to the criteria for building Gold tables in the Decision section.

#Rules: Distinguishing Additive from Non-Additive

The thing we had to be most careful about when designing Metric Views was the additivity of measures. An additive measure stays correct when you aggregate values from a finer granularity to a coarser granularity by summing them: summing daily play counts gives you the monthly play count. A non-additive measure becomes wrong if you do that. Summing daily unique user counts double-counts users who logged in on multiple days, so the result exceeds the monthly unique user count, and averaging daily medians doesn't give you the monthly median. This distinction determined whether a measure could be pre-aggregated, whether we had to note a summation warning in the measure's comment, and whether consumers could combine results again.

Type	Example metrics	Example formulas	Can it be summed?	Can it be pre-aggregated?
Additive	Play count, win count, total wait time	`count()`, `sum(...)`, `count() filter (where ...)`	Safe to sum at any granularity	Yes. Goes in the aggregated materialization
Non-additive	Unique user count, median wait time	`count(distinct ...)`, `percentile_approx(...)`	Wrong if summed. Must be recomputed from individual rows	No. Computed every time from the unaggregated materialization

#Define Averages as a Sum and a Count

As long as it's computed at query time, an average defined with avg() is accurate at any granularity. The problem arises when reusing pre-aggregated results. The aggregated cache in a materialization combines pre-aggregated values into a coarser granularity. But averages can't be combined. If you average two daily averages, days with many samples and days with few are weighted equally, and you get a value different from the monthly average. So avg() measures cannot use the aggregated cache and are recomputed from individual rows on every query. Consumers who take daily averages and average them again outside run into the same problem.

Sums and counts, on the other hand, can each be combined. So we defined an average as a sum measure and a count measure, and added a separate composed measure that divides the sum by the count. Combining the sum and count into a coarser granularity in the cache and then dividing gives an accurate average, and whatever granularity a consumer groups by, the division is redone at that granularity.

measures:
  - name: wait_time_seconds_sum
    expr: sum(case when is_success = 'Y' then wait_time_seconds end)
    comment: Total wait time (seconds) — successful matches
  - name: wait_sample_count
    expr: count(*) filter (where is_success = 'Y')
    comment: Wait time sample count — successful matches
  - name: wait_time_seconds_avg
    expr: measure(wait_time_seconds_sum) / nullif(measure(wait_sample_count), 0)
    comment: Average wait time (seconds)
  - name: wait_time_seconds_p50
    expr: percentile_approx(case when is_success = 'Y' then wait_time_seconds end, 0.50)
    comment: Median wait time (seconds) — successful matches [non-additive]

#Use Window Measures for Shares

For a metric like cookie pick rate — "this cookie's play count ÷ total play count for the mode" — the denominator sums over a wider scope than the numerator. Databricks provides window measures for this. If you exclude the cookie axis from the window definition of the denominator measure, the denominator is computed ignoring cookies no matter what granularity the consumer groups by.

dimensions:
  - name: mode_id
    expr: mode_id
  - name: cookie_id
    expr: cookie_id
measures:
  - name: mode_total_play_count
    expr: count(*)
    window:
      - order: cookie_id
        range: all
        semiadditive: last
    comment: Total play count for the mode — denominator ignoring cookies [LOD:exclude=cookie_id]
  - name: pick_ratio
    expr: count(*) * 100.0 / nullif(measure(mode_total_play_count), 0)
    comment: Cookie pick rate (%) — relative to the mode total [LOD:exclude=cookie_id]

However, if the dimension excluded from the denominator (here, cookie) isn't in the consuming query's group by, the numerator and denominator become equal and the result is silently 100% with no error, so we note that axis in the comment.

#What Goes in a Comment

We decided to write only three things in measure comments: what is being counted, population conditions not evident from the name, and a tag indicating whether it's safe to combine. We don't write the formula, because it's already in the definition itself and repeating it would be redundant, and because a longer comment makes it easier to miss the warnings that actually matter.

The reason we note "whether it's safe to combine" separately is that, unfortunately, Metric Views currently have no mechanism to prevent aggregations that shouldn't be combined. You can group by any defined dimension, and nothing checks whether that combination makes sense for the metric. If you group a unique game count, counted per game, by participants' skill tiers, a single game is counted in multiple tiers and the total exceeds the overall count — yet no error is raised, and you get a plausible-looking number. dbt's Semantic Layer lets you declare measures that become wrong when summed along the time axis with non_additive_dimension, and the engine blocks it. As of YAML spec 1.1, Metric Views don't offer such a declaration for regular measures. So we record this constraint in the comment. People and agents both write queries after reading comments, but this information matters especially for agents. A person can ask the author when the meaning is ambiguous, but an agent writes a query right away based only on the names, types, comments, and formulas of dimensions and measures. The formula alone doesn't tell you that summing a count(distinct ...) by month is wrong, so if it's in the comment, the agent re-groups over the monthly range, and if it isn't, the agent just makes its own call.

We used two tags in comments. [non-additive] means that combining aggregated values again gives the wrong result, and [LOD:exclude=<dim>] means that leaving the axis named in <dim> out of the group by gives the wrong result. We don't tag additive measures or ratios built from them. The absence of a tag is itself meant to signal "safe to combine." We put the same tag descriptions in the guide the analytics agent reads, so that the side building Metric Views and the side querying them follow the same rules.

#Decision: Gold Table or Metric View?

Once you introduce Metric Views, the question naturally follows: "If we're pre-aggregating anyway, how is this different from a Gold table, and which should we build?" We got this question for every metric too, and the criteria we used are summarized below.

Criterion	Gold table	Metric View
Query speed	Fastest, since it's already aggregated at a fixed granularity	Similar if an aggregated materialization matches; otherwise re-aggregated from the unaggregated materialization
Dimension flexibility	Can only be viewed at the granularity chosen at build time. A new axis requires a new table	Can be viewed in any combination within the defined dimensions
Non-additive metrics (distinct, quantiles)	Accurate only at the fixed granularity. Can't be re-aggregated to another granularity	Recomputed from individual rows whatever the group by
Consumers	Suited to queries at the same aggregation granularity. Multiple dashboards can share it if the granularity is the same	Suited to cases where multiple dashboards, analysts' ad hoc queries, and agents share the same definition regardless of granularity
Cost of adding a metric	Adding a column means reloading that column for the full history, and a new granularity requires a new table	Adding a measure to the YAML makes it usable immediately without reloading. Changing a dimension recomputes the entire cache
Relationship with agents	Columns hold precomputed values, so formulas must be written separately in the agent guide	The agent reads the Metric View's definition (formulas, comments) directly, so there's no need to copy formulas into the guide

As the table shows, metrics whose breakdown dimensions aren't decided in advance, and non-additive metrics that must be recomputed depending on granularity, are a fit for Metric Views. Conversely, for tables with a fixed aggregation granularity where speed matters most, a Gold table is simpler and faster. Looking at dashboards alone, Gold tables are sufficient for most charts. The reason we chose Metric Views anyway was the agent. The agent calls metrics with a different combination of dimensions for every question, so Gold tables with fixed granularity couldn't prepare for that in advance.

This also gives us our criterion for building Gold tables. The default is a Metric View on top of Silver. Only when a query that needs speed includes non-additive measures and computing it from the unaggregated materialization isn't fast enough do we turn that query into a Gold table with a fixed granularity. Queries that use only additive measures are covered well enough by the aggregated materialization, so they don't need Gold tables.

#Results: Comparing Costs Before and After the Migration

After introducing Metric Views, cases of the same metric being calculated with different definitions decreased. But did costs also go down compared to when we read logs directly? Since we had moved our dashboards to Metric Views as decided in the previous section, we compared before and after the move. We ran the original queries and the queries migrated to Metric Views once each, cold, with the same parameters, and compared compute and scan volume. For the heaviest dashboard, summing across its datasets, a single run showed a 41% reduction in compute and a 6.7x reduction in scan volume. The magnitude varied by dataset, though. Where log cleansing had been expensive, compute fell to as little as 30% of its previous level; where queries were already light, it was around 80%.

What about the agent? After restructuring the domain knowledge our in-house analytics agent references around Metric Views, we compared before and after using the same set of questions. The tokens and cost spent producing answers dropped by about half, and the number of turns taken to reach an answer dropped by 30%.

The "top 10 cookies by win rate" question mentioned earlier is a representative case. When reading logs directly, the agent first searched for which tables to use, decided on cleansing criteria and win conditions by itself, wrote a long query, and then, suspicious of the results, checked once more. After switching to Metric Views, it finished with a single call to a defined measure. Both answers got the order of the top cookies right, but in the answer from reading logs directly, the agent had chosen the cleansing criteria and win conditions, so the win rate numbers differed slightly from those shown on the dashboard; with Metric Views, the agent produced the same values as the dashboard.

#Follow-Up: What We Learned Operating Metric Views

Of the things we learned while revising Metric Views over the three months after deployment, here I'll summarize only those related to how Metric Views behave.

#If a Filter Consumers Always Apply Isn't in the Cache, the Cache Isn't Used

Nearly every dashboard dataset applied the normal-user flag in a where clause. However, that flag was missing from the aggregated materialization's dimension list. Databricks' query rewrite considers a materialization as a candidate only if it contains the dimensions used not just in the group by but also in the where. When we checked with explain extended, the pre-aggregated cache wasn't being applied, and everything was going to the unaggregated materialization. We added a rule to include filter dimensions that consumers always apply in the aggregated materialization.

#Changing a Dimension Rebuilds the Entire Cache

When the definition changes, a materialization recomputes the full history. Editing only measures doesn't trigger a recompute, but adding or changing a dimension recomputes the full history, so adding a single dimension costs more than you might think. This is because there's no concept of reloading by date as in a typical data mart. As a result, whether the source is a query that can recompute the full history became a design requirement, and when adding a dimension, we came to first weigh whether that axis is really necessary and what a full recompute would cost.

#Non-Additive Metrics Require a Cache of Individual Rows to Be Queryable

Unique user counts and quantiles must be recomputed from individual rows whatever the group by, so an aggregated cache can't cover them. That's why an unaggregated materialization that stores all individual rows is always needed; a setup with only aggregated materializations and no unaggregated one doesn't work. Any query that calls even one non-additive measure is computed entirely from the unaggregated materialization. So on dashboards, we separated frequently viewed non-additive metrics into different charts from additive ones. Dashboards can be split this way because which measures they call in which combinations is decided in advance, but the agent calls different measures for every question, so we couldn't use the same approach there.

#Embedding a Filter in a Measure Removes Consumer Choice

We embedded the condition "only successful matches with no bots" in the wait time measure. As a result, during periods with a high share of bot matches, we couldn't see wait times including bots. But pulling every condition out into dimensions would scatter the metric definition, so we decided to keep conditions that belong to the metric definition (successful matches only) inside the measure, and expose conditions that consumers may want to change depending on the situation (whether to include bots) as dimensions.

#Remaining Work

Once we finished introducing Metric Views, we found we had far too many measures. There are several measures with similar names, and for some, you can only tell which one to use by opening the definition. Early in the migration, we took the position that every number on a dashboard should be retrievable directly from a Metric View, and used the query results of the existing dashboards as the baseline. Every column in the results had to become a dimension or a measure, and our completion criterion was that values before and after the migration matched column by column. This let us migrate quickly and verify immediately, but as a result, the shape of our Metric Views simply followed the shape of the dashboard results. Starting from a single win rate, we got a new measure for every added condition: "win rate excluding bots," "win rate for a specific mode," "win rate based on first place." Anything derived from other measures, such as an index computed by dividing two measures, also became a measure.

In hindsight, it probably would have been better to keep these derived metrics outside the Metric View. A conditional win rate, after all, can be obtained by applying a filter to the win count and play count. So we plan to clean this up so that Metric Views keep only base measures that serve as building blocks for other metrics, and non-additive measures that must be recomputed depending on granularity.

One more thing: the rules we set in this work (additivity checks, comment tags, materialization placement, verification procedures) are hard for people to remember every time. So we're working on moving the process of creating and editing Metric Views itself into a skill for our coding agent. The goal is to mechanically catch YAML that violates the rules above, and to explicitly request review for items that require human judgment, such as the intent behind a metric definition. If the rules don't just live in a document but are checked automatically by a skill, we hope we can expect the same quality when applying the structure in this post to other games.

#Closing

When metric definitions are written separately in dashboard queries, in analysis queries, and in the skill documents agents read, there's no way to stop them from drifting apart little by little. So we gathered our metric definitions into Metric Views and had both dashboards and agents use those definitions as is. The cases where the numbers on a dashboard and the numbers an agent answered with didn't match, forcing us to check which one was right, dropped noticeably. Consolidating definitions was the goal, but by no longer repeating cleansing and aggregation every time, we were able to cut query costs as well.

If you're running into the problem of the same metric returning different values depending on where it's queried, I'd recommend first looking at how to consolidate your metric definitions in one place before fixing each query. Thank you.

Scaling Kafka Streams on Kubernetes: How We Cut Costs by 80%

Thu, 03 Sep 2026 00:00:00 GMT

Hello, I'm Taewan Kim from the Data Engineering Cell in the Tech Division.

In this post, I'll share how we designed a scaling policy suited to Kafka consumer workloads while migrating Log Transformer, a Kafka Streams application that ran on AWS EC2, to Kubernetes (k8s), and what came of it.

Here's what this post covers:

Why CPU-based HPA is a poor fit for Kafka consumer workloads
Designing scaling policies around workload-specific metrics with KEDA and consumer lag
Getting ahead of the midnight traffic spike with a cron trigger
Running multiple threads per pod to reduce JVM overhead and rebalance events
The result: more than 80% lower infrastructure costs compared with EC2

#Background: The Limits of Log Transformer on EC2

Log Transformer is a Kafka Streams application that reads raw logs from multiple Kafka topics, validates and transforms them, and then publishes the transformed logs to game-specific Kafka topics. Because it routes raw logs from multiple source topics into topics split by game and purpose, downstream pipelines can consume only the topics they need.

It ran on 72 c6g.2xlarge instances and had the following limitations.

Instance counts that didn't follow the traffic pattern: Originally, we attached a cron-based schedule to the ASG (Auto Scaling Group) so that it switched between 36 and 72 instances depending on the time of day. This approach was unstable, though, so we ended up pinning it at 72, the peak capacity, and kept that size even during low-traffic hours.
Rigid scaling: Keeping up with traffic changes meant someone had to change the ASG settings by hand each time.
A complex deployment pipeline: Spinnaker-based EC2 deployments had many stages and took a long time to roll out, so even a simple configuration change was a burden.

All of these limits came down to the lack of autoscaling that reacts to traffic in real time and of fast, simple deployments. On k8s, autoscalers such as HPA and KEDA can react to traffic changes automatically, and declarative manifests and GitOps deployments reduce the burden of configuration changes. So we decided to move Log Transformer to k8s.

As the first step of the migration, we split the single application that consumed multiple source topics together into independent Kafka Streams applications, one per source topic. Each topic has a different traffic pattern, so we wanted to apply scaling policies independently and isolate failures and deployments per topic. With that split done, we got down to designing the scaling policy.

#First Attempt: Why CPU-Based HPA Failed

After moving to k8s, the first thing we tried was a basic CPU-based HPA (Horizontal Pod Autoscaler). The replica counts and thresholds in the YAML in this post are example values for illustration.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: log-transformer-hpa
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

The result wasn't what we expected. Within a consumer group, each partition is assigned to only one consumer (here, a pod), so load spreads evenly only when partitions are divided evenly among consumers. In practice, partitions weren't distributed evenly across consumers, so while some pods hit 100% CPU, the average CPU stayed below the target and HPA never reacted.

#Partitions That Don't Divide Evenly, and Concentrated Load

Kafka Streams groups partitions into tasks and distributes them with StreamsPartitionAssignor, handing out tasks in proportion to the number of stream threads each instance has. The problem comes next. If the number of tasks isn't divisible by the total number of threads, the leftover tasks have to go somewhere, so some consumers get one extra task each and take on that much more load.

This isn't a weakness of any particular assignment strategy. As long as the number of tasks isn't divisible by the total number of threads, the imbalance remains no matter which assignment strategy you use.

[Example: 6 partitions, 4 consumers]

6 / 4 = 1 remainder 2 → 2 consumers each take one extra partition.

Consumer A: partition 0, 1  → CPU 80%
Consumer B: partition 2, 3  → CPU 80%
Consumer C: partition 4     → CPU 40%
Consumer D: partition 5     → CPU 40%

Average CPU: 60%
HPA target: 70%
→ Scaling condition not met. Consumers A and B fall behind.

#The Assumption Behind Average CPU

Here, lag means the number of messages in a topic that consumers haven't processed yet. On consumers that were assigned more partitions, processing falls behind and lag builds up, and those pods' CPU shoots up to 100%. But with many pods, the average CPU stays at or below the target, so HPA never reacts.

Average CPU is a valid metric only on the assumption that the workload is spread evenly across all of an application's pods. That assumption doesn't hold for workloads like Kafka consumers, where load is tied to partitions.

#Solution: Scaling on Consumer Lag with KEDA

The core of the problem was that CPU didn't properly represent the real load of this workload. We needed to scale on lag, the metric that directly reflects the load. HPA does support external metrics, but that requires attaching and operating a separate metrics adapter that exposes Kafka lag. So we adopted KEDA (Kubernetes Event-Driven Autoscaling), which can use a wide range of event sources as scaling criteria.

#Adopting KEDA and Setting lagThreshold

KEDA's Kafka scaler queries a consumer group's lag directly and scales based on that value.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: log-transformer
spec:
  scaleTargetRef:
    name: log-transformer
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker:9092
        consumerGroup: log-transformer-group
        topic: log-topic
        lagThreshold: '100'

lagThreshold is the amount of lag one replica is expected to handle. KEDA scales replicas to the total lag divided by this value, and the replica count never exceeds the number of partitions.

#Benefits of Lag-Based Scaling

Based on workload-specific metrics: Decisions are made on a metric that comes from the Kafka consumer workload itself rather than a general-purpose one.
Reacts even when partition counts are unbalanced: Because it uses the total lag rather than individual consumers' CPU, scaling decisions still happen when partitions are distributed unevenly and the average CPU looks low. If traffic concentrates on a single partition, however, adding replicas won't increase that partition's throughput, and the partition design itself needs another look.
Supports scale-to-zero: During hours when no data arrives on a source topic, the pod count can drop to zero, and pods start up to process messages once they begin to pile up again. We applied this to development-environment topics with intermittent traffic to eliminate idle costs, while production topics keep a minimum number of replicas to avoid processing delays.

After applying KEDA, the application ran reliably most of the time, but there was one window where lag-based scaling alone responded too late.

Many games reset features such as daily missions and daily check-ins at midnight (00:00), and at that moment related batch jobs and daily aggregation logs pour in all at once, causing a traffic spike. Consumer lag-based scaling reacts only after lag has built up, so it inevitably lags behind such a sharp traffic increase. On top of that, new pods only start processing after they become ready and the consumer group rebalance completes, plus JVM warm-up time. Until the scale-out finishes, lag keeps climbing, and log delivery to the downstream pipelines is delayed accordingly.

#Getting Ahead with a Cron Trigger

In addition to the Kafka scaler, KEDA supports a cron trigger. We solved the problem by increasing the pod count in advance, just before midnight.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: log-transformer
spec:
  scaleTargetRef:
    name: log-transformer
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker:9092
        consumerGroup: log-transformer-group
        topic: log-topic
        lagThreshold: '100'
    - type: cron
      metadata:
        timezone: Asia/Seoul
        start: '50 23 * * *' # scale out at 23:50
        end: '30 0 * * *' # end the cron trigger's active window at 00:30
        desiredReplicas: '8'

When the Kafka lag trigger and the cron trigger are used together, KEDA takes the higher replica count of the two. It scales out to 8 in advance at 23:50, absorbs the midnight traffic, and after 00:30 scales in naturally based on lag.

#Optimization: Fewer Rebalance Events Through Multiple Threads

With the scaling policy and the midnight response in place, we finally redesigned the pod configuration itself. A simple setup with one Kafka Streams thread per pod is easy to operate, but JVM startup overhead grows linearly with the number of pods, and pod lifecycle events (restarts, rolling deploys, evictions) directly trigger consumer group rebalances.

To reduce this, we configured each pod to run multiple Kafka Streams threads at once. Kafka Streams supports running multiple stream threads per instance (= pod) through the num.stream.threads setting, and each thread is independently assigned tasks (groups of partitions) to process (Kafka Streams Architecture). From the consumer group's point of view, each thread behaves like an independent consumer, so increasing the number of threads per pod leaves partition distribution and processing consistency unchanged. In other words, this change kept the total number of consumer threads the same and reduced only the number of pods, which let us lower the number of JVM instances, the memory overhead, and the absolute frequency of lifecycle events all at once.

However, as more threads are bundled into a single pod, the blast radius of a single pod failure grows. So we didn't increase threads per pod without limit, but capped it according to the CPU allocation rule below.

To avoid CPU contention between threads, we set the thread count at one CPU core per thread and sized the pod's CPU request/limit to match. For example, a pod running 6 threads is allocated 6 cores. Kafka Streams' capacity planning guide also recommends sizing the number of stream threads to the available CPU cores; we ran somewhat conservatively at one core per thread to make throughput predictable. This keeps per-pod throughput stable, and "a single pod's capacity to process the backlog," which lag-based scaling relies on, also converges to a predictable value.

#Results: Resources That Follow Traffic, and Lower Costs

#Cost Savings

Compared with running fixed EC2 instances, moving to k8s + KEDA cut infrastructure costs by more than 80%.

Item	EC2	k8s + KEDA
Scaling	Fixed at 72 (scheduled adjustment stopped)	Automatic (lag-based + cron)
Cost	Fixed 24-hour cost	Cost proportional to traffic
Operational burden	Infra work such as adding instances when a game is added	Done by deploying a config change

More important than the 80% figure itself is the change in how we operate. On EC2, we had to keep instances running at peak capacity at all times, but with KEDA the deployment shrinks to 2 pods overnight and scales out only at peaks, allocating just as many pods as the load at that moment requires. The 80% was measured under the current traffic pattern; the key point is that resources follow the traffic even as the pattern changes.

#Less Operational Burden

Adding a game only requires adding values to the Helm chart.
Scaling policies are managed in the Helm chart, so they can be reviewed and their history tracked.
Deployments go through ArgoCD with just an image tag and environment variable change, which greatly reduced deployment preparation and rollout time compared with the previous Spinnaker-based EC2 pipeline.

#Remaining Work: Distributing Partitions Evenly Down to the Thread Level

Lag-based scaling works stably in the current setup, but the underlying problem of partitions not being distributed evenly across consumers remains.

KEDA's ensureEvenDistributionOfPartitions option addresses this by limiting the replica count to divisors of the total number of partitions. However, this option only checks partitions / replicas. In a setup where each pod runs multiple threads, the units that actually divide up the partitions are threads, not pods, so the condition that has to hold is whether partitions / (replicas × threads per pod) is an integer. Because we had already split deployments by source topic, each pod is dedicated to a single topic, so the number of threads per pod here is the number of threads processing that topic.

If 24 partitions are processed with 4 threads per pod, the option allows replica counts of 1, 2, 3, 4, 6, 8, 12, and 24 (the divisors of 24), but only counts of 1, 2, 3, and 6 distribute tasks evenly across threads. The other values show two symptoms.

4 replicas: Each pod gets an even 6 tasks, but 24 tasks are split across 16 threads in total, so 8 threads get 2 tasks and the other 8 get 1. Throughput between threads can differ by up to 2x.
8 replicas or more: The number of threads exceeds the number of tasks. With 8 replicas, 8 of the 32 threads, and with 24 replicas, 72 of the 96 threads, receive no tasks at all and just spin through empty poll loops.

The same goes for the replica upper bound. KEDA limits the replica count to at most the number of partitions, but this upper bound is also calculated without knowing the number of threads per pod. Ultimately, extending even distribution down to the thread level requires KEDA to know the number of threads per pod when deciding replicas.

To fill this gap, we opened an issue proposing a new option in the KEDA repository, and a PR implementing an option named threadsPerConsumer is now under review. If you set this value to the number of threads per pod together with ensureEvenDistributionOfPartitions, KEDA picks replicas from the divisors of partitions / threadsPerConsumer rather than of partitions. In the earlier example, only the divisors of 6 instead of 24, that is 1, 2, 3, and 6, become candidates, guaranteeing even distribution at both the pod and the thread level.

More fundamentally, we're also writing a component in Rust that will replace Log Transformer, written in Scala, and take on an expanded role. We expect lower JVM startup overhead and GC latency, reduced CPU and memory usage, and higher throughput. We'll share the results in a separate post once they're in, along with the progress of the threadsPerConsumer option.

#Closing: Scaling Metrics That Fit the Workload

What this migration reconfirmed for us comes down to one thing.

Choose scaling metrics to match the load characteristics of your workload.

CPU utilization is a perfectly valid metric for workloads whose load spreads evenly across pods. But for workloads like Kafka consumers, where load is tied to partitions, average CPU alone made it hard to catch real processing delays. By using consumer lag as the criterion, we were able to make our scaling more accurate.

Beyond Kafka, KEDA provides scalers for a wide range of event sources, including RabbitMQ, AWS SQS, and Prometheus metrics. Being able to easily bring in metrics other than CPU as scaling criteria was its biggest advantage for us.

If you're wrestling with similar questions about scaling Kafka consumers, we recommend looking into consumer lag-based scaling.

Thank you.

Scala for Game Server Development

Mon, 24 Oct 2022 00:00:00 GMT

알림: 한국어로 번역된 글을 보시려면 여기를 클릭하세요.

At Studio Kingdom, we use the Scala programming language to bring our hit game CookieRun: Kingdom to life. Scala is the language used for developing the game servers. Even though it is not the most popular language out there, we think that it’s a perfect fit for us and game server development in general. Let’s see why.

In this article, we will discuss several aspects of game development where Scala shines particularly:

Game Logic correctness is very important. There is nothing worse for a player than seeing their efforts wasted by a programming bug. Depending on the game, this logic can be particularly large and complex.
Games are fast-evolving, and players are always eager for new features. That means the game server codebase needs to constantly grow and evolve. Being able to bring features at a steady pace is primordial in order to stay on top.
Designing a multi-player game means dealing with concurrency and scaling issues. Game servers will receive a lot of concurrent requests, some of them dealing with the same entities.

#Game logic correctness

#The Power of types

Scala is a statically-typed language, which means that types are defined and known at compile-time. This prevents a whole range of runtime errors common with dynamically-typed languages (such as Javascript’s TypeError). But compared to other statically-typed languages, Scala uses types to express a lot of different things.

A simple example of that is the type Option[A] that is used to represent an optional value of type A. Rather than returning A that can be null and add the proper null-handling logic everywhere at the risk of missing some places, the idiomatic Scala way is to return Option[A]. That forces the developer to always handle the case where A is missing, using pattern matching or functions such as getOrElse or fold.

// this returns Option[GnomeResearchLevel]
val currentLevel = getGnomeResearchLevel(researchDataId)

// we have to handle the case where there is no current level
// in this case, we take the initial level
val nextLevel = currentLevel.map(_.up).getOrElse(GnomeResearchLevel.initial)

We push this idea even further by using refined types to add constraints to the values types can have. For example, a “Kingdom Level” is an integer that can only be 1 or more. Instead of simply using Int, we use type PosInt that enforces that the value is positive. We know for sure that the value will never be 0, so we don’t need to handle that case or to be careful with things like division by 0.

type PosInt = Int Refined Positive // alias for a positive Int

val a: PosInt = 1
val b: PosInt = 0 // will fail to compile

The compiler is able to detect if static values are valid or not. For values coming from the client, an error will be raised immediately after parsing the client request if a value doesn’t respect the type constraint.

Using PosInt is great, but if a “Cookie Level” is also a PosInt, there is a risk that we pass the wrong level to a function. To avoid that, we use a newtype, which is basically a wrapper around our PosInt to give it a special meaning.

@newtype case class KingdomLevel(value: PosInt)

def receiveKingdomLevelRewards(level: KingdomLevel): List[Reward]

Now, the compiler will enforce that when we call receiveKingdomLevelReward, we pass a KindomLevel and nothing else.

In summary, by using types that describe precisely what they hold, we avoid a whole range of silly bugs and improve the quality of our code.

#Types as documentation

Just like returning an Option tells you that a function may or may not return a result, we use types that describe the kind of computation that a function runs. Here are a few examples:

Option[A]: a function that may or may not return a result of type A. It does not have any side effect.
Either[E, A]: a function that may either fail with an E or succeed with an A. It does not have any side effect.
ZIO[R, E, A]: a function that requires an environment R and may either fail with an E or succeed with an A. It may have some side effects (such as I/O). It is equivalent to R => Either[E, A] + the notion of side-effecting code.
ZPure[W, S1, S2, R, E, A]: a function that requires an environment R and an initial state S1 and may either fail with an E or succeed with an updated state S2 and an A along with a log with entries of type W. It does not have any side effect. It is equivalent to (R, S1) => (List[W], Either[E, (S2, A)]).

We use these various types to reflect exactly what a function is doing. You may have noticed that there are a lot of type parameters. But in practice we use type aliases to make it much shorter and convenient. Here are a few examples.

// a function that increments a metric
// the return type means it is side-effecting, and can never fail
def incrementInFlightMessagesMetric: UIO[Unit]

// a function that converts protobuf data into kingdom level
// the return type means that it may fail with a non-empty list of errors
// EitherNel[E, A] = Either[NonEmptyList[E], A]
def extractKingdomLevel(raw: Protos): EitherNel[ExtractionError, KingdomLevel]

// a function that returns the current pvp (a.k.a. "Kingdom Arena" in the game) state of the user
// the return type means it may access the user state but not modify it
def getCurrentPvpState: KingdomReader[Pvp]

Not only do these types enforce the behavior of a function, but they give you a precise idea of what a function is doing just by reading its type signature. All possible errors are clearly declared in return types (as well as the absence of error, which is just as useful), so we are never surprised by an exception thrown unexpectedly. The same goes with side effects.

#Make your own DSL

We’ve seen a few examples where we use types to constrain what a value can be or what a function can do, and rely on the compiler to catch any programming mistake early on. We can go even further and make the whole code constrained to only allow “valid” operations. We do that by creating custom DSLs for specific operations. Because Scala is both powerful and very concise, it excels at creating sub-languages. You can push it pretty far by making the code read almost like a spec document.

Without going that far, creating a DSL is useful when you want to restrict some parts of your code to only a small subset of operations. Let’s see an example: in our game, we support transactions between multiple entities. That means we can send commands to two different users (or a user and a guild), and commit all changes only if both commands are handled successfully.

We created a small DSL for transactions, with only the following operations:

identify an entity by ID: kingdom(userId) , guild(guildId)
send a command to an entity: entity ! command
combine sending operations: >>> for sequential, &&& for parallel

We can then write simple transactions like this:

transaction { context =>
  (kingdom(context.userId) ! SendFriendRequest(friendId)) &&&
    (kingdom(friendId) ! ReceiveFriendRequest(context.userId))
}

The transaction block defines where our DSL can be used. We won’t be able to use “normal code” within the boundaries of this block, but only what our DSL allows. It gives us access to a context that contains information about the current user. We will then send SendFriendRequest to the current user (that will check that this user is allowed to send a friend request, and save the pending request), and ReceiveFriendRequest to the other user (which will save the received request). These 2 operations will be done in parallel, and if any one of those 2 fail, nothing will be persisted.

This little transaction system does not support other types of operations. If we tried to persist something to Redis or Kafka within the transaction block, it wouldn’t be rollbacked in case of error. But because we use a custom DSL, it is actually impossible to call that kind of code (we would get a compiler error), because we are restricted by our DSL operations.

Most of our business logic is implemented using a DSL whose basic operations are:

succeed with a result
fail with a ValidationError
access the environment (that is a large object containing all the game configuration: the list of characters, levels, events, etc)
access the user state
lift an event (we use event sourcing), which will automatically update the user state

Thanks to this limited set of operations, it is impossible to do something unexpected in the middle of our business logic.

#Evolving fast

#Focus on what’s important

As we’ve just seen, writing DSLs constrain our code to a smaller subset of valid operations. It has another huge benefit, combined with the conciseness of Scala: it makes the code super short and readable.

Creating a DSL is a good way to hide the complexity of our program to focus on what really matters which is the business logic. Considering our transaction example above, it actually involves complex low-level code. We need to locate and send messages to entities that may be hosted on different nodes, using a particular messaging protocol. The persistence layer also needs to combine all writing operations into a single transaction that can be committed or rollbacked, communicating with an actual database. But we don’t want to care about any of this when we write our business logic.

The general idea behind this is to separate the “technical” layers from the business logic so that both of them can be implemented and tested entirely separately. That allows implementing business logic, which represents the majority of the work of adding new features, without any kind of interference, which is much more efficient.

Here’s a simplified example of business logic function used when a user joins a guild: we get some metadata from the game configuration, get the guild state and check that the request is valid. On success, we lift an event to record the fact that this user joined the guild. Note how we don’t care about messaging protocols, persistence, transactions, locks or whatever technical challenges that actually exist under the hood.

val joinGuild: KingdomProgram[Unit] =
  for {
	metadata    <- inquireGuildMetadata
    requesterId <- inquireRequesterId
    memberCount <- getGuild(_.members.size)
    ()          <- assertThat(
                     memberCount < metadata.maxMemberCount,
                     ValidationError.GuildFull
                   )
    ()          <- liftEvent(GuildMemberJoined(requesterId))
  } yield ()

An interesting consequence of this is that it’s easy to onboard new developers to our team: because there are only a few operations available, they will be able to understand the DSL and start writing code easily, without necessarily understanding how the DSL is implemented. They can focus on learning about the domain rather than technical concepts.

#Death to boilerplate!

Implementing new features usually means writing a lot of code: exposing new APIs to the client, implementing the persistence layer, converting data types between different layers, etc. One thing Scala is great at is keeping that boilerplate to a minimum, thanks to a powerful macro engine and the ability to write compiler plugins.

As it is commonly done in many languages, we generate server code from protobuf file definitions. However we don’t use the generated code in our business logic, because this generated code doesn’t use our refined types and we sometimes use more convenient data types than the ones generated (collections are usually an area where the choice of data type is quite significant). It means that we need to convert the proto-generated objects that we receive from the client into our own. We are talking about thousands of objects, which could become a huge source of boilerplate. However, thanks to Scala macros, these transformations are done automatically for the most part, and we can verify it at compile-time!

All we have to do is make sure we have an instance of Transformer for each data type. A macro is used to generate a Transformer automatically between 2 classes, as long as their field names are matching and that we have a Transformer between each corresponding field types.

Let’s see an example. We have a protobuf type defined like this:

message CraftProduction {
  string id = 1;
  repeated int32 cookie_data_ids = 2;
  int32 completed_job_count = 3;
}

The generated code will look like this:

case class CraftProduction(
  id: String,
  cookieDataIds: Seq[Int],
  completedJobCount: Int
)

But our own class with “enhanced” types is more like this:

case class CraftProduction(
  id: ActivityId,
  participants: NonEmptyList[CookieDataId],
  completedJobCount: Int
)

All IDs use more specific types such as ActivityId or CookieDataId so we don’t mix them up, we also use a NonEmptyList because there must be at least one participant, and finally the cookieDataIds field is named differently because it makes more sense in the game server code.

We’ll need transformer for the following types:

a Transformer between String and ActivityId
a Transformer between Int and CookieDataId
a Transformer between Seq and NonEmptyList

Transformer for IDs are very simple and can be created in a generic way since IDs are just wrapper types. A Transformer between Seq and NonEmptyList is also simple, failing if the Seq is empty, otherwise wrapping it into a NonEmptyList. The last problem we have is that the field cookieDataIds was renamed into participants.

In this case, we can create a transformer like this:

implicit val transformer =
  Transformer
    .define[entities.CraftProduction, protobuf.CraftProduction]
    .withFieldRenamed(_.participants, _.cookieDataIds)
    .buildTransformer

The macro is able to convert all fields automatically, and since we defined a mapping between the participants field and the cookieDataIds field, it is able to generate a transformer. Most of the time, fields are named the same way so we don’t need to write this kind of code at all. The great thing about this is that if anything is missing, we will get a compile error telling us which field couldn’t be transformed and why. We can be sure that if the code compiles, it will work as expected.

Another example of relying on macros and code generation to cut down the boilerplate is our usage of GraphQL. Macros allow us to generate a GraphQL schema from our server data classes automatically, and a compiler plugin automatically generates client code powered by Scala.js (Scala code that is converted into JavaScript). We get both a client and a server from the same code with almost no effort!

Most of the macros and compiler plugins mentioned here were not even developed by us: they are available in open source libraries. The Scala OSS ecosystem is very dynamic and offers a lot of options for cutting down the boilerplate.

#Safe refactoring

An important concept in the world of functional programming with Scala is referential transparency. A function that is referentially transparent means that we can replace that function with its value without changing the behavior of the program.

Let’s see an example.

def add(a: Int, b: Int) = a + b

val sum = add(1, 2)

Everywhere in our code, we can use sum or add(1, 2) interchangeably. It won’t change anything in the code whether we use one or the other.

On the other hand:

object Test {
  var total = 0

  def add(a: Int): Int = {
	total = total + a
	total
  }
}

val sum = Test.add(2)

Using sum or Test.add(2) will lead to totally different behavior: calling sum will always return the same result, however calling Test.add(2) will trigger another modification of total and return a different result each time. In this case we can say that add is NOT referentially transparent.

Why does this matter? If we are able to replace pieces of code by functions interchangeably, it means we can do heavy refactorings without risking to break the code’s behavior. In addition to that, static typing will ensure that moving code around always make sense and the compiler will tell us if types don’t match. The ease of refactoring means that our code will be easy to maintain and extend in the feature, allowing it to grow and evolve together with our feature set.

Referential transparency is not specific to Scala, but this style of programming where all functions are referentially transparent and use immutable data types is heavily encouraged and followed within the Scala community and its library ecosystem.

#Multi-player challenges

#Local concurrency

When building a game server for a multiplayer game, we usually need multiple concurrent requests to access the same data. We sometimes need to run tasks concurrently and gather the results. We also need to run tasks in the background for the whole lifecycle of our application. The list goes on. We need to be able to perform a lot of actions concurrently without exhausting the resources that we have.

Scala comes with the concept of Future but there are even better alternatives offered by libraries such as ZIO (which is what we use), Cats Effect or Monix. They all have in common to be built on top of fibers, which are lightweight “green threads” managed by a runtime system. Unlike traditional threads, they are very cheap to create and make concurrent programming super simple.

These libraries provide a bunch of operators to let you run computations in parallel: you can fork a computation to run it in a different fiber, join a fiber to wait for its result, interrupt a fiber, race multiple fibers, etc. Most of the time, you don’t even need to manipulate them as you can simply use high level operators such as ZIO.foreachPar or ZIO#zipPar. There are many convenient data types built on top of fibers such as Promise, Schedule or Queue.

Let’s see a simple example. Whenever a player open our game client, a gRPC stream is opened to the server and maintained for as long as the user uses the game. While this stream is alive, we want to regularly update the “last connection date” of the user in the database.

updateLastConnectionDate(userId)
  .repeat(Schedule.spaced(1 hour))
  .fork

That’s it. Running this code means that the updateLastConnectionDate function will be called every one hour. Because we use fork, the function will immediately return and run in the background. By default, a fiber is terminated when its parent fiber is terminated, so that background process will automatically be stopped when our gRPC stream is disconnected. If we wanted to keep the fiber running even after the termination of its parent, we could have used .forkDaemon.

Another super useful data type from ZIO is the Hub. Unlike a Queue where each item is given to a single consumer, items that are published to a Hub are broadcasted to all subscribers. Let’s imagine that we have global messages that we want to send to all connected users through that gRPC stream we just talked about. On each game server, we can start one fiber that consumes messages from Kafka (where global messages come from), and publishes each received messages to a hub. Then, each connected gRPC stream can simply subscribe to this hub and it will receive all the messages.

def createHub(consumer: Consumer): UIO[Hub[Message]] =
	Hub.unbounded[Message].flatMap { hub => // create a hub with no max size
    consumer
     .read                                // read from Kafka
     .mapZIO(event => hub.publish(event)) // publish all messages to the hub
     .runDrain                            // consume the stream to the end
     .fork                                // do this in a separate fiber
	 .as(hub)                             // return the hub
  }

// create a stream that will contain all messages published to the hub
val stream = ZStream.fromHub(hub)

At the end, our gRPC stream is just the combination of various streams coming from different sources like this one and merged together. This Hub data type is so convenient that we created our own variant, IndexedPubSub, that allows subscribing to particular keys only. Thanks to Hub but also Ref, Promise, Queue and a few others, local concurrency problems become really trivial with ZIO.

#Distributed concurrency

In a real application, concurrency problems are often not only local but also distributed. We want to able to scale out our game servers by simply adding more nodes. Having multiple nodes causes some problems: for example, if a guild only has 1 spot left but 2 players send a request to join that guild at the exact same time, only the first one should succeed and the other should fail.

A typical way to solve that is the single writer principle: at any given time, there should be only a single instance of each entity that can modify its state. In other words, there should be a single process handling messages for any given guild at any time. That way, this process will handle the 2 messages sequentially, succeeding the first one and failing the second.

There are several ways to implement this, and the Scala ecosystem is rich with options. One of them is using Kafka consumer groups to ensure only a single consumer handles all messages of a given entity ID. In our case, because of the “synchronous” nature of our workflows (the client expects an instant response for every request), we went another route and use the actor sharding pattern. This pattern was popularized by the Akka library which is ideal for getting started (we are now using our own custom implementation that follows similar principles).

An actor is basically a tiny process that consumes messages sequentially from a queue (called “mailbox”). Sharding means that actors are distributed across a group of nodes and that you can send messages to an actor just by knowing its ID. This concept is known as location transparency, meaning that you can send messages to an actor without knowing where the actor is actually located (it might be on the same node or another one).

Following the DSL introduced earlier to send message, 2 different nodes can call the following code concurrently:

guild(guildId) ! JoinGuild(inviteId)

Under the hood, the sharding system will find on which node the actor in charge of guildId is located, and send a request to that node containing the payload (JoinGuild). That node will forward those messages to the local actor handling this guild ID. The actor will handle the message that arrived first and respond with a success. It will then handle the second message and respond with a failure.

Thanks to a few powerful libraries, Scala makes it really easy to handle distributed and concurrent payloads without resorting to things like distributed locks that might clog response times and create issues like deadlocks. The actor pattern, whether it is implemented with Akka or ZIO, makes our life easier once again.

Through a handful of practical and simple examples, we’ve seen that Scala is a really interesting choice for building a game server. Combining its powerful type system with the best practices of functional programming, we are able to minimize the risk of errors and focus entirely on the business logic. Thanks to Scala’s rich ecosystem of open source libraries, we can cut down the boilerplate and resolve all kinds of concurrency problems.

Sounds attractive to you? Want to know more? We are hiring!