Understanding Kafka Partitioning and Murmur2 Hashing

How Kafka Ensures Ordering, Scalability, and Performance

Apache Kafka is one of the most widely used distributed streaming platforms in modern data architectures. Whether we are building real-time analytics, event-driven microservices, CDC pipelines, or streaming ETL systems, understanding Kafka partitioning and Murmur2 hashing is absolutely critical.

In this article, we will deeply understand:

• Kafka partitions
• Message keys
• Murmur2 hashing
• Modulo operation
• Ordering guarantees
• Consumer scalability
• Real-world production examples
• Best practices for Kafka partition design

What Is a Kafka Partition?

A Kafka topic is divided into multiple partitions.

Example:

Orders Topic
├── Partition 0
├── Partition 1
├── Partition 2

Partitions are the foundation of Kafka scalability.

They allow Kafka to:

• distribute data across brokers
• process messages in parallel
• scale horizontally
• support high throughput

Note – Without partitions, Kafka would not be able to process millions of events per second efficiently.

Why Kafka Uses Message Keys

When producing messages to Kafka, we can optionally provide a key.

Example:

ProducerRecord<String, String> record =
new ProducerRecord<>("orders", "1001", "Order Created");

— Where we have Topic = orders , Key = 1001 & Value = Order Created

The key is extremely important because Kafka uses it to decide, which partition the message should go to & how ordering should be preserved.

Why Ordering Matters in Kafka

Kafka mechanism guarantees ordering only within a partition.

Consider an e-commerce workflow:

Order Created --> Order Paid --> Order Shipped -->Order Delivered

If these events go to different partitions:

P0 → Order Created
P1 → Order Paid
P2 → Order Shipped

Consumers may process them out of sequence.

That can create serious data consistency problems.

To solve this, Kafka ensures all events for the same key go to the same partition.

Example:

Key = OrderID

Now all events for order 1001 land in the same partition.

How Kafka Determines the Partition

Kafka internally performs two major steps:

Step 1 — Murmur2 Hashing

Kafka first runs the key through the Murmur2 hashing algorithm.

Example: Key = "1001"

Murmur2 converts the key into a large integer: 85723921 - This is called the hash value.

What Is Murmur2 Hashing?

Murmur2 is a non-cryptographic hash function.

Kafka uses it because it is extremely fast, deterministic , evenly distributed, low collision probability as well as suitable for distributed systems

The important property is: Same key always produces the same hash.
Example:

Murmur2("1001") → 85723921
Murmur2("1001") → 85723921

Step 2 — Modulo Operation

After generating the hash value, Kafka calculates partition = hash % number_of_partitions

Example:

85723921 % 3 = 1

Meaning:

Partition 1

So all messages with key 1001 always go to Partition 1.

Understanding the Modulo (%) Operation

Our Basic match we have been doing for years, The % operator means Return the remainder after division.

Example:

14 % 3 Division: 14 ÷ 3 = 4 remainder 2

So: 14 % 3 = 2

Kafka uses modulo because it converts huge hash numbers into valid partition numbers.

Kafka Partitioning Example

Suppose: Partitions = 3

Possible partition IDs are:

0
1
2

Now Kafka performs: Hash Value Calculation Partition 1010 % 3 = 1P, 11414 % 3 = 2P, 21515 % 3 = 0P, 01616 % 3 = 1P, 11717 % 3 = 2P

Real-World Banking Example

Imagine a banking application.

Key: Account Number

Events: Deposit $100, Withdraw $50 , Transfer $25

If these events land in different partitions – balance inconsistencies occur, race conditions happen & transactions may process out of order

Kafka solves this by hashing the account number and routing all events to the same partition. This preserves event order.

Why Kafka Partitioning Is Critical

1. Ordering Guarantee

Kafka guarantees ordering within a partition. Using the same key ensures all related events remain ordered.

2. Scalability

Partitions allow parallel processing.

Example:

Partition 0 → Consumer 1
Partition 1 → Consumer 2
Partition 2 → Consumer 3

Now multiple consumers can process data simultaneously.

3. Load Balancing

Hashing distributes data evenly across partitions.

This prevents – partition hotspots, uneven broker utilization & consumer imbalance.

4. Consumer Group Parallelism

Kafka consumer groups operate partition-wise.

More partitions allow – higher throughput, better scaling & improved parallel processing

5. Stateful Stream Processing

Frameworks like- Kafka Streams, Apache Flink & Spark Streaming

Examples:

1- Shopping carts

2- User sessions

3- Bank balances

4- Inventory tracking

All related events must arrive at the same processing node. Kafka key hashing guarantees this.

Important Production Consideration :

Increasing Partition Count Changes Mapping

Suppose:

Old:
hash % 3 = 1

After increasing partitions:

New:
hash % 5 = 4

Now the same key may route to a different partition.

This can impact – ordering, state stores, stream joins as well as caching. Partition expansion in production environments requires careful planning.

What Happens If No Key Is Provided?

If the producer sends messages without a key:

new ProducerRecord<>("orders", value)

Kafka uses – round-robin partitioning & sticky partitioning

Benefits – better throughput & more even batching

Drawback: no ordering guarantee for related events

Kafka Partitioning Best Practices

Use Stable Business Keys

Some examples – (“customer_id”, “account_id”, “order_id”, “device_id”)

Avoid random keys unless ordering is not important.

Avoid Hot Partitions

Bad key examples – (country=”US”, status=”ACTIVE”)

These can overload a single partition.

Plan Partition Count Early

Changing partitions later can impact:

1-Ordering

2-Stateful processing

3-Joins

4- Stream applications

Monitor Partition Distribution

Use Kafka monitoring tools to check – skewed partitions, lag imbalance & throughput hotspots

Kafka Partitioning Architecture Flow

Producer
↓
Message Key
↓
Murmur2 Hashing
↓
Large Integer Hash
↓
Modulo Operation (% partitions)
↓
Partition Selected
↓
Consumer Processes Message

Final Thoughts

Kafka partitioning and Murmur2 hashing are foundational concepts for building scalable and reliable streaming systems.

Understanding how Kafka routes messages helps architects and engineers design:

1- Scalable event-driven systems

2- High-throughput CDC pipelines

3- Reliable microservices

4- Real-time analytics platforms

The simple formula: partition = hash(key) % partitions

This is one of the most important mechanisms behind Kafka’s scalability and ordering guarantees.