withsoon

What you'll have running

A local Kafka cluster with:

1 broker (KRaft mode — no Zookeeper)
Kafka UI at localhost:8080
A test topic you can produce to and consume from

Prerequisites

Docker Desktop installed and running
4GB RAM free

Step 1: docker-compose.yml

Create a new folder and add this file:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
    depends_on:
      - kafka

Step 2: Start everything

docker compose up -d

Wait ~15 seconds. Check it's up:

docker compose ps

Both kafka and kafka-ui should show running.

Step 3: Create a topic

docker exec kafka \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic my-first-topic \
  --partitions 3 \
  --replication-factor 1

Verify it exists:

docker exec kafka \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --list

Step 4: Produce messages

Open a producer console:

docker exec -it kafka \
  /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic my-first-topic

Type messages and hit Enter. Each line becomes one Kafka message.

hello world
kafka is fast
this is a test

Press Ctrl+C when done.

Step 5: Consume messages

In a new terminal:

docker exec -it kafka \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic my-first-topic \
  --from-beginning

You'll see all the messages you produced. Notice they may arrive slightly out of order across partitions — that's expected. Order is guaranteed per partition, not across partitions.

Step 6: Open Kafka UI

Visit http://localhost:8080 in your browser.

You'll see:

The local cluster
Your topic my-first-topic with 3 partitions
Message counts per partition
Consumer group lag (none yet)

What's actually happening

Producer → Leader partition (broker) → Followers replicate
Consumer ← Pulls from leader partition offset

With replication-factor 1 locally, there are no followers. In production you'd use 3.

The offset is just a number — position 0, 1, 2... per partition. Consumers track their own offset, stored in the __consumer_offsets internal topic.

Useful commands cheatsheet

# List topics
kafka-topics.sh --bootstrap-server localhost:9092 --list

# Describe a topic (partitions, replicas, leader)
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-first-topic

# Check consumer group lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

# Delete a topic
kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic my-first-topic

Tear down

docker compose down -v   # -v removes volumes (all data)

Next steps

Kafka system design deep dive — understand partitions, consumer groups, and production patterns
Add a Python producer using confluent-kafka or kafka-python
Set up Kafka Connect to stream data from a database

Kafka Local Setup — From Zero to Running in 10 Minutes