Using Apache Kafka with Docker

Let’s make a kafka container!

Requirements

• Docker Installed

Overview

Apache Kafka is a hot topic, especially when working with streaming data. However, if you’re like me and had no idea how to actually spin up a local version of Kafka this article should help you save a few minutes. This article will be a first part of a multipart article where we interact with kafka using the kafka-go library.

Article Goals

• Instantiate a highly accessible local version of Kafka on your own personal machine.

If you’re new to docker and need a quick grasp of how it works this article sums it up beautifully. https://vsupalov.com/6-docker-basics/

Docker Setup

For the sake of this project, I will be writing commands to my computer’s Desktop feel free to write to any directory.

We will be using bitnami/kafka Apache Kafka image, you can see the details here -> https://hub.docker.com/r/bitnami/kafka/

Pull the image

# create the kafka folder
cd Desktop
mkdir kafka
cd kafka
# pull the docker image and configure the .yml file
curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-kafka/master/docker-compose.yml > docker-compose.yml

After you run these commands you should see a docker-compose.yml in the Desktop/kafka directory.

Update docker-compose.yml

It all looks great but we actually need to update some environment variables inside of the .yml file, so that it will be compatible with the library in kafka we will use.

Modify the environment section of the docker-composer.yml, added values:

// Changes
kafka:
  image: docker.io/bitnami/kafka:3
  ports:
    - "9092:9092"
  volumes:
    - "kafka_data:/bitnami"
  environment:
    - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
    - ALLOW_PLAINTEXT_LISTENER=yes
// changes from here...
    - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP= CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT 
    - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
    -KAFKA_CFG_ADVERTISED_LISTENERS= CLIENT://localhost:9092,EXTERNAL://localhost:9093
    - KAFKA_INTER_BROKER_LISTENER_NAME=CLIENT
  
  depends_on:
    - zookeeper

This solution comes from the following github post -> https://github.com/segmentio/kafka-go/issues/591

build the image

Finally now we can go ahead and build the image!

docker-compose up -d

All done, you have successfully created an Apache Kafka instance locally on your machine. If you open docker you should see that it is running. The container should be living on http://localhost:9092.

This article wouldn’t have been possible without the resources below:

• https://github.com/segmentio/kafka-go/issues/591
• https://hub.docker.com/r/bitnami/kafka/
• https://vsupalov.com/6-docker-basics/