monkeybox_kubernetes/Workloads/kafka/readme.md

Kafka via Strimzi

for POC, see the strimzi_poc subdir.

Now lets build something more robust.

kubectl create namespace kafka
curl -o 1.strimzi.yaml 'https://strimzi.io/install/latest?namespace=kafka'
kubectl apply -f 1.strimzi.yaml -n kafka
curl -o 2.kafka-persistent.yaml 'https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.35.1/examples/kafka/kafka-persistent.yaml'
vim 2.kafka-persisent.yaml
# Change 'my-cluster' to 'mbox-cluster'
kubectl apply -f 2.kafka-persistent.yaml -n kafka
# Wait for it to be ready
kubectl wait kafka/mbox-cluster --for=condition=Ready --timeout=300s -n kafka

It didn't start. :(

Memory limits?

          resources:
            limits:
              memory: "500Mi"
              cpu: "500m"
            requests:
              memory: "100Mi"
              cpu: "250m"

That probably wasn't it.

Instead, I had to modify the end part:

  entityOperator:
    topicOperator: {}
    userOperator: {}
    template:
      topicOperatorContainer:
        env:
        - name: STRIMZI_USE_ZOOKEEPER_TOPIC_STORE
          value: "true"

This feels like a hack, but it's working.

Set up the external listener. Looks like this is smart and sets up multiples. See kubectl get service -n kafka to find the bootstrap IP.

# Terminal 1: (kcat on mac)
kafkacat -b 10.42.42.34:9092 -t new_topic -P -l /var/log/message
# Terminal 2:
kafkacat -b 10.42.42.34:9092 -t new_topic -C

Final test:

# from mac:
echo "Hello World" | kcat -P -b kafka.home.monkeybox.org:9092 -t test_topic
# from linux
kafkacat -C -b kafka.home.monkeybox.org:9092 -t test_topic