This repository demonstrates how to use Prometheus and Grafana for monitoring an Apache Kafka cluster.
-
Kafka (Confluent)
-
Zookeeper
-
Kafka Schema Registry (Confluent)
-
KSQLDB
-
Kafka Connect
-
Prometheus
-
Grafana
-
AKHQ (https://akhq.io/)
-
ZooNavigator (https://zoonavigator.elkozmon.com/en/stable/)
$ ./up -h -------------------------------------------------------------------------------- ____ __ _____ __ _ / __ \____ _/ /_____ _ / ___// /_________ ____ _____ ___ (_)___ ____ _ / / / / __ `/ __/ __ `/ \__ \/ __/ ___/ _ \/ __ `/ __ `__ \/ / __ \/ __ `/ / /_/ / /_/ / /_/ /_/ / ___/ / /_/ / / __/ /_/ / / / / / / / / / / /_/ / /_____/\__,_/\__/\__,_/ /____/\__/_/ \___/\__,_/_/ /_/ /_/_/_/ /_/\__, / /____/ Powered by Apache Kafka! -------------------------------------------------------------------------------- Usage: ./up [options] -n <[NAME]> : Name of the stack to deploy (required) -s <[SERVICE]> : Service names to deploy (run all services if empty) -h : Print this Help. The table below lists the available docker-compose stacks: NAME (./up -n <NAME>) |FILE (docker compose -f <FILE> up -d) ===================================+================================================== zk-kafka-multiple-nodes-sasl |zk-kafka-multiple-nodes-sasl-stack.yml zk-kafka-multiple-nodes |zk-kafka-multiple-nodes-stack.yml zk-kafka-single-node-full |zk-kafka-single-node-full-stack.yml zk-kafka-single-node-sasl |zk-kafka-single-node-sasl-stack.yml zk-kafka-single-node |zk-kafka-single-node-stack.yml zkless-kafka-multiple-nodes |zkless-kafka-multiple-nodes-stack.yml1. Clone the Kafka Monitoring Suite repository.
$ git clone https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose.git $ cd kafka-monitoring-stack-docker-compose2. Start Confluent/Kafka cluster.
Deploy one of the provided docker-compose stack:
Note: Depending on your network speed, this may take few minutes to download all images.
Start/Stop with:
$ ./zk-kafka-single-node-stack-start.sh $ ./zk-kafka-single-node-stack-stop.sh # or directly $ docker-compose -f zk-kafka-single-node-stack.yml up -d $ docker-compose -f zk-kafka-single-node-stack.yml down$ docker-compose -f zk-kafka-multiple-nodes-stack.yml up -d $ docker-compose -f zk-kafka-multiple-nodes-stack.yml down$ ./zk-kafka-single-node-sasl-stack-start.sh $ ./zk-kafka-single-node-sasl-stack-stop.sh$ docker-compose -f zk-kafka-multiple-nodes-sasl-stack.yml up -d $ docker-compose -f zk-kafka-multiple-nodes-sasl-stack.yml downStart/Stop with:
$ ./full-single-node-stack-start.sh $ ./full-single-node-stack-stop.sh # or directly $ docker-compose -f zk-kafka-single-node-full-stack.yml up -d $ docker-compose -f zk-kafka-single-node-full-stack.yml down3. Create Topic.
Create demo-topic with 6 partitions and 3 replicas.
$ docker exec -it kafka101 \ kafka-topics \ --create \ --partitions 6 \ --replication-factor 3 \ --topic demo-topic \ --bootstrap-server kafka101:290924. Produce messages.
Open a new terminal window, generate some message to simulate producer load.
$ docker exec -it kafka101 \ kafka-producer-perf-test \ --throughput 500 \ --num-records 100000000 \ --topic demo-topic \ --record-size 100 \ --producer-props bootstrap.servers=kafka101:290925. Consume messages.
Open a new terminal window, generate some message to simulate consumer load.
$ docker exec -it kafka101 \ kafka-consumer-perf-test \ --messages 100000000 \ --timeout 1000000 \ --topic demo-topic \ --reporting-interval 1000 \ --show-detailed-stats \ --bootstrap-server kafka101:290926. Open Grafana.
Open your favorite web browser and open one of the provided Grafana dashboards :
-
Kafka Cluster / Global Health Check
-
Kafka Cluster / Performance
-
Kafka Cluster / Zookeeper Connections
-
Kafka Cluster / JVM & OS
-
Kafka Cluster / Hard disk usage
-
Kafka Cluster / Topic Logs
Grafana is accessible at the address : http://localhost:3000
Security are :
-
user :
admin -
password :
kafka
Prometheus is accessible at the address : http://localhost:9090