centos7用docker安装kafka

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

参考之前的文章安装zookeeper集群 https://www.cnblogs.com/xiaohanlin/p/10124674.html

如果是测试环境也可以简单安装单节点的zookeeper 具体如下

docker pull docker.io/wurstmeister/zookeeper

启动zookeeper容器

docker pull docker.io/wurstmeister/zookeeper
docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper

2. 启动kafka容器

docker pull docker.io/wurstmeister/kafka:2.12-2.1.0

docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.0.168:2181,192.168.0.169:2181,192.168.0.170:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.0.170:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka:2.12-2.1.0

这里面主要设置了4个参数

KAFKA_BROKER_ID=0
KAFKA_ZOOKEEPER_CONNECT=192.168.0.168:2181,192.168.0.169:2181,192.168.0.170:2181
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.0.170:9092
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092

KAFKA_ZOOKEEPER_CONNECT 配置的是zookeeper的地址，可以单节点配置，也可以配置zookeeper集群多节点，用逗号隔开

中间两个参数的192.168.0.170改为宿主机器的IP地址，如果不这么设置，可能会导致在别的机器上访问不到kafka。

3. 测试kafka
进入kafka容器的命令行

docker exec -ti kafka /bin/bash

进入kafka所在目录

cd opt/kafka_2.12-2.1.0/

后面不再赘述，可参考前一篇文章https://blog.csdn.net/lblblblblzdx/article/details/80548157

4. 集群搭建
使用docker命令可快速在同一台机器，或者不同的机器搭建多个kafka，只需要改变brokerId和端口，集群的kafka的KAFKA_ZOOKEEPER_CONNECT需要配置为同一个zookeeper集群或者单节点

docker run -d --name kafka1 -p 9093:9093 -e KAFKA_BROKER_ID=1 -e KAFKA_ZOOKEEPER_CONNECT=192.168.0.168:2181,192.168.0.169:2181,192.168.0.170:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.0.169:9093 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9093 -t wurstmeister/kafka:2.12-2.1.0

docker run -d --name kafka2 -p 9094:9094 -e KAFKA_BROKER_ID=2 -e KAFKA_ZOOKEEPER_CONNECT=192.168.0.168:2181,192.168.0.169:2181,192.168.0.170:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.0.168:9094 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9094 -t wurstmeister/kafka:2.12-2.1.0

5. 创建Replication为3，Partition为3的topic
在kafka容器中的opt/kafka_2.12-2.1.0/目录下输入

bin/kafka-topics.sh --create --zookeeper 192.168.0.170:2181 --replication-factor 3 --partitions 3 --topic nova2_plantree_topic

6. 查看topic的状态
在kafka容器中的opt/kafka_2.12-2.1.0/目录下输入

bin/kafka-topics.sh --describe --zookeeper 192.168.0.170:2181 --topic nova2_plantree_topic

输出结果：

bash-4.4# ./kafka-topics.sh --describe --zookeeper 192.168.0.170:2181 --topic nova2_plantree_topic
Topic:nova2_plantree_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: nova2_plantree_topic Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: nova2_plantree_topic Partition: 1 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: nova2_plantree_topic Partition: 2 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2

显示每个分区的Leader机器为broker0，broker1，broker2，在broker0和1和2上具有备份，Isr代表存活的备份机器中存活的。

当停掉kafka1后，

docker stop kafka1

再查看topic状态，输出结果：

bash-4.4# ./kafka-topics.sh --describe --zookeeper 192.168.0.170:2181 --topic nova2_plantree_topic
Topic:nova2_plantree_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: nova2_plantree_topic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0
Topic: nova2_plantree_topic Partition: 1 Leader: 0 Replicas: 0,1,2 Isr: 0,2
Topic: nova2_plantree_topic Partition: 2 Leader: 2 Replicas: 1,2,0 Isr: 2,0

查看kafka的broker数量

docker exec -it zookeeper bash

[zk: localhost:2181(CONNECTED) 5] ls /brokers/ids
[2, 1, 0]

查看topic的数量

[zk: localhost:2181(CONNECTED) 6] ls /brokers/topics
[nova2_plantree_topic]

bash-4.4# ./kafka-topics.sh --zookeeper 192.168.0.170:2181 --list
nova2_plantree_topic

查看consumer组

bash-4.4# ./kafka-consumer-groups.sh --bootstrap-server 192.168.0.170:9092 --list
plantree_group

bash-4.4# ./kafka-consumer-groups.sh --bootstrap-server 192.168.0.170:9092,192.168.0.169:9093 --group plantree_group --describe

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
nova2_plantree_topic 0 7 7 0 consumer-1-330c0937-44db-4ec0-92da-352baf83043b /192.168.0.169 consumer-1
nova2_plantree_topic 1 5 5 0 consumer-1-330c0937-44db-4ec0-92da-352baf83043b /192.168.0.169 consumer-1
nova2_plantree_topic 2 5 5 0 consumer-1-71628b6f-10d1-4a0a-bc2b-0d9f17f6008f /192.168.0.168 consumer-1

模拟生产者

bin/kafka-console-producer.sh --broker-list 192.168.0.170:9092 --topic nova2nodepush

模拟消费者

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.170:9092 --topic nova2nodepush --from-beginning