9.3 k8s结合ELK实现日志收集 - 架构设计

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

数据流：

logfile -> filebeat > kafka（依赖zookeeper）-> logstash -> elasticsearch -> kibana

1.部署zookeeper集群

1.1 主机IP规划

hostname	IP
zk-kfk-1	192.168.2.26
zk-kfk-2	192.168.2.27
zk-kfk-3	192.168.2.28

3个服务器上都需要操作以下安装配置步骤

1.2 安装jdk11

apt install openjdk-11-jdk -y

1.3 下载安装包

wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

1.4 部署配置

tar xf apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local/ cd /usr/local ln -s apache-zookeeper-3.7.0-bin/ zookeepercd conf/ cp zoo_sample.cfg zoo.cfgmkdir -p /data/zookeeper-data# 配置文件修改如下： root@zk-kfk-1:/usr/local/zookeeper/conf# sed -e '/^#/d' -e '/^$/d' zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper-data clientPort=2181 maxClientCnxns=150 autopurge.snapRetainCount=3 autopurge.purgeInterval=1 server.1=192.168.2.26:2888:3888 server.2=192.168.2.27:2888:3888 server.3=192.168.2.28:2888:3888

1.5 启动服务并查看运行状态

# /usr/local/zookeeper/bin/zkServer.sh startroot@zk-kfk-1:# /usr/local/zookeeper/bin/zkServer.sh status /usr/bin/java ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: followerroot@zk-kfk-2:# /usr/local/zookeeper/bin/zkServer.sh status ... Mode: followerroot@zk-kfk-3:# /usr/local/zookeeper/bin/zkServer.sh status ... Mode: leader

2.部署kafka

复用zookeeper集群3台服务器部署，以下操作3台服务器上都需要执行

2.1 下载安装包并解压

# 下载 wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.0.0/kafka_2.13-3.0.0.tg# 解压 tar xf kafka_2.13-3.0.0.tgz -C /usr/local/ cd /usr/local ln -s kafka_2.13-3.0.0/ kafka# 创建数据目录 mkdir -p /data/kafka-logs

2.2 修改配置文件

# vim /usr/local/kafka/config/server.properties broker.id=26 listeners=PLAINTEXT://192.168.2.26:9092 log.dirs=/data/kafka-logs zookeeper.connect=192.168.2.26:2181,192.168.2.27:2181,192.168.2.28:2181

2.3 启动服务

cd /usr/local/kafka && ./bin/kafka-server-start.sh -daemon config/server.properties

使用绝对路径启动会报错：

# /usr/local/kafka/bin/kafka-server-start.sh --daemon /usr/local/kafka/config/server.properties [2021-10-28 08:51:14,097] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [2021-10-28 08:51:14,455] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) ...

3.部署ES集群

复用zookeeper,kafka 3台服务器

3.1 下载安装包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.1-amd64.deb

3.2 安装

dpkg -i elasticsearch-7.15.1-amd64.deb

3.3 修改配置

集群中的另外2个节点的node.name，network.host 2个配置需按实际情况修改

# es node-1的配置 root@zk-kfk-1:/data# cat /etc/elasticsearch/elasticsearch.yml cluster.name: k8s-elk node.name: node-1 path.data: /data/es/data path.logs: /data/es/logs network.host: 192.168.2.26 http.port: 9200 discovery.seed_hosts: ["192.168.2.26", "192.168.2.27","192.168.2.28"] cluster.initial_master_nodes: ["192.168.2.26", "192.168.2.27","192.168.2.28"] action.destructive_requires_name: true# 创建数据和日志目录 mkdir -p /data/es/data mkdir -p /data/es/logs# 修改目录权限 chown -R elasticsearch:elasticsearch /data/es/

3.4 启动服务

systemctl start elasticsearch.service

3.5 使用elasticsearch-head 查看es集群状态

chrome浏览器加载elasticsearch-head 插件参考：

https://blog.csdn.net/xiaolinzi176/article/details/105187311

4.pod中启动filebeat实现日志收集并输出到kafka

采用将filebeat和nginx放到一个容器里面方案

4.1 制作镜像

Dockerfile

FROM nginx:1.21.1 COPY nginx.conf /etc/nginx COPY default.conf /etc/nginx/conf.d COPY filebeat-7.15.1-amd64.deb /root COPY start_filebeat.sh /docker-entrypoint.d RUN rm -f /var/log/nginx/* \ && dpkg -i /root/filebeat-7.15.1-amd64.deb \ && rm -f /root/filebeat-7.15.1-amd64.deb COPY filebeat.yml /etc/filebeat

filebeat.yaml配置文件

filebeat.inputs: - type: log paths: - /var/log/nginx/access.log tail_files: true tags: ["nginx-access"] fields: log_topics: nginx-accesssetup.kibana: host: "192.168.2.29:5601"output.kafka: enabled: true hosts: ["192.168.2.26:9092","192.168.2.27:9092","192.168.2.28:9092"] topic: '%{[fields][log_topics]}'

filebeat运行脚本

cat start_filebeat.sh #/bin/bash nohup /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat 2>&1 &

镜像目录文件列表

filebeat-7.15.1-amd64.deb 可以在官网下载

总用量 35016 drwxr-xr-x 2 root root 4096 10月 29 14:42 ./ drwxr-xr-x 4 root root 4096 10月 28 18:59 ../ -rwxr-xr-x 1 root root 137 10月 29 14:39 build_command.sh* -rw-r--r-- 1 root root 619 10月 28 18:59 default.conf -rw-r--r-- 1 root root 317 10月 29 14:42 Dockerfile -rw-r--r-- 1 root root 35821726 10月 28 18:59 filebeat-7.15.1-amd64.deb -rw-r--r-- 1 root root 518 10月 29 14:27 filebeat.yml -rw-r--r-- 1 root root 1270 10月 28 19:01 nginx.conf -rwxr-xr-x 1 root root 207 10月 29 11:24 start_filebeat.sh*

build_command.sh 脚本

#!/bin/bash TAG=$1 docker build -t 192.168.1.110/web/nginx-filebeat:${TAG} . sleep 1 docker push 192.168.1.110/web/nginx-filebeat:${TAG}

生成镜像文件并上传到本地harbor

# ./build_command.sh 2021.10.29-144201# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.1.110/web/nginx-filebeat 2021.10.29-144201 8f011ec312e5 4 hours ago 312MB

使用docker run运行测试

# docker run -d -p 81:80 192.168.1.110/web/nginx-filebeat:2021.10.29-144201root@k8-deploy:~/k8s-yaml/web/nginx_2/dockerfile# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 40dff03b2e4a 192.168.1.110/web/nginx-filebeat:2021.10.29-144201 "/docker-entrypoint.…" 4 hours ago Up 4 hours 0.0.0.0:81->80/tcp exciting_perlman

进入容器并查看nginx log

root@40dff03b2e4a:/# tail -f /var/log/nginx/*.log ==> /var/log/nginx/access.log <== 29/Oct/2021:06:44:57 +0000|172.17.0.1|localhost|GET / HTTP/1.1|200|612|0.000|905|76|-|-|curl/7.68.0 29/Oct/2021:06:51:43 +0000|127.0.0.1|localhost|GET / HTTP/1.1|200|612|0.000|905|73|-|-|curl/7.64.0

使用kafka客户端工具（offsetexplorer_64bit.exe）查看kafka中是否已经有数据

5.部署logstash将kakfa数据输出至elasticsearch集群

5.1 下载logstash安装包

https://artifacts.elastic.co/downloads/logstash/logstash-7.15.1-amd64.deb

5.2 安装

dpkg -i logstash-7.15.1-amd64.deb

5.3 编写logstash处理nginx日志的配置文件

配置文件路径：/etc/logstash/conf.d/kafka-to-es.conf

input { kafka { bootstrap_servers => "192.168.2.26:9092,192.168.2.27:9092,192.168.2.28:9092" auto_offset_reset => "latest" topics=> ["nginx-access"] consumer_threads => 2 decorate_events => "true" } }#============================================================================= filter { json { source => "message" } mutate{ split => ["message","|"] add_field => { "time_local" => "%{[message][0]}" } add_field => { "client_ip" => "%{[message][1]}" } add_field => { "domain_name" => "%{[message][2]}" } add_field => { "request" => "%{[message][3]}" } add_field => { "status" => "%{[message][4]}" } add_field => { "body_bytes_sent" => "%{[message][5]}" } add_field => { "request_time" => "%{[message][6]}" } add_field => { "bytes_sent" => "%{[message][7]}" } add_field => { "request_length" => "%{[message][8]}" } add_field => { "upstream_response_time" => "%{[message][9]}" } add_field => { "http_referer" => "%{[message][10]}" } add_field => { "user_agent" => "%{[message][11]}" } remove_field => ["message", "log", "file", "offset", "ecs", "version", "path", "agent", "beat", "@version", "input", "prospector", "source", "offset"] } date { match => [ "time_local","dd/MMM/YYYY:HH:mm:ss Z" ] target => "@timestamp" } }#============================================================================= #调试用 output { stdout { codec => rubydebug } }# 生产环境使用 #output { # elasticsearch { # hosts => [ "http://192.168.2.26:9200","http://192.168.2.27:9200","http://192.168.2.28:9200"] # index => "%{tags}-%{+YYYY.MM.dd}" # } #}

测试配置文件

root@zk-kfk-1:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka-to-es.conf -t Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO ] 2021-10-29 19:05:14.181 [main] runner - Starting Logstash {"logstash.version"=>"7.15.1", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.12+7 on 11.0.12+7 +indy +jit [linux-x86_64]"} [INFO ] 2021-10-29 19:05:14.293 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [INFO ] 2021-10-29 19:05:14.330 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [WARN ] 2021-10-29 19:05:15.140 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2021-10-29 19:05:18.207 [LogStash::Runner] Reflections - Reflections took 166 ms to scan 1 urls, producing 120 keys and 417 values [WARN ] 2021-10-29 19:05:20.035 [LogStash::Runner] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. Configuration OK

5.5 使用调试模式将logstash处理后的日志直接打印到屏

root@zk-kfk-1:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka-to-es.conf ...

5.6 在nginx容器内重新写几条数据进行测试

root@40dff03b2e4a:/# curl 127.0.0.1

5.7 观察logstash是否有相关日志的输出

{ "request" => "GET / HTTP/1.1", "client_ip" => "127.0.0.1", "status" => "200", "@timestamp" => 2021-10-29T11:07:16.000Z, "request_length" => "73", "http_referer" => "-", "domain_name" => "localhost", "user_agent" => "curl/7.64.0", "bytes_sent" => "905", "request_time" => "0.000", "time_local" => "29/Oct/2021:11:07:16 +0000", "fields" => { "log_topics" => "nginx-access" }, "host" => { "name" => "40dff03b2e4a" }, "upstream_response_time" => "-", "tags" => [ [0] "nginx-access" ], "body_bytes_sent" => "612" }

5.7 测试没问题后，更改logstash的配置文件，将日志写入es集群，并重新以服务的方式启动logstash

# 修改配置文件 kafka-to-es.conf #output { # stdout { codec => rubydebug } #} output { elasticsearch { hosts => [ "http://192.168.2.26:9200","http://192.168.2.27:9200","http://192.168.2.28:9200"] index => "%{tags}-%{+YYYY.MM.dd}" } }# systemctl start logstash.service

6.部署kibana并验证数据

6.1 下载安装包并安装

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.15.1-amd64.debdpkg -i kibana-7.15.1-amd64.deb

6.2 修改配置文件如下参数

# 配置文件 /etc/kibana/kibana.yml server.port: 5601 server.host: "192.168.2.27" elasticsearch.hosts: ["http://192.168.2.29:9200"] i18n.locale: "zh-CN"

6.3 查看kibana监听端口

root@zk-kfk-2:~# netstat -ntlp |grep 5601 tcp 0 0 192.168.2.27:5601 0.0.0.0:* LISTEN 31969/node