Docker 部署ELK之Sentinl日志报警

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

前篇文章简单介绍了Docker 部署ELK，以及使用filebeat收集java日志。这篇我们介绍下日志报警配置，这里我们使用Sentinl插件。

1、修改kibana参数

进入elk容器，修改对应参数

[root@centos-mq ~]# docker exec -it elk /bin/bash
root@70f05fc990bd:/# vim /opt/kibana/config/kibana.yml
sentinl:
settings:
    email:
      active: true
#ssl: true ## 云服务器时打开这注释，因为云服务器会禁用25端口
#port：465
      user: *****@163.com## 发件人
      password: ****## 授权码(不是密码)
      host: smtp.163.com
    report:
      active: false

2、安装Sentinl插件

Sentinl版本要选择与kibana版本一致，否则会安装失败

root@70f05fc990bd:/# /opt/kibana/bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.6.0-0/sentinl-v6.6.0.zip
Attempting to transfer from https://github.com/sirensolutions/sentinl/releases/download/tag-6.6.0-0/sentinl-v6.6.0.zip
Transferring 134770542 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete
root@70f05fc990bd:~# /etc/init.d/kibana restart

在docker里面下载总是失败，我从宿主机下载，然后拷贝到docker容器里：

[root@localhost duan]# pwd
/home/duan
[root@localhost duan]# docker cp /home/duan/sentinl-v6.6.0.zip elk:/opt
[root@localhost duan]# docker exec -it elk sh
# cd opt
# ls
elasticsearch kibana logstash sentinl-v6.6.0.zip

安装时指定的是本地文件：

# /opt/kibana/bin/kibana-plugin install file:////opt/sentinl-v6.6.0.zip
Attempting to transfer from file:////opt/sentinl-v6.6.0.zip
Transferring 134770542 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation was unsuccessful due to error "Command failed: /opt/kibana/node/bin/node /opt/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false

FATAL CLI ERROR YAMLException: can not read an implicit mapping pair; a colon is missed at line 106, column 5:
#ssl: true ## 云服务器时打开这注释，因 ...
^
at generateError (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:160:10)
at throwError (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:166:9)
at readBlockMapping (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1018:11)
at composeNode (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1315:12)
at readDocument (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1478:3)
at loadDocuments (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1538:5)
at load (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1555:19)
at Object.safeLoad (/opt/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1573:10)
at readYaml (/opt/kibana/src/core/server/config/read_config.js:25:38)
at Object.exports.getConfigFromFiles (/opt/kibana/src/core/server/config/read_config.js:50:22)
"
#

上面的配置失败了，因为kibana.yml文件的ssl配置项

#ops.interval:
sentinl:
settings:
email:
active: true
user: xxxx@.com
password: xxxxx
host: smtp..com
ssl: false
report:
active: true
# vi kibana.yml

修改配置文件，删除sentinl插件重新安装插件：

cd /opt/kibana/bin
./kibana-plugin remove sentinl
./kibana-plugin install file:///opt/sentinl-v6.6.0.zip

安装重启完，浏览器访问kibana界面，即可看到Sentinl插件菜单

3、配置报警

Sentinl >> New >> Watcher Advanced

点击保存，会创建一个报警模板，修改模板内容如下：

{
"actions": {
"邮件告警": {
"name": "日志异常",
"throttle_period": "0h2m0s",
"email_html": {
"stateless": false,
"subject": "evolut-api-gateway模块--ERROR日志",
"priority": "medium",
"html": "<p><i>Hi,各位同事请注意下面有 {{payload.hits.total}} 条错误信息，请查看并处理！！</i>.</p>\n<div style=\"color:grey;\">\n <hr />\n</div>\n<div>\n<br>{{#payload.hits.hits}} <li style='color:red'><b>source:</b> {{_source.source}} </li><br><li><b>message</b>: {{_source.message}}</li><br><br>{{/payload.hits.hits}} \n</div>",
"to": "xiong@xxx.com",
"from": "e@126.com"
}
},
"钉钉告警模板": {
"name": "webhook告警",
"throttle_period": "0h2m0s",
"webhook": {
"priority": "medium",
"stateless": false,
"method": "POST",
"host": "oapi.dingtalk.com",
"port": "",
"path": "/robot/send?access_token=bdf86156bcded8b10727ceff898b943ef726baaebd797f760336",
"body": "{\r\n \"msgtype\": \"markdown\",\r\n \"at\": {\r\n \"isAtAll\": \"True\"\r\n },\r\n \"markdown\": {\r\n \"title\": \"异常消息\",\r\n \"text\": \" evolut-api-gateway模块-错误日志: \\n {{#payload.hits.hits}} {{_source.message}} \r\n{{/payload.hits.hits}}\"\r\n }\r\n}",
"params": {
"watcher": "{{watcher.title}}",
"payload_count": "{{payload.hits.total}}"
},
"headers": {
"Content-Type": "application/json"
},
"message": "生产环境异常",
"use_https": true
}
}
},
"input": {
"search": {
"request": {
"index": [
"prd-evolut-api-gateway*"
],
"body": {
"query": {
"bool": {
"must": {
"match": {
"message": "ERROR"
}
},
"filter": {
"range": {
"@timestamp": {
"gte": "now-5m/m",
"lte": "now/m",
"format": "epoch_millis"
}
}
}
}
},
"size": ,
"aggs": {
"dateAgg": {
"date_histogram": {
"field": "@timestamp",
"time_zone": "Asia/Shanghai",
"interval": "1m",
"min_doc_count":
}
}
}
}
}
}
},
"condition": {
"script": {
"script": "payload.hits.total >= 1"
}
},
"transform": {},
"trigger": {
"schedule": {
"later": "every 2 minutes"
}
},
"disable": false,
"report": false,
"title": "evolut-api-gateway"
}

配置完成后，等待设置对应的时间，是要触发设置的报警机制，会看到报警日志发送至设定的邮箱

钉钉：

在sentinl里面加一个watcher：

Input填入：

{
"search": {
"request": {
"index": [
"*"
],
"body": {
"query": {
"bool": {
"must": [
{
"query_string": {
"analyze_wildcard": true,
"query": "\"error\""
}
},
{
"range": {
"@timestamp": {
"gte": "now-10m",
"lte": "now",
"format": "epoch_millis"
}
}
}
],
"must_not": []
}
}
}
}
}
}
condition填入：
{
"script": {
"script": "payload.hits.total > 1"
}
}
添加一个webhook的action：