正文

监控prometheus

IP云
IP云V管理员/3 阅读
12/02
提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

一、prometheus-webhook-daingtalak

github地址：[Releases · timonwong/prometheus-webhook-dingtalk · GitHub](https://github.com/timonwong/prometheus-webhook-dingtalk/releases)
下载地址：[](https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v0.3.0/prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz)

自己去GitHub上下载需要的版本，然后解压： 

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v0.3.0/prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz

tar xf prometheus-webhook-dingtalk-0.3..linux-amd64.tar.gz -C /data; cd /data

mv prometheus-webhook-dingtalk-0.3..linux-amd64 prometheus-webhook-dingtalk

修改配置文件:
# cat default.tmpl 

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}

**Labels**

{{ range .Labels.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}

{{ end }}

**Annotations**

{{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}

{{ end }}

**Source:** [{{ .GeneratorURL }}]({{ .GeneratorURL }})

{{ end }}{{ end }}

{{ define "ding.link.title" }}{{ template "__subject" . }}{{ end }}

{{ define "ding.link.content" }}#### \[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**

{{ template "__text_alert_list" .Alerts.Firing }}

{{ end }}

启动服务：
# cat prometheus-webhook-dingtalk.sh 

#!/bin/bash

nohup prometheus-webhook-dingtalk --web.listen-address="0.0.0.0:8060" --ding.profile="test=https://oapi.dingtalk.com/robot/send?access_token=89f3cedfb3c3cdb031bdf10f8fc52bf1add575e9b5fb6f462a8cca6859af4" >>/data/prometheus-webhook-daingtalak/nohub.out >& &

--ding.profile是钉钉机器人生成的，自己创建个钉钉机器人。

二、Alertmanager
github地址：[Releases · prometheus/alertmanager · GitHub](https://github.com/prometheus/alertmanager/releases)

下载地址：[Releases · prometheus/alertmanager · GitHub](https://github.com/prometheus/alertmanager/releases)

自己去GitHub上下载需要的版本，然后解压： 

wget https://github.com/prometheus/alertmanager/releases/download/v0.15.1/alertmanager-0.15.1.linux-amd64.tar.gz

tar xf alertmanager-0.15..linux-amd64.tar.gz -C /data ;cd /data

mv alertmanager-0.15..linux-amd64 alertmanager

修改配置文件，由于我自己使用的是钉钉告警，所以本文使用的钉钉：
# cat alertmanager.yml 

global:

  resolve_timeout: 5m

route:

  group_by: ['alertname']

  group_wait: 10s

  group_interval: 10s

  repeat_interval: 1h

  receiver: 'test'

receivers:

- name: 'test'

  webhook_configs:

   - url: "http://127.0.0.1:8060/dingtalk/test/send"

     send_resolved: true

此处的url是prometheus-webhook-daingtalak的地址，用于将告警信息转换成钉钉可以接受的消息格式。

启动alertmanager：
# cat alertmanager.sh 

#!/bin/bash

nohup alertmanager --config.file="/data/alertmanager/alertmanager.yml" --storage.path="/data/alertmanager/data" --web.listen-address="0.0.0.0:9093" >>/data/alertmanager/nohub.out >& &

alertmanager访问地址：
http://ip:9093

三、Prometheus

github地址：[Releases · prometheus/prometheus · GitHub](https://github.com/prometheus/prometheus/releases)

1、prometheus组成
1）prometheus：主程序，主要负责采集数据以及数据存储，并且对外提供PromQL实现监控数据的查询以及聚合分析；
2）*_exporter：于向Prometheus Server暴露数据采集的endpoint,Prometheus轮训这些Exporter采集并且保存数据；
3）alertManager: 负责实现告警，结合邮件或钉钉
4）pushgateway: Prometheus为一些临时存在的进程，如批处理任务，提供了Push Gateway，这些客户端可以将数据push到Push Gateway中，然后由Push Gateway提供pull接口将数据暴露给PrometheusServer。

5）prometheus主要通过pull的方式获取数据，这样就大大减少了被监控端的压力和系统资源的占用。

2、安装
下载地址：[Releases · prometheus/prometheus · GitHub](https://github.com/prometheus/prometheus/releases)
自己去GitHub上下载需要的版本，然后解压： 

wget https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz

tar xf prometheus-2.3..linux-amd64.tar.gz -C /data ;cd /data

mv prometheus-2.3..linux-amd64 prometheus

然后修改配置文件，定义相应的监控项job:
# cat prometheus.yml 

# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every  seconds. Default is every  minute.

  evaluation_interval: 15s # Evaluate rules every  seconds. The default is every  minute.

  # scrape_timeout is set to the global default (10s).

#remote_write:

#  - url: "http://10.2.79.208:9201/write"

#remote_read:

#  - url: "http://10.2.79.208:9201/read"

# Alertmanager configuration

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      - 127.0.0.1:

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

  # - "first_rules.yml"

  # - "second_rules.yml"

  - "/data/prometheus/mongodb-rules.yml"

  - "/data/prometheus/consul-rules.yml"

  - "/data/prometheus/redis-rules.yml"

  - "/data/prometheus/nginx-rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

    static_configs:

    - targets: ['localhost:9090']

  - job_name: 'mongodb1'

    static_configs:

    - targets: ['10.10.8.70:9218']

  - job_name: 'mongodb1-system'

    static_configs:

    - targets: ['10.10.8.70:9100']

  - job_name: 'mongodb2'

    static_configs:

    - targets: ['10.10.5.108:9218']

rule_files:指定告警规则文件的路径，可以定义自己的告警规则

# cat consul-rules.yml 

---

groups:

- name: consul

  rules:

  - alert: consul_catalog_service_node_healthy

    expr: consul_catalog_service_node_healthy <

    for: 60s

    labels:

      serverity: critical

    annotations:

      descrition: '{{ $labels.node }}  {{ $labels.service_id }} is Unhealth'

      summary: 'some service is unhealth,you must chek it out by consul'

  - alert: consul_node_health

    expr: consul_exporter_build_info <

    for: 60s

    labels:

       serverity: critical

    annotations:

       descrition: '{{ $labels.instance }} consul server is down '

       summary: 'consul server is down'

  - alert: consul_health_service_status

    expr: consul_health_service_status <

    for: 60s

    labels:

      serverity: critical

    annotations:

      descrition: '{{ $labels.node }}  {{ $labels.service_id }} is Unhealth'

      summary: 'some service is unhealth,you must chek it out by consul'

# cat mongodb-rules.yml 

---

groups:

- name: mongodb

  rules:

  - alert: mongodb_mongod_connections

    expr: mongodb_mongod_connections{state='current'} and  mongodb_mongod_connections <

    for: 10s

    labels:

      serverity: critical

    annotations:

      description: '{{ $labels.instance }}   of      {{ $labels.job }}   connections is low  11'

      summary: 'connections is too Low,Mongodb mybe is Down!'

  - alert: mongodb_mongod_connections

    expr: mongodb_mongod_connections{state='current'} and  mongodb_mongod_connections >

    for: 10s

    labels:

      serverity: warning

    annotations:

      description: '{{ $labels.instance }}   of      {{ $labels.job }}   connections is high  570'

      summary: 'connections is too much'

  - alert: mongodb_mongod_memory

    expr:  mongodb_mongod_memory{type='virtual'} and mongodb_mongod_memory <

    for: 5s

    labels:

      serverity: critical

    annotations:

      description: '{{ $labels.instance }} of  {{ $labels.job }} {{ $labels.type }}   is too low'

      summary: 'mongodb mybe is down'

  - alert: mongodb_mongod_replset_member_health

    expr: mongodb_mongod_replset_member_health !=

    for: 5s

    labels:

      serverity: critical

    annotations:

      description: ' {{ $labels.name }}  {{ $labels.state}} is down'

      summary: 'one of replsets node is down'

  - alert: mongodb_mongod_replset_my_state

    expr: mongodb_mongod_replset_my_state{job='mongodb3'} and mongodb_mongod_replset_my_state !=

    for: 5s

    labels:

      serverity: critical

    annotations:

      description: ' replsets master have been  changed, {{ $labels.job }}  is not master'

      summary: 'mongodb3 master is down,chek the status'

#cat redis-rules.yml 

---

groups:

- name: redis

  rules:

  - alert: redis_instantaneous_ops_per_sec

    expr: redis_instantaneous_ops_per_sec <

    for: 120s

    labels:

      serverity: critical

    annotations:

      descrition: '{{ $labels.job }}   is Unhealth'

      summary: 'redis-prod options/sec is too low,redis maybe traffic jam ,you must check it out by "redis-cli slowlog get"'

#cat nginx-rules.yml 

---

groups:

- name: nginx-exporter

  rules:

  - alert: status_code_499

    expr: status_code_499 >

    for: 60s

    labels:

      serverity: critical

    annotations:

      descrition: ' status_code_499:{{ status_code_499 }}'

      summary: 'nginx status code 499 is too much,check loadbalance /var/log/nginx/share.log'

  - alert: status_code_400

    expr: status_code_400 >

    for: 60s

    labels:

      serverity: critical

    annotations:

      descrition: 'status_code_400: {{ status_code_400 }}'

      summary: 'nginx status code 400 is too much,check loadbalance /var/log/nginx/share.log'

nginx是我自己写的一个exportor，地址：https://github.com/cuishuaigit/nginx_exporter

启动：
# cat prometheus.sh 

#!/bin/bash

nohup prometheus --config.file="/data/prometheus/prometheus.yml" --web.listen-address="0.0.0.0:9090"  --storage.tsdb.path="/data/prometheus/data"  --web.console.libraries="/data/prometheus/console_libraries"  --web.console.templates="/data/prometheus/consoles"  --web.enable-admin-api --log.level=info >>/data/prometheus/nohub.out >& &

prometheus_ui访问：
http://ip:9090

四、exporter

1、https://github.com/prometheus/node_exporter

2、https://github.com/prometheus/influxdb_exporter

3、https://github.com/prometheus/mysqld_exporter

4、https://github.com/prometheus/jmx_exporter

5、https://github.com/prometheus/consul_exporter

6、https://github.com/prometheus/haproxy_exporter

相关阅读

目录[+]