实现监控机器CPU、内存、磁盘，接口可用性探测监控并异常报警

所需组件：prometheus+blackbox_exporter+alertmanager+prometheus-webhook-dingtalk+node_exporter+grafana

组件说明：

prometheus 监控端
blackbox_exporter http探测端
alertmanager 告警端
prometheus-webhook-dingtalk 钉钉告警组件
node_exporter 采集端
grafana 展示端

组件部署：

prometheus：

下载地址：

https://github.com/prometheus/prometheus/releases/download/

部署步骤：

tar -zxvf prometheus-2.25.0.linux-amd64.tar.gz
mv prometheus-2.25.0.linux-amd64 prometheus

启动命令：

vi /usr/lib/systemd/system/prometheus.service

添加：
[Unit]
Description=https://prometheus.io/

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --log.format=json

[Install]
WantedBy=multi-user.target
启动：systemctl start prometheus.service

配置文件：

/usr/local/prometheus/prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 172.16*.*:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
    - "/etc/prometheus/*.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

#linux主机监控
  - job_name: '172.21.*.*'
    static_configs:
    - targets: ['172.21.*.*:9100']




  - job_name: 'http_duogan'
    metrics_path: /probe
    params:
      module: [http_2xx]  # 仅检查返回2xx状态码的响应i
    scrape_interval: 3m  # 设置探测频率为3分钟
    scrape_timeout: 30s  # 设置探测超时时间为30秒
    static_configs:
      - targets:
          - 'https://www.digitalpole.shujijian.com/multi/alive/alive'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.16.*.*:9115  # 使用Blackbox Exporter进行探测


# redis监控
  - job_name: "redis"
    static_configs:
      - targets: ["172.16.*.*:9121"]


#mysql 监控
  - job_name: 'mysql_monitor'
    static_configs:
      - targets: ['172.21.*.*:9104']
#es监控

  - job_name: 'es_monitor'
    static_configs:
      - targets: ['172.16.*.*:9114']

/etc/prometheus/java.yml 接口探测告警规则

groups:
  - name: http_alerts_duogan
    rules:
      - alert: 系统接口探测异常！
        expr: probe_success{job="http_duogan"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{
   { $labels.instance }}  系统接口探测异常！"
          description: "{
   { $labels.instance }}  系统接口探测异常，接口状态为不可用！"

/etc/prometheus/duogan.yml cpu、内存、磁盘告警规则

groups:
- name: test
  rules:
  - alert: CPU使用率告警
    expr:  1 - avg(irate(node_cpu_seconds_total{mode="idle",job!~"(IDC-GPU|hw-nodes-prod-ES|nodes-test-GPU|nodes-dev-GPU)"}[30m])) by (instance) > 0.8
    for: 1m  
    labels:
      level: warning
    annotations:
      summary: "{
   { $labels.instance }} CPU使用率告警 "
      description: "{
   {$labels.instance}} CPU使用率超过80%(当前值: {
   { $value }} )"

  - alert: CPU使用率告警 
    expr: 1 - avg(irate(node_cpu_seconds_total{mode="idle",job=~"(IDC-GPU|hw-nodes-prod-ES)"}[30m])) by (instance) > 0.75
    for: 1m
    labels:
      level: warning
    annotations:
      summary: "{
   { $labels.instance }} CPU负载告警 "
      description: "{
   {$labels.instance}} CPU使用率超过75%(当前值: {
   { $value }})"

  - alert:  内存使用率告警
    expr:  (1-node_memory_MemAvailable_bytes{job!="IDC-GPU"} /  node_memory_MemTotal_bytes{job!="IDC-GPU"}) * 100 > 80
    labels:
      level: critical
    annotations:
      summary: "{
   { $labels.instance }} 可用内存不足告警"
      description: "{
   {$labels.instance}} 内存使用率已达80% (当前值: {
   { $value }})"

  - alert:  内存使用率告警
    expr:  (1-node_memory_MemAvailable_bytes{job!="IDC-GPU"} /  node_memory_MemTotal_bytes{job!="IDC-GPU"}) * 100 > 75
    labels:
      level: critical
    annotations:
      summary: "{
   { $labels.instance }} 可用内存不足告警"
      description: "{
   {$labels.instance}} 内存使用率已达75% (当前值: {
   { $value }})"



  - alert:  磁盘使用率告警
    expr: 100 - (node_filesystem_avail_bytes{fstype=~"ext4|xfs", mountpoint !~ "/var/lib/[kubelet|rancher].*" } / node_filesystem_size_bytes{fstype=~"ext4|xfs", mountpoint !~ "/var/lib/[kubelet|rancher].*"}) * 100 > 80
    labels:
      level: warning
    annotations:
      summary: "{
   { $labels.instance }} 磁盘使用率告警"
      description: "{
   {$labels.instance}} 磁盘使用率已超过80% (当前值: {
   { $value }})"

  - alert:  磁盘使用率告警
    expr: 100 - (node_filesystem_avail_bytes{fstype=~"ext4|xfs", mountpoint !~ "/var/lib/[kubelet|rancher].*" } / node_filesystem_size_bytes{fstype=~"ext4|xfs", mountpoint !~ "/var/lib/[kubelet|rancher].*"}) * 100 > 75
    labels:
      level: warning
    annotations:
      summary: "{
   { $labels.instance }} 磁盘使用率告警"
      description: "{
   {$labels.instance}} 磁盘使用率已超过75% (当前值: {
   { $value }})"

blackbox_exporter:

下载地址：

https://github.com/prometheus/blackbox_exporter/

部署步骤：

tar -zxvf blackbox_exporter-0.24.0.linux-amd64.tar.gz
mv blackbox_exporter-0.24.0.linux-amd64 blackbox_exporter

启动命令：

nohup ./blackbox_exporter --config.file=blackbox.yml &

配置文件：

/usr/local/blackbox_exporter/blackbox.yml

modules:
  http_2xx:
    prober: http
    http:
      preferred_ip_protocol: "ip4"
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  grpc:
    prober: grpc
    grpc:
      tls: true
      preferred_ip_protocol: "ip4"
  grpc_plain:
    prober: grpc
    grpc:
      tls: false
      service: "service1"
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
      - send: "SSH-2.0-blackbox-ssh-check"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
  icmp_ttl5:
    prober: icmp
    timeout: 5s
    icmp:
      ttl: 5

alertmanager:

下载地址：

https://github.com/prometheus/alertmanager

部署步骤：

tar -zvf alertmanager-0.25.0.linux-amd64.tar.gz
mv alertmanager-0.25.0.linux-amd64.tar.gz alertmanager

启动命令：

nohup ./alertmanager &

配置文件：

/usr/local/alertmanager/alertmanager.yml

route:
  group_by: ['alertname']
  group_wait: 30s    #当传入的警报创建了一组新的警报时，请至少等待多少秒发送初始通知。默认30s
  group_interval: 5m  #发送一次告警后，等待多久再发送同组新增的告警。默认5m
  repeat_interval: 10m  #如果警报已成功发送，请等待多少小时以重新发送警报。默认4h
  receiver: 'webhook1'
receivers:
- name: 'webhook1'
  webhook_configs:
  - url: 'http://172.16.*.*:8060/dingtalk/webhook1/send'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

prometheus-webhook-dingtalk:

下载地址：

https://github.com/timonwong/prometheus-webhook-dingtalk

部署步骤：

tar -zxvf prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz

mv prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz prometheus-webhook-dingtalk

启动命令：

nohup ./prometheus-webhook-dingtalk --config.file=config.example.yml &

配置文件：

/usr/local/prometheus-webhook-dingtalk/config.example.yml

## Request timeout
# timeout: 5s

## Uncomment following line in order to write template from scratch (be careful!)
#no_builtin_template: true

## Customizable templates path
templates:
   - contrib/templates/*.tmpl

## You can also override default template using `default_message`
## The following example to use the 'legacy' template from v0.3.0
#default_message:
#  title: '{
   { template "legacy.title" . }}'
#  text: '{
   { template "legacy.content" . }}'

## Targets, previously was known as "profiles"
targets:
  webhook1:
   #钉钉机器人地址
    url: https://oapi.dingtalk.com/robot/send?access_token=05989278
    # secret for signature
    secret: 钉钉机器人se
  webhook2:
    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
  webhook_legacy:
    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
    # Customize template content
    message:
      # Use legacy template
      title: '{
   { template "legacy.title" . }}'
      text: '{
   { template "legacy.content" . }}'
  webhook_mention_all:
    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
    mention:
      all: true
  webhook_mention_users:
    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
    mention:
      mobiles: ['156xxxx8827', '189xxxx8325']

钉钉告警模版：

/usr/local/prometheus-webhook-dingtalk/contrib/templates/default.tmpl

...
...
{
   {/* Firing */}}

{
   { define "default.__text_alert_list" }}{
   { range . }}

**触发时间:** {
   { dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}

**摘要:** {
   { .Annotations.summary }}

**描述:** {
   { .Annotations.description }}

**监控:** [grafana](http://grafana_ip:8000/grafana/d/GuJ5DHMnz/fu-wu-qi-jian-kong-tu-biao?orgId=1)

**详情:**
{
   { range .Labels.SortedPairs }}{
   { if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {
   { .Name }}: {
   { .Value | markdown | html }}
{
   { end }}{
   { end }}
{
   { end }}{
   { end }}

{
   {/* Resolved */}}

{
   { define "default.__text_resolved_list" }}{
   { range . }}

**触发时间:** {
   { dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}

**解除时间:** {
   { dateInZone "2006.01.02 15:04:05" (.EndsAt) "Asia/Shanghai" }}

**摘要:** {
   { .Annotations.summary }}

**监控:** [grafana](http://grafana_ip:8000/grafana/d/GuJ5DHMnz/fu-wu-qi-jian-kong-tu-biao?orgId=1)

**详情:**
{
   { range .Labels.SortedPairs }}{
   { if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {
   { .Name }}: {
   { .Value | markdown | html }}
{
   { end }}{
   { end }}
{
   { end }}{
   { end }}
...
...

node_exporter:

下载地址：

https://github.com/prometheus/node_exporter

部署步骤：

tar -zxvf node_exporter-1.1.2.linux-amd64.tar.gz
mv node_exporter-1.1.2.linux-amd64.tar.gz node_exporter

启动命令：

systemctl start node_exporter.service

添加

[Unit]
Description=node_exporter
After=network.target

[Service]
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动：systemctl start node_exporter.service

配置文件：

无配置文件

grafana：

下载地址：

http://www.rpmfind.net/linux/rpm2html/search.php?query=grafana&submit=Search+…&system=&arch=

部署步骤：

yum install -y initscripts fontconfig
yum install -y grafana-7.5.7-2.el8.x86_64.rpm

启动命令：

systemctl start grafana-server.service

启动后访问地址：ip:3000初始用户名和密码都是admin
第一次登陆需要修改密码

添加prometheus数据源：Configuration -> Data Sources ->add data source -> Prometheus