1.主机监控

(1).部署node_exporter

[email protected] src]# tar xf node_exporter-0.18.1.linux-amd64.tar.gz
[[email protected] src]# mv node_exporter-0.18.1.linux-amd64 /usr/local/node_exporter-0.18.1
[[email protected] src]# ln -s /usr/local/node_exporter-0.18.1/ /usr/local/node_exporter

//修改node_exporter配置文件
[[email protected] ~]# cat /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node_exporter
[Service]
User=nobody
ExecStart=/usr/local/node_exporter/node_exporter --log.level=error
ExecStop=/usr/bin/killall node_exporter
[Install]
WantedBy=default.target

//启动node_exporter
[[email protected] x]# systemctl start node_exporter

(2).修改prometheus 文件并重启

//修改prometheus配置，添加监控节点
[[email protected] x]# vim /usr/local/prometheus/prometheus.yml
- job_name: 'host_monitor'
	static_configs:
	- targets: ['localhost:9100'] ##新增9100 端口主机监控

//检查prometheus配置语法
[[email protected] prometheus]# cd /usr/local/prometheus/
[[email protected] prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
SUCCESS: 0 rule files found

//重启prometheus
[[email protected] x]# systemctl restart prometheus

导入主机模板8919

2.MySQL 单机监控

(1).部署mysql_exporter

[[email protected] src]# wget -c
https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-
0.12.1.linux-amd64.tar.gz

[[email protected] src]# tar xf mysqld_exporter-0.12.1.linux-amd64.tar.gz
[[email protected] src]# mv mysqld_exporter-0.12.1.linux-amd64 /usr/local/mysqld_exporter-0.12.1
[[email protected] src]# ln -s /usr/local/mysqld_exporter-0.12.1/ /usr/local/mysqld_exporter

//通过systemd 方式管理
[[email protected] ~]# cat /usr/lib/systemd/system/mysqld_exporter.service
[Unit]
Description=mysql Monitoring System
Documentation=mysql Monitoring System

[Service]
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter \
		--collect.info_schema.processlist \
		--collect.info_schema.innodb_tablespaces \
		--collect.info_schema.innodb_metrics \
		--collect.perf_schema.tableiowaits \
		--collect.perf_schema.indexiowaits \
		--collect.perf_schema.tablelocks \
		--collect.engine_innodb_status \
		--collect.perf_schema.file_events \
		--collect.binlog_size \
		--collect.info_schema.clientstats \
		--collect.perf_schema.eventswaits \
		--config.my-cnf=/usr/local/mysqld_exporter/my.cnf

[Install]
WantedBy=multi-user.target

(2).增加配置文件

[[email protected] src]# cat /usr/local/mysqld_exporter/my.cnf
[client]
# host=localhost
user=exporter
password=123456

(3).mysql 添加授权账户

db02 [(none)]>GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON *.* TO
'exporter'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)
db02 [(none)]>flush privileges;

[[email protected] ~]# systemctl start mysqld_exporter

http://ip:9104/metrics

mysql_up 1 ##代表mysql 被监控并且已经启动

(4).修改prometheus 文件并重启

- job_name: 'mysql_monitor'
	static_configs:
	- targets: ['172.16.0.8:9104']

[[email protected] src]# systemctl restart prometheus

Mysql 状态监控模板7362

3.MySQL 主从监控

(1).主从配置

环境准备

[[email protected] src]# cat /data/mysql/mysql3306/my3306.cnf
[mysql]
prompt="\[email protected]\h [\d]>"
[mysqld]
user = mysql
basedir = /usr/local/mysql
datadir = /data/mysql/mysql3306/data
log-error=/data/mysql/mysql3306/data/error_3306.log
server_id = 19
port = 3306
log_bin=/data/mysql/mysql3306/binlog/mysql-bin
binlog_format=row
gtid-mode=on
enforce-gtid-consistency=true
socket = /tmp/mysql3306.sock[[email protected] system]#


[[email protected] system]# cat /data/mysql/mysql3306/my3306.cnf
[mysqld]
user = mysql
basedir = /usr/local/mysql
datadir = /data/mysql/mysql3306/data
log_bin= /data/mysql/mysql3306/binlog/mysql-bin
server_id = 18
gtid-mode=on
enforce-gtid-consistency=true
port = 3306
socket = /tmp/mysql3306.sock

//主库
grant replication slave on *.* to [email protected]'172.16.0.%' identified by '123456';

//从库
CHANGE MASTER TO MASTER_HOST='172.16.0.8',
MASTER_USER='repl',
MASTER_PASSWORD='123456',
MASTER_PORT=3306,
MASTER_AUTO_POSITION=1;

mysql> start slave;
mysql> show slave status\G;

从库增加mysql_exporter 监控，过程和主从步骤一致
从库查看

验证从库指标
mysql_slave_status_slave_io_running

(2).修改prometheus 文件并重启

修改prometheus 配置
- job_name: 'mysql_monitor'
	static_configs:
	- targets: ['172.16.0.8:9104','localhost:9104']

[[email protected] src]# systemctl restart prometheus

主从模板7371

(3).添加MySQL 告警规则

[[email protected] rules]# cat /usr/local/prometheus/rules/mysql.yml
groups:
- name: MySQL-rules
	rules:
	- alert: MySQL Status
		expr: up == 0
		for: 5s
		labels:
			severity: warning
		annotations:
			summary: "{
    {
    $labels.instance}}: MySQL has stop "
			description: "MySQL 数据库挂了,请检查"
	- alert: MySQL Slave IO Thread Status
		expr: mysql_slave_status_slave_io_running == 0
		for: 5s
		labels:
			severity: warning
		annotations:
			summary: "{
    {
    $labels.instance}}: MySQL Slave IO Thread has stop "
			description: "检测MySQL 主从IO 线程运行状态"
	- alert: MySQL Slave SQL Thread Status
		expr: mysql_slave_status_slave_sql_running == 0
		for: 5s
		labels:
			severity: warning
		annotations:
			summary: "{
    {
    $labels.instance}}: MySQL Slave SQL Thread has stop "
			description: "检测MySQL 主从SQL 线程运行状态"

停止从库观察
[[email protected] rules]# systemctl stop mysqld3306

停止从库sql 线程观察
mysql> stop slave sql_thread;

4.Redis 监控

(1).部署redis_exporter

[[email protected] src]# wget https://github.com/oliver006/redis_exporter/releases/download/v0.30.0/redis_exporter-v0.30.0.linux-amd64.tar.gz
[[email protected] src]# mkdir /usr/local/redis_exporter
[[email protected] src]# tar xf redis_exporter-v0.30.0.linux-amd64.tar.gz
[[email protected] src]# mv redis_exporter /usr/local/redis_exporter/

解压后只有一个二进制程序就叫redis_exporter 通过-h 可以获取到帮助信息，下面列出一些常用的选项：

• redis.addr ： 指明一个或多个Redis 节点的地址， 多个节点使用逗号分隔， 默认为
  redis://localhost:6379
• redis.password：验证Redis 时使用的密码；
• redis.file：包含一个或多个redis 节点的文件路径，每行一个节点，此选项与-redis.addr 互
  斥。
• web.listen-address：监听的地址和端口，默认为0.0.0.0:9121

运行redis_exporter 服务
直接启动

• 无密码
  ./redis_exporter redis//172.16.0.9:6379 &
• 有密码
  redis_exporter -redis.addr 172.16.0.9:6379 -redis.password 123456

Systemd 方式启动

vim /usr/lib/systemd/system/redis_exporter.service
[Unit]
Description=redis_exporter
Documentation=https://github.com/oliver006/redis_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/redis_exporter/redis_exporter -redis.addr 172.16.0.9:6379
Restart=on-failure
[Install]
WantedBy=multi-user.target

[[email protected] src]# useradd prometheus -s /sbin/nologin -M

(2).修改prometheus 文件并重启

- job_name: 'redis_exporter'
	scrape_interval: 10s
	static_configs:
	- targets: ['172.16.0.9:9121']

[[email protected] src]# /usr/local/prometheus/promtool check config /usr/local/prometheus/prometheus.yml
[[email protected] src]# systemctl restart prometheus

导入redis 监控模板763


这里注意：如果redis 没有配置内存最大可用值

127.0.0.1:6379> CONFIG GET maxmemory
1) "maxmemory"
2) "0"

则该内存值在grafana 界面显示是0
配置参数如下
maxmemory 128m

(3).添加Redis 告警规则

[[email protected] rules]# cat redis.yml

groups:
- name: redis_instance
	rules:
	#redis 实例宕机危险等级: 5
	- alert: RedisInstanceDown
		expr: redis_up == 0
		for: 10s
		labels:
			severity: warning
		annotations:
			summary: "Redis down (export {
    { $labels.instance }})"
			description: "Redis instance is down\n VALUE = {
    { $value }}\n INSTANCE: {
    { $labels.addr }} {
    { $labels.alias }}"
			
	#redis 内存占用过多危险等级: 4
	- alert: RedisOutofMemory
		expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 60
		for: 3m
		labels:
			severity: warning
		annotations:
			summary: "Out of memory (export {
    { $labels.instance }})"
			description: "Redis is running out of memory > 80%\n VALUE= {
    { $value }}\n INSTANCE:{
    { $labels.addr }} {
    { $labels.alias }}"

	# redis 连接数过多危险等级: 3
	- alert: RedisTooManyConnections
		expr: redis_connected_clients > 2000
		for: 3m
		labels:
			severity: warning
		annotations:
			summary: "Too many connections (export {
    { $labels.instance}})"
			description: "Redis instance has too many connections\n value = {
    {
    $value}}\n INSTANCE:{
    { $labels.addr }} {
    { $labels.alias }}"

[[email protected] rules]# /usr/local/prometheus/promtool check config
/usr/local/prometheus/prometheus.yml
[[email protected] rules]# systemctl restart prometheus

停掉redis 观察

恢复观察

5.elasticsearch 集群监控

(1).Es 集群环境准备

安装java

yum install -y java-1.8.0-openjdk.x86_64

1.安装软件

rpm -ivh elasticsearch-6.6.0.rpm

2.修改配置文件

[[email protected] elasticsearch]# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: Linux
node.name: node-2
path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 172.16.0.7,127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.0.7", "172.16.0.8"]
discovery.zen.minimum_master_nodes: 2

3.修改内存锁定

[[email protected] ~]# systemctl edit elasticsearch
[Service]
LimitMEMLOCK=infinity

4.创建数据目录并授权

mkidr /data/elasticsearch
chown =R elasticsearch:elasticsearch /data/elasticsearch

5.重启服务

systemctl daemon-reload
systemctl start elasticsearch

6.查看日志和端口

tail -f /var/log/elasticsearch/Linux.log
netstat -lntup:grep 9200

(2).部署es_export

wget https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.1.0/elasticsearch_exporter-1.1.0.linux-amd64.tar.gz
tar -xvf elasticsearch_exporter-1.1.0.linux-amd64.tar.gz
mv elasticsearch_exporter-1.1.0.linux-amd64 /us r/local/elasticsearch_exporter-1.1.0
ln -s /usr/local/elasticsearch_exporter-1.1.0 /usr/local/elasticsearch_exporter

运行es_export服务
直接启动
nohup ./elasticsearch_exporter --es.uri http://172.16.0.7:9200 &
–es.uri 默认http://localhost:9200，连接到的Elasticsearch 节点的地址（主机和端口）

Systemd 方式启动

cat /etc/systemd/system/elasticsearch_exporter.service
[Unit]
Description=Elasticsearch stats exporter for Prometheus
Documentation=Prometheus exporter for various metrics
[Service]
ExecStart=/usr/local/elasticsearch_exporter/elasticsearch_exporter --es.uri http://ip:9200
[Install]
WantedBy=multi-user.target

http://ip:9114/metrics/ 查看采集到的信息

(3).修改prometheus 文件并重启

- job_name: 'elasticsearch_exporter'
	scrape_interval: 10s
	metrics_path: "/_prometheus/metrics"
	static_configs:
	- targets: ['172.16.0.5:9114','172.16.0.6:9114','172.16.0.7:9114',]

导入granfana 模板2322/266

(4).添加es 告警规则

集群状态，green（ 所有的主分片和副本分片都正常运行）、yellow（所有的主分片都正常运行，但不是所有的副本分片都正常运行）red（有主分片没能正常运行）

groups:
- name: es
	rules:
	- alert: esclusterwrong
		expr: elasticsearch_cluster_health_status{
    color="green"} != 1
		for: 10s
		labels:
			severity: critical
		annotations:
			description: "elasticsearch cluster {
    {
    $labels.server}} 异常"
	- alert: esDown
		expr: elasticsearch_cluster_health_number_of_nodes != 3
		for: 10s
		labels:
			severity: critical
		annotations:
			description: "elasticsearch service {
    {
    $labels.instance}} down"

6.Docker 监控

cAdvisor 将容器统计信息公开为 Prometheus 指标。
默认情况下，这些指标在/metrics HTTP 端点下提供。
可以通过设置-prometheus_endpoint 命令行标志来自定义此端点。
要使用 Prometheus 监控 cAdvisor，只需在 Prometheus 中配置一个或多个作业，这些作业会
在该指标端点处刮取相关的 cAdvisor 流程。

(1).Docker 环境准备

CentOS 7（使用 yum 进行安装）

# step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
sudo yum-config-manager --add-repo
https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step 3: 更新并安装Docker-CE
sudo yum makecache fast
sudo yum -y install docker-ce
# Step 4: 开启Docker 服务
sudo service docker start

# 下载测试镜像
[[email protected] redis-5.0.8]# docker pull busybox

# 生成容器
[[email protected] redis-5.0.8]# docker run -itd --name bb1 busybox
[[email protected] redis-5.0.8]# docker run -itd --name bb2 busybox

docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest

验证采集的数据

[[email protected] network-scripts]# curl http://172.16.0.8:8080/metrics

(2).修改prometheus 文件并重启

# Prometheus 增加docker 监控
- job_name: 'docker'
	static_configs:
	- targets: ['172.16.0.8:8080']
[[email protected] rules]# /usr/local/prometheus/promtool check config /usr/local/prometheus/prometheus.yml
[[email protected] rules]# systemctl restart prometheus

• 容器CPU 使用率:
  sum(irate(container_cpu_usage_seconds_total{image!=“”}[1m])) without (cpu)
• 查询容器内存使用量（单位：字节）:
  container_memory_usage_bytes{image!=“”}
• 查询容器网络接收量速率（单位：字节/秒）：
  sum(rate(container_network_receive_bytes_total{image!=“”}[1m])) without (interface)
• 查询容器网络传输量速率（单位：字节/秒）：
  sum(rate(container_network_transmit_bytes_total{image!=“”}[1m])) without (interface)
• 查询容器文件系统读取速率（单位：字节/秒）：
  sum(rate(container_fs_reads_bytes_total{image!=“”}[1m])) without (device)
• 查询容器文件系统写入速率（单位：字节/秒）：
  sum(rate(container_fs_writes_bytes_total{image!=“”}[1m])) without (device)

grafana 模板：193 模板：