前言：
blackbox_exporter是监控Prometheus 官方提供的 exporter 之一,主要提供http 、dns
、实现tcp、监控icmp 的源码下载实现监控数据采集。
Consul主要提供
，监控服务发现 ，实现健康检查，监控等功能，云计算实现本次集成主要使用到服务发现功能 。监控
本文主要实现，实现基于consul_sd_config & consul 的监控 prometheus 服务发现
，实现网路设备ping监控
，服务器租用实现站点可用行监控，监控以及证书相关信息监控 。实现
安装环境 ：
k8sconsulPrometheusblackbox_exporter1: Consul 安装
1.1 ：使用helm 安装 consul 复制Bash
# 添加 consul helm 源
helm repo add hashicorp https://helm.releases.hashicorp.com # 安装consul
helm - n consul install \
--set storageClass=alicloud-disk-efficiency \consul hashicorp/ consul \
--version=0.32.11.2.3.4.5.6.7.8. 1.2 ：查看服务安装状态 复制Bash
[root@xxxxxxxx consul_install]# kubectl - n consul get pods
NAME READY STATUS RESTARTS AGE
consul-consul-9lxfc 1/1 Running 0 6 d1h
consul-consul-ntqcf 1/1 Running 0 6 d1h
consul-consul-q7c6f 1/1 Running 0 6 d1h
consul-consul-server-0 1/1 Running 0 6 d1h
consul-consul-server-1 1/1 Running 0 6 d1h
consul-consul-server-2 1/1 Running 0 6d1h1.2.3.4.5.6.7.8.9. 1.3 ：nginx-ingress consulconsul_ingress.yml 复制Bash
# consul.xxxxxx.cn -----> 替换为正确域名apiVersion: networking.k8s.io/ v1
kind: Ingress
metadata: name: consul- ingress
namespace: consul
annotations: kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /spec: rules: - host: consul.xxxxxx.cn http: paths: - path: / pathType: Prefix
backend: service: name: consul-consul- ui
port: number: 801.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23. 执行部署 复制Bash
kubectl apply -f consul_ingress.yml1.2. 1.4：访问测试2: Blackbox_export
2.1：blackbox 安装blackbox-exporter-config.yaml 复制Bash
apiVersion: v1
kind: ConfigMap
metadata: name: blackbox- exporter
labels: app: blackbox- exporter
data: blackbox.yml: |- modules: ## ----------- DNS 检测配置 ----------- dns_tcp: prober: dns
dns: transport_protocol: "tcp" preferred_ip_protocol: "ip4" query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的监控网址
query_type: "A" ## ----------- TCP 检测模块配置 ----------- tcp_connect: prober: tcp
timeout: 5 s
## ----------- ICMP 检测配置 ----------- ping: prober: icmp
timeout: 5 s
icmp: preferred_ip_protocol: "ip4" ## ----------- HTTP GET 2xx 检测模块配置 ----------- http_get_2xx: prober: http
timeout: 10 s
http: method: GET
preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1","HTTP/2"] valid_status_codes: [200] # 验证的HTTP状态码, 默认为2xx
no_follow_redirects: false # 是高防服务器否不跟随重定向
## ----------- HTTP GET 3xx 检测模块配置 ----------- http_get_3xx: prober: http
timeout: 10 s
http: method: GET
preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1","HTTP/2"] valid_status_codes: [301,302,304,305,306,307] # 验证的HTTP状态码, 默认为2xx
no_follow_redirects: false # 是否不跟随重定向
## ----------- HTTP POST 监测模块 ----------- http_post_2xx: prober: http
timeout: 10 s
http: method: POST
preferred_ip_protocol: "ip4" valid_http_versions: ["HTTP/1.1", "HTTP/2"] #headers: # HTTP头设置
# Content-Type: application/ json
#body: { } # 请求体设置1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.46.47.48.49.50.51.52.53.54.55.56.57.58.59. blackbox-exporter-deploy.yaml 复制Bash
apiVersion: v1
kind: Service
metadata: name: blackbox- exporter
labels: k8s-app: blackbox- exporter
spec: type: ClusterIP
ports: - name: http
port: 9115 targetPort: 9115 selector: k8s-app: blackbox- exporter
---apiVersion: apps/ v1
kind: Deployment
metadata: name: blackbox- exporter
labels: k8s-app: blackbox- exporter
spec: replicas: 1 selector: matchLabels: k8s-app: blackbox- exporter
template: metadata: labels: k8s-app: blackbox- exporter
spec: containers: - name: blackbox- exporter
image: prom/blackbox-exporter:v0.19.0 args: - --config.file=/etc/blackbox_exporter/blackbox.yml - --web.listen-address=:9115 - --log.level=info ports: - name: http
containerPort: 9115 resources: limits: cpu: 3 memory: 6000 Mi
requests: cpu: 100 m
memory: 50 Mi
livenessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 readinessProbe: tcpSocket: port: 9115 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 volumeMounts: - name: config
mountPath: /etc/ blackbox_exporter
volumes: - name: config
configMap: name: blackbox- exporter
defaultMode: 4201.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45.46.47.48.49.50.51.52.53.54.55.56.57.58.59.60.61.62.63.64.65.66.67.68.69.70.71.72.73. 执行安装 复制Bash
kubectl apply -f blackbox-exporter-deploy.yamlkubectl apply -f blackbox-exporter-config.yaml1.2.3. 2.2 ：nginx ingress blackbox-exporter • blackbox_ingress.yml 复制Bash
apiVersion: networking.k8s.io/ v1
kind: Ingress
metadata: name: blackbox- ingress
namespace: monitoring
annotations: kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /spec: rules: - host: blackbox-devops.lululemon.cn http: paths: - path: / pathType: Prefix
backend: service: name: blackbox- exporter
port: number: 91151.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21. 执行安装 复制Bash
kubectl apply -f blackbox_ingress.yml1.2. 3: rometheus 添加 服务动态发现
复制Bash
##### http_get_2xx 数据获取
- job_name: http_get_2xx
params: module: - http_get_2xx
scrape_interval: 2 s
scrape_timeout: 2 s
metrics_path: / probe
consul_sd_configs: # consul 服务地址
- server: consul-consul-server.consul.svc.cluster.local:8500 tag_separator: , services: - http_get_2xx
relabel_configs: - source_labels: [__meta_consul_service_address] target_label: __param_target
- source_labels: [__meta_consul_service_address] target_label: instance
- target_label: __address__
## blackbox- export 地址
replacement: blackbox-exporter.monitoring.svc.cluster.local:9115 ####### icmp 配置
- job_name: blackbox_icmp
params: module: - ping
scrape_interval: 2 s
scrape_timeout: 2 s
metrics_path: / probe
consul_sd_configs: # consul 服务地址
- server: consul-consul-server.consul.svc.cluster.local:8500 tag_separator: , services: - ping
relabel_configs: - source_labels: [__meta_consul_service_address] target_label: __param_target
- source_labels: [__meta_consul_service_address] target_label: instance
- target_label: __address__
## blackbox- export 地址
replacement: blackbox-exporter.monitoring.svc.cluster.local:91151.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.42.43.44.45. 4：添加 icmp 监控
4.1 ：添加监控地址到consulicmp_list 复制Bash
192.168.1.1192.168.1.21.2.3. add_consul_service_icmp.sh 复制Bash
#!/usr/bin/ env bash
ip_addr= $1
if test "$ip_addr"; then
curl -X PUT -d { "id": "icmp_${ ip_addr}", "name": "ping", "address": "${ ip_addr}", "port": 443, "Meta": { "env": "prod", "team": "network", "project": "network", "owner": "Mike" }, "tags": ["node"], "checks": [{ "http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]} \
http://consul-consul-server:8500/v1/agent/service/ register
else
echo "请输入参数"fi1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24. 添加service ping 复制Bash
for i in `cat icmp_list`;do bash add_consul_service_icmp.sh $i;done1.2. 4.2：查看consul 服务
4.3：删除ping 监控地址脚本 复制Bash
#!/usr/bin/ env bash
ip_addr= $1
curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ ip_addr}1.2.3.4.5. 5: 添加http_get_2xx
5.1：添加监控域名domain_name_list 复制Bash
wwww.baidu.comwwww.1111.comwwww.2222.com1.2.3.4. add_consul_service_http_get_2xx.sh 复制Bash
#!/usr/bin/ env bash
service_name= $1
if test "$service_name"; then
curl -X PUT -d { "id": "http_get_2xx_${ service_name}", "name": "http_get_2xx", "address": "https://${ service_name}", "port": 443, "Meta": { "env": "prod", "team": "web", "project": "web", "owner": "Devops" }, "tags": ["node"], "checks": [{ "http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]} \
http://consul-consul-server:8500/v1/agent/service/ register
else
echo "请输入参数"fi1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24. 添加 service http_get_2xx 复制Bash
for i in `cat domain_name_list`;do bash add_consul_service_http_get_2xx.sh $i;done1.2. 5.2 ：查看consul 服务
5.3 ：删除域名监控脚本del_consul_service_http_get_2xx.sh 复制Bash
#!/usr/bin/ env bash
ip_addr= $1
curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ ip_addr}1.2.3.4.5. 6
：查看prometheus 监控
总结 ：
使用上述方案
，黑盒监控与自建cmdb 平台很容易进行集成
，使其监控自动化 ，亿华云不需要过多的人工干预
，可以省去大量的人工成本
，grafana 的模板下载配置这里就不进行过多介绍，自行通过谷歌完成
。