企业 DevOps + 协同办公 + 可观测性 + 网络基础设施平台-Part 15.26 Enterprise Logging & Observability Platform Role

关注

原创最新推荐文章于 2026-10-28 00:00:00 发布 202 阅读 AI 写同款· GEO 优化 ›

【信息科学与工程学】【数据中心】第三十五篇云计算数据中心的学科知识04 编号学科（课程）核心知识点在云计算/云存储/云网络/云安全/云MaaS中的作用代表教材/资料/论文 + 数学方程式列表工业界应用D1421云原生数据库：TiDBHTAP（混合事务/分析处理）、分布式SQL、水平扩展、强一致（Raft）、自动故障恢复、与MySQL兼容、TiFlash列式引擎提供弹性扩展的分布式数据库；支撑高并发在线交易与实时分析教材：《TiDB in Action》PingCAP（2020）论文：《TiDB: A Raft-based HTAP Database》（2017）阅读详情

本部分生成企业级 Logging & Observability Platform 自动化部署 Ansible Role

覆盖企业生产可观测平台：

Elasticsearch Cluster
Logstash
Kibana
Filebeat
Fluent Bit
Loki
Promtail
Grafana
Prometheus
AlertManager
Thanos
OpenTelemetry Collector
Jaeger Distributed Tracing
Grafana Tempo
Uptime Monitor
Audit Log Platform

适用于：

Ubuntu 22.04 / 24.04

CentOS Stream 9

Docker

Kubernetes

Helm

Ansible >=2.15

企业生产环境

15.26.1 Observability整体架构

                         Application


                              |


             +----------------+----------------+

             |                |                |

          Logs            Metrics           Trace


             |                |                |


        Fluent-bit       Prometheus       OpenTelemetry


             |                |                |


     +-------+-------+        |        +------+------+

     |               |        |        |             |

 Elasticsearch     Loki      |      Jaeger       Tempo


     |               |        |        |             |


     +---------------+--------+--------+-------------+


                         |


                    Grafana


                         |


              Dashboard / Alert / Query


                         |


                    AlertManager


                         |


                 Email / Slack / Webhook

15.26.2 Role目录结构

创建：

roles/observability-platform/


├── defaults/

│   └── main.yml


├── tasks/


│   ├── main.yml
│   ├── elasticsearch.yml
│   ├── logstash.yml
│   ├── kibana.yml
│   ├── filebeat.yml
│   ├── fluent-bit.yml
│   ├── loki.yml
│   ├── promtail.yml
│   ├── prometheus.yml
│   ├── alertmanager.yml
│   ├── grafana.yml
│   ├── thanos.yml
│   ├── opentelemetry.yml
│   ├── jaeger.yml
│   ├── tempo.yml
│   ├── uptime.yml
│   └── audit.yml



├── templates/


│
│   ├── elasticsearch.yml.j2
│   ├── logstash.conf.j2
│   ├── kibana.yml.j2
│   ├── prometheus.yml.j2
│   ├── alertmanager.yml.j2
│   ├── loki.yaml.j2
│   ├── promtail.yaml.j2
│   ├── grafana.ini.j2
│   ├── thanos.yml.j2
│   └── otel-config.yaml.j2



└── handlers/


    └── main.yml

15.26.3 默认变量

文件：

roles/observability-platform/defaults/main.yml

---


observability_root:

 /mnt/observability



##################
# Elasticsearch
##################


elasticsearch_version:

 8.15



elastic_heap:

 4g



##################
# Loki
##################


loki_version:

 3.2



##################
# Prometheus
##################


prometheus_version:

 v2.55



##################
# Grafana
##################


grafana_version:

 11.2



##################
# AlertManager
##################


alertmanager_version:

 v0.27



##################
# Thanos
##################


thanos_version:

 v0.37



##################
# OpenTelemetry
##################


otel_version:

 0.110



##################
# Jaeger
##################


jaeger_version:

 1.62



##################
# Tempo
##################


tempo_version:

 2.6

15.26.4 Inventory

新增：

inventory/production/hosts.yml

---

observability:


 children:


  monitoring:


   hosts:


    monitor01:


      ansible_host:

       10.10.40.10



    monitor02:


      ansible_host:

       10.10.40.11

15.26.5 Playbook

文件：

playbooks/observability.yml

---

- name:

    Deploy Observability Platform


  hosts:

    monitoring


  become:

    true


  roles:


    - observability-platform

15.26.6 Role入口

文件：

tasks/main.yml

---


- import_tasks:

    elasticsearch.yml



- import_tasks:

    logstash.yml



- import_tasks:

    kibana.yml



- import_tasks:

    filebeat.yml



- import_tasks:

    fluent-bit.yml



- import_tasks:

    loki.yml



- import_tasks:

    promtail.yml



- import_tasks:

    prometheus.yml



- import_tasks:

    alertmanager.yml



- import_tasks:

    grafana.yml



- import_tasks:

    thanos.yml



- import_tasks:

    opentelemetry.yml



- import_tasks:

    jaeger.yml



- import_tasks:

    tempo.yml



- import_tasks:

    uptime.yml



- import_tasks:

    audit.yml

15.26.7 Elasticsearch Cluster

架构


          Application Logs


                 |


             Filebeat


                 |


        +--------+--------+

        |        |        |

      ES01     ES02     ES03


        Elasticsearch Cluster


                 |


              Kibana

Docker Compose:

services:


 elasticsearch:


  image:

   docker.elastic.co/elasticsearch/elasticsearch:8.15.0



  environment:


   discovery.type:

    multi-node


   cluster.name:

    production-es



   ES_JAVA_OPTS:

    "-Xms4g -Xmx4g"



  volumes:


   - /mnt/observability/es:/usr/share/elasticsearch/data


  ports:


   - 9200:9200

检查：

curl http://localhost:9200

输出：

{
 "cluster_name":"production-es"
}

15.26.8 Logstash

作用：

Input

 |

Filter

 |

Output

模板：

templates/logstash.conf.j2

input {


 beats {


  port => 5044


 }


}



filter {


 json {


  source => "message"


 }


}



output {


 elasticsearch {


 hosts =>

 ["http://elasticsearch:9200"]


 index =>

 "application-%{+YYYY.MM.dd}"


 }


}

启动：

docker compose up -d logstash

15.26.9 Kibana

部署：

services:


 kibana:


  image:

   docker.elastic.co/kibana/kibana:8.15.0


  ports:


   - 5601:5601


  environment:


   ELASTICSEARCH_HOSTS:


    http://elasticsearch:9200

访问：

http://monitor:5601

15.26.10 Filebeat

用途：

采集：

Linux日志
Docker日志
Kubernetes日志

配置：

filebeat.inputs:


- type:

  container


  paths:


  - /var/lib/docker/containers/*/*.log



output.elasticsearch:


 hosts:


 - elasticsearch:9200

启动：

systemctl restart filebeat

15.26.11 Fluent Bit

Kubernetes推荐。

DaemonSet：

apiVersion:

apps/v1


kind:

DaemonSet


metadata:


name:

fluent-bit


spec:


template:


spec:


containers:


- name:

 fluent-bit


 image:

 fluent/fluent-bit:3.1

输出：

Pod Log

 |

Fluent Bit

 |

Loki

15.26.12 Loki

架构：


Logs


 |

Promtail


 |

Loki


 |

Grafana

部署：

services:


 loki:


  image:

   grafana/loki:3.2.0


  ports:


   - 3100:3100


  volumes:


   - /mnt/observability/loki:/loki

测试：

curl localhost:3100/ready

15.26.13 Promtail

配置：

server:


 port:

 9080



clients:


- url:


 http://loki:3100/loki/api/v1/push



positions:


 filename:

 /tmp/positions.yaml



scrape_configs:


- job_name:


 kubernetes

15.26.14 Prometheus

架构：


Exporter


 |

Prometheus


 |

Grafana

配置：

模板：

global:


 scrape_interval:

 15s



scrape_configs:



- job_name:

 node


 static_configs:


 - targets:


   - node-exporter:9100



- job_name:

 kubernetes


 kubernetes_sd_configs:


 - role:

   pod

部署：

services:


 prometheus:


  image:

   prom/prometheus:v2.55


  ports:


   - 9090:9090

检查：

curl localhost:9090/-/healthy

15.26.15 Node Exporter

安装：

- name:

 install node exporter


docker_container:


 name:

 node-exporter


 image:

 prom/node-exporter

指标：

CPU

Memory

Disk

Network

Filesystem

15.26.16 AlertManager

配置：

模板：

global:


 smtp_smarthost:

  smtp.company.com:25



route:


 receiver:

  email



receivers:


- name:

 email


 email_configs:


 - to:


   admin@company.com

启动：

docker compose up -d alertmanager

15.26.17 Grafana

部署：

services:


 grafana:


  image:

   grafana/grafana:11.2


  ports:


   - 3000:3000


  volumes:


   - /mnt/observability/grafana:/var/lib/grafana

自动添加Datasource：

datasources:


- name:

 Prometheus


 type:

 prometheus


 url:

 http://prometheus:9090



- name:

 Loki


 type:

 loki


 url:

 http://loki:3100

15.26.18 Thanos

用途：

Prometheus长期存储。

架构：


Prometheus


 |

Sidecar


 |

Object Storage


 |

Thanos Query


 |

Grafana

部署：

services:


 thanos-query:


  image:

   quay.io/thanos/thanos:v0.37


  command:


   query


  ports:


   - 10902:10902

支持：

MinIO
S3
OSS
Ceph RGW

15.26.19 OpenTelemetry Collector

统一采集：

Trace

Metric

Log

配置：

receivers:


 otlp:


  protocols:


   grpc:


   http:



exporters:


 prometheus:


 jaeger:



service:


 pipelines:


  traces:


  metrics:


  logs:

15.26.20 Jaeger

部署：

services:


 jaeger:


  image:

   jaegertracing/all-in-one:1.62


  ports:


   - 16686:16686


   - 4317:4317

访问：

http://monitor:16686

15.26.21 Grafana Tempo

部署：

services:


 tempo:


  image:

   grafana/tempo:2.6


  ports:


   - 3200:3200

Trace查询：

Grafana

 |

Tempo

15.26.22 Uptime Monitor

部署：

services:


 uptime-kuma:


  image:

   louislam/uptime-kuma


  ports:


   - 3001:3001

监控：

HTTP

TCP

Ping

SSL

Docker

15.26.23 Audit Log Platform

统一审计：

采集：


Linux auditd

Kubernetes Audit

IAM Audit

Application Audit

配置：

audit:


sources:


 - kubernetes


 - linux


 - keycloak


 - vault


retention:

365d

15.26.24 Monitoring健康检查

文件：

scripts/check-observability.sh

#!/bin/bash


echo Elasticsearch

curl localhost:9200



echo Loki

curl localhost:3100/ready



echo Prometheus

curl localhost:9090/-/healthy



echo Grafana

curl localhost:3000/api/health



echo AlertManager

curl localhost:9093/-/healthy



echo Thanos

curl localhost:10902/-/healthy



echo Jaeger

curl localhost:16686

Part 15.26完成

✅ Elasticsearch Cluster
✅ Logstash
✅ Kibana
✅ Filebeat
✅ Fluent Bit
✅ Loki
✅ Promtail
✅ Prometheus
✅ AlertManager
✅ Grafana
✅ Thanos长期存储
✅ OpenTelemetry Collector
✅ Jaeger
✅ Grafana Tempo
✅ Uptime Monitor
✅ Audit Platform
✅ Monitoring Health Check