Prometheus录制规则与告警策略实践

概述目标：通过Recording Rules降低查询成本、统一指标命名，并以Alerting Rules实现分级告警与抑制。适用：服务延迟/错误率、资源使用率、队列滞后等核心指标治理。核心与实战录制规则示例（`rules/recording.yml`）：groups: - name: service-latency rules: - record: job:http_request_duration_seconds:p95 expr: percentile_over_time(0.95, http_request_duration_seconds_bucket[5m]) - record: job:error_rate:ratio expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) 告警规则示例（`rules/alerts.yml`）：groups: - name: service-alerts rules: - alert: HighErrorRate expr: job:error_rate:ratio > 0.05 for: 10m labels: severity: critical service: api annotations: summary: "API错误率升高" description: "错误率超过5%持续10分钟" - alert: HighLatencyP95 expr: job:http_request_duration_seconds:p95 > 0.8 for: 5m labels: severity: warning service: api annotations: summary: "API P95 延迟高" description: "P95超过800ms持续5分钟" 示例Prometheus加载规则：rule_files: - "rules/recording.yml" - "rules/alerts.yml" Alertmanager路由与抑制（`alertmanager.yml`）：route: receiver: default group_by: ['alertname','service'] group_wait: 30s group_interval: 5m repeat_interval: 2h routes: - matchers: - severity="critical" receiver: pager receivers: - name: default webhook_configs: [{ url: "http://ops:8080/alerts" }] - name: pager slack_configs: [{ channel: "#alerts", send_resolved: true }] 验证与监控规则校验：promtool check rules rules/recording.yml promtool check rules rules/alerts.yml 运行时检查：curl -s http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.type=="recording")' curl -s http://prometheus:9090/api/v1/alerts | jq 告警流转：amtool alert 常见误区录制规则命名不规范导致下游混乱；应使用层级化前缀与用途后缀。`for`过短造成抖动与告警风暴；需结合历史数据设定合理持续时间。路由与抑制未配置导致重复通知；应按`severity/service`分组并抑制相关联告警。结语通过录制规则与分级告警策略，Prometheus与Alertmanager可实现稳定、可维护的监控体系，并以工具校验保证质量。