每日一Go-70、Prometheus + Grafana 从采集到告警的完整实战（Go + Kind）

Prometheus 是一个以时间序列为核心、通过 Pull 模型采集指标、用 PromQL 做聚合分析、最终通过告警驱动运维决策的监控系统。Grafana 是一个把 Prometheus 里的“冰冷指标”变成你一眼能看懂、能做决策的可视化与告警平台。一、在Kind 集群里装 Prometheus Grafana1. 安装Helm打开Windows终端winget install Helm.Helm 已找到 Helm[Helm.Helm]版本4.0.4此应用程序由其所有者授权给你。 Microsoft 对第三方程序包概不负责也不向第三方程序包授予任何许可证。 正在下载 https://get.helm.sh/helm-v4.0.4-windows-amd64.zip ██████████████████████████████19.5MB/19.5MB已成功验证安装程序哈希 正在提取存档...已成功提取存档 正在启动程序包安装...已修改路径环境变量重启 shell 以使用新值。 添加了命令行别名helm已成功安装2. 添加Helm仓库helm repoaddprometheus-communityhttps://prometheus-community.github.io/helm-charts//国内用这个helm repoaddaliyunhttps://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts helm repo update3. 创建 namespacekubectl createnamespacemonitoring4. 安装 kube-prometheus-stackhelm install monitoring prometheus-community/kube-prometheus-stack \ -n monitoring等待几分钟kubectlgetpods-n monitoring5. 本地访问 Grafanakubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring// PowerShell中输入命令以获取grafana的密码kubectl get secret monitoring-grafana-n monitoring-o jsonpath{.data.admin-password}|ForEach-Object{[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_))}浏览器访问 http://localhost:3000,输入用户名和密码 admin / vy6GVpIgUvmzslR57JDJViyP5x7SLpt1ydlP0upK二、让 Go 服务暴露 Prometheus 指标1. Go 服务代码// 包声明定义包名为 mainpackagemain// 导入依赖包import(fmtstrconvgithub.com/gin-gonic/gingithub.com/prometheus/client_golang/prometheusgithub.com/prometheus/client_golang/prometheus/promhttp)var(httpTotalprometheus.NewCounterVec(prometheus.CounterOpts{Name:http_requests_total,Help:Total HTTP requests,},[]string{method,path,status},))// Prometheus中间件funcprometheusMiddleware()gin.HandlerFunc{returnfunc(c*gin.Context){c.Next()// 先执行后续逻辑// 请求完成后记录指标status:strconv.Itoa(c.Writer.Status())httpTotal.WithLabelValues(c.Request.Method,c.FullPath(),status).Inc()}}// 主函数程序入口funcmain(){fmt.Println(Hello, Codee君!)// 注册指标prometheus.MustRegister(httpTotal)// 创建默认的 Gin 路由引擎r:gin.Default()// 使用中间件所有路由都会被监控r.Use(prometheusMiddleware())// 定义 GET 路由 /ping处理函数返回 JSON 格式的响应r.GET(/ping,func(c*gin.Context){c.JSON(200,gin.H{message:pong,})})// 暴露 metrics 接口r.GET(/metrics,func(c*gin.Context){promhttp.Handler().ServeHTTP(c.Writer,c.Request)})// 启动 HTTP 服务器监听 8080 端口r.Run(:8080)}2. Dockerfile# 第一阶段编译阶段 FROM golang:1.25.5-alpine AS builder # 设置环境变量国内环境建议开启代理加快下载速度 ENV GO111MODULEon \ GOPROXYhttps://goproxy.cn,direct # 设置工作目录 WORKDIR /app # 复制依赖文件并下载利用 Docker 缓存 COPY go.mod go.sum ./ RUN go mod download # 复制源代码并编译 COPY . . # CGO_ENABLED0 确保静态链接能在 alpine 下运行 RUN CGO_ENABLED0 GOOSlinux go build -o main . # 第二阶段运行阶段 FROM alpine:latest # 安装基础工具可选 RUN apk --no-cache add ca-certificates WORKDIR /root/ # 从编译阶段复制二进制文件 COPY --frombuilder /app/main . # 暴露 Gin 和 Prometheus 监听的端口 EXPOSE 8080 # 运行应用 CMD [./main]4. 容器和 部署到 Kindkubectl createnamespacecodee-junapiVersion: apps/v1 kind: Deployment metadata: name: go-metrics-demo namespace: codee-jun spec: replicas: 1 selector: matchLabels: app: go-metrics-demo template: metadata: labels: app: go-metrics-demo spec: containers: - name: app image: imoowi/golang_per_day:day70 ports: - containerPort: 8080apiVersion: v1 kind: Service metadata: name: go-metrics-demo namespace: codee-jun labels: app: go-metrics-demo # ✅ 给 Service 打 label必须 spec: selector: app: go-metrics-demo ports: - name: http port: 8080 targetPort: 8080kubectl apply-f deployment.yaml kubectl apply-f service.yaml$ kubectl.exe get pods-n codee-jun NAME READYSTATUSRESTARTS AGE go-metrics-demo-7c5cfffb4d-nrmlj1/1Running069s$ kubectl.exe get svc-n codee-jun NAMETYPECLUSTER-IP EXTERNAL-IP PORT(S)AGE go-metrics-demo ClusterIP10.96.138.1388080/TCP11s三、让 Prometheus 抓到你的Go服务1. 创建 ServiceMonitorapiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: go-metrics-demo namespace: monitoring labels: release: monitoring spec: namespaceSelector: matchNames: - codee-jun selector: matchLabels: app: go-metrics-demo endpoints: - port: http path: /metrics interval: 15s2. 给 Service 加上 port nameports:-name:httpport:8080targetPort:80803. 验证 Prometheus 是否抓到kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090 -n monitoring访问 http://localhost:9090搜索指标http_requests_total看到数据说明链路通了。四、Grafana Dashboard 实战1. 添加 Prometheus 数据源kube-prometheus-stack 已自动配置好不需要手动加。2. 创建核心面板在Grafana里添加新Dashboard然后添加以下三个面板2.1 QPSsum(rate(http_requests_total[1m]))2.2 错误率Error Ratesum(rate(http_requests_total{status~5..}[5m]))/ sum(rate(http_requests_total[5m]))2.3 P99 延迟histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m]))by(le))五、添加告警1. 配置文件apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: go-service-alerts labels: release: monitoring spec: groups: - name: go-service.rules rules: - alert: HighErrorRate expr:|sum(rate(http_requests_total{status~5..}[5m]))/ sum(rate(http_requests_total[5m]))0.05for: 2m labels: severity: critical annotations: summary:Go 服务错误率过高2. 发布配置kubectl apply -f prometheusrule.alert.yaml3. 在Prometheus里查看结果4. 制造500错误$ kubectl.exe get pods-n codee-jun-o wide NAME READYSTATUSRESTARTS AGE IP NODE NOMINATED NODE READINESS GATES go-metrics-demo-66698549d7-ggfwx1/1Running018m10.244.1.7golang-per-day-workerkubectl.exe exec-it go-metrics-demo-66698549d7-ggfwx-n codee-jun--sh~# apkaddcurl(1/10)Installing brotli-libs(1.2.0-r0)(2/10)Installing c-ares(1.34.6-r0)(3/10)Installinglibunistring(1.4.1-r0)(4/10)Installinglibidn2(2.3.8-r0)(5/10)Installing nghttp2-libs(1.68.0-r0)(6/10)Installingnghttp3(1.13.1-r0)(7/10)Installinglibpsl(0.21.5-r3)(8/10)Installing zstd-libs(1.5.7-r2)(9/10)Installinglibcurl(8.17.0-r1)(10/10)Installingcurl(8.17.0-r1)Executing busybox-1.37.0-r30.triggerOK:13.4MiBin27packages~#whiletrue;docurlhttp://localhost:8080/error;done2分钟后就可以在Prometheus Alert页面看见友情链接:加班费计算器(vx小程序搜索“加班计”)*源码地址*评论区要如果您喜欢这篇文章请您点赞、分享、亮爱心万分感谢