写在前面
Spring AI Agent系列第七篇。前面把CRUD、MCP、Redis、RabbitMQ、Security、ES全接了Agent。现在解决最后一块拼图——出了事谁第一个知道。

大部分团队的故障响应流程：用户投诉"系统好慢" → 运维去看监控 → 发现某个接口耗时从50ms涨到了3秒 → 排查原因 → 发现数据库连接池满了 → 扩容。从故障发生到人工介入，平均30分钟。

我们方案：Agent每10秒扫一遍Prometheus指标，发现异常自动执行预定义的诊断流程——查日志、看堆栈、判断是否需要重启或扩容。能自动修的自己修，修不了的发告警附诊断报告。上线两个月，平均故障响应时间从30分钟降到了2分钟。

环境：Spring Boot 3.3.0 + Micrometer + Prometheus + Grafana + MCP协议。

一、先把指标暴露出来
pom.xml：

xml

org.springframework.boot
spring-boot-starter-actuator


io.micrometer
micrometer-registry-prometheus

application.yml：

yaml
management:
endpoints:
web:
exposure:
include: health,info,prometheus,metrics,threaddump,heapdump
endpoint:
health:
show-details: always
metrics:
export:
prometheus:
enabled: true
tags:
application: ${spring.application.name}
二、自定义业务指标
java
@Component
public class BusinessMetrics {

private final MeterRegistry meterRegistry; private final Counter orderCreated; private final Counter orderFailed; private final Timer paymentTimer; private final AtomicInteger activeUsers; public BusinessMetrics(MeterRegistry meterRegistry) { this.meterRegistry = meterRegistry; this.orderCreated = Counter.builder("business.orders.created") .description("订单创建总数") .tag("type", "order") .register(meterRegistry); this.orderFailed = Counter.builder("business.orders.failed") .description("订单创建失败数") .tag("type", "order") .register(meterRegistry); this.paymentTimer = Timer.builder("business.payment.duration") .description("支付处理耗时") .register(meterRegistry); this.activeUsers = meterRegistry.gauge("business.users.active", new AtomicInteger(0)); } public void recordOrderSuccess() { orderCreated.increment(); } public void recordOrderFailure() { orderFailed.increment(); } public void recordPayment(long durationMs) { paymentTimer.record(durationMs, TimeUnit.MILLISECONDS); } public void setActiveUsers(int count) { activeUsers.set(count); }

}
Prometheus配置：

yaml
scrape_configs:

• job_name: ‘spring-boot-app’
  metrics_path: ‘/actuator/prometheus’
  scrape_interval: 10s
  static_configs:

  • targets: [‘localhost:8080’]
    labels:
    service: ‘my-app’
    env: ‘production’
    三、监控数据Tool
    java
    @Component
    public class PrometheusMonitorTool {

  private final MeterRegistry meterRegistry;

  @Tool(description = “查询系统核心健康指标：CPU、内存、线程、GC、接口QPS和耗时、错误率”)
  public String checkSystemHealth() {
  StringBuilder report = new StringBuilder(“系统健康报告：\n\n”);

  double heapUsed = meterRegistry.get("jvm.memory.used") .tag("area", "heap").gauge().value(); double heapMax = meterRegistry.get("jvm.memory.max") .tag("area", "heap").gauge().value(); double heapUsage = (heapUsed / heapMax) * 100; report.append(String.format("【JVM堆内存】%.0f MB / %.0f MB (%.1f%%)\n", heapUsed / 1024 / 1024, heapMax / 1024 / 1024, heapUsage)); if (heapUsage > 85) report.append(" ⚠️ 堆内存使用率超过85%，可能有内存泄漏\n"); double gcCount = meterRegistry.get("jvm.gc.pause").functionCounter().count(); report.append(String.format("【GC次数】%.0f 次\n", gcCount)); double threadCount = meterRegistry.get("jvm.threads.live").gauge().value(); report.append(String.format("【活跃线程】%.0f\n", threadCount)); if (threadCount > 500) report.append(" ⚠️ 活跃线程超过500，检查线程泄漏\n"); return report.toString();

  }

  @Tool(description = “检查数据库连接池：活跃连接、等待连接、超时次数”)
  public String checkDatabasePool() {
  StringBuilder report = new StringBuilder(“数据库连接池报告：\n\n”);

  double activeConnections = meterRegistry.get("hikaricp.connections.active").gauge().value(); double idleConnections = meterRegistry.get("hikaricp.connections.idle").gauge().value(); double pendingConnections = meterRegistry.get("hikaricp.connections.pending").gauge().value(); double maxConnections = meterRegistry.get("hikaricp.connections.max").gauge().value(); double timeoutCount = meterRegistry.get("hikaricp.connections.timeout").functionCounter().count(); report.append(String.format("活跃连接：%.0f / %.0f\n", activeConnections, maxConnections)); report.append(String.format("空闲连接：%.0f\n", idleConnections)); report.append(String.format("等待连接：%.0f\n", pendingConnections)); report.append(String.format("超时次数：%.0f\n", timeoutCount)); if (pendingConnections > 0) report.append("⚠️ 有连接等待，考虑扩容连接池\n"); if (timeoutCount > 0) report.append("⚠️ 发生过连接超时\n"); return report.toString();

  }

  @Tool(description = “获取最近N分钟的错误统计，按类型分组”)
  public String getErrorStats(
  @ToolParam(description = “最近N分钟”) int lastMinutes) {

  double error5xx = meterRegistry.get("http.server.requests") .tag("status", "500").functionCounter().count(); double totalRequests = meterRegistry.get("http.server.requests") .functionCounter().count(); double errorRate = error5xx / Math.max(1, totalRequests) * 100; StringBuilder report = new StringBuilder("最近" + lastMinutes + "分钟错误统计：\n"); report.append("5xx错误：" + error5xx + " 次\n"); report.append(String.format("错误率：%.2f%%\n", errorRate)); if (errorRate > 5) report.append("⚠️ 错误率超过5%，紧急排查！\n"); else if (errorRate > 1) report.append("⚡ 错误率偏高\n"); else report.append("✅ 正常\n"); return report.toString();

  }
  }
  四、智能告警Tool
  java
  @Component
  public class AlertManagementTool {

  @Tool(description = “根据监控数据自动判断故障级别并处理。” +
  “P0-致命、P1-严重（自动修复）、P2-警告（记录待查）”)
  public String assessAndHandle(String healthReport) {

  if (healthReport.contains("使用率超过85%") && healthReport.contains("错误率超过5%")) { return handleP0(); } if (healthReport.contains("等待连接") || healthReport.contains("超时次数")) { return handleP1(); } return handleP2();

  }

  private String handleP0() {
  return “🚨 P0致命故障：\n” +
  “1. 已生成thread dump和heap dump\n” +
  “2. 已向运维发送紧急告警\n” +
  “3. 建议：先重启恢复服务，再分析dump定位根因”;
  }

  private String handleP1() {
  return “⚡ P1严重故障，Agent已尝试自动修复：\n” +
  “- 连接池不足→建议扩容\n” +
  “- 线程过多→已生成thread dump”;
  }

  private String handleP2() {
  return “📝 P2警告，已记录监控驾驶舱，下一巡检窗口复核。”;
  }
  }
  五、定时巡检
  java
  @Component
  public class MonitoringCronJob {

  @Autowired private PrometheusMonitorTool monitorTool;
  @Autowired private AlertManagementTool alertTool;

  @Scheduled(fixedRate = 10000) // 每10秒
  public void healthCheck() {
  String health = monitorTool.checkSystemHealth();
  String action = alertTool.assessAndHandle(health);
  if (action.startsWith(“🚨”) || action.startsWith(“⚡”)) {
  sendAlert(action); // 钉钉/飞书/企微
  }
  }

  @Scheduled(fixedRate = 60000) // 每分钟
  public void poolCheck() {
  String report = monitorTool.checkDatabasePool();
  if (report.contains(“⚠️”)) sendAlert(“数据库连接池告警\n” + report);
  }
  }
  六、踩坑记录
  坑1：Prometheus拉取间隔太短。 设1秒一次，/actuator/prometheus 返回几百个指标数据量很大，等于自己DOS自己。生产10-15秒足够。

坑2：Metrics标签爆炸。 给每个Controller方法加了自定义标签，50个URI=50个时间序列，Prometheus内存起飞。只给核心接口加。

坑3：告警风暴。 Agent每10秒扫一次，同问题每次都发告警——一分钟6条一模一样的。必须加告警去重：同类型5分钟内只发一次。

七、总结
Prometheus采集指标 + Agent分析决策 + Grafana给人看驾驶舱。故障响应从手动翻监控的30分钟变成Agent自动处理的2分钟。

没用监控的，先把 /actuator/prometheus 配好——门槛最低、收益最大。

有用的话点赞收藏。