Cilium Hubble 可观测层部署实战启用 Relay、安装 CLI、验证 API 访问与故障排查【免费下载链接】ciliumeBPF-based Networking, Security, and Observability项目地址: https://gitcode.com/GitHub_Trending/ci/ciliumHubble 是 Cilium 的可观测层用于获取 Kubernetes 集群在网络与安全层面的集群级流量可视化能力。本文基于 Cilium 官方文档Documentation/observability/hubble/setup.rst的完整脉络展开覆盖从启用 Hubble Relay、安装 Hubble CLI、验证 Hubble API 访问到部署故障排查的全套流程并结合仓库中的 CLI 与 Helm 源码说明各命令背后的实际行为帮助你把 Hubble 稳定跑在真实集群上。前置条件确认 Cilium 已正确安装本指南假设 Cilium 已经正确安装在你的 Kubernetes 集群中安装方式参见Documentation/gettingstarted/k8s-install-default.rst对应的安装文档。如果不确定先运行$ cilium status确认 Cilium 处于 up and running 状态后再进入 Hubble 的启用流程。启用 Hubble 与 Hubble Relay关键前提TCP 4244 端口必须在所有节点放行启用 Hubble 有一个硬性要求所有运行 Cilium 的节点必须开放 TCP 端口 4244。该端口是各节点上 Hubble agent 对外暴露 Hubble API 的端口Hubble Relay 需要通过节点上的这个端口经由hubble-peerService汇聚整个集群的流数据。如果防火墙/安全组未放行 4244Relay 将无法连接各节点的 Hubble API。方式一Cilium CLIcilium hubble enable$ cilium hubble enable Found existing CA in secret cilium-ca ✨ Patching ConfigMap cilium-config to enable Hubble... ♻️ Restarted Cilium pods Generating certificates for Relay... 2021/04/13 17:11:23 [INFO] generate received request 2021/04/13 17:11:23 [INFO] received CSR 2021/04/13 17:11:23 [INFO] generating key: ecdsa-256 2021/04/13 17:11:23 [INFO] encoded CSR 2021/04/13 17:11:23 [INFO] signed certificate with serial number 365589302067830033295858933512588007090526050046 2021/04/13 17:11:24 [INFO] generate received request 2021/04/13 17:11:24 [INFO] received CSR 2021/04/13 17:11:24 [INFO] generating key: ecdsa-256 2021/04/13 17:11:24 [INFO] encoded CSR 2021/04/13 17:11:24 [INFO] signed certificate with serial number 644167683731852948186644541769558498727586273511 ✨ Deploying Relay...从输出可以看到cilium hubble enable依次完成了四件事查找/复用 CA、打补丁cilium-configConfigMap 以开启 Hubble、重启 Cilium Pod 使配置生效、为 Relay 生成 TLS 证书并部署 Relay。从源码可以印证这条命令的行为cilium hubble enable定义在 cilium-cli/cli/hubble.go其核心实现是 cilium-cli/hubble/hubble.go 中的EnableWithHelm——它本质上执行一次 Helm upgrade把hubble.relay.enabled和hubble.ui.enabled两个值注入 chart并复用既有 valuesReuseValues: true。命令还支持两个常用开关见 addCommonHubbleEnableFlags--relay默认true是否部署 Hubble Relay--ui默认false是否同时启用 Hubble UI。方式二Helm如果 Cilium 是通过helm install安装的Hubble 本身默认已启用。此时只需再开启 Hubble Relayhelm upgrade cilium install/kubernetes/cilium \ --namespace kube-system \ --reuse-values \ --set hubble.relay.enabledtrue对应的 Helm 值定义在 install/kubernetes/cilium/values.yaml 的hubble配置块中hubble.relay、hubble.ui、hubble.tls等小节其中 relay 相关配置从 第 1766 行 附近开始。Hubble 各组件的角色从源码结构看启用 Hubble 后集群中涉及的可观测组件及其职责如下均可在仓库中对应到源码与清单组件部署形态职责端口Hubble agent内嵌在 cilium agent 中采集本节点 eBPF 捕获的流flow并保留在 ring buffer4244节点上对外暴露 Hubble APIHubble Relay独立 Deploymenthubble-relay汇聚所有节点的 Hubble API向客户端提供集群级查询4245gRPC、4222gRPC healthhubble-peerService指向各节点 cilium Pod 的 headless ServiceRelay 通过hubble-peer.ns.svc.cluster.local:443连接各节点的 Hubble API4244443 → 4244Hubble CLI本地命令行工具查询/订阅 Hubble API连接 127.0.0.1:4245经 port-forwardRelay 的服务端实现位于 hubble-relay/main.go 与 hubble-relay/cmd/serve/serve.go。Helm chart 中也为这些组件预置了独立的 ServiceAccounthubble-relay、hubble-ui以及用于 TLS 证书自动生成的hubble-generate-certs见 install/kubernetes/cilium/values.yaml 中serviceAccounts.relay/serviceAccounts.ui/serviceAccounts.hubblecertgen小节。一个值得注意的特性是Hubble 是运行在 Cilium Agent 内的非关键系统。即使 Hubble 启动失败Cilium Pod 本身仍会保持 Running 和健康状态——这一设计在故障排查时非常关键后文详述。验证cilium statusHubble 已启用且运行正常启用后运行cilium status验证$ cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: OK \__/¯¯\__/ Hubble Relay: OK \__/ ClusterMesh: disabled DaemonSet cilium Desired: 1, Ready: 1/1, Available: 1/1 DaemonSet cilium-envoy Desired: 1, Ready: 1/1, Available: 1/1 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 1 cilium-envoy Running: 1 cilium-operator Running: 1 clustermesh-apiserver hubble-relay Running: 1 Cluster Pods: 8/8 managed by CiliumHubble Relay: OK与hubble-relayDeploymentReady: 1/1即表示 Hubble 已启用且 Relay 正常运行。安装 Hubble CLI要访问 Hubble 采集的可观测数据需要在本机安装 Hubble CLI。以下为官方文档给出的三种平台安装方式均下载最新 release 并校验 SHA256LinuxHUBBLE_VERSION$(curl -s https://raw.githubusercontent.com/cilium/hubble/main/stable.txt) HUBBLE_ARCHamd64 if [ $(uname -m) aarch64 ]; then HUBBLE_ARCHarm64; fi curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum} sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}macOSHUBBLE_VERSION$(curl -s https://raw.githubusercontent.com/cilium/hubble/main/stable.txt) HUBBLE_ARCHamd64 if [ $(uname -m) arm64 ]; then HUBBLE_ARCHarm64; fi curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-${HUBBLE_ARCH}.tar.gz{,.sha256sum} shasum -a 256 -c hubble-darwin-${HUBBLE_ARCH}.tar.gz.sha256sum sudo tar xzvfC hubble-darwin-${HUBBLE_ARCH}.tar.gz /usr/local/bin rm hubble-darwin-${HUBBLE_ARCH}.tar.gz{,.sha256sum}注意 Linux 与 macOS 的架构检测差异Linux 判断aarch64macOS 判断arm64uname -m在两个平台上返回不同字串。WindowsPowerShell/cmd curlcurl -LO https://raw.githubusercontent.com/cilium/hubble/main/stable.txt set /p HUBBLE_VERSIONstable.txt curl -L --fail -O https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz curl -L --fail -O https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum certutil -hashfile hubble-windows-amd64.tar.gz SHA256 type hubble-windows-amd64.tar.gz.sha256sum :: verify that the checksum from the two commands above match tar zxf hubble-windows-amd64.tar.gz解压后需将hubble.exe移动到%PATH%环境变量列出的某个目录中即可全局使用。验证 Hubble API 访问建立端口转发Hubble CLI 通过 gRPC 连接 Hubble Relay 的 4245 端口。由于 Relay 是集群内 Deployment本地需要先建立端口转发。以下命令均使用-P--port-forward标志自动从本机将 Hubble Relay 服务转发到本地4245端口详见 Documentation/observability/hubble/port-forward.rst$ hubble status -P Healthcheck (via 127.0.0.1:4245): Ok Current/Max Flows: 11917/12288 (96.98%) Flows/s: 11.74 Connected Nodes: 3/3也可以省略-P标志手动建立端口转发。Cilium CLI 方式$ cilium hubble port-forward ℹ️ Hubble Relay is available at 127.0.0.1:4245或者用 kubectl$ kubectl -n kube-system port-forward service/hubble-relay 4245:80 Forwarding from 127.0.0.1:4245 - 4245 Forwarding from [::1]:4245 - 4245从源码看cilium hubble port-forward实现在 cilium-cli/cli/hubble.go它通过 RelayPortForwardCommand 对命名空间内的hubble-relayService 执行 port-forward--port-forward标志默认值为4245传0则随机选取端口成功后打印Hubble Relay is available at 127.0.0.1:port。此外源码中还定义了cilium hubble ui命令默认转发到本地12000端口--open-browser默认开启用于直接打开 Hubble UI 网页。查询流数据确认健康检查通过后可以直接查询流flowAPI$ hubble observe -P Feb 12 19:13:58.111: kube-system/hubble-relay-6467f4f4d-xrxfs:47550 (ID:95552) - 172.18.0.2:4244 (host) to-stack FORWARDED (TCP Flags: ACK, PSH) ...hubble status输出中的几个关键字段含义Healthcheck经本地转发端口127.0.0.1:4245对 Relay 的 gRPC 健康检查Current/Max FlowsRelay 侧 ring buffer 中当前保留的流数与容量上限示例中 11917/12288约 97%Flows/s当前流采集速率Connected NodesRelay 已连上的节点数示例为 3/3。自定义服务器地址与更多选项如果你把端口转发到了4245之外的端口例如使用--port-forward-port PORT做自动端口转发必须用--server标志或HUBBLE_SERVER环境变量指定 Hubble 服务器地址默认值localhost:4245运行hubble help status、hubble help observe查看子命令帮助运行hubble config查看/配置 Hubble CLI 的全部参数。如果集群已启用 Hubble TLShubble.tls相关配置访问 Hubble API 时还需额外提供 TLS 证书/密钥等标志参见仓库中 Hubble TLS 相关文档。故障排查cilium status Pod 状态 日志总体判断原则先运行cilium status定位问题范围。注意两条规则Hubble Relay 已启用时cilium status中Hubble Relay一行应显示OK否则会出现 errors/warningsHubble 已启用时Cilium一行应显示OK否则会出现 errors/warnings。由于 Hubble 是非关键系统Hubble 失败不会导致 Cilium Pod 本身崩溃——Cilium Pod 依然 Running/Ready。如果Cilium与Hubble Relay同时报 warning/error往往说明 Hubble 配置有误或 Hubble 子系统整体启动失败。场景一Hubble Relay 异常cilium status报告 Relay 错误时的典型输出$ cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: OK \__/¯¯\__/ Hubble Relay: 1 errors, 2 warnings \__/ ClusterMesh: disabled DaemonSet cilium Desired: 1, Ready: 1/1, Available: 1/1 Deployment hubble-relay Desired: 1, Unavailable: 1/1 ... Errors: hubble-relay hubble-relay 1 pods of Deployment hubble-relay are not ready Warnings: hubble-relay hubble-relay-85f98cc7df-s2lkq pod is pending按以下步骤定位查看 Relay Pod 状态$ kubectl -n kube-system get pods -l k8s-apphubble-relay NAME READY STATUS RESTARTS AGE hubble-relay-6467f4f4d-x825b 0/1 CrashLoopBackOff 5 (19s ago) 7m28s若 Pod 处于Pending用kubectl describe查看调度/资源问题$ kubectl describe -n kube-system pod/hubble-relay-6467f4f4d-x825b若 Pod 未Running或 CrashLoopBackOff查看日志$ kubectl -n kube-system logs hubble-relay-6467f4f4d-x825b日志中可以看到 Relay 启动的关键信息gRPC 健康服务监听:4222gRPC 服务器监听:4245并尝试连接peerTarget:hubble-peer.kube-system.svc.cluster.local.:443建立 peer 变更通知。若日志出现connection refused例如dial tcp 10.96.49.4:443: connect: connection refused说明 Hubble Relay 无法通过hubble-peerService 连到 Cilium agent 暴露的 Hubble API——常见根因就是节点未放行 4244 端口或节点上的 Cilium agent Hubble 功能未启用。TLS 相关错误参见仓库中 Hubble TLS 故障排查章节。场景二Hubbleagent 侧异常Cilium一行报 warning 的典型输出$ cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: 1 warnings \__/¯¯\__/ Operator: OK ... Errors: hubble-relay hubble-relay 1 pods of Deployment hubble-relay are not ready Warnings: cilium cilium-5bjkq Hubble: failed to setup metrics: metric unknown-metric does not exist注意 warning 直接指明原因Hubble: failed to setup metrics: metric unknown-metric does not exist。排查步骤检查 Cilium Pod 状态$ kubectl -n kube-system get pods -l k8s-appcilium NAME READY STATUS RESTARTS AGE cilium-5bjkq 1/1 Running 1 (18m ago) 33mPending 的 Pod 用kubectl describe -n kube-system pod/cilium-5bjkq排查未 Running 或反复重启的 Pod过滤 Hubble 子系统日志$ kubectl logs -n kube-system -c cilium-agent -l k8s-appcilium --tail-1 | grep subsyshubble time2025-02-12T22:12:01.227357082Z levelinfo msgStarting Hubble Metrics server address:9965 metricsunknown-metric subsyshubble tlsfalse time2025-02-12T22:12:01.22740229Z levelerror msgFailed to launch hubble errorfailed to setup metrics: metric unknown-metric does not exist subsyshubble该案例中hubble.metrics配置了不存在的指标名unknown-metric导致 Hubble 启动失败。修复方式是回到cilium-config或 Helm values 的hubble.metrics核对指标名后重启。小结与延伸阅读本文按照 Cilium 官方文档Documentation/observability/hubble/setup.rst的完整流程梳理了 Hubble 部署的四个环节启用CLI/Helm注意 4244 端口、安装 Hubble CLI三平台 校验和、验证 API 访问port-forward hubble status/hubble observe、故障排查cilium status分层定位 Relay 与 agent 两侧问题。关键操作要点回顾节点必须放行 TCP 4244Relay 经hubble-peerService 聚合各节点流量Relay 对客户端监听 4245gRPC、4222健康检查CLI 经 port-forward 到127.0.0.1:4245非 4245 转发端口需配合--server或HUBBLE_SERVER使用Hubble 失败不影响 Cilium 数据面但要借助subsyshubble日志与cilium status的 errors/warnings 定位。后续可继续阅读仓库中的相关文档Hubble CLI 使用Hubble UIHubble 配置参考端口转发说明【免费下载链接】ciliumeBPF-based Networking, Security, and Observability项目地址: https://gitcode.com/GitHub_Trending/ci/cilium创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考