Calico IPIP CrossSubnet 与 IPIP 默认模式对比模式介
使用场景
参考官网文档
部署流程
本文分别部署默认 IPIP 模式与 IPIP CrossSubnet 模式，分别在请求同网段、不同网段时进行抓包对比
1.通过脚本快速生成 IPIP 默认模式
#!/bin/bash set -v # 1. Prepare NoCNI environment cat <<EOF | HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= kind create cluster --name=calico-ipip --image=burlyluo/kindest:v1.27.3 --config=- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true podSubnet: "10.244.0.0/16" nodes: - role: control-plane kubeadmConfigPatches: - | kind: InitConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.5.10 - role: worker kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.5.11 - role: worker kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.8.10 - role: worker kubeadmConfigPatches: - | kind: JoinConfiguration nodeRegistration: kubeletExtraArgs: node-ip: 10.1.8.11 EOF # 2. Remove taints controller_node_ip=`kubectl get node -o wide --no-headers | grep -E "control-plane|bpf1" | awk -F " " '{print $6}'` kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule- kubectl get nodes -o wide ./2-setup-clab.sh # 3. Collect startup message controller_node_name=$(kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep control-plane) if [ -n "$controller_node_name" ]; then timeout 1 docker exec -t $controller_node_name bash -c 'cat << EOF > /root/monitor_startup.sh #!/bin/bash ip -ts monitor all > /root/startup_monitor.txt 2>&1 EOF chmod +x /root/monitor_startup.sh && /root/monitor_startup.sh' else echo "No such controller_node!" fi # 4. Install CNI[Calico v3.23.2] kubectl apply -f calico.yaml其中 2-setup-clab.sh 的作用是通过 containerlab 创建四个容器，给他们设置 IP 后分别与 kind 创建的四个容器共享网络命名空间，这样 k8s 集群就能使用 kind 参数指定的 node-ip 了：
#!/bin/bash set -v for br in br-pool0 br-pool1; do ip link set $br down > /dev/null 2>&1 ip link delete $br ip link add $br type bridge ip link set $br up done cat << EOF > clab.yaml | containerlab destroy -t clab.yaml --cleanup - name: calico-ipip topology: nodes: gw0: kind: linux image: hub.deepflow.yunshan.net/network-demo/vyos:1.4.9 cmd: /sbin/init binds: - /lib/modules:/lib/modules - ./startup-conf/gw0-boot.cfg:/opt/vyatta/etc/config/config.boot br-pool0: kind: bridge br-pool1: kind: bridge server1: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-control-plane exec: - ip addr add 10.1.5.10/24 dev net0 - ip route replace default via 10.1.5.1 server2: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-worker exec: - ip addr add 10.1.5.11/24 dev net0 - ip route replace default via 10.1.5.1 server3: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-worker2 exec: - ip addr add 10.1.8.10/24 dev net0 - ip route replace default via 10.1.8.1 server4: kind: linux image: hub.deepflow.yunshan.net/network-demo/nettool network-mode: container:calico-ipip-worker3 exec: - ip addr add 10.1.8.11/24 dev net0 - ip route replace default via 10.1.8.1 links: - endpoints: ["br-pool0:br-pool0-net0", "server1:net0"] mtu: 1500 - endpoints: ["br-pool0:br-pool0-net1", "server2:net0"] mtu: 1500 - endpoints: ["br-pool1:br-pool1-net0", "server3:net0"] mtu: 1500 - endpoints: ["br-pool1:br-pool1-net1", "server4:net0"] mtu: 1500 - endpoints: ["gw0:eth1", "br-pool0:br-pool0-net2"] mtu: 1500 - endpoints: ["gw0:eth2", "br-pool1:br-pool1-net2"] mtu: 1500 EOFgw0 中 startup-conf/gw0-boot.cfg 文件的作用就是让 10.1.5.0/24 和 10.1.8.0/24 两个子网能互通（两个子网的默认网关都在 gw0 上，gw0 直接转发就行）：
interfaces { ethernet eth1 { address "10.1.5.1/24" duplex "auto" speed "auto" } ethernet eth2 { address "10.1.8.1/24" duplex "auto" speed "auto" } loopback lo { } } nat { source { rule 100 { outbound-interface { name "eth0" } source { address "10.1.0.0/16" } translation { address "masquerade" } } } } system { config-management { commit-revisions "100" } console { device ttyS0 { speed "9600" } } host-name "gw0" login { user vyos { authentication { encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/" plaintext-password "" } } } time-zone "UTC" }## calico yaml # Auto-detect the BGP IP address. - name: IP value: "autodetect" # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: "Always" # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: "Never" # Enable or Disable VXLAN on the default IPv6 IP pool. - name: CALICO_IPV6POOL_VXLAN value: "Never"2.通过脚本快速生成 IPIP CrossSubnet 模式
其余部署脚本一致，仅在 calico CALICO_IPV4POOL_IPIP 模式中有差异：
## calico yaml # Auto-detect the BGP IP address. - name: IP value: "autodetect" # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: "CrossSubnet" # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: "Never" # Enable or Disable VXLAN on the default IPv6 IP pool. - name: CALICO_IPV6POOL_VXLAN value: "Never"创建测试 Pod
本质是 Nginx，用于后续请求抓包使用
apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: nginx name: pod spec: replicas: 4 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: burlyluo/nettool:latest name: nettoolbox env: - name: NETTOOL_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: privileged: true affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: nginx topologyKey: kubernetes.io/hostname查询部署结果
1.查询 IPIP 默认模式部署结果
root@network-demo:~# docker ps --format '{{.Names}}' clab-calico-ipip-server2 clab-calico-ipip-server4 clab-calico-ipip-server1 clab-calico-ipip-server3 clab-calico-ipip-gw0 calico-ipip-worker calico-ipip-worker2 calico-ipip-control-plane calico-ipip-worker3在主机上看到创建的 br-pool0-net0 网卡与 containerlab 创建的容器中 net0 网卡对应。在 kind 生成的 docker 容器中也能看到相同的网卡，说明已经共享了同一个网络空间：
root@network-demo:~# ip -d link show br-pool0-net0 198: br-pool0-net0@if197: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-pool0 state UP mode DEFAULT group default link/ether aa:c1:ab:1c:c9:1c brd ff:ff:ff:ff:ff:ff link-netns clab-calico-ipip-server1 promiscuity 1 allmulti 1 minmtu 68 maxmtu 65535 veth bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.c6:58:98:9d:5f:ea designated_root 8000.c6:58:98:9d:5f:ea hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on bcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off locked off addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536 root@network-demo:~# docker exec -it clab-calico-ipip-server1 ip -d link show net0 197: net0@if198: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:bd:45:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535 veth addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 root@network-demo:~# docker exec -it calico-ipip-control-plane ip -d link show net0 197: net0@if198: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether aa:c1:ab:bd:45:17 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535 veth addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535root@network-demo:~# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system calico-kube-controllers 1/1 Running 0 16m 10.244.51.196 calico-ipip-control-plane kube-system calico-node-64f6p 1/1 Running 0 16m 10.1.5.10 calico-ipip-control-plane kube-system calico-node-p4ks7 1/1 Running 0 16m 10.1.5.11 calico-ipip-worker kube-system calico-node-pjbc7 1/1 Running 0 16m 10.1.8.11 calico-ipip-worker3 kube-system calico-node-r6rk2 1/1 Running 0 16m 10.1.8.10 calico-ipip-worker2 kube-system coredns-5d78c9869d-jx4lx 1/1 Running 0 17m 10.244.51.194 calico-ipip-control-plane kube-system coredns-5d78c9869d-mrf2d 1/1 Running 0 17m 10.244.51.195 calico-ipip-control-plane kube-system etcd-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-apiserver-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-controller-manager-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-proxy-4svbw 1/1 Running 0 17m 10.1.8.10 calico-ipip-worker2 kube-system kube-proxy-4zw9q 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane kube-system kube-proxy-5nnfn 1/1 Running 0 17m 10.1.8.11 calico-ipip-worker3 kube-system kube-proxy-b69xp 1/1 Running 0 17m 10.1.5.11 calico-ipip-worker kube-system kube-scheduler-calico-ipip 1/1 Running 0 17m 10.1.5.10 calico-ipip-control-plane root@network-demo:~# kubectl describe pods -n kube-system calico-node-64f6p | grep 'CALICO_IPV4POOL' CALICO_IPV4POOL_IPIP: Always