Ping不通layer2模式服务的外部IP #341

bluven · 2023-07-04T08:36:52Z

bluven
Jul 4, 2023

根据这个文档Use OpenELB in Layer 2 Mode做实验，机器环境如下：

kubectl get nodes -o wide
NAME       STATUS   ROLES                            AGE     VERSION                    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
edge1      Ready    agent,edge                       5h27m   v1.22.6-kubeedge-v1.12.2   10.22.48.13   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3
edge2      Ready    agent,edge                       5h24m   v1.22.6-kubeedge-v1.12.2   10.22.48.27   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3
node1      Ready    <none>                           5h42m   v1.22.5                    10.22.48.12   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3
shanghai   Ready    connector,control-plane,master   5h56m   v1.22.5                    10.22.48.21   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3

其中edge1, edge2是边缘节点。

openELB的相关pod状况如下:

[root@shanghai openelb]# kubectl get po -o wide
NAME                                READY   STATUS             RESTARTS         AGE   IP             NODE       NOMINATED NODE   READINESS GATES
layer2-openelb-7b4fdf6f85-554ds     1/1     Running            0                44m   10.233.66.16   edge1      <none>           <none>
layer2-openelb-7b4fdf6f85-nnszz     1/1     Running            0                44m   10.233.67.18   edge2      <none>           <none>
openelb-admission-create--1-4xpb5   0/1     Completed          0                53m   10.233.64.21   shanghai   <none>           <none>
openelb-admission-patch--1-dflkw    0/1     Completed          2                53m   10.233.64.20   shanghai   <none>           <none>
openelb-keepalive-vip-b8gkb         1/1     Running            0                51m   10.22.48.12    node1      <none>           <none>
openelb-keepalive-vip-j8jqq         0/1     CrashLoopBackOff   14 (3m57s ago)   51m   10.22.48.13    edge1      <none>           <none>
openelb-keepalive-vip-v9xmf         0/1     CrashLoopBackOff   14 (4m19s ago)   51m   10.22.48.27    edge2      <none>           <none>
openelb-keepalive-vip-xg4kd         1/1     Running            0                51m   10.22.48.21    shanghai   <none>           <none>
openelb-manager-76f5d5cf65-897hq    1/1     Running            0                53m   10.22.48.12    node1      <none>           <none>

EIP的配置如下:

kubectl describe eip layer2-eip -n openelb-system
Name:         layer2-eip
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  network.kubesphere.io/v1alpha2
Kind:         Eip
Metadata:
  Creation Timestamp:  2023-07-04T07:41:35Z
  Finalizers:
    finalizer.ipam.kubesphere.io/v1alpha1
Spec:
  Address:    10.22.48.200-10.22.48.210
  Interface:  eth0
  Protocol:   layer2
Status:
  First IP:   10.22.48.200
  Last IP:    10.22.48.210
  Pool Size:  11
  Ready:      true
  Usage:      1
  Used:
    10.22.48.200:  openelb-system/layer2-svc
  v4:              true
Events:            <none>

layer-svc2的状态如下:

[root@shanghai ~]# kubectl get svc
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
layer2-svc          LoadBalancer   10.233.38.100   10.22.48.200   80:31221/TCP   40m

[root@shanghai ~]# kubectl describe svc layer2-svc
Name:                     layer2-svc
Namespace:                openelb-system
Labels:                   eip.openelb.kubesphere.io/v1alpha2=layer2-eip
Annotations:              eip.openelb.kubesphere.io/v1alpha2: layer2-eip
                          layer2.openelb.kubesphere.io/v1alpha1: shanghai
                          lb.kubesphere.io/v1alpha1: openelb
                          protocol.openelb.kubesphere.io/v1alpha1: layer2
Selector:                 app=layer2-openelb
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.233.38.100
IPs:                      10.233.38.100
LoadBalancer Ingress:     10.22.48.200
Port:                     http  80/TCP
TargetPort:               8080/TCP
NodePort:                 http  31221/TCP
Endpoints:                10.233.66.16:8080,10.233.67.18:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

在另一个客户端机器(10.22.48.20)ping 10.22.48.200时，发现ping 不通

[root@gateway ~]# ping 10.22.48.200
PING 10.22.48.200 (10.22.48.200) 56(84) bytes of data.
From 10.22.48.21: icmp_seq=2 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=3 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=4 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=5 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=6 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=8 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=11 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=17 Redirect Host(New nexthop: 10.22.48.200)

在shanghai(10.22.48.21)机器上抓包结果如下:

bash-5.1# tcpdump -nn -i any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:34:06.368538 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 1, length 64
08:34:06.368681 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 1, length 64
08:34:07.367277 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 2, length 64
08:34:07.367362 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:07.367396 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 2, length 64
08:34:08.367326 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 3, length 64
08:34:08.367371 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:08.367396 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 3, length 64
08:34:09.367295 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 4, length 64
08:34:09.367358 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:09.367384 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 4, length 64
08:34:10.367306 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 5, length 64
08:34:10.367371 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:10.367404 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 5, length 64
08:34:11.367324 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 6, length 64
08:34:11.367388 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:11.367416 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 6, length 64
08:34:12.367302 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 7, length 64
08:34:12.367358 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 7, length 64
08:34:13.367285 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 8, length 64
08:34:13.367350 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:13.367377 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 8, length 64
08:34:14.367347 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 9, length 64
08:34:14.367406 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 9, length 64

需要一提的是，虽然ping的结果不理想，但服务还是通了

[root@gateway ~]# curl 10.22.48.200
You've hit layer2-openelb-7b4fdf6f85-nnszz
[root@gateway ~]# curl 10.22.48.200
You've hit layer2-openelb-7b4fdf6f85-554ds

用 ip neigh查看发现10.22.48.200相关的记录是FAILED状态，

[root@gateway ~]# ip neigh
10.22.48.200 dev eth0  FAILED
10.22.48.21 dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
10.22.48.11 dev eth0 lladdr fa:16:3e:ba:d4:52 STALE
10.22.48.254 dev eth0 lladdr 00:1e:08:0f:3b:ba REACHABLE
fe80::ea30:60b3:32e3:36f4 dev eth0 lladdr fa:16:3e:a2:8e:b3 STALE
2003::66:d dev eth0 lladdr fa:16:3e:15:62:a9 STALE
2003::66:5 dev eth0 lladdr fa:16:3e:86:96:86 STALE
fe80::5880:9aff:fe6c:da29 dev eth0 lladdr 5a:80:9a:6c:da:29 STALE
fe80::cac:79d1:cfe7:f69c dev eth0 lladdr fa:16:3e:15:62:a9 STALE
fe80::24f2:4d6c:a01e:7fbe dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
fe80::a4e5:68ff:fe4c:6730 dev eth0 lladdr a6:e5:68:4c:67:30 STALE

但如果是在执行curl 10.22.48.200后再执行ip neigh，就会发现

[root@gateway ~]# ip neigh
10.22.48.200 dev eth0 lladdr fa:16:3e:70:d5:b3 REACHABLE
10.22.48.21 dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
10.22.48.11 dev eth0 lladdr fa:16:3e:ba:d4:52 STALE
10.22.48.254 dev eth0 lladdr 00:1e:08:0f:3b:ba REACHABLE
fe80::ea30:60b3:32e3:36f4 dev eth0 lladdr fa:16:3e:a2:8e:b3 STALE
2003::66:d dev eth0 lladdr fa:16:3e:15:62:a9 STALE
2003::66:5 dev eth0 lladdr fa:16:3e:86:96:86 STALE
fe80::5880:9aff:fe6c:da29 dev eth0 lladdr 5a:80:9a:6c:da:29 STALE
fe80::cac:79d1:cfe7:f69c dev eth0 lladdr fa:16:3e:15:62:a9 STALE
fe80::24f2:4d6c:a01e:7fbe dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
fe80::a4e5:68ff:fe4c:6730 dev eth0 lladdr a6:e5:68:4c:67:30 STALE

Answered by renyunkang

Jul 5, 2023

这个是符合预期的，openelb 对于分配的 loadbalancer ip 不会绑定到物理网卡中，而 svc 本身也应用于四层的转发，因此对于 icmp 协议无法做出回应

View full answer

renyunkang · 2023-07-05T02:19:48Z

renyunkang
Jul 5, 2023
Maintainer

这个是符合预期的，openelb 对于分配的 loadbalancer ip 不会绑定到物理网卡中，而 svc 本身也应用于四层的转发，因此对于 icmp 协议无法做出回应

2 replies

bluven Jul 5, 2023
Author

那为什么你们的例子看起来很正常？

renyunkang Jul 5, 2023
Maintainer

感谢提醒，这块文档上确实有一些不完善。
我们默认的使用的是 ipvs 的 kube-proxy，对于 svc 的 ip，会写到 kube-ipvs0 网卡上，因此会响应 icmp 协议。

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ping不通layer2模式服务的外部IP #341

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Ping不通layer2模式服务的外部IP #341

bluven Jul 4, 2023

Replies: 1 comment · 2 replies

renyunkang Jul 5, 2023 Maintainer

bluven Jul 5, 2023 Author

renyunkang Jul 5, 2023 Maintainer

bluven
Jul 4, 2023

Replies: 1 comment 2 replies

renyunkang
Jul 5, 2023
Maintainer

bluven Jul 5, 2023
Author

renyunkang Jul 5, 2023
Maintainer