root@desistdaydream:~# kubectl logs -n kube-system kube-proxy-4thfl | more
E0507 06:05:09.433545       1 proxier.go:1445] can't open "nodePort for mysql/mysql-bj-net:mysql" (:33306/tcp), skipping this nodePort: listen tcp4 :33306: bind: address already in use
E0507 06:05:09.602044       1 proxier.go:1445] can't open "nodePort for mysql/mysql-bj-net:mysql" (:33306/tcp), skipping this nodePort: listen tcp4 :33306: bind: address already in use
E0507 06:05:39.333119       1 proxier.go:1445] can't open "nodePort for mysql/mysql-bj-net:mysql" (:33306/tcp), skipping this nodePort: listen tcp4 :33306: bind: address already in use
E0507 06:06:09.494578       1 proxier.go:1445] can't open "nodePort for mysql/mysql-bj-net:mysql" (:33306/tcp), skipping this nodePort: listen tcp4 :33306: bind: address already in use

故障排查

这个 kube-proxy 在 master-2 上，去 master-2 上看，发现根本没有人占用 33306。反倒是 kube-apiserver 作为客户端，使用 33306 端口，与 etcd 的 2379 进行互联

[root@master-2 ~]# ss -ntap | grep 33306
ESTAB      0      0      127.0.0.1:33306              127.0.0.1:2379                users:(("kube-apiserver",pid=2746,fd=77))
ESTAB      0      0      127.0.0.1:2379               127.0.0.1:33306               users:(("etcd",pid=2768,fd=100))

故障处理

将 kube-apiserver 的 manifest 从 /etc/kubernetes/manifests 目录中移出。待 kube-proxy 创建完端口后，再移回 manifest

故障分析

kubernetes 这样设计，这不是给 Local Process 挖坑吗，内核在随机选择本地端口的时候，很可能会命中 kubernetes svc 的端口号呀。其实 Kubernetes 已经尽力了。当创建 nodePort 类型的 svc 时，kube-proxy 除了会下发 iptables 规则，还会创建一个监听 Socket，该 Socket 监听的端口号就是 nodePort，因此：

当内核指定 bind 该端口号时，会返回端口已使用
当内核随机选择本地端口号时，不会命中该端口

因此，正常情况下，Local Process 不会进坑。但是，如果 Local Process 先启动，kube-proxy 后启动，则会出现上文描述的情况。此时，kube-proxy 仍然会下发 iptables 规则，并且尝试 bind 该端口号，但会不成功，因为已经被 Local Process 占用了。

    E0729 01:48:43.034098       1 proxier.go:1072] can't open "nodePort for default/nginx:" (:31325/tcp), skipping this nodePort: listen tcp :31325: bind: address already in use
    E0729 01:49:13.064492       1 proxier.go:1072] can't open "nodePort for default/nginx:" (:31325/tcp), skipping this nodePort: listen tcp :31325: bind: address already in use
    E0729 01:49:43.094846       1 proxier.go:1072] can't open "nodePort for default/nginx:" (:31325/tcp), skipping this nodePort: listen tcp :31325: bind: address already in use