二进制部署 k8s 集群 1.23.6 版本

发表于 更新于 分类于 ， 阅读次数： 本文字数： 3.9k 阅读时长 ≈ 14 分钟

通过本文的指导，读者可以了解如何通过二进制的方式部署 Kubernetes 1.23.6 版本集群。二进制部署可以加深对 Kubernetes 各组件的理解，可以灵活地将各个组件部署到不同的机器，以满足自身的要求。但是需要注意的是，二进制部署需要手动配置各个组件，需要一定的技术水平和经验。

二进制部署 k8s 集群 1.23.6 版本

1. 环境介绍

虽然 kubeadm, kops, kubespray 以及 rke, kubesphere 等工具可以快速部署 K8s 集群，但是依然会有很多人热衷与使用二进制部署 K8s 集群。

二进制部署可以加深对 K8s 各组件的理解，可以灵活地将各个组件部署到不同的机器，以满足自身的要求。还可以生成一个超长时间自签证书，比如 99 年，免去忘记更新证书过期带来的生产事故。

1.1 书写约定

  • 命令行输入，均以 ➜ 符号表示
  • 注释使用 # 或 // 表示
  • 执行命令输出结果，以空行分隔

1.2 规划

角色 主机名 IP 组件
master cnode0 10.128.170.20 etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, kubelet
node1 cnode1 10.128.170.21 kubelet, kube-proxy

1.3 环境配置

  • 关闭防火墙

    1
    2
    ➜ systemctl stop firewalld
    ➜ systemctl disable firewalld

  • 关闭 selinux

    1
    2
    3
    4
    # 临时
    ➜ setenforce 0
    # 永久
    ➜ sed -i 's/enforcing/disabled/' /etc/selinux/config

  • 关闭 swap

    1
    2
    3
    4
    # 临时
    ➜ swapoff -a
    # 永久
    ➜ sed -ri 's/.*swap.*/#&/' /etc/fstab

    使用 -r 选项可以使用扩展正则表达式，这提供了一种更强大和灵活的方式来匹配文本中的模式。

    使用正则表达式 .*swap.* 匹配包含 swap 字符串的行，并在行首添加 # 符号，& 表示匹配到的整个字符串。

  • 设置主机名

    1
    2
    3
    4
    # 10.128.170.20 主机
    ➜ hostnamectl set-hostname cnode0
    # 10.128.170.21 主机
    ➜ hostnamectl set-hostname cnode1

  • 时间同步

    1
    2
    3
    4
    5
    # 设置时区
    ➜ timedatectl set-timezone Asia/Shanghai
    # 安装时间同步服务
    ➜ yum install chrony -y
    ➜ systemctl enable --now chronyd

  • 主机名解析

    1
    2
    3
    4
    ➜ cat >> /etc/hosts << EOF
    10.128.170.20 cnode0 cnode0.com
    10.128.170.21 cnode1 cnode1.com
    EOF

  • 将桥接的 IPv4 流量传递到 iptables 的链

    1
    2
    3
    4
    5
    6
    7
    ➜ cat > /etc/sysctl.d/kubernetes.conf << EOF
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
    EOF
    # 生效
    ➜ sysctl --system

  • 设置文件描述符限制

    1
    2
    3
    4
    # 临时
    ➜ ulimit -SHn 65535
    # 永久
    ➜ echo "*      -      nofile   65535" >>/etc/security/limits.conf

    用于设置当前用户的最大文件描述符数限制。具体来说，它的作用是将当前用户的软限制和硬限制都设置为 65535。

  • 更新 epel 源

    1
    ➜ yum install epel-release -y

    如需换源，请参考：https://developer.aliyun.com/mirror/epel

  • 加载 ipvs 模块

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    ➜ yum install ipset ipvsadm -y
    ➜ cat > /etc/sysconfig/modules/ipvs.modules << "EOF"
    #!/bin/bash
    modprobe -- ip_vs
    modprobe -- ip_vs_rr
    modprobe -- ip_vs_wrr
    modprobe -- ip_vs_sh
    modprobe -- nf_conntrack_ipv4
    EOF
    ➜ chmod +x /etc/sysconfig/modules/ipvs.modules
    ➜ /bin/bash /etc/sysconfig/modules/ipvs.modules
    ➜ lsmod | grep -e ip_vs -e nf_conntrack_ipv4
    • modprobe -- ip_vs: 加载 ip_vs 内核模块，该模块提供了 Linux 内核中的 IP 负载均衡功能。
    • modprobe -- ip_vs_rr: 加载 ip_vs_rr 内核模块，该模块提供了基于轮询算法的 IP 负载均衡策略。
    • modprobe -- ip_vs_wrr: 加载 ip_vs_wrr 内核模块，该模块提供了基于加权轮询算法的 IP 负载均衡策略。
    • modprobe -- ip_vs_sh: 加载 ip_vs_sh 内核模块，该模块提供了基于哈希算法的 IP 负载均衡策略。
    • modprobe -- nf_conntrack_ipv4: 加载 nf_conntrack_ipv4 内核模块，该模块提供了 Linux 内核中的网络连接跟踪功能，用于跟踪网络连接的状态。

    这些命令通常用于配置 Linux 系统中的负载均衡和网络连接跟踪功能。在加载这些内核模块之后，就可以使用相应的工具和命令来配置和管理负载均衡和网络连接跟踪。例如，可以使用 ipvsadm 命令来配置 IP 负载均衡，使用 conntrack 命令来查看和管理网络连接跟踪表。

    如果提示如下错误：

    1
    "modprobe: FATAL: Module nf_conntrack_ipv4 not found in directory /lib/modules/4.18.0-372.9.1.el8.x86_64"

    则需要将 nf_conntrack_ipv4 修改为 nf_conntrack，然后重新执行命令，因为在高版本内核中已经把 nf_conntrack_ipv4 替换为 nf_conntrack。

    nf_conntrack_ipv4 和 nf_conntrack 都是 Linux 内核中的网络连接跟踪模块，用于跟踪网络连接的状态。它们的区别在于：

    • nf_conntrack_ipv4 模块只能跟踪 IPv4 协议的网络连接，而 nf_conntrack 模块可以跟踪 IPv4 和 IPv6 协议的网络连接。
    • nf_conntrack_ipv4 模块是 nf_conntrack 模块的一个子模块，它提供了 IPv4 协议的网络连接跟踪功能。因此，如果要使用 nf_conntrack_ipv4 模块，必须先加载 nf_conntrack 模块。

    这两个模块通常用于 Linux 系统中的网络安全和网络性能优化。它们可以被用于防火墙、负载均衡、网络流量分析等场景中，以便对网络连接进行跟踪、监控和控制。例如，可以使用 iptables 命令和 nf_conntrack 模块来实现基于连接状态的防火墙规则，或者使用 ipvsadm 命令和 nf_conntrack 模块来实现 IP 负载均衡。

  • 免密登录

    1
    2
    3
    4
    5
    # 为了便捷操作,在 cnode0 上创建免密登录其他节点
    ➜ ssh-keygen -t rsa
    ➜ ssh-copy-id cnode1
    ➜ ssh-copy-id cnode2
    ➜ ssh-copy-id cnode3

  • 创建 kubernetes 证书存放目录

    1
    ➜ mkdir -p /etc/kubernetes/pki

  • 重启

    1
    ➜ reboot

1.4 下载 k8s 二进制程序

从官方发布地址下载二进制包 下载地址

下载 Server Binaries 即可，这个包含了所有所需的二进制文件。解压后，复制二进制 kube-apiserver, kube-scheduler, kube-controller-manager, kube-proxy,kubelet, kubectl 到 master 节点 /usr/local/bin 目录下，复制二进制 kube-proxy,kubelet 到 worker 节点 /usr/local/bin 目录下。

1
2
3
4
5
6
7
8
➜ ll /usr/local/bin/kube*

-rwxr-xr-x 1 root root 128516096 Dec 29 14:59 /usr/local/bin/kube-apiserver
-rwxr-xr-x 1 root root 118489088 Dec 29 14:59 /usr/local/bin/kube-controller-manager
-rwxr-xr-x 1 root root  46202880 Dec 29 14:59 /usr/local/bin/kubectl
-rwxr-xr-x 1 root root 122352824 Dec 29 14:59 /usr/local/bin/kubelet
-rwxr-xr-x 1 root root  43581440 Dec 29 14:59 /usr/local/bin/kube-proxy
-rwxr-xr-x 1 root root  49020928 Dec 29 14:59 /usr/local/bin/kube-scheduler

2. 安装 docker

参考地址 安装docker，Docker 需要在各个节点上安装

  • 切换镜像源

    1
    wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

  • 查看当前镜像源中支持的 docker 版本

    1
    yum list docker-ce --showduplicates

  • 安装特定版本的 docker

    1
    yum install --setopt=obsoletes=0 docker-ce-20.10.14 -y

    --setopt=obsoletes=0 是 yum 包管理器的一个选项，它的作用是禁用软件包依赖关系中的版本升级。

  • 添加配置文件

    Docker 在默认情况下使用的 Cgroup Driver 为 cgroupfs，而 Kubernetes 推荐使用 systemd 来替代 cgroupfs

    1
    2
    3
    4
    5
    6
    7
    mkdir /etc/docker
    cat > /etc/docker/daemon.json << EOF
    {
      "exec-opts": ["native.cgroupdriver=systemd"],
      "registry-mirrors": ["https://kn0t2bca.mirror.aliyuncs.com"]
    }
    EOF

  • 启动 dokcer

    1
    2
    3
    systemctl daemon-reload
    systemctl restart docker
    systemctl enable docker

3. 创建 ca 证书

3.1 安装 cfssl

cfssl 是一款证书签署工具，使用 cfssl 工具可以很简化证书签署过程，方便颁发自签证书。

CloudFlare's distributes cfssl source code on github page and binaries on cfssl website.

Our documentation assumes that you will run cfssl on your local x86_64 Linux host.

1
2
3
curl -s -L -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x /usr/local/bin/{cfssl,cfssljson}

离线安装的情况，直接把两个文件下载下来重命名即可

3.2 创建 ca 证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 创建的证书统一放到 /etc/kubernetes/ssl 目录，创建后复制到 /etc/kubernetes/pki 目录
➜ mkdir /etc/kubernetes/ssl

➜ cd /etc/kubernetes/ssl

# ca 证书创建申请
➜ cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "k8s",
            "OU": "system"
        }
    ],
    "ca": {
        "expiry": "87600h"
    }
}
EOF

# 创建 ca 证书
➜ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

# 验证结果，会生成两个证书文件
➜ ll ca*pem

-rw------- 1 haxi haxi 1675 Dec 30 11:32 ca-key.pem
-rw-rw-r-- 1 haxi haxi 1314 Dec 30 11:32 ca.pem

# 复制 ca 证书到 /etc/kubernetes/pki
➜ cp ca*pem /etc/kubernetes/pki

ca-csr.json 这个文件是 Kubernetes 集群中使用的证书颁发机构 (CA) 证书签名请求 (CSR) 配置文件，用于定义 CA 的证书签名请求配置。

在这个配置文件中，CN 字段指定了证书的通用名称为 kubernetes，key 字段指定了证书的密钥算法为 RSA，密钥长度为 2048 位。names 字段定义了证书的其他信息，如国家、省份、城市、组织和组织单位等。ca 字段指定了证书的过期时间为 87600 小时（即 10 年）。

这个配置文件用于创建 Kubernetes 集群中的 CA 证书，以便对集群中的其他证书进行签名和认证。

  • CN(Common Name): kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)
  • names[].O(Organization): kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)

由于这里是 CA 证书，是签发其它证书的根证书，这个证书密钥不会分发出去作为 client 证书，所有组件使用的 client 证书都是由 CA 证书签发而来，所以 CA 证书的 CN 和 O 的名称并不重要，后续其它签发出来的证书的 CN 和 O 的名称才是有用的。

3.3 创建签发配置文件

由于各个组件都需要配置证书，并且依赖 CA 证书来签发证书，所以我们首先要生成好 CA 证书以及后续的签发配置文件。

创建用于签发其它证书的配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 证书签发配置文件
➜ cat > ca-config.json << EOF
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "kubernetes": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "87600h"
            }
        }
    }
}
EOF

ca-config.json 这个文件是签发其它证书的配置文件，用于定义签名配置和证书配置。其中，signing 字段定义了签名配置，profiles 字段定义了不同场景下的证书配置。

在这个配置文件中，default 配置指定了默认的证书过期时间为 87600 小时（即 10 年），profiles 配置定义了一个名为 kubernetes 的证书配置，它指定了证书的用途（签名、密钥加密、服务器认证和客户端认证）和过期时间。

这个配置文件用于创建 Kubernetes 集群中的证书和密钥，以便对集群进行安全认证和加密通信。

  • signing：定义了签名配置，包括默认的签名过期时间和各个证书配置的签名过期时间。
  • profiles：定义了不同场景下的证书配置，包括证书的用途、过期时间和其他属性。

在使用 cfssl gencert 命令生成证书时，可以使用 -config 参数指定配置文件，以便根据配置文件中的规则生成符合要求的证书。如果不指定 -config 参数，则 cfssl gencert 命令将使用默认的配置文件。

4. 部署 etcd

etcd 版本选择的是最新版本 3.5.1，下载二进制 etcd下载链接

4.1 颁发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# etcd 证书签署申请
# hosts 字段中，IP 为所有 etcd 集群节点地址，这里可以做好规划，预留几个 IP，以备以后扩容。
➜ cat > etcd-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "10.128.170.20",
        "10.128.170.21",
        "10.128.170.22",
        "10.128.170.23",
        "10.128.170.24",
        "10.128.170.25",
        "cnode0",
        "cnode1",
        "cnode2",
        "cnode3",
        "cnode4",
        "cnode5",
        "cnode0.com",
        "cnode1.com",
        "cnode2.com",
        "cnode3.com",
        "cnode4.com",
        "cnode5.com"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "k8s",
            "OU": "system"
        }
    ]
}
EOF

# 签署 etcd 证书
➜ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

# 验证结果，会生成两个证书文件
➜ ll etcd*pem

-rw------- 1 haxi haxi 1679 Dec 30 11:32 etcd-key.pem
-rw-rw-r-- 1 haxi haxi 1440 Dec 30 11:32 etcd.pem

# 复制 etcd 证书到 /etc/kubernetes/pki
➜ cp etcd*pem /etc/kubernetes/pki

4.2 部署 etcd

下载二进制 etcd下载链接并解压，将二进制程序 etcd etcdctl 复制到 /usr/local/bin 目录下

1
2
3
4
➜ ll /usr/local/bin/etcd*

-rwxrwxr-x 1 root root 21823488 Dec 29 14:13 /usr/local/bin/etcd
-rwxrwxr-x 1 root root 16711680 Dec 29 14:13 /usr/local/bin/etcdctl

编写服务配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
➜ mkdir /etc/etcd

➜ cat > /etc/etcd/etcd.conf << EOF
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.128.170.20:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.128.170.20:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.128.170.20:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.128.170.20:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://10.128.170.20:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

# 配置文件解释
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR： 数据保存目录
ETCD_LISTEN_PEER_URLS：集群内部通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址列表
ETCD_INITIAL_CLUSTER_TOKEN：集群通信token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

编写服务启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 创建数据目录
➜ mkdir -p /var/lib/etcd

# 创建系统服务
➜ cat > /lib/systemd/system/etcd.service << "EOF"
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/kubernetes/pki/etcd.pem \
  --key-file=/etc/kubernetes/pki/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/pki/ca.pem \
  --peer-cert-file=/etc/kubernetes/pki/etcd.pem \
  --peer-key-file=/etc/kubernetes/pki/etcd-key.pem \
  --peer-trusted-ca-file=/etc/kubernetes/pki/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

启动 etcd 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now etcd

# 验证结果
➜ systemctl status etcd

# 查看日志
➜ journalctl -u etcd

5. 部署 kube-apiserver

5.1 颁发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
# kube-apiserver 证书签署申请
# hosts 字段中，IP 为所有 kube-apiserver 节点地址，这里可以做好规划，预留几个 IP，以备以后扩容。我这里写 6 个 IP
# 10.128.170.20 10.128.170.21 10.128.170.22 10.128.170.23 10.128.170.24 10.128.170.25
# 10.96.0.1 是 service 网段的第一个 IP
# kubernetes.default.svc.cluster.local 这一串是 kube-apiserver 的 service 域名
➜ cat > kube-apiserver-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
        "127.0.0.1",
        "10.128.170.20",
        "10.128.170.21",
        "10.128.170.22",
        "10.128.170.23",
        "10.128.170.24",
        "10.128.170.25",
        "cnode0",
        "cnode1",
        "cnode2",
        "cnode3",
        "cnode4",
        "cnode5",
        "cnode0.com",
        "cnode1.com",
        "cnode2.com",
        "cnode3.com",
        "cnode4.com",
        "cnode5.com",
        "10.96.0.1",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "k8s",
            "OU": "system"
        }
    ]
}
EOF

# 签署 kube-apiserver 证书
➜ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

# 验证结果，会生成两个证书文件
➜ ll kube-apiserver*pem

-rw------- 1 haxi haxi 1675 Dec 30 11:33 kube-apiserver-key.pem
-rw-rw-r-- 1 haxi haxi 1590 Dec 30 11:33 kube-apiserver.pem

# 复制 kube-apiserver 证书到 /etc/kubernetes/pki
➜ cp kube-apiserver*pem /etc/kubernetes/pki

5.2 部署 kube-apiserver

编写服务配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
➜ cat > /etc/kubernetes/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --insecure-port=0 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem \
  --tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/pki/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/pki/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/pki/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/kubernetes/pki/ca.pem \
  --etcd-certfile=/etc/kubernetes/pki/etcd.pem \
  --etcd-keyfile=/etc/kubernetes/pki/etcd-key.pem \
  --etcd-servers=https://10.128.170.21:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=1 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

如果 etcd 是一个集群，则 --etcd-servers 可以添加多个，例如：--etcd-servers=https://10.128.170.21:2379,https://10.128.170.22:2379,https://10.128.170.23:2379

生成 token 文件

1
2
3
➜ cat > /etc/kubernetes/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

在这个命令中，head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成了一个 16 字节的随机字符串，并将其转换为十六进制格式。这个字符串将作为令牌的值。

  • kubelet-bootstrap 是令牌的用户名
  • 10001 是令牌的 UID，
  • system:node-bootstrapper 是令牌的组名。

这些值将用于 kubelet 节点的身份验证和授权。

编写服务启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
➜ cat > /usr/lib/systemd/system/kube-apiserver.service << "EOF"
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

启动 kube-apiserver 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kube-apiserver

# 验证结果
➜ systemctl status kube-apiserver

# 查看日志
➜ journalctl -u kube-apiserver

6. 部署 kubectl

部署完 kube-apiserver 后，就可以部署 kubectl 了，因为 kubectl 可以验证 kube-apiserver 是否已经正常工作了。

6.1 颁发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# kubectl 证书签署申请
# O 参数的值必须为 system:masters，因为这是 kube-apiserver 一个内置好的角色，拥有集群管理的权限
➜ cat > kubectl-csr.json << EOF
{
    "CN": "clusteradmin",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "system:masters",
            "OU": "system"
        }
    ]
}
EOF

# 签署 kubectl 证书
➜ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubectl-csr.json | cfssljson -bare kubectl

# 验证结果，会生成两个证书文件
➜ ll kubectl*pem

-rw------- 1 haxi haxi 1675 Dec 30 11:34 kubectl-key.pem
-rw-rw-r-- 1 haxi haxi 1415 Dec 30 11:34 kubectl.pem

6.2 生成配置文件

1
2
3
4
5
6
7
8
9
10
11
➜ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.128.170.20:6443 --kubeconfig=kube.config

➜ kubectl config set-credentials clusteradmin --client-certificate=kubectl.pem --client-key=kubectl-key.pem --embed-certs=true --kubeconfig=kube.config

➜ kubectl config set-context kubernetes --cluster=kubernetes --user=clusteradmin --kubeconfig=kube.config

➜ kubectl config use-context kubernetes --kubeconfig=kube.config

➜ mkdir -p ~/.kube

➜ cp kube.config ~/.kube/config

以上命令用于在本地创建一个 Kubernetes 配置文件 kube.config，并将其复制到 ~/.kube/config 文件中，以便使用 kubectl 命令与 Kubernetes 集群进行交互。

kubectl config set-cluster 命令设置了一个名为 kubernetes 的集群，指定了以下参数：

  • --certificate-authority=ca.pem：指定 CA 证书文件的路径。
  • --embed-certs=true：将 CA 证书嵌入到配置文件中。
  • --server=https://10.128.170.20:6443：指定 API Server 的地址和端口。
  • --kubeconfig=kube.config：指定要写入的配置文件路径。

这些参数将用于创建一个名为 kubernetes 的集群配置，并将其写入到 kube.config 文件中。

kubectl config set-credentials 命令设置了一个名为 clusteradmin 的用户，指定了以下参数：

  • --client-certificate=kubectl.pem：指定客户端证书文件的路径。
  • --client-key=kubectl-key.pem：指定客户端私钥文件的路径。
  • --embed-certs=true：将客户端证书和私钥嵌入到配置文件中。
  • --kubeconfig=kube.config：指定要写入的配置文件路径。

这些参数将用于创建一个名为 clusteradmin 的用户配置，并将其写入到 kube.config 文件中。

kubectl config set-context 命令设置了一个名为 kubernetes 的上下文，指定了以下参数：

  • --cluster=kubernetes：指定要使用的集群。
  • --user=clusteradmin：指定要使用的用户。
  • --kubeconfig=kube.config：指定要写入的配置文件路径。

这些参数将用于创建一个名为 kubernetes 的上下文配置，并将其写入到 kube.config 文件中。

kubectl config use-context 命令将当前上下文设置为 kubernetes，指定了以下参数：

  • --kubeconfig=kube.config：指定要使用的配置文件路径。

这个命令将当前上下文设置为 kubernetes，以便 kubectl 命令可以使用 kube.config 文件与 Kubernetes 集群进行交互。

6.3 获取集群信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
➜ kubectl cluster-info

Kubernetes control plane is running at https://10.128.170.20:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

➜ kubectl get all -A

NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   22m

➜ kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get "https://127.0.0.1:10259/healthz": dial tcp 127.0.0.1:10259: connect: connection refused
controller-manager   Unhealthy   Get "https://127.0.0.1:10257/healthz": dial tcp 127.0.0.1:10257: connect: connection refused
etcd-0               Healthy     {"health": "true"}

6.4 设置 kubectl 自动补全

查看 kubectl 命令自动补全帮助：

1
2
3
4
5
6
7
kubectl completion --help
````

安装 bash-completion：

```shell
yum install bash-completion -y

设置 kubectl 自动补全配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
echo "source <(kubectl completion bash)" >> ~/.bashrc
````

## 7. 部署 kube-controller-manager

### 7.1 颁发证书

```shell
# kube-controller-manager 证书签署申请
# hosts 字段中，IP 为所有节点地址，这里可以做好规划，预留几个 IP，以备以后扩容。我这里写 6 个 IP
➜ cat > kube-controller-manager-csr.json << EOF
{
    "CN": "system:kube-controller-manager",
    "hosts": [
        "127.0.0.1",
        "10.128.170.20",
        "10.128.170.21",
        "10.128.170.22",
        "10.128.170.23",
        "10.128.170.24",
        "10.128.170.25",
        "cnode0",
        "cnode1",
        "cnode2",
        "cnode3",
        "cnode4",
        "cnode5",
        "cnode0.com",
        "cnode1.com",
        "cnode2.com",
        "cnode3.com",
        "cnode4.com",
        "cnode5.com"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "system:kube-controller-manager",
            "OU": "system"
        }
    ]
}
EOF

# 签署 kube-controller-manager 证书
➜ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

# 验证结果，会生成两个证书文件
➜ ll kube-controller-manager*pem

-rw------- 1 haxi haxi 1679 Dec 30 12:13 kube-controller-manager-key.pem
-rw-rw-r-- 1 haxi haxi 1513 Dec 30 12:13 kube-controller-manager.pem

# 复制 kube-controler-manager 证书到 /etc/kubernetes/pki
➜ cp kube-controller-manager*pem /etc/kubernetes/pki

system:kube-controller-manager 是 Kubernetes 中的一个预定义 RBAC 角色，用于授权 kube-controller-manager 组件对 Kubernetes API 的访问。详细介绍请参考官方文档：https://kubernetes.io/zh-cn/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings

7.2 部署 kube-controller-manager

编写服务配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
➜ cat > /etc/kubernetes/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--port=0 \
  --secure-port=10257 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.96.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --cluster-signing-duration=87600h \
  --tls-cert-file=/etc/kubernetes/pki/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/pki/kube-controller-manager-key.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --leader-elect=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --use-service-account-credentials=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.240.0.0/12 \
  --v=4"
EOF

生成 kubeconfig

1
2
3
4
5
6
7
8
9
➜ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.128.170.20:6443 --kubeconfig=kube-controller-manager.kubeconfig

➜ kubectl config set-credentials kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig

➜ kubectl config set-context default --cluster=kubernetes --user=kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

➜ kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

➜ cp kube-controller-manager.kubeconfig /etc/kubernetes/

编写服务启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
➜ cat > /usr/lib/systemd/system/kube-controller-manager.service << "EOF"
[Unit]
Description=Kubernetes controller manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

启动 kube-controller-manager 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kube-controller-manager

# 验证结果
➜ systemctl status kube-controller-manager

# 查看日志
➜ journalctl -u kube-controller-manager

查看组件状态

1
2
3
4
5
6
7
➜ kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get "https://127.0.0.1:10259/healthz": dial tcp 127.0.0.1:10259: connect: connection refused
controller-manager   Healthy     ok
etcd-0               Healthy     {"health": "true"}

8. 部署 kube-scheduler

8.1 颁发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# kube-scheduler 证书签署申请
# hosts 字段中，IP 为所有节点地址，这里可以做好规划，预留几个 IP，以备以后扩容。我这里写 6 个 IP
➜ cat > kube-scheduler-csr.json << EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
        "127.0.0.1",
        "10.128.170.20",
        "10.128.170.21",
        "10.128.170.22",
        "10.128.170.23",
        "10.128.170.24",
        "10.128.170.25",
        "cnode0",
        "cnode1",
        "cnode2",
        "cnode3",
        "cnode4",
        "cnode5",
        "cnode0.com",
        "cnode1.com",
        "cnode2.com",
        "cnode3.com",
        "cnode4.com",
        "cnode5.com"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "system:kube-scheduler",
            "OU": "system"
        }
    ]
}
EOF

# 签署 kube-scheduler 证书
➜ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

# 验证结果，会生成两个证书文件
➜ ll kube-scheduler*pem

-rw------- 1 haxi haxi 1679 Dec 30 13:19 kube-scheduler-key.pem
-rw-rw-r-- 1 haxi haxi 1489 Dec 30 13:19 kube-scheduler.pem

# 复制 kube-scheduler 证书到 /etc/kubernetes/pki
➜ cp kube-scheduler*pem /etc/kubernetes/pki

8.2 部署 kube-scheduler

编写服务配置文件

1
2
3
4
5
6
7
8
9
➜ cat > /etc/kubernetes/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
  --leader-elect=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

生成 kubeconfig

1
2
3
4
5
6
7
8
9
➜ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.128.170.20:6443 --kubeconfig=kube-scheduler.kubeconfig

➜ kubectl config set-credentials kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig

➜ kubectl config set-context default --cluster=kubernetes --user=kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

➜ kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

➜ cp kube-scheduler.kubeconfig /etc/kubernetes/

编写服务启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
➜ cat > /usr/lib/systemd/system/kube-scheduler.service << "EOF"
[Unit]
Description=Kubernetes scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

启动 kube-scheduler 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kube-scheduler

# 验证结果
➜ systemctl status kube-scheduler

# 查看日志
➜ journalctl -u kube-scheduler

查看组件状态

1
2
3
4
5
6
7
➜ kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

9. 部署 kubelet

master 节点上部署 kubelet 是可选的，一旦部署 kubelet，master 节点也可以运行 Pod，如果不希望 master 节点上运行 Pod，则可以给 master 节点打上污点。

master 节点部署 kubelet 是有好处的，一是可以通过诸如 kubectl get node 等命令查看节点信息，二是可以在上面部署监控系统，日志采集系统等。

9.1 授权 kubelet 允许请求证书

授权 kubelet-bootstrap 用户允许请求证书

1
2
3
4
5
➜ kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

9.2 部署 kubelet

编写服务配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
➜ cat > /etc/kubernetes/kubelet.conf << EOF
KUBELET_OPTS="--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --config=/etc/kubernetes/kubelet.yaml \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --cert-dir=/etc/kubernetes/pki \
  --network-plugin=cni \
  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 \
  --logtostderr=false \
  --v=4 \
  --log-dir=/var/log/kubernetes \
  --fail-swap-on=false"
EOF

➜ cat > /etc/kubernetes/kubelet.yaml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 0
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
healthzBindAddress: 127.0.0.1
healthzPort: 10248
rotateCertificates: true
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

生成 kubeconfig

1
2
3
4
5
6
7
8
9
➜ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.128.170.20:6443 --kubeconfig=kubelet-bootstrap.kubeconfig

➜ kubectl config set-credentials kubelet-bootstrap --token=$(awk -F, '{print $1}' /etc/kubernetes/token.csv) --kubeconfig=kubelet-bootstrap.kubeconfig

➜ kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

➜ kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

➜ cp kubelet-bootstrap.kubeconfig /etc/kubernetes/

编写服务启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
➜ cat > /usr/lib/systemd/system/kubelet.service << "EOF"
[Unit]
Description=Kubernetes kubelet
After=network.target network-online.targer docker.service
Wants=docker.service

[Service]
EnvironmentFile=/etc/kubernetes/kubelet.conf
ExecStart=/usr/local/bin/kubelet $KUBELET_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

启动 kubelet 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kubelet

# 验证结果
➜ systemctl status kubelet

# 查看日志
➜ journalctl -u kubelet

批准节点加入集群

1
2
3
4
5
6
7
8
9
10
11
12
13
➜ kubectl get csr

NAME        AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
csr-dtprn   11s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending

➜ kubectl certificate approve csr-dtprn

certificatesigningrequest.certificates.k8s.io/csr-dtprn approved

➜ kubectl get csr

NAME        AGE    SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
csr-dtprn   113s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued

查看节点

1
2
3
4
5
6
➜ kubectl get node

NAME     STATUS     ROLES    AGE   VERSION
cnode0   NotReady   <none>   57s   v1.23.6

# 此时节点状态还是 NotReady，因为还没有安装网络插件，正确安装网络插件后，状态会变为 Ready.

10. 部署 kube-proxy

10.1 颁发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# kube-proxy 证书签署申请
➜ cat > kube-proxy-csr.json << EOF
{
    "CN": "system:kube-proxy",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangDong",
            "L": "ShenZhen",
            "O": "k8s",
            "OU": "system"
        }
    ]
}
EOF

# 签署 kube-proxy 证书
➜ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

# 验证结果，会生成两个证书文件
➜ ll kube-proxy*pem

-rw------- 1 haxi haxi 1679 Dec 31 10:26 kube-proxy-key.pem
-rw-rw-r-- 1 haxi haxi 1407 Dec 31 10:26 kube-proxy.pem

# 复制 kube-proxy 证书到 /etc/kubernetes/pki
➜ cp kube-proxy*pem /etc/kubernetes/pki

10.2 部署 kube-proxy

编写服务配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
➜ cat > /etc/kubernetes/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--config=/etc/kubernetes/kube-proxy.yaml \
  --logtostderr=false \
  --v=4 \
  --log-dir=/var/log/kubernetes"
EOF

➜ cat > /etc/kubernetes/kube-proxy.yaml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
bindAddress: 0.0.0.0
clusterCIDR: 10.240.0.0/12
healthzBindAddress: 0.0.0.0:10256
metricsBindAddress: 0.0.0.0:10249
mode: ipvs
ipvs:
  scheduler: "rr"
EOF

生成 kubeconfig

1
2
3
4
5
6
7
8
9
➜ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://10.128.170.20:6443 --kubeconfig=kube-proxy.kubeconfig

➜ kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

➜ kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

➜ kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

➜ cp kube-proxy.kubeconfig /etc/kubernetes/

编写服务启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
➜ cat > /usr/lib/systemd/system/kube-proxy.service << "EOF"
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=-/etc/kubernetes/kube-proxy.conf
ExecStart=/usr/local/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
RestartSec=5
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

启动 kube-proxy 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kube-proxy

# 验证结果
➜ systemctl status kube-proxy

# 查看日志
➜ journalctl -u kube-proxy

11. 部署网络

11.1 部署 calico

参考地址 calico

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
➜ curl https://projectcalico.docs.tigera.io/v3.23/manifests/calico.yaml -O
# 修改 Pod IP 地址段，找到 CALICO_IPV4POOL_CIDR 变量，取消注释并修改如下
            - name: CALICO_IPV4POOL_CIDR
              value: "10.240.0.0/12"

➜ kubectl apply -f calico.yaml

configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

# 查看网络 pod
➜ kubectl get deploy,pod -n kube-system

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1/1     1            1           24m

NAME                                          READY   STATUS    RESTARTS   AGE
pod/calico-kube-controllers-6b77fff45-mxxkg   1/1     Running   0          24m
pod/calico-node-ld4sg                         1/1     Running   0          24m

# 查看 node 状态
➜ kubectl get node

NAME     STATUS   ROLES    AGE   VERSION
cnode0   Ready    <none>   38m   v1.23.6

# 查看 ipvs 模式
➜ ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr
  -> 10.128.170.20:6443           Masq    1      5          0

如果 node 状态仍然是 NotReady，基本上是镜像未拉取完成或拉取失败导致的，如果一段时间后仍拉取失败，则尝试手动拉取镜像。

11.2 授权 kube-apiserver 访问 kubelet

Using RBAC Authorization

应用场景：例如 kubectl exec/run/logs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
➜ cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

➜ kubectl apply -f apiserver-to-kubelet-rbac.yaml

clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

➜ kubectl logs calico-kube-controllers-6b77fff45-mxxkg -n kube-system

11.3 部署 coredns

参考地址 coredns

coredns.yaml.sed 原始文件见附录章节 "16.1 coredns.yaml.sed"，该 yaml 指定使用的 coredns 的版本是 1.9.2。

下载 yaml 文件

1
➜ curl https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed -o coredns.yaml

对 yaml 文件做如下修改：

  • CLUSTER_DOMAIN 改为 cluster.local
  • REVERSE_CIDRS 改为 in-addr.arpa ip6.arpa
  • UPSTREAMNAMESERVER 改为 /etc/resolv.conf，如果报错，则改成当前网络所使用的 DNS 地址
  • 删除 STUBDOMAINS
  • CLUSTER_DNS_IP 改为 10.96.0.10（应与 /etc/kubernetes/kubelet.yaml 中配置的 clusterDNS 保持一致）
1
➜ kubectl apply -f coredns.yaml

验证（如果 calico 的 pod 未就绪，请检查是否是镜像拉取未完成或镜像拉取失败）

1
2
3
4
5
6
7
8
9
10
11
12
➜ kubectl get deploy,pod,svc -n kube-system
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1/1     1            1           43m
deployment.apps/coredns                   1/1     1            1           3m8s

NAME                                          READY   STATUS    RESTARTS   AGE
pod/calico-kube-controllers-6b77fff45-mxxkg   1/1     Running   0          43m
pod/calico-node-ld4sg                         1/1     Running   0          43m
pod/coredns-799bc9dbc6-qqh7h                  1/1     Running   0          3m8s

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   3m8s

dig 测试

1
2
3
4
5
6
7
➜ yum install bind-utils -y

➜ dig -t A www.baidu.com @10.96.0.10 +short

www.a.shifen.com.
182.61.200.6
182.61.200.7

pod 测试

Kubernetes busybox nslookup问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
➜ kubectl run -it --rm --image=busybox:1.28.3 -- sh

If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
/ # ping -c 4 www.baidu.com
PING www.baidu.com (182.61.200.6): 56 data bytes
64 bytes from 182.61.200.6: seq=0 ttl=52 time=6.860 ms
64 bytes from 182.61.200.6: seq=1 ttl=52 time=6.592 ms
64 bytes from 182.61.200.6: seq=2 ttl=52 time=6.488 ms
64 bytes from 182.61.200.6: seq=3 ttl=52 time=7.288 ms

--- www.baidu.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 6.488/6.807/7.288 ms

12. 添加 worker 节点

worker 节点需要部署两个组件 kubelet, kube-proxy.

（master 节点执行）从 master 节点上复制以下文件到 worker 节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
➜ scp /etc/kubernetes/pki/ca.pem \
       /etc/kubernetes/pki/kube-proxy.pem \
       /etc/kubernetes/pki/kube-proxy-key.pem \
       root@cnode1:/etc/kubernetes/pki/

➜ scp /etc/kubernetes/kubelet.conf \
       /etc/kubernetes/kubelet.yaml \
       /etc/kubernetes/kubelet-bootstrap.kubeconfig \
       /etc/kubernetes/kube-proxy.conf \
       /etc/kubernetes/kube-proxy.yaml \
       /etc/kubernetes/kube-proxy.kubeconfig \
       root@cnode1:/etc/kubernetes/

➜ scp /usr/lib/systemd/system/kubelet.service \
       /usr/lib/systemd/system/kube-proxy.service \
       root@cnode1:/usr/lib/systemd/system/

（master 节点执行）复制 kubelet, kube-proxy 二进制程序到 /usr/local/bin

1
2
3
➜ scp /usr/local/bin/kubelet \
       /usr/local/bin/kube-proxy \
       root@cnode1:/usr/local/bin/

（worker 节点执行）worker 节点启动 kube-proxy 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kube-proxy

# 验证结果
➜ systemctl status kube-proxy

# 查看日志
➜ journalctl -u kube-proxy

（worker 节点执行）worker 节点启动 kubelet 服务

1
2
3
4
5
6
7
8
9
➜ systemctl daemon-reload

➜ systemctl enable --now kubelet

# 验证结果
➜ systemctl status kubelet

# 查看日志
➜ journalctl -u kubelet

（master 节点执行）批准 worker 节点加入集群

1
2
3
4
5
6
7
8
9
10
11
12
13
➜ kubectl get csr

NAME        AGE   SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
csr-9mvtn   23s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Pending

➜ kubectl certificate approve csr-9mvtn

certificatesigningrequest.certificates.k8s.io/csr-9mvtn approved

➜ kubectl get csr

NAME        AGE     SIGNERNAME                                    REQUESTOR           REQUESTEDDURATION   CONDITION
csr-9mvtn   3m16s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   <none>              Approved,Issued

（master 节点执行）查看节点

1
2
3
4
5
➜ kubectl get node

NAME     STATUS   ROLES    AGE     VERSION
cnode0   Ready    <none>   4h31m   v1.23.6
cnode1   Ready    <none>   105m    v1.23.6

如果 cnode1 的状态仍是 NotReady，请检查是否是镜像拉取未完成或镜像拉取失败。

13. 禁止 master 节点运行 pod

至此 1 master 1 worker 的 k8s 二进制集群已搭建完毕。

此外，还可以给节点打上角色标签，使得查看节点信息更加直观

1
2
3
4
5
6
7
8
# 给 master 节点打上 controlplane,etcd 角色标签
➜ kubectl label node cnode0 node-role.kubernetes.io/controlplane=true node-role.kubernetes.io/etcd=true

# 给 worker 节点打上 worker 角色标签
➜ kubectl label node cnode1 node-role.kubernetes.io/worker=true

# 查看节点标签
➜ kubectl get node --show-labels

如果不希望 master 节点运行 Pod，则给 master 打上污点

1
➜ kubectl taint node cnode0 node-role.kubernetes.io/controlplane=true:NoSchedule

后续可以新增 2 个 etcd 节点组成 etcd 集群，新增 2 个控制平面，避免单点故障。

14. 测试应用服务部署

创建 namespace

1
2
3
4
5
6
7
8
9
10
11
12
➜ kubectl create namespace dev

namespace/dev created

➜ kubectl get namespace

NAME              STATUS   AGE
default           Active   15h
dev               Active   15s
kube-node-lease   Active   15h
kube-public       Active   15h
kube-system       Active   15h

创建 deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
➜ mkdir -p /etc/kubernetes/resources

➜ cat > /etc/kubernetes/resources/nginx-deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-pod
  template:
    metadata:
      labels:
        app: nginx-pod
    spec:
      containers:
      - name: nginx
        image: nginx:latest
EOF

➜ kubectl apply -f /etc/kubernetes/resources/nginx-deployment.yaml

deployment.apps/nginx-deployment created

➜ kubectl get pod -n dev

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-7d4578b56c-cndrb   1/1     Running   0          48s

创建 service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
➜ cat > /etc/kubernetes/resources/nginx-service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: dev
spec:
  selector:
    app: nginx-pod
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30001
EOF

➜ kubectl apply -f /etc/kubernetes/resources/nginx-service.yaml

service/nginx-service created

➜ kubectl get svc -n dev

NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx-service   NodePort   10.96.221.226   <none>        80:30001/TCP   17s

测试服务访问

1
2
3
4
5
6
7
8
9
10
11
12
➜ curl 10.128.170.20:30001 -I

HTTP/1.1 200 OK
Server: nginx/1.21.5
Date: Sat, 21 May 2022 08:06:25 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 Dec 2021 15:28:38 GMT
Connection: keep-alive
ETag: "61cb2d26-267"
Accept-Ranges: bytes

15. 部署 Dashboard

在 Kubernetes 社区中，有一个很受欢迎的 Dashboard 项目，它可以给用户提供一个可视化的 Web 界面来查看当前集群的各种信息。用户可以用 Kubernetes Dashboard 部署容器化的应用、监控应用的状态、执行故障排查任务以及管理 Kubernetes 各种资源。

官方参考文档：https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

使用 nodeport 方式将 dashboard 服务暴露在集群外，指定使用 30443 端口。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# 下载相关 yaml 文件
➜ curl https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml -O

# 修改 Service 部分
➜ vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort  # 新增
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30443  # 新增
  selector:
    k8s-app: kubernetes-dashboard

# 部署
➜ kubectl apply -f recommended.yaml

namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

# 查看 kubernetes-dashboard 下的资源
➜ kubectl get deploy -n kubernetes-dashboard

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
dashboard-metrics-scraper   1/1     1            1           12m
kubernetes-dashboard        1/1     1            1           12m

➜ kubectl get pod -n kubernetes-dashboard

NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-799d786dbf-xpvcc   1/1     Running   0          13m
kubernetes-dashboard-546cbc58cd-hzvhr        1/1     Running   0          13m

➜ kubectl get svc -n kubernetes-dashboard

NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.96.73.62     <none>        8000/TCP        13m
kubernetes-dashboard        NodePort    10.96.148.106   <none>        443:30443/TCP   13m

如果 kubernetes-dashboard 下的资源一直未就绪，请检查是否是正在拉取镜像或者镜像一直拉取失败。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
➜ kubectl describe pod kubernetes-dashboard-546cbc58cd-hzvhr -n kubernetes-dashboard

...
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  6m20s  default-scheduler  Successfully assigned kubernetes-dashboard/kubernetes-dashboard-546cbc58cd-hzvhr to cnode1
  Normal  Pulling    6m20s  kubelet            Pulling image "kubernetesui/dashboard:v2.5.0"

➜ kubectl describe pod kubernetes-dashboard-546cbc58cd-hzvhr -n kubernetes-dashboard

Events:
  Type     Reason          Age                 From               Message
  ----     ------          ----                ----               -------
  Normal   Scheduled       10m                 default-scheduler  Successfully assigned kubernetes-dashboard/kubernetes-dashboard-546cbc58cd-hzvhr to cnode1
  Warning  Failed          2m1s                kubelet            Failed to pull image "kubernetesui/dashboard:v2.5.0": rpc error: code = Unknown desc = dial tcp 104.18.124.25:443: i/o timeout
  Warning  Failed          2m1s                kubelet            Error: ErrImagePull
  Normal   SandboxChanged  2m                  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         118s (x3 over 2m)   kubelet            Back-off pulling image "kubernetesui/dashboard:v2.5.0"
  Warning  Failed          118s (x3 over 2m)   kubelet            Error: ImagePullBackOff
  Normal   Pulling         106s (x2 over 10m)  kubelet            Pulling image "kubernetesui/dashboard:v2.5.0"
  Normal   Pulled          25s                 kubelet            Successfully pulled image "kubernetesui/dashboard:v2.5.0" in 1m21.608630166s
  Normal   Created         22s                 kubelet            Created container kubernetes-dashboard
  Normal   Started         21s                 kubelet            Started container kubernetes-dashboard

创建 service account 并绑定默认 cluster-admin 管理员集群角色

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# 下面创建了一个叫 admin-user 的服务账号，放在 kubernetes-dashboard 命名空间下，并将 cluster-admin 角色绑定到 admin-user 账户，这样 admin-user 账户就有了管理员的权限。
# 默认情况下，kubeadm 创建集群时已经创建了 cluster-admin 角色，我们直接绑定即可。
➜ cat > dashboard-admin-user.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

# 应用资源配置清单
➜ kubectl apply -f dashboard-admin-user.yaml

serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created

查看 admin-user 账户的 token

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
➜ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

Name:         admin-user-token-vmbfm
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: bc3c111d-947e-4444-8fc0-2ff69abada00

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1367 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Ikd6alY4Rm1QWFRiRk9VUDlta1U1QnFVM2VyUXpXSkUwRzRKek9QX2pxbUkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXZtYmZtIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYzNjMTExZC05NDdlLTQ0NDQtOGZjMC0yZmY2OWFiYWRhMDAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.e-fkl4gBppWnwMy5b1PvHNf5RiBL_uAT0o_QFTy4YWwkcBSnn9JqlCBJzy6Vblw2mIekkMwuOGux-vU8V9nPuAHczsKr1Kq2leZKR0rnNrFwqge-IO-U4pkY8sfWYWPo7j5Oop1dNAKt9q33WyenjpA6T_IF-TmdtsX4AhLeZp67RrDotq80dpSjmSBzDU2rZ6gwknCfCwum_Crn1uruNiGGP4dkFifIK78RfDyCIMdMvYuwoa9hYPTFVNPZQcTRecmdtOmfXyVHpS7FfKf3YTCm9vbyqrBLyzHinYf-dBBr5ivktJKOepuqbKSoQ68Q1KnxjeG9ouWaYa3jiukArw

使用输出的 token 登录 Dashboard

16. 附录

16.1 coredns.yaml.sed

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    - pods
    - namespaces
    verbs:
    - list
    - watch
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes CLUSTER_DOMAIN REVERSE_CIDRS {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . UPSTREAMNAMESERVER {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }STUBDOMAINS
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. Default is 1.
  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
               matchExpressions:
               - key: k8s-app
                 operator: In
                 values: ["kube-dns"]
             topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: coredns/coredns:1.9.2
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: CLUSTER_DNS_IP
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP

References

二进制部署 K8s 集群 1.23.1 版本

部署一套完整的企业级K8s集群