Kubernetes (k8s) 环境搭建指南

·
瀚星空 云原生 k8s June 8, 2020

Kubernetes——让容器（Container）应用进入大规模工业生产。

Kubernetes是容器编排系统的事实标准

在单机上运行容器，无法发挥它的最大效能，只有形成集群，才能最大程度发挥容器的良好隔离、资源分配与编排管理的优势，而对于容器的编排管理，Swarm、Mesos和Kubernetes的大战已经基本宣告结束，Kubernetes成为了无可争议的赢家。本文对生产环境中 Kubernetes 搭建的过程进行简单记录。

1. 安装 Docker

该部分参考 Kubernetes 官方文档：https://kubernetes.io/docs/setup/production-environment/container-runtimes/

首先，新建 install_docker.sh 文件，并修改文件内容如下

# Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install \
  apt-transport-https ca-certificates curl software-properties-common
### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add
# Add the Docker apt repository:
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

# Install Docker CE
apt-get update && apt-get install -y \
  containerd.io=1.2.13-1 \
  docker-ce=5:19.03.8~3-0~ubuntu-$(lsb_release -cs) \
  docker-ce-cli=5:19.03.8~3-0~ubuntu-$(lsb_release -cs)

# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
systemctl daemon-reload
systemctl restart docker

之后，为该文件授予执行权限，并执行

chmod +x install_docker.sh
sudo ./install_docker.sh

等待脚本（Script）结束，完成 docker 的安装

2. 安装 Kubeadm

该部分可以参考官方文档：https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

安装前检查

  • One or more machines running one of:
  • Ubuntu 16.04+
  • Debian 9+
  • CentOS 7
  • Red Hat Enterprise Linux (RHEL) 7
  • Fedora 25+
  • HypriotOS v1.0.1+
  • Container Linux (tested with 1800.6.0)
  • 2 GB or more of RAM per machine (any less will leave little room for your apps)
  • 2 CPUs or more
  • Full network connectivity between all machines in the cluster (public or private network is fine)
  • Unique hostname, MAC address, and product_uuid for every node. See here for more details.
  • Certain ports are open on your machines. See here for more details.
  • Swap disabled. You MUST disable swap in order for the kubelet to work properly

  1. 按照官方要求，关闭 swap 交换分区：

    首先执行命令：

    sudo swapoff -a

    之后编辑文件：sudo vim /etc/fstab 将含有 swap.img 的那行注释掉

  2. 修改每个主机（Host）的 hostname ，确保唯一，以便区分

    sudo hostnamectl set-hostname <设定的 hostname>

  3. 保证网络（Network）路由的正确性，要求 iptables 对网桥 bridge 的数据进行处理。在 ssh 中执行下面的代码

    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

  4. 检查端口占用情况

    k8s 需要保证以下端口畅通：

    Control-plane node(s)

    Protocol Direction Port Range Purpose Used By
    TCP Inbound 6443* Kubernetes API server All
    TCP Inbound 2379-2380 etcd server client API kube-apiserver, etcd
    TCP Inbound 10250 Kubelet API Self, Control plane
    TCP Inbound 10251 kube-scheduler Self
    TCP Inbound 10252 kube-controller-manager Self

    Worker node(s)

    Protocol Direction Port Range Purpose Used By
    TCP Inbound 10250 Kubelet API Self, Control plane
    TCP Inbound 30000-32767 NodePort Services† All

安装 kubeadm，kubelet 和 kubectl

  • kubeadm：用来初始化集群的指令。
  • kubelet：在集群中的每个节点上用来启动 pod 和容器等。
  • kubectl：用来与集群通信的命令行（Command-line）工具。

对于master节点以及 worker 节点，都需要安装这些工具，执行下方的代码：

sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

安装完成之后，shell 中有如下提示：

3. 使用 kubeadm 创建集群

初始化集群

在 master 机器中执行如下命令：

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

该命令需要从官网下载资源，需要保证外网畅通，否则耗时极长，命令执行完成后，shell 会出现如下的提示：

[init] Using Kubernetes version: vX.Y.Z
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubeadm-cp localhost] and IPs [10.138.0.4 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubeadm-cp localhost] and IPs [10.138.0.4 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubeadm-cp kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.138.0.4]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 31.501735 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-X.Y" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kubeadm-cp" as an annotation
[mark-control-plane] Marking the node kubeadm-cp as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kubeadm-cp as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: <token>
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  /docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

可以看到，执行成功后，终端（Terminal）会打印出一条连接命令，大致形式如下：

kubeadm join 你的IP地址:6443 --token 你的TOKEN --discovery-token-ca-cert-hash sha256:你的CA证书[哈希]^(Hash)

先保存该命令，备用。后续worker pods 需要使用该命令与 master 连接。若该命令没有保存，或 TOKEN 已经过期，可以使用下方命令重新查看 TOKEN 以及 CA 证书哈希：

token. 通过命令Kubeadm token list找回
ca-cert. 执行命令openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'找回

设置当前用户可使用 Kubernetes

安装 Kubernetes 时，使用了 root 身份，为了让当前用户也可以执行 kubectl 等管理命令，需要为当前用户增加权限，执行以下代码：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

若该步骤没有执行，后续操作中，可能会报如下错误：

The connection to the server localhost:8080 was refused - did you specify the right host or port?

若当前用户本身为 root 用户 （如 centos）执行以下命令：

export KUBECONFIG=/etc/kubernetes/admin.conf

安装网络插件

为了使得 Pod 之间可以相互通信，必须要部署一个 Pod 网络插件

部署 Calico 插件的命令如下：

kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml

安装Pod网络后，可以通过检查 CoreDNS Pod是否Running在中来确认其正常工作 kubectl get pods --all-namespaces

4. 新增 pod 加入节点

  1. 修改 hostname

    sudo hostnamectl set-hostname zjh-worker1

  2. 执行加入命令

    sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

    如果没有 token ，可以在 master 主机中运行以下命令来获取token

    kubeadm token list

    默认情况下，token 会在24小时后过期。如果要在当前 token 过期后将节点加入集群，则可以通过在控制平面节点上运行以下命令来创建新 token：

    kubeadm token create

    CA 证书的哈希值通过以下命令获得：

    openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
  openssl dgst -sha256 -hex | sed 's/^.* //'

加入成功之后，pod 终端提示如下内容：

回到 master 终端，执行：

kubectl get nodes

可以看到，nodes 的状态均为 Ready

5. 安装可视化控制面板 Kubernetes Dashboard

Dashboard 的官方库位于：https://github.com/kubernetes/dashboard#kubernetes-dashboard

若只需要本机访问 Dashboard，而不需要网络中的其他机器通过 ip 地址访问该 Dashboard 的话，直接按照 readme 中的配置说明安装即可。

由于笔者的 k8s 集群安装在虚拟机中，且通过内网中的其他设备访问，若按照官方的方法安装，使用 kubectl proxy 的方式访问时，会提示当前的网络环境不安全，需要通过 localhost 或者 HTTPS 的方法访问，因此，需要暴露出相关的端口，使用 https 的方式才能登录。

  1. 下载 yaml 配置

    wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/recommended.yaml

  2. 对该 yaml 文件进行修改

    • 暴露端口：寻找到下述配置区域，在对应位置加入行 type:NodePort

      kind: Service
apiVersion: v1
metadata:
labels:
 k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
 - port: 443
   targetPort: 8443
type: NodePort #加入该行
selector:
 k8s-app: kubernetes-dashboard

    • 配置自签名证书位置：由于默认的证书无法使用 https 连接，需要我们手动生成自己的证书，在配置文件（Profile）中，首先修改证书相关设置：

      kind: Deployment
apiVersion: apps/v1
metadata:
labels:
 k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
 matchLabels:
   k8s-app: kubernetes-dashboard
template:
 metadata:
   labels:
     k8s-app: kubernetes-dashboard
 spec:
   containers:
     - name: kubernetes-dashboard
       image: kubernetesui/dashboard:v2.0.1
       imagePullPolicy: Always
       ports:
         - containerPort: 8443
           protocol: TCP
       args:
         - --tls-cert-file=dashboard.crt #加入该行
         - --tls-key-file=dashboard.key #加入该行
           # - --auto-generate-certificates #注释掉该行
         - --namespace=kubernetes-dashboard
         # Uncomment the following line to manually specify Kubernetes API server Host
         # If not specified, Dashboard will attempt to auto discover the API server and connect
         # to it. Uncomment only if the default does not work.
         # - --apiserver-host=http://my-address:port

    • 增加 admin 用户，因为安全原因，默认的用户权限很少，学习使用时，可以加入高权限的 admin 用户，方便管理：

      ---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
---

      该部分参考：https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

  3. 安装 Kubernetes Dashboard

    sudo kubectl apply -f recommended.yaml

  4. 生成自定义证书：

    cd $HOME/kubernetes/
mkdir certs
# Generate Certs
openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=kubernetes-dashboard"
openssl x509 -req -sha256 -days 365 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt

    删除原始证书 secret，并添加新证书：

    # Add certs to dashboard
kubectl delete secret kubernetes-dashboard-certs --namespace=kubernetes-dashboard
kubectl create secret generic kubernetes-dashboard-certs --from-file=$HOME/kubernetes/certs -n kubernetes-dashboard

  5. 查看运行状态：

    kubectl  get pod,deploy,svc -n kubernetes-dashboard

    查看日志信息：

    kubectl logs kubernetes-dashboard-697dbf9547-hf9pp --namespace=kubernetes-dashboard

  6. 无误后，通过浏览器访问 dashboard

    从运行状态中，可以看到当前 dashboard 暴露的端口，比如笔者的 443 端口映射到了 31382 端口上，因此直接访问 https://master-ip:端口

  7. 生成token

    kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

    之后终端会显示超级长一串 token，将该 token 粘贴到浏览器，就可以进入 Dashboard

参考资料：

Kubernetes Handbook——Kubernetes中文指南/云原生应用架构实践手册：https://jimmysong.io/kubernetes-handbook/

kubernetes-dashboard(1.8.3)部署与踩坑：https://www.cnblogs.com/rainingnight/p/deploying-k8s-dashboard-ui.html

k8s dashboard 的http接口（Interface）改造：https://juejin.im/post/5d749d95518825770c43a7e9

kubernetes dashboard在ssl的各种场景下的手动部署：https://www.servicemesher.com/blog/general-kubernetes-dashboard/

简单了解一下K8S，并搭建自己的集群: https://zhuanlan.zhihu.com/p/97605697?utm_source=qq&utm_medium=social&utm_oi=58758625492992

  • 在线代码编写环境——云端的vscode：code-server
  • Coming soon
_(:3 」∠)_
(抱歉)
这里不能评论哦

Title
安装前检查
安装 kubeadm，kubelet 和 kubectl
初始化集群
设置当前用户可使用 Kubernetes
安装网络插件

© 2023 夏末阁. Using Typecho & Moricolor.

陕ICP备16018536号-1