评论

收藏

[Linux] 企业级k8s高可用集群搭建(无坑版,如有坑请联系解决)

服务系统 服务系统 发布于:2021-07-06 16:28 | 阅读数:414 | 评论:0

  

  • 目录
    一.服务器规划
    二.资源准备(系统配置,内核优化,yum源配置)(所有节点操作)
    三.keepalived配置(master节点操作)
    四.haproxy配置(master节点操作)
    五.docker安装(所有节点操作)
    六.安装kubeadm,kubelet kubectl(所有节点操作)
    七.k8s集群安装(在具有vip的master上操作)
    八.安装集群网络(master节点操作)
    九.其他节点加入集群
    十.部署dashboard(k8s-master-01)
    十一.部署ingress(0.30.0)
    十二.部署metric
    十三.部署kubernetes dns缓存
    十四.k8s缩容扩容维护
    十五.k8s常见操作十六.参考下一篇,关于应用部署实际应用案例
  一.服务器规划(基于腾讯云服务器的规划)
  k8s-master-01 10.206.16.14 master
  k8s-master-02 10.206.16.15 master
  k8s-master-03 10.206.16.16 master
  k8s-node-01 10.206.16.8 node
  k8s-node-02 10.206.16.9  node
  k8s-harbor 10.206.16.4  harbor
  vip              10.206.16.18   (https://cloud.tencent.com/document/product/215/36694 腾讯云vip申请办法)
  服务器系统centos8  k8s版本1.16  机器配置4核16G
  二.资源准备
  1.设置主机的名字
  hostnamectl set-hostname k8s-master-01(分别在对应的主机按照规划设置名字)
  2.编辑hosts文件(在所有主机操作)
  cat <<EOF >>/etc/hosts
  10.206.16.18 master.k8s.io k8s-vip
  10.206.16.14 master01.k8s.io k8s-master-01
  10.206.16.15 master02.k8s.io k8s-master-02
  10.206.16.16 master03.k8s.io k8s-master-03
  10.206.16.8 node01.k8s.io k8s-node-01
  10.206.16.9 node02.k8s.io k8s-node-02
  10.206.16.4 k8s-harbor
  EOF
  3.关闭防火墙,selinux,swap(在所有主机操作)
systemctl stop firewalld && systemctl disable firewalld && setenforce 0 && swapoff -a
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
  4.配置内核参数(所有主机操作)
cat >/etc/sysctl.d/k8s.conf <<EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables =1
net.bridge.bridge-nf-call-iptables =1
EOF
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
  5.设置资源配置文件(所有主机操作)
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf
  6.配置yum源(所有主机操作)
yum install -y wget  
mkdir /etc/yum.repos.d/bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak  
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos8_base.repo  
wget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo  
yum clean all && yum makecache
  7.配置kubernetes源(所有主机操作)
cat <<EOF >/etc/yum.repos.d/kubernetes.repo  
[kubernetes]  
name=Kubernetes  
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/  
enabled=1  
gpgcheck=1  
repo_gpgcheck=1  
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
  8.配置docker源(所有主机操作)
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
  9.安装相关包(所有主机操作)
yum install -y conntrack-tools libseccomp libtool-ltdl
  三.部署keepalived(在3台master机器操作)
  1.安装keepalived(在3台master操作)
yum install -y conntrack-tools libseccomp libtool-ltdl
  2.配置(另外的两台master配置和上面类似,只需要修改对应的state配置为BACKUP,priority权重值不同即可,配置中的其他字段这里不做说明)
  k8s-master-01配置:
cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived
global_defs {
   router_id k8s
}
vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 3
  weight -2
  fall 10
  rise 2
}
vrrp_instance VI_1 {
  state MASTER 
  interface eth0 
  virtual_router_id 51
  priority 250
  nopreempt   #设置非抢占模式
  preempt_delay 10  #抢占延时10分钟
  advert_int 1    #检查间隔默认1s
  authentication {
    auth_type PASS
    auth_pass ceb1b3ec013d66163d6ab11
  }
  unicast_src_ip 10.206.16.14  #设置本机内网ip
  unicast_peer{  #其他两台master ip
    10.206.16.15
    10.206.16.16
  }
 
  virtual_ipaddress {
    10.206.16.18
  }
  track_script {
    check_haproxy
  }
}
EOF
  k8s-master-02配置
cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived
global_defs {
   router_id k8s
}
vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 3
  weight -2
  fall 10
  rise 2
}
vrrp_instance VI_1 {
  state BACKUP 
  interface eth0
  virtual_router_id 51
  nopreempt
  preempt_delay 10
  priority 200
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass ceb1b3ec013d66163d6ab11
  }
  unicast_src_ip 10.206.16.15  #设置本机内网ip
  unicast_peer{
    10.206.16.14
    10.206.16.16
  }
  virtual_ipaddress {
     10.206.16.18
  }
  track_script {
    check_haproxy
  }
}
EOF
  k8s-master-03配置:
cat > /etc/keepalived/keepalived.conf <<EOF 
! Configuration File for keepalived
global_defs {
   router_id k8s
}
vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 3
  weight -2
  fall 10
  rise 2
}
vrrp_instance VI_1 {
  state BACKUP
  interface eth0
  virtual_router_id 51
  priority 150
  nopreempt
  preempt_delay 10
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass ceb1b3ec013d66163d6ab11
  }
  unicast_src_ip 10.206.16.16
  unicast_peer{
    10.206.16.14
    10.206.16.15
  }
  virtual_ipaddress {
     10.206.16.18
  }
  track_script {
    check_haproxy
  }
}
EOF
  3.启动检查
  设置开机启动
systemctl enable keepalived.service
  启动
systemctl start keepalived.service
  查看启动状态
systemctl status keepalived.service
  启动后查看k8s-master-01网卡信息
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
  link/ether 52:54:00:f2:57:46 brd ff:ff:ff:ff:ff:ff
  inet 10.206.16.14/20 brd 10.206.31.255 scope global noprefixroute eth0
     valid_lft forever preferred_lft forever
  inet 10.206.16.18/32 scope global eth0
     valid_lft forever preferred_lft forever
  inet6 fe80::5054:ff:fef2:5746/64 scope link 
     valid_lft forever preferred_lft forever
  尝试停掉k8s-master-01的keepalived服务,查看vip是否能漂移到其他的master,并且重新启动k8s-master-01的keepalived服务,查看vip是否能正常漂移回来,证明配置没有问题。
  四.haproxy搭建(三台master上操作)
  1.安装haproxy(三台master执行)
yum install -y haproxy
  2.配置(三台master执行)
cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
  # to have these messages end up in /var/log/haproxy.log you will
  # need to:
  # 1) configure syslog to accept network log events.  This is done
  #  by adding the '-r' option to the SYSLOGD_OPTIONS in
  #  /etc/sysconfig/syslog
  # 2) configure local2 events to go to the /var/log/haproxy.log
  #   file. A line like the following can be added to
  #   /etc/sysconfig/syslog
  #
  #  local2.*             /var/log/haproxy.log
  #
  log     127.0.0.1 local2
  
  chroot    /var/lib/haproxy
  pidfile   /var/run/haproxy.pid
  maxconn   4000
  user    haproxy
  group     haproxy
  daemon 
     
  # turn on stats unix socket
  stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------  
defaults
  mode          http
  log           global
  option          httplog
  option          dontlognull
  option http-server-close
  option forwardfor     except 127.0.0.0/8
  option          redispatch
  retries         3
  timeout http-request  10s
  timeout queue       1m
  timeout connect     10s
  timeout client      1m
  timeout server      1m
  timeout http-keep-alive 10s
  timeout check       10s
  maxconn         3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#--------------------------------------------------------------------- 
frontend kubernetes-apiserver
  mode         tcp
  bind         *:16443
  option         tcplog
  default_backend    kubernetes-apiserver  
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
  mode    tcp
  balance   roundrobin
  server    master01.k8s.io   10.206.16.14:6443 check
  server    master02.k8s.io   10.206.16.15:6443 check
  server    master03.k8s.io   10.206.16.16:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
  bind         *:1080
  stats auth       admin:awesomePassword
  stats refresh    5s
  stats realm      HAProxy\ Statistics
  stats uri      /admin?stats
EOF
  3.启动和检查(三台master执行)
  # 设置开机启动 systemctl enable haproxy
  # 开启haproxy  systemctl start haproxy
  # 查看启动状态  systemctl status haproxy
  4.检查服务端口是否启动
[root@VM-16-14-centos yum.repos.d]# netstat -lntup|grep haproxy
tcp    0    0 0.0.0.0:1080      0.0.0.0:*         LISTEN    37567/haproxy     
tcp    0    0 0.0.0.0:16443       0.0.0.0:*         LISTEN    37567/haproxy     
udp    0    0 0.0.0.0:48413       0.0.0.0:*               37565/haproxy
   五.安装docker(所有节点)
  1.安装
  # step 1: 安装必要的一些系统工具
  $ yum install -y yum-utils device-mapper-persistent-data lvm2
  # Step 2: 添加软件源信息
  $ sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  # Step 3: 查找Docker-CE的版本:
  $ yum list docker-ce.x86_64 --showduplicates | sort -r
  # Step 4: 安装指定版本的Docker-CE
  $ yum makecache
  $ yum install -y docker-ce
  2.配置
  修改docker的配置文件,目前k8s推荐使用的docker文件驱动是systemd
  mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
  "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
  "overlay2.override_kernel_check=true"
  ],
  "insecure-registries" : ["10.206.16.4"]
}
EOF
  修改docker的服务配置文件,指定docker的数据目录为外挂的磁盘--graph /data/docker
mkdir /data/docker
sed -i "s#containerd.sock#containerd.sock --graph /data/docker#g" /lib/systemd/system/docker.service
  添加如下执行语句:(如果pod之间无法通信的问题)
mkdir -p /etc/systemd/system/docker.service.d/
cat>/etc/systemd/system/docker.service.d/10-docker.conf<<EOF
[Service]
ExecStartPost=/sbin/iptables --wait -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecStopPost=/bin/bash -c '/sbin/iptables --wait -D FORWARD -s 0.0.0.0/0 -j ACCEPT &> /dev/null || :'
ExecStartPost=/sbin/iptables --wait -I INPUT -i cni0 -j ACCEPT
ExecStopPost=/bin/bash -c '/sbin/iptables --wait -D INPUT -i cni0 -j ACCEPT &> /dev/null || :'
EOF
  3.启动docker
$ systemctl daemon-reload
$ systemctl start docker.service
$ systemctl enable docker.service
$ systemctl status docker.service
  六.安装kubeadm,kubelet,和kubectl(所有节点操作)
  1.安装
yum install -y kubelet-1.16.3 kubeadm-1.16.3 kubectl-1.16.3
systemctl enable kubelet
  2.配置kubectl自动补全功能
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
  七.安装k8s集群(在具有vip的k8s-master-01上操作)
  1.创建配置文件:
  mkdir /usr/local/kubernetes/manifests -p
  cd /usr/local/kubernetes/manifests/
cat > kubeadm-config.yaml <<EOF
apiServer:
  certSANs:
  - k8s-master-01
  - k8s-master-02
  - k8s-master-03
  - master.k8s.io
  - 10.206.16.14
  - 10.206.16.15
  - 10.206.16.16
  - 10.206.16.18
  - 127.0.0.1
  extraArgs:
  authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns: 
  type: CoreDNS
etcd:
  local:  
  dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.16.3
networking: 
  dnsDomain: cluster.local  
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.1.0.0/16
scheduler: {}
EOF
  2.初始化master节点
[root@VM-16-14-centos manifests]# kubeadm init --config kubeadm-config.yaml 
[init] Using Kubernetes version: v1.16.3
[preflight] Running pre-flight checks
[WARNING FileExisting-tc]: tc not found in system path
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.7. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master.k8s.io k8s-master-01 k8s-master-02 k8s-master-03 master.k8s.io] and IPs [10.1.0.1 10.206.16.14 10.206.16.14 10.206.16.15 10.206.16.16 10.206.16.18 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-01 localhost] and IPs [10.206.16.14 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-01 localhost] and IPs [10.206.16.14 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 36.002615 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master-01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ee3zom.l4xeahsfqcj9uvvz
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:
  kubeadm join master.k8s.io:16443 --token ee3zom.l4xeahsfqcj9uvvz \
  --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b \
  --control-plane   
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join master.k8s.io:16443 --token ee3zom.l4xeahsfqcj9uvvz \
  --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b
  3.按照提示配置环境变量
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  4.查看集群状态
[root@VM-16-14-centos manifests]# kubectl get cs
NAME         AGE
scheduler      <unknown>
controller-manager   <unknown>
etcd-0         <unknown>
[root@VM-16-14-centos manifests]# kubectl get pods -n kube-system
NAME                  READY   STATUS  RESTARTS   AGE
coredns-58cc8c89f4-ch8pn        0/1   Pending   0      2m45s
coredns-58cc8c89f4-qdz7t        0/1   Pending   0      2m45s
etcd-k8s-master-01            1/1   Running   0      113s
kube-apiserver-k8s-master-01      1/1   Running   0      99s
kube-controller-manager-k8s-master-01   1/1   Running   0      98s
kube-proxy-wvp9b            1/1   Running   0      2m45s
kube-scheduler-k8s-master-01      1/1   Running   0      2m6s
  里处于pending状态的原因是因为还没有安装网络组件
  九.安装集群网络(master操作)
  1.安装flannel插件
[root@VM-16-14-centos flannel]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
  2.检查
[root@VM-16-14-centos flannel]# kubectl get pods -n kube-system
NAME                  READY   STATUS  RESTARTS   AGE
coredns-58cc8c89f4-ch8pn        1/1   Running   0      32m
coredns-58cc8c89f4-qdz7t        1/1   Running   0      32m
etcd-k8s-master-01            1/1   Running   0      31m
kube-apiserver-k8s-master-01      1/1   Running   0      31m
kube-controller-manager-k8s-master-01   1/1   Running   0      31m
kube-flannel-ds-qljzc           1/1   Running   0      63s
kube-proxy-wvp9b            1/1   Running   0      32m
kube-scheduler-k8s-master-01      1/1   Running   0      32m
  十.其他节点加入集群(master,node都需要加入)
1.master加入集群
1.1复制密钥和相关文件(k8s-master-01执行)
建立免登录
ssh-keygen -t rsa
ssh-copy-id root@10.206.16.15
ssh-copy-id root@10.206.16.16
复制文件到k8s-master-02
ssh root@10.206.16.15 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@10.206.16.15:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@10.206.16.15:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@10.206.16.15:/etc/kubernetes/pki/etcd
  复制文件到k8s-master-03
ssh root@10.206.16.16 mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/admin.conf root@10.206.16.16:/etc/kubernetes
scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@10.206.16.16:/etc/kubernetes/pki
scp /etc/kubernetes/pki/etcd/ca.* root@10.206.16.16:/etc/kubernetes/pki/etcd
  1.2 master加入集群
  分别在其他两个master ,k8s-master-02,k8s-master-03上执行k8s-master-01 init后输出的join命令,如果找不到可以在k8s-master-01执行  kubeadm token create --print-join-command
  在k8s-master-02上执行操作,需要带上参数--control-plane表示把master控制节点加入到集群
kubeadm join master.k8s.io:16443 --token 13dqfw.8vteayxksdn03mve   --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b  --control-plane
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  在k8s-master-03上执行join命令
kubeadm join master.k8s.io:16443 --token 13dqfw.8vteayxksdn03mve   --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b  --control-plane
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  1.3检查master是否加入成功
root@VM-16-14-centos flannel]#  kubectl get node
NAME      STATUS   ROLES  AGE   VERSION
k8s-master-01   Ready  master   74m   v1.16.3
k8s-master-02   Ready  master   18m   v1.16.3
k8s-master-03   Ready  master   87s   v1.16.3
[root@VM-16-14-centos flannel]# kubectl get pods --all-namespaces
NAMESPACE   NAME                  READY   STATUS  RESTARTS   AGE
kube-system   coredns-58cc8c89f4-ch8pn        1/1   Running   0      74m
kube-system   coredns-58cc8c89f4-qdz7t        1/1   Running   0      74m
kube-system   etcd-k8s-master-01            1/1   Running   0      73m
kube-system   etcd-k8s-master-02            1/1   Running   0      19m
kube-system   etcd-k8s-master-03            1/1   Running   0      93s
kube-system   kube-apiserver-k8s-master-01      1/1   Running   0      73m
kube-system   kube-apiserver-k8s-master-02      1/1   Running   0      19m
kube-system   kube-apiserver-k8s-master-03      1/1   Running   0      94s
kube-system   kube-controller-manager-k8s-master-01   1/1   Running   1      73m
kube-system   kube-controller-manager-k8s-master-02   1/1   Running   0      19m
kube-system   kube-controller-manager-k8s-master-03   1/1   Running   0      94s
kube-system   kube-flannel-ds-965w9           1/1   Running   0      94s
kube-system   kube-flannel-ds-qljzc           1/1   Running   0      42m
kube-system   kube-flannel-ds-vjn8d           1/1   Running   1      19m
kube-system   kube-proxy-6w9ch            1/1   Running   0      19m
kube-system   kube-proxy-p4mt8            1/1   Running   0      94s
kube-system   kube-proxy-wvp9b            1/1   Running   0      74m
kube-system   kube-scheduler-k8s-master-01      1/1   Running   1      73m
kube-system   kube-scheduler-k8s-master-02      1/1   Running   0      19m
kube-system   kube-scheduler-k8s-master-03      1/1   Running   0      94s
  2.node加入到集群(分别在两个node上执行)
  2.1配置:
kubeadm join master.k8s.io:16443 --token hx67nu.7nlxcsvcsa8uy46o   --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b
  2.2检查:
[root@VM-16-14-centos flannel]#  kubectl get node
NAME      STATUS   ROLES  AGE   VERSION
k8s-master-01   Ready  master   80m   v1.16.3
k8s-master-02   Ready  master   24m   v1.16.3
k8s-master-03   Ready  master   7m11s   v1.16.3
k8s-node-01   Ready  <none>   2m58s   v1.16.3
k8s-node-02   Ready  <none>   101s  v1.16.3
[root@VM-16-14-centos flannel]#  kubectl get pods -n kube-system
NAME                  READY   STATUS  RESTARTS   AGE
coredns-58cc8c89f4-ch8pn        1/1   Running   0      80m
coredns-58cc8c89f4-qdz7t        1/1   Running   0      80m
etcd-k8s-master-01            1/1   Running   0      79m
etcd-k8s-master-02            1/1   Running   0      24m
etcd-k8s-master-03            1/1   Running   0      7m20s
kube-apiserver-k8s-master-01      1/1   Running   0      79m
kube-apiserver-k8s-master-02      1/1   Running   0      24m
kube-apiserver-k8s-master-03      1/1   Running   0      7m21s
kube-controller-manager-k8s-master-01   1/1   Running   1      79m
kube-controller-manager-k8s-master-02   1/1   Running   0      24m
kube-controller-manager-k8s-master-03   1/1   Running   0      7m21s
kube-flannel-ds-965w9           1/1   Running   0      7m21s
kube-flannel-ds-nvdhl           1/1   Running   0      111s
kube-flannel-ds-qljzc           1/1   Running   0      48m
kube-flannel-ds-vjn8d           1/1   Running   1      24m
kube-flannel-ds-z9zc2           1/1   Running   0      3m8s
kube-proxy-6w9ch            1/1   Running   0      24m
kube-proxy-fswvz            1/1   Running   0      111s
kube-proxy-p4mt8            1/1   Running   0      7m21s
kube-proxy-wvp9b            1/1   Running   0      80m
kube-proxy-z27lw            1/1   Running   0      3m8s
kube-scheduler-k8s-master-01      1/1   Running   1      79m
kube-scheduler-k8s-master-02      1/1   Running   0      24m
kube-scheduler-k8s-master-03      1/1   Running   0      7m21s
  十一.部署dashboard(k8s-master-01执行)
  部署最新版本v2.0.0-beta6,下载yaml
  cd /usr/local/kubernetes/manifests/
  mkdir dashboard && cd dashboard
  wget -c https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta6/aio/deploy/recommended.yaml
  # 修改service类型为nodeport
  vim recommended.yaml
...
kind: Service
apiVersion: v1
metadata:
  labels:
  k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
  - port: 443
    targetPort: 8443
    nodePort: 30001
  selector:
  k8s-app: kubernetes-dashboard
...
[root@k8s-master-01 dashboard]# kubectl apply -f recommended.yaml 
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[root@k8s-master-01 dashboard]# kubectl get pods -n kubernetes-dashboard 
NAME                     READY   STATUS  RESTARTS   AGE
dashboard-metrics-scraper-76585494d8-62vp9   1/1   Running   0      6m47s
kubernetes-dashboard-b65488c4-5t57x      1/1   Running   0      6m48s
[root@k8s-master-01 dashboard]# kubectl get svc -n kubernetes-dashboard
NAME            TYPE    CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
dashboard-metrics-scraper   ClusterIP   10.1.207.27  <none>    8000/TCP    7m6s
kubernetes-dashboard    NodePort  10.1.207.168   <none>    443:30001/TCP   7m7s
# 在node上通过https://nodeip:30001访问是否正常,注意在firefox浏览器执行,使用非安全模式进入
  2.创建service account并绑定默认cluster-admin管理员集群角色
  vim dashboard-adminuser.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
[root@VM-16-14-centos dashboard]# kubectl apply -f dashboard-adminuser.yaml 
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
获取token,dashboard通过这个token进入系统
[root@VM-16-14-centos dashboard]# kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
Name:     admin-user-token-p7wgc
Namespace:  kubernetes-dashboard
Labels:     <none>
Annotations:  kubernetes.io/service-account.name: admin-user
        kubernetes.io/service-account.uid: 0e9f5406-3c26-4141-a233-ff4eaa841401
Type:  kubernetes.io/service-account-token
Data
====
namespace:  20 bytes
token:    eyJhbGciOiJSUzI1NiIsImtpZCI6InBDdkJ1VWZFZENYbEd3ZGVrc3FldlhXWG94QU0ySjN1M1Y4ZVRJOUZPd1UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXA3d2djIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwZTlmNTQwNi0zYzI2LTQxNDEtYTIzMy1mZjRlYWE4NDE0MDEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.jcCnl8hHWrZcFtd5H17gJdoaDiFsPUos_4oNYQexiXSjFDgy972Bk1qgYV-zHZhu7o_UZyESMRLTlzRFl3W5Eqbhq9fouD0j0DH_qnTGewNTEuByQj5n6uPLloPG5VNCOs1y3TINVj8LdG5q_n6DWfozfn76eNhU9eAnSJZVZ97dGKy_LDykpM9QtJQQkpaF9jSnDPCeoSnSd_1ud1FoQlNS3PAenB54khOmL5gbD6Pf4uJOVUjzxoHk_--gKDW7juVAsaDPbbGftuiM1mIfQ3K02VoNMiG1VB2hlzJ5kWeUn7wpqZpmngzrqBtVj5DJWSpnHAZZef_FFCakKMp5TA
ca.crt:   1025 bytes
  十二.部署ingress控制器 mandatory.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
 data:  #为了让pod获取真实ip
  compute-full-forwarded-for: 'true'
  use-forwarded-headers: 'true'   
  
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
    - ""
  resources:
    - configmaps
    - endpoints
    - nodes
    - pods
    - secrets
  verbs:
    - list
    - watch
  - apiGroups:
    - ""
  resources:
    - nodes
  verbs:
    - get
  - apiGroups:
    - ""
  resources:
    - services
  verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
  resources:
    - events
  verbs:
    - create
    - patch
  - apiGroups:
    - "extensions"
    - "networking.k8s.io"
  resources:
    - ingresses
  verbs:
    - get
    - list
    - watch
  - apiGroups:
    - "extensions"
    - "networking.k8s.io"
  resources:
    - ingresses/status
  verbs:
    - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
    - ""
  resources:
    - configmaps
    - pods
    - secrets
    - namespaces
  verbs:
    - get
  - apiGroups:
    - ""
  resources:
    - configmaps
  resourceNames:
    # Defaults to "<election-id>-<ingress-class>"
    # Here: "<ingress-controller-leader>-<nginx>"
    # This has to be adapted if you change either parameter
    # when launching the nginx-ingress-controller.
    - "ingress-controller-leader-nginx"
  verbs:
    - get
    - update
  - apiGroups:
    - ""
  resources:
    - configmaps
  verbs:
    - create
  - apiGroups:
    - ""
  resources:
    - endpoints
  verbs:
    - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
---
apiVersion: apps/v1
kind: DaemonSet 
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
  app.kubernetes.io/name: ingress-nginx
  app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
  matchLabels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  template:
  metadata:
    labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    annotations:
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
  spec:
    serviceAccountName: nginx-ingress-serviceaccount
    hostNetwork: true
    containers:
    - name: nginx-ingress-controller
      image: lizhenliang/nginx-ingress-controller:0.20.0 
      args:
      - /nginx-ingress-controller
      - --configmap=$(POD_NAMESPACE)/nginx-configuration
      - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      - --publish-service=$(POD_NAMESPACE)/ingress-nginx
      - --annotations-prefix=nginx.ingress.kubernetes.io
      securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE
      # www-data -> 33
      runAsUser: 33
      env:
      - name: POD_NAME
        valueFrom:
        fieldRef:
          fieldPath: metadata.name
      - name: POD_NAMESPACE
        valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
      ports:
      - name: http
        containerPort: 80
      - name: https
        containerPort: 443
      livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: 10254
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
      readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: 10254
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
---
  十三.部署metric(v0.3.6)  mandatory.yaml
## ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
## ClusterRole aggregated-metrics-reader
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
  rbac.authorization.k8s.io/aggregate-to-view: "true"
  rbac.authorization.k8s.io/aggregate-to-edit: "true"
  rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods","nodes"]
  verbs: ["get","list","watch"]
---
## ClusterRole metrics-server
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups: [""]
  resources: ["pods","nodes","nodes/stats","namespaces","configmaps"]
  verbs: ["get","list","watch"]
---
## ClusterRoleBinding auth-delegator
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
## RoleBinding metrics-server-auth-reader
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
## ClusterRoleBinding system:metrics-server
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
 
## APIService
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
  name: metrics-server
  namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
 
 
## Service
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
  kubernetes.io/name: "Metrics-server"
  kubernetes.io/cluster-service: "true"
spec:
  selector:
  k8s-app: metrics-server
  ports:
  - port: 443
  targetPort: 4443
---
## Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
  k8s-app: metrics-server
spec:
  selector:
  matchLabels:
    k8s-app: metrics-server
  template:
  metadata:
    name: metrics-server
    labels:
    k8s-app: metrics-server
  spec:
    hostNetwork: true
    serviceAccountName: metrics-server
    containers:
    - name: metrics-server
    ## 修改镜像源地址
    image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
    imagePullPolicy: IfNotPresent
    args:
      - --cert-dir=/tmp
      - --secure-port=4443
      - --kubelet-insecure-tls  ## 增加
      - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname ## 增加
    ports:
    - name: main-port
      containerPort: 4443
      protocol: TCP
    securityContext:
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
    resources:
      limits:
      memory: 1Gi
      cpu: 1000m
      requests:
      memory: 1Gi
      cpu: 1000m
    volumeMounts:
    - name: tmp-dir
      mountPath: /tmp
    - name: localtime
      readOnly: true
      mountPath: /etc/localtime
    volumes:
    - name: tmp-dir
    emptyDir: {}
    - name: localtime
    hostPath:
      type: File
      path: /etc/localtime
    nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: "amd64"
  十四.安装kubernetes dns缓存,避免延迟问题
  kubectl apply -f https://github.com/feiskyer/kubernetes-handbook/raw/master/examples/nodelocaldns/nodelocaldns-kubenet.yaml
  (参考:https://mp.weixin.qq.com/s/t7nt87JPJnWEVCNBS-sBpw)
  十五.    集群的扩容,缩容
  1.集群扩容
  默认情况下加入集群的token是24小时过期,24小时后如果是想要新的node加入到集群,需要重新生成一个token,命令如下
# 显示获取token列表
$ kubeadm token list
# 生成新的token
$ kubeadm token create
  除token外,join命令还需要一个sha256的值,通过以下方法计算
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
  用上面输出的token和sha256的值或者是利用kubeadm token create --print-join-command拼接join命令即可
  2.集群的缩容
  kubectl cordon <node name>  #设置为不可调度
  kubectl drain <node name> --delete-local-data --force --ignore-daemonsets  驱逐节点上的pod
  kubectl delete node <node name>
  3.初始化重新加入
  需要把原来的配置清空
  kubeadm reset
  systemctl stop kubelet
  systemctl stop docker
  rm -rf /var/lib/cni/
  rm -rf /var/lib/kubelet/*
  rm -rf /etc/cni/
  ifconfig cni0 down
  ifconfig flannel.1 down
  ifconfig docker0 down
  ip link delete cni0
  ip link delete flannel.1
  systemctl start docker
  
  
  
  

  
关注下面的标签,发现更多相似文章