K8s+Jenkins+GitLab-自动化部署项目 一、部署流程
IP 角色 172.25.0.30 master1、Jenkins 172.25.0.31 node1、Gitlab 172.25.0.32 node2、Harbor、Jenkins-slave
二、K8s 安装 1. 安装要求 ² 一台或多台机器，操作系统 CentOS7.x-86_x64² 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多² 可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点² 禁止swap分区2. 准备环境 角色色 IP k8s 版本 k8s-master 172.25.0.30 kubernetes1.21.0 k8s-node1 172.25.0.31 kubernetes1.21.0 k8s-node2 172.25.0.32 kubernetes1.21.0
systemctl stop firewalldsystemctl disable firewalldswapoff -a # 临时sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久hostnamectl set-hostname <hostname>hostnamectl set-hostname <hostname>cat >> /etc/hosts << EOFEOFcat > /etc/sysctl.d/k8s.conf << EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1vm.swappiness=0net.ipv4.ip_forward=1net.ipv4.tcp_tw_recycle=0EOFsysctl --system # 生效yum install ntpdate -yntpdate -u pool.ntp.orgcrontab -l*/20 * * * * /sbin/ntpdate -u pool.ntp.org > /dev/null 2>&13. 所有节点安装Docker/kubeadm/kubelet 3.1 安装Docker #yum install -y yum-utils \ device-mapper-persistent-data \ lvm2#yum-config-manager \ --add-repo \ https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo#yum install docker-ce -y#cat > /etc/docker/daemon.json << EOF {"registry-mirrors": ["https://h***3j.mirror.aliyuncs.com"]}EOF3.2 添加阿里云YUM软件源 $ cat > /etc/yum.repos.d/kubernetes.repo << EOF[kubernetes]name=Kubernetesbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=0repo_gpgcheck=0gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgEOF3.3 安装kubeadm，kubelet和kubectl $ yum install -y kubelet-1.21.0 kubeadm-1.21.0 kubectl-1.21.0$ systemctl enable kubelet4. 部署Kubernetes Master $ kubeadm init \ --apiserver-advertise-address=172.25.0.30 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.21.0 \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 --upload-certsdocker pull registry.aliyuncs.com/google_containers/coredns:1.8.0docker tag registry.aliyuncs.com/google_containers/coredns:1.8.0 registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config$ kubectl get nodes5. 加入Kubernetes Node $ kubeadm join 172.25.0.30:6443 --token amdbyn.a02my1ugmoblwy4q --discovery-token-ca-cert-hash sha256:18462463a7db86052399e97b18efe3f12edc5999293abdccf7529669df0ad3fakubeadm token create --print-join-command6. 部署CNI网络插件 wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlkubectl apply -f kube-flannel.ymlkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlkubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcoredns-545d6fc579-2cgr8 1/1 Running 0 72s
7. 测试kubernetes集群 $ kubectl create deployment nginx --image=nginx$ kubectl expose deployment nginx --port=80 --type=NodePort$ kubectl get pod,svc三、部署gitlab 1、使用docker方式部署 docker run -d --hostname gitlab.xxx.cn \--publish 443:443 --publish 80:80 --publish 2222:22 \--name gitlab --restart always --volume /srv/gitlab/config:/etc/gitlab \--volume /srv/gitlab/logs:/var/log/gitlab \--volume /srv/gitlab/data:/var/opt/gitlab \gitlab/gitlab-ce:latest2、配置gitlab vim /srv/gitlab/config/gitlab.rbexternal_url 'http://172.25.0.31'gitlab_rails['gitlab_ssh_host'] = '172.25.0.31'gitlab_rails['gitlab_shell_ssh_port'] = 2222 # 此端口是run时22端口映射的2222端口sed -i 's/# puma['worker_processes'] = 2/puma['worker_processes'] = 2/g' /srv/gitlab/config/gitlab.rbsed -i 's/# postgresql['shared_buffers'] = "256MB"/postgresql['shared_buffers'] = "64MB"/g' /srv/gitlab/config/gitlab.rbsed -i 's/# postgresql['max_worker_processes'] = 8/postgresql['max_worker_processes'] = 5/g' /srv/gitlab/config/gitlab.rbsed -i 's/# sidekiq['max_concurrency'] = 50/sidekiq['max_concurrency'] = 10/g' /srv/gitlab/config/gitlab.rbsed -i 's/# sidekiq['min_concurrency'] = nil/sidekiq['min_concurrency'] = 5/g' /srv/gitlab/config/gitlab.rb$ docker restart gitlab3、配置管理员passwd 1、进入容器 docker exec -it gitlab /bin/bash启用 gitlab的 rubygitlab-rails console -e production2、进入管理员用户 user = User.where(id: 1).first3、更改password user.password = 'abcd1234'user.password_confirmation = 'abcd1234'sxi4、保存 user.save!
4、访问gitlab 登陆，如下图所示： 账号： root password：abcd1234
四、部署harbor镜像仓库 1、docker-compose 安装 curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-`uname -s`-`uname -m` > /usr/bin/docker-composechmod a+x /usr/bin/docker-compose2、Harbor配置、ssl证书生产 1、下载包，这里是v2.2.3： wget https://github.com/goharbor/harbor/releases/download/v2.2.3/harbor-offline-installer-v2.2.3.tgz2、解压harbor tar -zxvf harbor-offline-installer-v2.2.3.tgz && mv harbor /usr/local/
3、配置并安装 cp harbor.yml.tmpl harbor.ymlvim harbor.ymlhostname: hub.sx.com #也可以是iphttp: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80# https related confighttps: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /data/cert/server.crt private_key: /data/cert/server.keyharbor_admin_password: harbor12345database: # The password for the root user of Harbor DB. Change this before any production use. password: root123 data_volume: /data/harbor #自定义路径mkdir -p /data/{cert,harbor}4、证书生成 mkdir -p /data/cert && cd /data/certopenssl genrsa -des3 -out server.key 2048
openssl req -new -key server.key -out server.csrcp server.key server.key.orgopenssl rsa -in server.key.org -out server.keyopenssl x509 -req -days 3650 -in server.csr -signkey server.key -out server.crtchmod a+x *3、harbor安装 1、运行prepare文件 cd /usr/local/harbor./prepare
2、执行脚本安装 ./install.sh
4、登录访问 1、页面登录 harbor.yaml 的 hostname ) 。
2、docker登录 echo 172.25.0.32 hub.sx.com >> /etc/hostsdocker login https://hub.sx.commkdir -p /etc/docker/certs.d/hub.sx.comrsync -avz root@172.25.0.32:/data/cert/server.crt /etc/docker/certs.d/hub.sx.com/
五、部署Jenkins-slave分布式环境 xiaozhagn/jenkins-slave 一、Master-Slave文件下载：
二、制作镜像 cd jenkins-dockerdocker build -t jenkins-slave:latest .
三、推送镜像到harbor docker tag jenkins-slave hub.sx.com/ops/jenkins-slavedocker push hub.sx.com/ops/jenkins-slave
六、k8s部署Jenkins 1、创建nfs yum install nfs-utils -ymkdir -p /data/nfs/jenkinsvim /etc/exports/data/nfs/jenkins 172.25.0.30/24(sync,rw,no_root_squash)systemctl restart nfs rpcbindsystemctl enable nfs rpcbindyum install nfs-utils -ysystemctl start nfs && systemctl enable nfsshowmount -e 172.25.0.30
2、yaml部署jenkins 1、创建namespace，用于安装jenkins kubectl create namespace jenkinskubectl get namespace
2、创建 PV & PVC vim jenkins-storage.yamlapiVersion: v1kind: PersistentVolumemetadata: name: jenkins labels: app: jenkinsspec: capacity: storage: 20Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain mountOptions: #NFS挂载选项 - hard - nfsvers=4.1 nfs: #NFS设置 path: /data/nfs/jenkins # 根据上一步的地址来定 server: 172.25.0.30 # 根据nfs server ip实际来设置---kind: PersistentVolumeClaimapiVersion: v1metadata: name: jenkins namespace: jenkins spec: accessModes: - ReadWriteMany resources: requests: storage: 20Gi #存储空间大小 selector: matchLabels: app: jenkins-n ：指定 namespace Yaml 文件指定了 namespce 运行时可以不指定namespce kubectl apply -f jenkins-storage.yaml -n jenkinskubectl get pvkubectl get pvc -n jenkins
3、创建 ServiceAccount & ClusterRoleBinding 这里为了方便，将 cluster-admin 绑定到 ServiceAccount 来保证 Jenkins 拥有足够的权限。 vim jenkins-rbac.yamlapiVersion: v1kind: ServiceAccountmetadata: name: jenkins-admin #ServiceAccount用户名称，（注意不要改） namespace: jenkins #指定namespace，一定要修改成你自己的namespace labels: name: jenkins---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: jenkins-adminnamespace: jenkins labels: name: jenkinssubjects: - kind: ServiceAccount name: jenkins-admin namespace: jenkins #这里的表示在jenkins空间授予cluster-admin的权限roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.iokubectl apply -f jenkins-rbac.yaml -n jenkins4、创建 Service & Deployment vim jenkins-deployment.yamlapiVersion: v1kind: Servicemetadata: name: jenkinsnamespace: jenkins labels: app: jenkinsspec: type: NodePort ports: - name: http port: 8080 #服务端口 targetPort: 8080 nodePort: 32000 #NodePort方式暴露 Jenkins 端口 - name: jnlp port: 50000 #代理端口 targetPort: 50000 nodePort: 32001 selector: app: jenkins---apiVersion: apps/v1kind: Deploymentmetadata: name: jenkins namespace: jenkins labels: app: jenkinsspec: selector: matchLabels: app: jenkins replicas: 1 template: metadata: labels: app: jenkins spec: nodeName: k8s-master #是在运行在K8s-master上 serviceAccountName: jenkins-admin containers: - name: jenkins image: jenkins/jenkins:lts securityContext: runAsUser: 0 #设置以ROOT用户运行容器 privileged: true #拥有特权 ports: - name: http containerPort: 8080 - name: jnlp containerPort: 50000 resources: limits: memory: 2Gi cpu: "1000m" requests: memory: 512Mi cpu: "500m" env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: "JAVA_OPTS" #设置变量，指定时区和 jenkins slave 执行者设置 value: " -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai " - name: "JENKINS_OPTS" value: "--prefix=/jenkins" #设置路径前缀加上 Jenkins volumeMounts: #设置要挂在的目录 - name: data mountPath: /var/jenkins_home volumes: - name: data persistentVolumeClaim: claimName: jenkins #设置PVC² JAVA_OPTS ： JVM 参数设置 ² JENKINS_OPTS ： Jenkins 参数设置 ² 其它参数： 默认情况下， Jenkins 生成代理是保守的。例如，如果队列中有两个构建，它不会立即生成两个执行器。它将生成一个执行器，并等待某个时间释放第一个执行器，然后再决定生成第二个执行器。 Jenkins 确保它生成的每个执行器都得到了最大限度的利用。如果你想覆盖这个行为，并生成一个执行器为每个构建队列立即不等待，所以在 Jenkins 启动时候添加这些参数 : -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 有了上面的部署文件后，再将 Jenkins 部署到 Kuberntes 中： kubectl apply -f jenkins-deployment.yaml -n jenkinskubectl describe pod $(kubectl get pods -n jenkins | awk '{print $1}' | grep jenkins) -n jenkins
nodeName: k8s-master
resources: limits: memory: 1Gi cpu: "1000m" requests: memory: 512Mi cpu: "500m"5、配置私有仓库的镜像下载与访问 docker login hub.sx.com# cat .docker/config.json |base64 -w 0ewoJImF1dGhzIjogewoJCSJodWIuc3guY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmFHRnlZbTl5TVRJek5EVT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjggKGxpbnV4KSIKCX0KfQ==vim registry-pull-secret.yamlapiVersion: v1kind: Secretmetadata: name: registry-pull-secret namespace: jenkinsdata: .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJodWIuc3guY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmFHRnlZbTl5TVRJek5EVT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjggKGxpbnV4KSIKCX0KfQ==type: kubernetes.io/dockerconfigjsonkubectl apply -f registry-pull-secret.yaml -n jenkinskubectl get secret
3、获取 Jenkins 生成的 Token 在安装 Jenkins 时候，它默认生成一段随机字符串，用于安装验证。这里访问它的安装日志，获取它生成的 Token 字符串。 (1) 、查看 Jenkins Pod 启动日志 kubectl logs $(kubectl get pods -n jenkins | awk '{print $1}' | grep jenkins) -n jenkins
(2) 、查看日志中生成的 Token 字符串 查看日志，默认给的 token为： bf0c6fcb10b845239a06ff9362a6843c4、访问Jenkins
token
七、Jenkins、gitlab配置 一、安装k8s的相关插件 1、jenkins 设置中文显示
2、安装项目插件 Kubernetes Continuous Deploy、Kubernetes、Gitlab Hook 、GitLab、Build Authorization Token、Pipeline: Stage Step、Pipeline: Basic Steps、Pipeline 的插件1、页面安装插件
2、手动上传安装
docker restart （k8s_jenkins_jenkins容器）二、构建项目
三、设置jenkins挂钩k8s的环境 1、创建kubernets云
2、配置kubernets云
² Kubernetes 地址：https://kubernetes.default 或者 k8s的地址https://172.25.0.30:6443² 空间名：jenkins （自己创建的空间名）² Jenkins 地址：http://jenkins.jenkins.svc.cluster.local:8080
Error testing connection https://172.25.0.30:6443: java.io.FileNotFoundException: /root/.kube/config (No such file or directory) 四、jenkins添加凭证
1、添加私钥认证
2、添加k8s认证
五、gitlab添加凭证
八、jenkins+gitlab挂钩
1、test项目配置。
2、配置gitlab
URl：http://172.25.0.30:32000/jenkins/project/testSecret token：c07ebaae20ced8e74c8c0d93890708b03、配置添加webhook报错处理
九、部署代码 1、拉去测试代码，修改并推送到demo
2、查看jenkins任务是否已经触发并构建成功
3、访问测试
十、jenkins报错以及处理 1、报错一： Failed to connect to http://jenkins.jenkins.svc.cluster.local:8080/jenkins/tcpSlaveAgentListener/: jenkins.jenkins.svc.cluster.local
kubectl delete -f kube-flannel.ymlkubectl create -f kube-flannel.ymlkubectl get pods -n kube-system -oname |grep coredns |xargs kubectl delete -n kube-systemkubectl edit deploy coredns -n kube-systemimage: registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0image: registry.aliyuncs.com/google_containers/coredns:1.8.02、报错二： Illegal tunneling parameter: http://jenkins.jenkins.svc.cluster.local:50000
3、报错三： with name:[null] in namespace:[kube-ops] failed.
4、报错四： ERROR:ERROR:java.lang.RuntimeException:io.kubernetes.client.openapi.ApiException: Bad Request Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: Bad Request
各位客官，编写实在不易，点个赞再走呗 完～
关注公众号，了解更多 
服务系统
发布于：2021-12-15 16:08
|
阅读数：1507
|
评论：0
QQ好友和群
QQ空间
