ビビリフクロウの足跡

いつもお世話になっているインターネットへの恩返し

kubeadmを用いたKubernetes HAクラスタ on AWS

AWS Kubernetes

AWS上でkubeadmを使い、Kubernetes HAクラスタを作ってみたときのメモです。

構成

  • OS: CentOS7
  • サーバ群
    • Master + etcd
      • 3台(aws-k8s-master[01-03])
    • Node
      • 3台(aws-k8s-node[01-03])

前提

  • 作業ユーザはroot

手順

Dockerのインストール(aws-k8s-master[01-03]、aws-k8s-node[01-03])

yum install -y docker
systemctl enable docker && systemctl start docker

kubeadm、kubelet、kubectlのインストール(aws-k8s-master[01-03]、aws-k8s-node[01-03])

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

cfssl、cfssljsonのインストール(aws-k8s-master[01-03])

curl -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -o /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x /usr/local/bin/cfssl*

etcdサーバのためのCA証明書作成(aws-k8s-master01)

mkdir -p /etc/etcd/pki
cd /etc/etcd/pki
cat >ca-config.json <<EOF
{
   "signing": {
       "default": {
           "expiry": "175200h"
       },
       "profiles": {
           "server": {
               "expiry": "175200h",
               "usages": [
                   "signing",
                   "key encipherment",
                   "server auth",
                   "client auth"
               ]
           },
           "client": {
               "expiry": "175200h",
               "usages": [
                   "signing",
                   "key encipherment",
                   "client auth"
               ]
           },
           "peer": {
               "expiry": "175200h",
               "usages": [
                   "signing",
                   "key encipherment",
                   "server auth",
                   "client auth"
               ]
           }
       }
   }
}
EOF
# 証明書の有効期限は20年
cat >ca-csr.json <<EOF
{
   "CN": "etcd",
   "key": {
       "algo": "rsa",
       "size": 2048
   }
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

etcdのクライアント証明書の作成(aws-k8s-master01)

cat >client.json <<EOF
{
  "CN": "client",
  "key": {
      "algo": "ecdsa",
      "size": 256
  }
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

環境変数のエクスポート(aws-k8s-master[01-03])

export PEER_NAME=$(hostname)
export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet \K[\d.]+')

SSHキーペアの作成(aws-k8s-master02, aws-k8s-master03)

ssh-keygen -t rsa -b 4096 -C ""
cat ~/.ssh/id_rsa.pub
# 出力された内容をaws-k8s-master01の/root/.ssh/authorized_keysのエントリを追加する

CA証明書とクライアント証明書の配布(aws-k8s-master02, aws-k8s-master03)

mkdir -p /etc/etcd/pki
cd /etc/etcd/pki
scp root@<aws-k8s-master01のIP>:/etc/etcd/pki/ca.pem .
scp root@<aws-k8s-master01のIP>:/etc/etcd/pki/ca-key.pem .
scp root@<aws-k8s-master01のIP>:/etc/etcd/pki/client.pem .
scp root@<aws-k8s-master01のIP>:/etc/etcd/pki/client-key.pem .
scp root@<aws-k8s-master01のIP>:/etc/etcd/pki/ca-config.json .

etcdサーバ証明書とpeer証明書の作成(aws-k8s-master[01-03])

cfssl print-defaults csr > config.json
sed -i '0,/CN/{s/example\.net/'"$PEER_NAME"'/}' config.json
sed -i 's/www\.example\.net/'"$PRIVATE_IP"'/' config.json
sed -i 's/example\.net/'"$PEER_NAME"'/' config.json
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server config.json | cfssljson -bare server
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer config.json | cfssljson -bare peer

etcdサーバのsystemd登録と起動設定(aws-k8s-master[01-03])

export ETCD_VERSION="v3.1.12"
curl -sSL https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz | tar -xzv --strip-components=1 -C /usr/local/bin/
touch /etc/etcd.env
echo "PEER_NAME=${PEER_NAME}" >> /etc/etcd.env
echo "PRIVATE_IP=${PRIVATE_IP}" >> /etc/etcd.env
cat >/etc/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd.env
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/usr/local/bin/etcd --name <name> --data-dir /var/lib/etcd --listen-client-urls https://<my_ip>:2379 --advertise-client-urls https://<my_ip>:2379 --listen-peer-urls https://<my_ip>:2380 --initial-advertise-peer-urls https://<my_ip>:2380 --cert-file=/etc/etcd/pki/server.pem --key-file=/etc/etcd/pki/server-key.pem --client-cert-auth --trusted-ca-file=/etc/etcd/pki/ca.pem --peer-cert-file=/etc/etcd/pki/peer.pem --peer-key-file=/etc/etcd/pki/peer-key.pem --peer-client-cert-auth --peer-trusted-ca-file=/etc/etcd/pki/ca.pem --initial-cluster aws-k8s-master01=https://<aws-k8s-master01のIP>:2380,aws-k8s-master02=https://<aws-k8s-master02のIP>:2380,aws-k8s-master03=https://<aws-k8s-master03のIP>:2380 --initial-cluster-token my-etcd-token --initial-cluster-state new
# <name>にはaws-k8s-master[01-03]が入る

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

NLBの作成(AWS ManagementConsole等)

(6443/TCPのリスナを作成、aws-k8s-master01のインスタンスをターゲットグループに登録する)
(NLB作成後、DNS名にpingを発行し、NLBのIPを確認する)

kubeadmの実行(aws-k8s-master01)

cat >config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://<aws-k8s-master01のIP>:2379
  - https://<aws-k8s-master02のIP>:2379
  - https://<aws-k8s-master03のIP>:2379
  caFile: /etc/etcd/pki/ca.pem
  certFile: /etc/etcd/pki/client.pem
  keyFile: /etc/etcd/pki/client-key.pem
networking:
  podSubnet: 192.168.0.0/16
apiServerCertSANs:
- <NLBのIP>
- <my_ip>
apiServerExtraArgs:
  apiserver-count: "3"
EOF
kubeadm init --config=config.yaml
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

kubeadmの実行(aws-k8s-master02,aws-k8s-master03)

mkdir /etc/kubernetes/pki
scp root@<aws-k8s-master01のIP>:/etc/kubernetes/pki/* /etc/kubernetes/pki
rm -f /etc/kubernetes/pki/apiserver*
cat >config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://<aws-k8s-master01のIP>:2379
  - https://<aws-k8s-master02のIP>:2379
  - https://<aws-k8s-master03のIP>:2379
  caFile: /etc/etcd/pki/ca.pem
  certFile: /etc/etcd/pki/client.pem
  keyFile: /etc/etcd/pki/client-key.pem
networking:
  podSubnet: 192.168.0.0/16
apiServerCertSANs:
- <NLBのIP>
- <my_ip>
apiServerExtraArgs:
  apiserver-count: "3"
EOF
kubeadm init --config=config.yaml
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

NLBへ残りのMasterを追加(AWS ManagementConsole等)

(NLBのターゲットグループにaws-k8s-master02、aws-k8s-master03を追加)

calicoのインストール(aws-k8s-master01)

kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Nodeのクラスタ参加(aws-k8s-node[01-03])

kubeadm join --token <token> <aws-k8s-master01のIP>:6443 --discovery-token-ca-cert-hash sha256:<hash>
# <token>、<hash>はkubeadm initをaws-k8s-master01で実行した際に表示されたものを使用

kube-proxyの再設定(aws-k8s-master01)

kubectl get configmap -n kube-system kube-proxy -o yaml > kube-proxy-cm.yaml
sed -i 's#server:.*#server: https://<NLBのIP>:6443#g' kube-proxy-cm.yaml
kubectl apply -f kube-proxy-cm.yaml --force
kubectl delete pod -n kube-system -l k8s-app=kube-proxy

kubeletの再設定(aws-k8s-node[01-03])

sed -i 's#server:.*#server: https://<NLBのIP>:6443#g' /etc/kubernetes/kubelet.conf
systemctl restart kubelet