Haproxy Ingress Controllerの導入
OpenShiftではおなじみのroute機能を、Kubernetes上でもIngressを使って再現してみたいと思います。
まずはDNSワイルドカードの有効化から。今回は*.ha-kubernetes.internal
というアドレスをhaproxyポッドに割り当てます。
DNSサーバの設定
# vi /etc/dnsmasq.conf
address=/ha-kubernetes.internal/192.168.0.89
# systemctl restart dnsmasq
例のごとくマルチマスターKubernetes環境を前提とするので、フロントエンドのロードバランサ(これもhaproxy)に対し80や443ポートにアクセスが来たら、マスターの同一ポートにリクエストを流すように設定します。
フロントエンドロードバランサの設定(haproxy-*)
# vi /etc/haproxy/haproxy.cfg
... frontend router_http *:80 default_backend router_http frontend router_https *:443 default_backend router_https frontend router_stat *:1936 default_backend router_stat ... backend router_http balance leastconn server master1 kube-master-01.bbrfkr.mydns.jp:80 check server master2 kube-master-02.bbrfkr.mydns.jp:80 check backup server master3 kube-master-03.bbrfkr.mydns.jp:80 check backup backend router_https balance leastconn server master1 kube-master-01.bbrfkr.mydns.jp:443 check server master2 kube-master-02.bbrfkr.mydns.jp:443 check backup server master3 kube-master-03.bbrfkr.mydns.jp:443 check backup backend router_stat balance leastconn server master1 kube-master-01.bbrfkr.mydns.jp:1936 check server master2 kube-master-02.bbrfkr.mydns.jp:1936 check backup server master3 kube-master-03.bbrfkr.mydns.jp:1936 check backup
# systemctl restart haproxy
デフォルトでkube-apiserverはnodePortに30000-32767の間しか指定できないので、この縛りを緩くします。
nodeportレンジの変更(kube-master-*)
# sed -i 's/KUBE_API_ARGS="/KUBE_API_ARGS="--service-node-port-range 1-32767 /g' /etc/kubernetes/apiserver # systemctl restart kube-apiserver
これからが実際のHaproxy Ingress Controllerの設定です。
haproxy ingress controllerの作成
- デフォルトページ(404)作成
# vi ingress-default-deployment.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: run: ingress-default-backend name: ingress-default-backend namespace: kube-system spec: replicas: 1 selector: matchLabels: run: ingress-default-backend template: metadata: labels: run: ingress-default-backend spec: containers: - name: ingress-default-backend image: gcr.io/google_containers/defaultbackend:1.0 ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: labels: run: ingress-default-backend name: ingress-default-backend namespace: kube-system spec: ports: - name: port-1 port: 8080 protocol: TCP targetPort: 8080 selector: run: ingress-default-backend
# kubectl create -f ingress-default-deployment.yaml
- controllerのコンフィグ作成
# vi haproxy-configmap.yaml
apiVersion: v1 data: dynamic-scaling: "true" backend-server-slots-increment: "4" kind: ConfigMap metadata: name: haproxy-configmap namespace: kube-system
# kubectl create -f haproxy-configmap.yaml
- ingress controllerのデプロイ
# vi haproxy-ingress-deployment.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: haproxy-ingress namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: haproxy-ingress-clusterrole rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: haproxy-ingress-role namespace: kube-system rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<haproxy>" # This has to be adapted if you change either parameter # when launching the haproxy-ingress-controller. - "ingress-controller-leader-haproxy" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: haproxy-ingress-role-nisa-binding namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: haproxy-ingress-role subjects: - kind: ServiceAccount name: haproxy-ingress namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: haproxy-ingress-clusterrole-nisa-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: haproxy-ingress-clusterrole subjects: - kind: ServiceAccount name: haproxy-ingress namespace: kube-system --- apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: run: haproxy-ingress name: haproxy-ingress namespace: kube-system spec: replicas: 1 selector: matchLabels: run: haproxy-ingress template: metadata: labels: run: haproxy-ingress spec: containers: - name: haproxy-ingress image: quay.io/jcmoraisjr/haproxy-ingress args: - --default-backend-service=kube-system/ingress-default-backend - --default-ssl-certificate=kube-system/tls-secret - --configmap=$(POD_NAMESPACE)/haproxy-configmap - --reload-strategy=native ports: - name: http containerPort: 80 - name: https containerPort: 443 - name: stat containerPort: 1936 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace serviceAccountName: haproxy-ingress
# kubectl create -f haproxy-ingress-deployment.yaml
- ingress controllerの外部公開サービス作成
# vi haproxy-ingress-svc.yaml
apiVersion: v1 kind: Service metadata: labels: run: haproxy-ingress name: haproxy-ingress namespace: kube-system spec: type: NodePort ports: - name: port-1 port: 80 protocol: TCP targetPort: 80 nodePort: 80 - name: port-2 port: 443 protocol: TCP targetPort: 443 nodePort: 443 - name: port-3 port: 1936 protocol: TCP targetPort: 1936 nodePort: 1936 selector: run: haproxy-ingress
# kubectl create -f haproxy-ingress-svc.yaml
ここまでで、Haproxy Ingress Controllerの構成はおしまいです。実際に動作確認をしてみましょう。
動作確認
- エンドポイントサービスの作成
# vi httpd-deployment.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: run: httpd name: httpd spec: replicas: 1 selector: matchLabels: run: httpd template: metadata: labels: run: httpd spec: containers: - name: httpd image: httpd ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: labels: run: httpd name: httpd namespace: default spec: ports: - name: port-1 port: 80 protocol: TCP targetPort: 80 selector: run: httpd
# kubectl create -f httpd-deployment.yaml
- Ingressの作成
# vi httpd-ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: httpd spec: rules: - host: test.ha-kubernetes.internal http: paths: - path: / backend: serviceName: httpd servicePort: 80
# kubectl create -f httpd-ingress.yaml
- ブラウザでtest.ha-kubernetes.internalにアクセス