ビビリフクロウの足跡

いつもお世話になっているインターネットへの恩返し

Haproxy Ingress Controllerの導入

OpenShiftではおなじみのroute機能を、Kubernetes上でもIngressを使って再現してみたいと思います。

まずはDNSワイルドカードの有効化から。今回は*.ha-kubernetes.internalというアドレスをhaproxyポッドに割り当てます。

DNSサーバの設定

# vi /etc/dnsmasq.conf
address=/ha-kubernetes.internal/192.168.0.89
# systemctl restart dnsmasq

例のごとくマルチマスターKubernetes環境を前提とするので、フロントエンドのロードバランサ(これもhaproxy)に対し80や443ポートにアクセスが来たら、マスターの同一ポートにリクエストを流すように設定します。

フロントエンドロードバランサの設定(haproxy-*)

# vi /etc/haproxy/haproxy.cfg
...
frontend router_http *:80
    default_backend    router_http

frontend router_https *:443
    default_backend    router_https

frontend router_stat *:1936
    default_backend    router_stat
...
backend router_http
    balance   leastconn
    server    master1  kube-master-01.bbrfkr.mydns.jp:80  check
    server    master2  kube-master-02.bbrfkr.mydns.jp:80  check  backup
    server    master3  kube-master-03.bbrfkr.mydns.jp:80  check  backup

backend router_https
    balance   leastconn
    server    master1  kube-master-01.bbrfkr.mydns.jp:443  check
    server    master2  kube-master-02.bbrfkr.mydns.jp:443  check  backup
    server    master3  kube-master-03.bbrfkr.mydns.jp:443  check  backup

backend router_stat
    balance   leastconn
    server    master1  kube-master-01.bbrfkr.mydns.jp:1936  check
    server    master2  kube-master-02.bbrfkr.mydns.jp:1936  check  backup
    server    master3  kube-master-03.bbrfkr.mydns.jp:1936  check  backup
# systemctl restart haproxy

デフォルトでkube-apiserverはnodePortに30000-32767の間しか指定できないので、この縛りを緩くします。

nodeportレンジの変更(kube-master-*)

# sed -i 's/KUBE_API_ARGS="/KUBE_API_ARGS="--service-node-port-range 1-32767 /g' /etc/kubernetes/apiserver
# systemctl restart kube-apiserver

これからが実際のHaproxy Ingress Controllerの設定です。

haproxy ingress controllerの作成

  • デフォルトページ(404)作成
# vi ingress-default-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: ingress-default-backend
  name: ingress-default-backend
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      run: ingress-default-backend
  template:
    metadata:
      labels:
        run: ingress-default-backend
    spec:
      containers:
        - name: ingress-default-backend
          image: gcr.io/google_containers/defaultbackend:1.0
          ports:
          - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: ingress-default-backend
  name: ingress-default-backend
  namespace: kube-system
spec:
  ports:
  - name: port-1
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: ingress-default-backend
# kubectl create -f ingress-default-deployment.yaml
  • controllerのコンフィグ作成
# vi haproxy-configmap.yaml
apiVersion: v1
data:
    dynamic-scaling: "true"
    backend-server-slots-increment: "4"
kind: ConfigMap
metadata:
  name: haproxy-configmap
  namespace: kube-system
# kubectl create -f haproxy-configmap.yaml
  • ingress controllerのデプロイ
# vi haproxy-ingress-deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: haproxy-ingress
  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: haproxy-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: haproxy-ingress-role
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<haproxy>"
      # This has to be adapted if you change either parameter
      # when launching the haproxy-ingress-controller.
      - "ingress-controller-leader-haproxy"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: haproxy-ingress-role-nisa-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: haproxy-ingress-role
subjects:
  - kind: ServiceAccount
    name: haproxy-ingress
    namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: haproxy-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: haproxy-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: haproxy-ingress
    namespace: kube-system

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      containers:
      - name: haproxy-ingress
        image: quay.io/jcmoraisjr/haproxy-ingress
        args:
        - --default-backend-service=kube-system/ingress-default-backend
        - --default-ssl-certificate=kube-system/tls-secret
        - --configmap=$(POD_NAMESPACE)/haproxy-configmap
        - --reload-strategy=native
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: stat
          containerPort: 1936
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      serviceAccountName: haproxy-ingress 
# kubectl create -f haproxy-ingress-deployment.yaml
  • ingress controllerの外部公開サービス作成
# vi haproxy-ingress-svc.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: kube-system
spec:
  type:
    NodePort
  ports:
  - name: port-1
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 80
  - name: port-2
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 443
  - name: port-3
    port: 1936
    protocol: TCP
    targetPort: 1936
    nodePort: 1936
  selector:
    run: haproxy-ingress
# kubectl create -f haproxy-ingress-svc.yaml

ここまでで、Haproxy Ingress Controllerの構成はおしまいです。実際に動作確認をしてみましょう。

動作確認

  • エンドポイントサービスの作成
# vi httpd-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: httpd
  name: httpd
spec:
  replicas: 1
  selector:
    matchLabels:
      run: httpd
  template:
    metadata:
      labels:
        run: httpd
    spec:
      containers:
        - name: httpd
          image: httpd
          ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: httpd
  name: httpd
  namespace: default
spec:
  ports:
    - name: port-1
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    run: httpd
# kubectl create -f httpd-deployment.yaml
# vi httpd-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: httpd
spec:
  rules:
  - host: test.ha-kubernetes.internal
    http:
      paths:
      - path: /
        backend:
          serviceName: httpd
          servicePort: 80
# kubectl create -f httpd-ingress.yaml
  • ブラウザでtest.ha-kubernetes.internalにアクセス