Implementing a proper master/worker split in k8s.py

cluster/juju/layers/kubernetes/README.md

@@ -32,12 +32,45 @@ juju add-relation kubernetes etcd
 For your convenience this charm supports some configuration options to set up
 a Kuberentes cluster that works in your environment:  
 
-**version**: Set the version of the Kubernetes containers to deploy.
-The default value is "v1.0.6".  Changing the version causes the all the
-Kubernetes containers to be restarted.
+**version**: Set the version of the Kubernetes containers to deploy. The
+version string must be in the following format "v#.#.#" where the numbers
+match with the
+[kubernetes release labels](https://github.com/kubernetes/kubernetes/releases)
+of the [kubernetes github project](https://github.com/kubernetes/kubernetes).
+Changing the version causes the all the Kubernetes containers to be restarted.
 
 **cidr**: Set the IP range for the Kubernetes cluster. eg: 10.1.0.0/16
 
+# Storage
+The kubernetes charm is built to handle multiple storage devices if the cloud
+provider works with
+[Juju storage](https://jujucharms.com/docs/devel/charms-storage).
+
+The 16.04 (xenial) release introduced [ZFS](https://en.wikipedia.org/wiki/ZFS)
+to Ubuntu. The xenial charm can use ZFS witha raidz pool. A raidz pool
+distributes parity along with the data (similar to a raid5 pool) and can suffer
+the loss of one drive while still retaining data. The raidz pool requires a
+minimum of 3 disks, but will accept more if they are provided.
+
+You can add storage to the kubernetes charm in increments of 3 or greater:
+
+```
+juju add-storage kubernetes/0 disk-pool=ebs,3,1G
+```
+
+**Note**: Due to a limitation of raidz you can not add individual disks to an
+existing pool. Should you need to expand the storage of the raidz pool, the
+additional add-storage commands must be the same number of disks as the original
+command. At this point the charm will have two raidz pools added together, both
+of which could handle the  loss of one disk each.
+
+The storage code handles the addition of devices to the charm and when it
+recieves three disks creates a raidz pool that is mounted at the /srv/kubernetes
+directory by default. If you need the storage in another location you must
+change the `mount-point` value in layer.yaml before the charms is deployed.
+
+To avoid data loss you must attach the storage before making the connection to
+the etcd cluster.
 
 ## State Events
 While this charm is meant to be a top layer, it can be used to build other

cluster/juju/layers/kubernetes/config.yaml

 options:
   version:
     type: string
-    default: "v1.1.7"
+    default: "v1.2.3"
     description: |
-        The version of Kubernetes to use in this charm.  The version is
-        inserted in the configuration files that specify the hyperkube
-        container to use when starting a Kubernetes cluster.  Changing this
-        value will restart the Kubernetes cluster.
+      The version of Kubernetes to use in this charm. The version is inserted
+      in the configuration files that specify the hyperkube container to use
+      when starting a Kubernetes cluster. Changing this value will restart the
+      Kubernetes cluster.
   cidr:
     type: string
     default: 10.1.0.0/16
     description: |
-      Network CIDR to assign to K8s service groups
+      Network CIDR to assign to Kubernetes service groups. This must not
+      overlap with any IP ranges assigned to nodes for pods.
+  dns_domain:
+    type: string
+    default: cluster.local
+    description: |
+      The domain name to use for the Kubernetes cluster by the
+      skydns service.

cluster/juju/layers/kubernetes/layer.yaml

-includes: ['layer:docker', 'layer:flannel', 'layer:tls', 'interface:etcd']
+includes: ['layer:leadership', 'layer:docker', 'layer:flannel', 'layer:storage', 'layer:tls', 'interface:etcd']
+repo: https://github.com/mbruzek/layer-k8s.git
+options:
+  storage:
+    storage-driver: zfs
+    mount-point: '/srv/kubernetes'

cluster/juju/layers/kubernetes/metadata.yaml

@@ -15,3 +15,5 @@ subordinate: false
 requires:
    etcd:
      interface: etcd
+series: 
+  - 'trusty'

cluster/juju/layers/kubernetes/templates/docker-compose.yml

-# https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker.md
+# http://kubernetes.io/docs/getting-started-guides/docker/
 
+# # Start kubelet and then start master components as pods
 # docker run \
-#     --volume=/:/rootfs:ro \
-#     --volume=/sys:/sys:ro \
-#     --volume=/dev:/dev \
-#     --volume=/var/lib/docker/:/var/lib/docker:rw \
-#     --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
-#     --volume=/var/run:/var/run:rw \
-#     --volume=/var/lib/juju/agents/unit-k8s-0/charm/files/manifests:/etc/kubernetes/manifests:rw \
-#     --volume=/srv/kubernetes:/srv/kubernetes \
 #     --net=host \
 #     --pid=host \
-#     --privileged=true \
-#     -ti \
-#     gcr.io/google_containers/hyperkube:v1.0.6 \
-#     /hyperkube kubelet --containerized --hostname-override="127.0.0.1" \
-#     --address="0.0.0.0" --api-servers=http://localhost:8080 \
-#     --config=/etc/kubernetes/manifests
+#     --privileged \
+#     --restart=on-failure \
+#     -d \
+#     -v /sys:/sys:ro \
+#     -v /var/run:/var/run:rw \
+#     -v /:/rootfs:ro \
+#     -v /var/lib/docker/:/var/lib/docker:rw \
+#     -v /var/lib/kubelet/:/var/lib/kubelet:rw \
+#     gcr.io/google_containers/hyperkube-${ARCH}:v${K8S_VERSION} \
+#     /hyperkube kubelet \
+#         --address=0.0.0.0 \
+#         --allow-privileged=true \
+#         --enable-server \
+#         --api-servers=http://localhost:8080 \
+#         --config=/etc/kubernetes/manifests-multi \
+#         --cluster-dns=10.0.0.10 \
+#         --cluster-domain=cluster.local \
+#         --containerized \
+#         --v=2
 
-kubelet:
-  image: gcr.io/google_containers/hyperkube:{{version}}
+master:
+  image: gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}
   net: host
   pid: host
   privileged: true
@@ -27,42 +33,92 @@ kubelet:
   volumes:
     - /:/rootfs:ro
     - /sys:/sys:ro
-    - /dev:/dev
     - /var/lib/docker/:/var/lib/docker:rw
     - /var/lib/kubelet/:/var/lib/kubelet:rw
     - /var/run:/var/run:rw
-    - {{manifest_directory}}:/etc/kubernetes/manifests:rw
+    - {{ manifest_directory }}:/etc/kubernetes/manifests:rw
     - /srv/kubernetes:/srv/kubernetes
   command: |
-    /hyperkube kubelet --containerized --hostname-override="{{private_address}}"
-    --address="0.0.0.0" --api-servers=http://localhost:8080
-    --config=/etc/kubernetes/manifests {% if dns_server %}
-    --cluster-dns={{dns_server}} --cluster-domain=cluster.local {% endif %}
+    /hyperkube kubelet
+    --address="0.0.0.0"
+    --allow-privileged=true
+    --api-servers=http://localhost:8080
+    --cluster-dns={{ pillar['dns_server'] }}
+    --cluster-domain={{ pillar['dns_domain'] }}
+    --config=/etc/kubernetes/manifests
+    --containerized
+    --hostname-override="{{ private_address }}"
     --tls-cert-file="/srv/kubernetes/server.crt"
     --tls-private-key-file="/srv/kubernetes/server.key"
+    --v=2
 
-# docker run --net=host -d gcr.io/google_containers/etcd:2.0.12 \
-#   /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 \
-#   --data-dir=/var/etcd/data
-etcd:
+# Start kubelet without the config option and only kubelet starts.
+# kubelet gets the tls credentials from /var/lib/kubelet/kubeconfig
+# docker run \
+#     --net=host \
+#     --pid=host \
+#     --privileged \
+#     --restart=on-failure \
+#     -d \
+#     -v /sys:/sys:ro \
+#     -v /var/run:/var/run:rw  \
+#     -v /:/rootfs:ro \
+#     -v /var/lib/docker/:/var/lib/docker:rw \
+#     -v /var/lib/kubelet/:/var/lib/kubelet:rw \
+#     gcr.io/google_containers/hyperkube-${ARCH}:v${K8S_VERSION} \
+#     /hyperkube kubelet \
+#         --allow-privileged=true \
+#         --api-servers=http://${MASTER_IP}:8080 \
+#         --address=0.0.0.0 \
+#         --enable-server \
+#         --cluster-dns=10.0.0.10 \
+#         --cluster-domain=cluster.local \
+#         --containerized \
+#         --v=2
+
+
+kubelet:
+  image: gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}
   net: host
-  image: gcr.io/google_containers/etcd:2.0.12
+  pid: host
+  privileged: true
+  restart: always
+  volumes:
+    - /:/rootfs:ro
+    - /sys:/sys:ro
+    - /var/lib/docker/:/var/lib/docker:rw
+    - /var/lib/kubelet/:/var/lib/kubelet:rw
+    - /var/run:/var/run:rw
+    - /srv/kubernetes:/srv/kubernetes
   command: |
-    /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001
-    --data-dir=/var/etcd/data
+    /hyperkube kubelet
+    --address="0.0.0.0"
+    --allow-privileged=true
+    --api-servers=https://{{ master_address }}:6443
+    --cluster-dns={{ pillar['dns_server'] }}
+    --cluster-domain={{ pillar['dns_domain'] }}
+    --containerized
+    --hostname-override="{{ private_address }}"
+    --v=2
 
 # docker run \
 #     -d \
 #     --net=host \
 #     --privileged \
-#     gcr.io/google_containers/hyperkube:v${K8S_VERSION} \
-#     /hyperkube proxy --master=http://127.0.0.1:8080 --v=2
+#     --restart=on-failure \
+#     gcr.io/google_containers/hyperkube-${ARCH}:v${K8S_VERSION} \
+#     /hyperkube proxy \
+#         --master=http://${MASTER_IP}:8080 \
+#         --v=2
 proxy:
   net: host
   privileged: true
   restart: always
-  image: gcr.io/google_containers/hyperkube:{{version}}
-  command: /hyperkube proxy --master=http://127.0.0.1:8080 --v=2
+  image: gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}
+  command: |
+    /hyperkube proxy
+    --master=http://{{ master_address }}:8080
+    --v=2
 
 # cAdvisor (Container Advisor) provides container users an understanding of
 # the resource usage and performance characteristics of their running containers.

cluster/juju/layers/kubernetes/templates/master.json

@@ -7,55 +7,81 @@
   "containers":[
     {
       "name": "controller-manager",
-      "image": "gcr.io/google_containers/hyperkube:{{version}}",
+      "image": "gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}",
       "command": [
               "/hyperkube",
               "controller-manager",
               "--master=127.0.0.1:8080",
+              "--service-account-private-key-file=/srv/kubernetes/server.key",
+              "--root-ca-file=/srv/kubernetes/ca.crt",
+              "--min-resync-period=3m",
               "--v=2"
-        ]
+      ],
+      "volumeMounts": [
+        {
+          "name": "data",
+          "mountPath": "/srv/kubernetes"
+        }
+      ]
     },
     {
       "name": "apiserver",
-      "image": "gcr.io/google_containers/hyperkube:{{version}}",
+      "image": "gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}",
       "command": [
               "/hyperkube",
               "apiserver",
-              "--address=0.0.0.0",
-              "--client_ca_file=/srv/kubernetes/ca.crt",
-              "--cluster-name=kubernetes",
-              "--etcd-servers={{connection_string}}",
-              "--service-cluster-ip-range={{cidr}}",
+              "--service-cluster-ip-range={{ cidr }}",
+              "--insecure-bind-address=0.0.0.0",
+              "--etcd-servers={{ connection_string }}",
+              "--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota",
+              "--client-ca-file=/srv/kubernetes/ca.crt",
+              "--basic-auth-file=/srv/kubernetes/basic_auth.csv",
+              "--min-request-timeout=300",
               "--tls-cert-file=/srv/kubernetes/server.crt",
               "--tls-private-key-file=/srv/kubernetes/server.key",
-              "--v=2"
-        ],
+              "--token-auth-file=/srv/kubernetes/known_tokens.csv",
+              "--allow-privileged=true",
+              "--v=4"
+      ],
       "volumeMounts": [
-          {
-              "mountPath": "/srv/kubernetes",
-              "name": "certs-kubernetes",
-              "readOnly": true
-          }
-        ]
+        {
+          "name": "data",
+          "mountPath": "/srv/kubernetes"
+        }
+      ]
     },
     {
       "name": "scheduler",
-      "image": "gcr.io/google_containers/hyperkube:{{version}}",
+      "image": "gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}",
       "command": [
               "/hyperkube",
               "scheduler",
               "--master=127.0.0.1:8080",
               "--v=2"
         ]
+    },
+    {
+      "name": "setup",
+      "image": "gcr.io/google_containers/hyperkube-{{ arch }}:{{ version }}",
+      "command": [
+              "/setup-files.sh",
+              "IP:{{ private_address }},IP:{{ public_address }},DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local"
+      ],
+      "volumeMounts": [
+        {
+          "name": "data",
+          "mountPath": "/data"
+        }
+      ]
     }
   ],
   "volumes": [
-      {
-          "hostPath": {
-              "path": "/srv/kubernetes"
-          },
-          "name": "certs-kubernetes"
-      }
-    ]
+    {
+      "hostPath": {
+          "path": "/srv/kubernetes"
+      },
+      "name": "data"
+    }
+  ]
  }
 }

cluster/juju/layers/kubernetes/templates/skydns-rc.yml

 apiVersion: v1
 kind: ReplicationController
 metadata:
-  name: kube-dns-v8
+  name: kube-dns-v11
   namespace: kube-system
   labels:
     k8s-app: kube-dns
-    version: v8
+    version: v11
     kubernetes.io/cluster-service: "true"
 spec:
- {% if dns_replicas -%} replicas: {{ dns_replicas }} {% else %} replicas: 1 {% endif %}
+  replicas: {{ pillar['dns_replicas'] }}
   selector:
     k8s-app: kube-dns
-    version: v8
+    version: v11
   template:
     metadata:
       labels:
         k8s-app: kube-dns
-        version: v8
+        version: v11
         kubernetes.io/cluster-service: "true"
     spec:
       containers:
       - name: etcd
-        image: gcr.io/google_containers/etcd:2.0.9
+        image: gcr.io/google_containers/etcd-{{ arch }}:2.2.1
         resources:
+          # TODO: Set memory limits when we've profiled the container for large
+          # clusters, then set request = limit to keep this container in
+          # guaranteed class. Currently, this container falls into the
+          # "burstable" category so the kubelet doesn't backoff from restarting it.
           limits:
             cpu: 100m
+            memory: 500Mi
+          requests:
+            cpu: 100m
             memory: 50Mi
         command:
         - /usr/local/bin/etcd
@@ -40,26 +47,60 @@ spec:
         - name: etcd-storage
           mountPath: /var/etcd/data
       - name: kube2sky
-        image: gcr.io/google_containers/kube2sky:1.11
+        image: gcr.io/google_containers/kube2sky:1.14
         resources:
+          # TODO: Set memory limits when we've profiled the container for large
+          # clusters, then set request = limit to keep this container in
+          # guaranteed class. Currently, this container falls into the
+          # "burstable" category so the kubelet doesn't backoff from restarting it.
           limits:
             cpu: 100m
+            # Kube2sky watches all pods.
+            memory: 200Mi
+          requests:
+            cpu: 100m
             memory: 50Mi
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+            scheme: HTTP
+          initialDelaySeconds: 60
+          timeoutSeconds: 5
+          successThreshold: 1
+          failureThreshold: 5
+        readinessProbe:
+          httpGet:
+            path: /readiness
+            port: 8081
+            scheme: HTTP
+          # we poll on pod startup for the Kubernetes master service and
+          # only setup the /readiness HTTP server once that's available.
+          initialDelaySeconds: 30
+          timeoutSeconds: 5
         args:
         # command = "/kube2sky"
-       {% if dns_domain -%}- -domain={{ dns_domain }} {% else %} - -domain=cluster.local {% endif %}
-        - -kube_master_url=http://{{ private_address }}:8080
+        - --domain={{ pillar['dns_domain'] }}
+        - --kube-master-url=http://{{ private_address }}:8080
       - name: skydns
-        image: gcr.io/google_containers/skydns:2015-03-11-001
+        image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
         resources:
+          # TODO: Set memory limits when we've profiled the container for large
+          # clusters, then set request = limit to keep this container in
+          # guaranteed class. Currently, this container falls into the
+          # "burstable" category so the kubelet doesn't backoff from restarting it.
           limits:
             cpu: 100m
+            memory: 200Mi
+          requests:
+            cpu: 100m
             memory: 50Mi
         args:
         # command = "/skydns"
-        - -machines=http://localhost:4001
+        - -machines=http://127.0.0.1:4001
         - -addr=0.0.0.0:53
-       {% if dns_domain -%}- -domain={{ dns_domain }}. {% else %}  - -domain=cluster.local. {% endif %}
+        - -ns-rotate=false
+        - -domain={{ pillar['dns_domain'] }}.
         ports:
         - containerPort: 53
           name: dns
@@ -67,21 +108,18 @@ spec:
         - containerPort: 53
           name: dns-tcp
           protocol: TCP
-        livenessProbe:
-          httpGet:
-            path: /healthz
-            port: 8080
-            scheme: HTTP
-          initialDelaySeconds: 30
-          timeoutSeconds: 5
       - name: healthz
         image: gcr.io/google_containers/exechealthz:1.0
         resources:
+          # keep request = limit to keep this container in guaranteed class
           limits:
             cpu: 10m
             memory: 20Mi
+          requests:
+            cpu: 10m
+            memory: 20Mi
         args:
-       {% if dns_domain -%}- -cmd=nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} localhost >/dev/null {% else %} - -cmd=nslookup kubernetes.default.svc.kubernetes.local localhost >/dev/null {% endif %}
+        - -cmd=nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1 >/dev/null
         - -port=8080
         ports:
         - containerPort: 8080

cluster/juju/layers/kubernetes/templates/skydns-svc.yml

@@ -10,7 +10,7 @@ metadata:
 spec:
   selector:
     k8s-app: kube-dns
-  clusterIP:  {{ dns_server }}
+  clusterIP:  {{ pillar['dns_server'] }}
   ports:
   - name: dns
     port: 53

hack/verify-flags/exceptions.txt

@@ -19,11 +19,11 @@ cluster/gce/gci/configure-helper.sh:  local api_servers="--master=https://${KUBE
 cluster/gce/gci/configure-helper.sh:  sed -i -e "s@{{pillar\['allow_privileged'\]}}@true@g" "${src_file}"
 cluster/gce/trusty/configure-helper.sh:  sed -i -e "s@{{pillar\['allow_privileged'\]}}@true@g" "${src_file}"
 cluster/gce/util.sh:    local node_ip=$(gcloud compute instances describe --project "${PROJECT}" --zone "${ZONE}" \
-cluster/juju/layers/kubernetes/reactive/k8s.py:        check_call(split(cmd.format(directory, cluster_name, public_address,
-cluster/juju/layers/kubernetes/reactive/k8s.py:        check_call(split(cmd.format(directory, context, cluster_name, user)))
 cluster/juju/layers/kubernetes/reactive/k8s.py:    cluster_name = 'kubernetes'
-cluster/juju/layers/kubernetes/templates/master.json:              "--client_ca_file=/srv/kubernetes/ca.crt",
-cluster/juju/layers/kubernetes/templates/skydns-rc.yml:        - -kube_master_url=http://{{ private_address }}:8080
+cluster/juju/layers/kubernetes/reactive/k8s.py:    check_call(split(cmd.format(kubeconfig, cluster_name, server, ca)))
+cluster/juju/layers/kubernetes/reactive/k8s.py:    check_call(split(cmd.format(kubeconfig, context, cluster_name, user)))
+cluster/juju/layers/kubernetes/reactive/k8s.py:    client_key = '/srv/kubernetes/client.key'
+cluster/juju/layers/kubernetes/reactive/k8s.py:    tlslib.client_key(None, client_key, user='ubuntu', group='ubuntu')
 cluster/lib/logging.sh:      local source_file=${BASH_SOURCE[$frame_no]}
 cluster/lib/logging.sh:  local source_file=${BASH_SOURCE[$stack_skip]}
 cluster/log-dump.sh:    for node_name in "${NODE_NAMES[@]}"; do