Skip to content

K
k3s
Commit 6bdc0bfd authored by mikebrow's avatar mikebrow
Browse files
Options

address issue #1488; clean up linewrap and some minor editing issues in the docs/design/* tree

Signed-off-by: 's avatarmikebrow <brownwm@us.ibm.com>
parent 4638f2f3
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 790 additions and 472 deletions
docs/design/README.md
View file @ 6bdc0bfd
...@@ -34,19 +34,59 @@ Documentation for other releases can be found at ...@@ -34,19 +34,59 @@ Documentation for other releases can be found at
# Kubernetes Design Overview # Kubernetes Design Overview
Kubernetes is a system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. Kubernetes is a system for managing containerized applications across multiple
hosts, providing basic mechanisms for deployment, maintenance, and scaling of
Kubernetes establishes robust declarative primitives for maintaining the desired state requested by the user. We see these primitives as the main value added by Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers require active controllers, not just imperative orchestration. applications.
Kubernetes is primarily targeted at applications composed of multiple containers, such as elastic, distributed micro-services. It is also designed to facilitate migration of non-containerized application stacks to Kubernetes. It therefore includes abstractions for grouping containers in both loosely coupled and tightly coupled formations, and provides ways for containers to find and communicate with each other in relatively familiar ways. Kubernetes establishes robust declarative primitives for maintaining the desired
state requested by the user. We see these primitives as the main value added by
Kubernetes enables users to ask a cluster to run a set of containers. The system automatically chooses hosts to run those containers on. While Kubernetes's scheduler is currently very simple, we expect it to grow in sophistication over time. Scheduling is a policy-rich, topology-aware, workload-specific function that significantly impacts availability, performance, and capacity. The scheduler needs to take into account individual and collective resource requirements, quality of service requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, deadlines, and so on. Workload-specific requirements will be exposed through the API as necessary. Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and
replicating containers require active controllers, not just imperative
Kubernetes is intended to run on a number of cloud providers, as well as on physical hosts. orchestration.
A single Kubernetes cluster is not intended to span multiple availability zones. Instead, we recommend building a higher-level layer to replicate complete deployments of highly available applications across multiple zones (see [the multi-cluster doc](../admin/multi-cluster.md) and [cluster federation proposal](../proposals/federation.md) for more details). Kubernetes is primarily targeted at applications composed of multiple
containers, such as elastic, distributed micro-services. It is also designed to
Finally, Kubernetes aspires to be an extensible, pluggable, building-block OSS platform and toolkit. Therefore, architecturally, we want Kubernetes to be built as a collection of pluggable components and layers, with the ability to use alternative schedulers, controllers, storage systems, and distribution mechanisms, and we're evolving its current code in that direction. Furthermore, we want others to be able to extend Kubernetes functionality, such as with higher-level PaaS functionality or multi-cluster layers, without modification of core Kubernetes source. Therefore, its API isn't just (or even necessarily mainly) targeted at end users, but at tool and extension developers. Its APIs are intended to serve as the foundation for an open ecosystem of tools, automation systems, and higher-level API layers. Consequently, there are no "internal" inter-component APIs. All APIs are visible and available, including the APIs used by the scheduler, the node controller, the replication-controller manager, Kubelet's API, etc. There's no glass to break -- in order to handle more complex use cases, one can just access the lower-level APIs in a fully transparent, composable manner. facilitate migration of non-containerized application stacks to Kubernetes. It
therefore includes abstractions for grouping containers in both loosely coupled
and tightly coupled formations, and provides ways for containers to find and
communicate with each other in relatively familiar ways.
Kubernetes enables users to ask a cluster to run a set of containers. The system
automatically chooses hosts to run those containers on. While Kubernetes's
scheduler is currently very simple, we expect it to grow in sophistication over
time. Scheduling is a policy-rich, topology-aware, workload-specific function
that significantly impacts availability, performance, and capacity. The
scheduler needs to take into account individual and collective resource
requirements, quality of service requirements, hardware/software/policy
constraints, affinity and anti-affinity specifications, data locality,
inter-workload interference, deadlines, and so on. Workload-specific
requirements will be exposed through the API as necessary.
Kubernetes is intended to run on a number of cloud providers, as well as on
physical hosts.
A single Kubernetes cluster is not intended to span multiple availability zones.
Instead, we recommend building a higher-level layer to replicate complete
deployments of highly available applications across multiple zones (see
[the multi-cluster doc](../admin/multi-cluster.md) and [cluster federation proposal](../proposals/federation.md)
for more details).
Finally, Kubernetes aspires to be an extensible, pluggable, building-block OSS
platform and toolkit. Therefore, architecturally, we want Kubernetes to be built
as a collection of pluggable components and layers, with the ability to use
alternative schedulers, controllers, storage systems, and distribution
mechanisms, and we're evolving its current code in that direction. Furthermore,
we want others to be able to extend Kubernetes functionality, such as with
higher-level PaaS functionality or multi-cluster layers, without modification of
core Kubernetes source. Therefore, its API isn't just (or even necessarily
mainly) targeted at end users, but at tool and extension developers. Its APIs
are intended to serve as the foundation for an open ecosystem of tools,
automation systems, and higher-level API layers. Consequently, there are no
"internal" inter-component APIs. All APIs are visible and available, including
the APIs used by the scheduler, the node controller, the replication-controller
manager, Kubelet's API, etc. There's no glass to break -- in order to handle
more complex use cases, one can just access the lower-level APIs in a fully
transparent, composable manner.
For more about the Kubernetes architecture, see [architecture](architecture.md). For more about the Kubernetes architecture, see [architecture](architecture.md).
......
docs/design/admission_control.md
View file @ 6bdc0bfd
...@@ -43,24 +43,30 @@ Documentation for other releases can be found at ...@@ -43,24 +43,30 @@ Documentation for other releases can be found at
## Background ## Background
High level goals: High level goals:
* Enable an easy-to-use mechanism to provide admission control to cluster.
* Enable a provider to support multiple admission control strategies or author
their own.
* Ensure any rejected request can propagate errors back to the caller with why
the request failed.
* Enable an easy-to-use mechanism to provide admission control to cluster Authorization via policy is focused on answering if a user is authorized to
* Enable a provider to support multiple admission control strategies or author their own perform an action.
* Ensure any rejected request can propagate errors back to the caller with why the request failed
Authorization via policy is focused on answering if a user is authorized to perform an action.
Admission Control is focused on if the system will accept an authorized action. Admission Control is focused on if the system will accept an authorized action.
Kubernetes may choose to dismiss an authorized action based on any number of admission control strategies. Kubernetes may choose to dismiss an authorized action based on any number of
admission control strategies.
This proposal documents the basic design, and describes how any number of admission control plug-ins could be injected. This proposal documents the basic design, and describes how any number of
admission control plug-ins could be injected.
Implementation of specific admission control strategies are handled in separate documents. Implementation of specific admission control strategies are handled in separate
documents.
## kube-apiserver ## kube-apiserver
The kube-apiserver takes the following OPTIONAL arguments to enable admission control The kube-apiserver takes the following OPTIONAL arguments to enable admission
control:
| Option | Behavior | | Option | Behavior |
| ------ | -------- | | ------ | -------- |
...@@ -72,7 +78,8 @@ An **AdmissionControl** plug-in is an implementation of the following interface: ...@@ -72,7 +78,8 @@ An **AdmissionControl** plug-in is an implementation of the following interface:
```go ```go
package admission package admission
// Attributes is an interface used by a plug-in to make an admission decision on a individual request. // Attributes is an interface used by a plug-in to make an admission decision
// on a individual request.
type Attributes interface { type Attributes interface {
GetNamespace() string GetNamespace() string
GetKind() string GetKind() string
...@@ -88,8 +95,8 @@ type Interface interface { ...@@ -88,8 +95,8 @@ type Interface interface {
} }
``` ```
A **plug-in** must be compiled with the binary, and is registered as an available option by providing a name, and implementation A **plug-in** must be compiled with the binary, and is registered as an
of admission.Interface. available option by providing a name, and implementation of admission.Interface.
```go ```go
func init() { func init() {
...@@ -97,9 +104,12 @@ func init() { ...@@ -97,9 +104,12 @@ func init() {
} }
``` ```
Invocation of admission control is handled by the **APIServer** and not individual **RESTStorage** implementations. Invocation of admission control is handled by the **APIServer** and not
individual **RESTStorage** implementations.
This design assumes that **Issue 297** is adopted, and as a consequence, the general framework of the APIServer request/response flow will ensure the following: This design assumes that **Issue 297** is adopted, and as a consequence, the
general framework of the APIServer request/response flow will ensure the
following:
1. Incoming request 1. Incoming request
2. Authenticate user 2. Authenticate user
......
docs/design/admission_control_limit_range.md
View file @ 6bdc0bfd
...@@ -36,7 +36,8 @@ Documentation for other releases can be found at ...@@ -36,7 +36,8 @@ Documentation for other releases can be found at
## Background ## Background
This document proposes a system for enforcing resource requirements constraints as part of admission control. This document proposes a system for enforcing resource requirements constraints
as part of admission control.
## Use cases ## Use cases
...@@ -64,7 +65,8 @@ const ( ...@@ -64,7 +65,8 @@ const (
LimitTypeContainer LimitType = "Container" LimitTypeContainer LimitType = "Container"
) )
// LimitRangeItem defines a min/max usage limit for any resource that matches on kind. // LimitRangeItem defines a min/max usage limit for any resource that matches
// on kind.
type LimitRangeItem struct { type LimitRangeItem struct {
// Type of resource that this limit applies to. // Type of resource that this limit applies to.
Type LimitType `json:"type,omitempty"` Type LimitType `json:"type,omitempty"`
...@@ -72,29 +74,38 @@ type LimitRangeItem struct { ...@@ -72,29 +74,38 @@ type LimitRangeItem struct {
Max ResourceList `json:"max,omitempty"` Max ResourceList `json:"max,omitempty"`
// Min usage constraints on this kind by resource name. // Min usage constraints on this kind by resource name.
Min ResourceList `json:"min,omitempty"` Min ResourceList `json:"min,omitempty"`
// Default resource requirement limit value by resource name if resource limit is omitted. // Default resource requirement limit value by resource name if resource limit
// is omitted.
Default ResourceList `json:"default,omitempty"` Default ResourceList `json:"default,omitempty"`
// DefaultRequest is the default resource requirement request value by resource name if resource request is omitted. // DefaultRequest is the default resource requirement request value by
// resource name if resource request is omitted.
DefaultRequest ResourceList `json:"defaultRequest,omitempty"` DefaultRequest ResourceList `json:"defaultRequest,omitempty"`
// MaxLimitRequestRatio if specified, the named resource must have a request and limit that are both non-zero where limit divided by request is less than or equal to the enumerated value; this represents the max burst for the named resource. // MaxLimitRequestRatio if specified, the named resource must have a request
// and limit that are both non-zero where limit divided by request is less
// than or equal to the enumerated value; this represents the max burst for
// the named resource.
MaxLimitRequestRatio ResourceList `json:"maxLimitRequestRatio,omitempty"` MaxLimitRequestRatio ResourceList `json:"maxLimitRequestRatio,omitempty"`
} }
// LimitRangeSpec defines a min/max usage limit for resources that match on kind. // LimitRangeSpec defines a min/max usage limit for resources that match
// on kind.
type LimitRangeSpec struct { type LimitRangeSpec struct {
// Limits is the list of LimitRangeItem objects that are enforced. // Limits is the list of LimitRangeItem objects that are enforced.
Limits []LimitRangeItem `json:"limits"` Limits []LimitRangeItem `json:"limits"`
} }
// LimitRange sets resource usage limits for each kind of resource in a Namespace. // LimitRange sets resource usage limits for each kind of resource in a
// Namespace.
type LimitRange struct { type LimitRange struct {
TypeMeta `json:",inline"` TypeMeta `json:",inline"`
// Standard object's metadata. // Standard object's metadata.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata // More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
ObjectMeta `json:"metadata,omitempty"` ObjectMeta `json:"metadata,omitempty"`
// Spec defines the limits enforced. // Spec defines the limits enforced.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status // More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status
Spec LimitRangeSpec `json:"spec,omitempty"` Spec LimitRangeSpec `json:"spec,omitempty"`
} }
...@@ -102,24 +113,29 @@ type LimitRange struct { ...@@ -102,24 +113,29 @@ type LimitRange struct {
type LimitRangeList struct { type LimitRangeList struct {
TypeMeta `json:",inline"` TypeMeta `json:",inline"`
// Standard list metadata. // Standard list metadata.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds // More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds
ListMeta `json:"metadata,omitempty"` ListMeta `json:"metadata,omitempty"`
// Items is a list of LimitRange objects. // Items is a list of LimitRange objects.
// More info: http://releases.k8s.io/HEAD/docs/design/admission_control_limit_range.md // More info:
// http://releases.k8s.io/HEAD/docs/design/admission_control_limit_range.md
Items []LimitRange `json:"items"` Items []LimitRange `json:"items"`
} }
``` ```
### Validation ### Validation
Validation of a **LimitRange** enforces that for a given named resource the following rules apply: Validation of a **LimitRange** enforces that for a given named resource the
following rules apply:
Min (if specified) <= DefaultRequest (if specified) <= Default (if specified) <= Max (if specified) Min (if specified) <= DefaultRequest (if specified) <= Default (if specified)
<= Max (if specified)
### Default Value Behavior ### Default Value Behavior
The following default value behaviors are applied to a LimitRange for a given named resource. The following default value behaviors are applied to a LimitRange for a given
named resource.
``` ```
if LimitRangeItem.Default[resourceName] is undefined if LimitRangeItem.Default[resourceName] is undefined
...@@ -137,11 +153,14 @@ if LimitRangeItem.DefaultRequest[resourceName] is undefined ...@@ -137,11 +153,14 @@ if LimitRangeItem.DefaultRequest[resourceName] is undefined
## AdmissionControl plugin: LimitRanger ## AdmissionControl plugin: LimitRanger
The **LimitRanger** plug-in introspects all incoming pod requests and evaluates the constraints defined on a LimitRange. The **LimitRanger** plug-in introspects all incoming pod requests and evaluates
the constraints defined on a LimitRange.
If a constraint is not specified for an enumerated resource, it is not enforced or tracked. If a constraint is not specified for an enumerated resource, it is not enforced
or tracked.
To enable the plug-in and support for LimitRange, the kube-apiserver must be configured as follows: To enable the plug-in and support for LimitRange, the kube-apiserver must be
configured as follows:
```console ```console
$ kube-apiserver --admission-control=LimitRanger $ kube-apiserver --admission-control=LimitRanger
...@@ -158,7 +177,7 @@ Supported Resources: ...@@ -158,7 +177,7 @@ Supported Resources:
Supported Constraints: Supported Constraints:
Per container, the following must hold true Per container, the following must hold true:
| Constraint | Behavior | | Constraint | Behavior |
| ---------- | -------- | | ---------- | -------- |
...@@ -168,8 +187,10 @@ Per container, the following must hold true ...@@ -168,8 +187,10 @@ Per container, the following must hold true
Supported Defaults: Supported Defaults:
1. Default - if the named resource has no enumerated value, the Limit is equal to the Default 1. Default - if the named resource has no enumerated value, the Limit is equal
2. DefaultRequest - if the named resource has no enumerated value, the Request is equal to the DefaultRequest to the Default
2. DefaultRequest - if the named resource has no enumerated value, the Request
is equal to the DefaultRequest
**Type: Pod** **Type: Pod**
...@@ -190,7 +211,8 @@ Across all containers in pod, the following must hold true ...@@ -190,7 +211,8 @@ Across all containers in pod, the following must hold true
## Run-time configuration ## Run-time configuration
The default ```LimitRange``` that is applied via Salt configuration will be updated as follows: The default ```LimitRange``` that is applied via Salt configuration will be
updated as follows:
``` ```
apiVersion: "v1" apiVersion: "v1"
...@@ -219,7 +241,8 @@ the following would happen. ...@@ -219,7 +241,8 @@ the following would happen.
1. The incoming container cpu would request 250m with a limit of 500m. 1. The incoming container cpu would request 250m with a limit of 500m.
2. The incoming container memory would request 250Mi with a limit of 500Mi 2. The incoming container memory would request 250Mi with a limit of 500Mi
3. If the container is later resized, it's cpu would be constrained to between .1 and 1 and the ratio of limit to request could not exceed 4. 3. If the container is later resized, it's cpu would be constrained to between
.1 and 1 and the ratio of limit to request could not exceed 4.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/admission_control_limit_range.md?pixel)]() [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/admission_control_limit_range.md?pixel)]()
......
docs/design/admission_control_resource_quota.md
View file @ 6bdc0bfd
...@@ -36,7 +36,8 @@ Documentation for other releases can be found at ...@@ -36,7 +36,8 @@ Documentation for other releases can be found at
## Background ## Background
This document describes a system for enforcing hard resource usage limits per namespace as part of admission control. This document describes a system for enforcing hard resource usage limits per
namespace as part of admission control.
## Use cases ## Use cases
...@@ -103,7 +104,7 @@ type ResourceQuotaList struct { ...@@ -103,7 +104,7 @@ type ResourceQuotaList struct {
## Quota Tracked Resources ## Quota Tracked Resources
The following resources are supported by the quota system. The following resources are supported by the quota system:
| Resource | Description | | Resource | Description |
| ------------ | ----------- | | ------------ | ----------- |
...@@ -116,16 +117,19 @@ The following resources are supported by the quota system. ...@@ -116,16 +117,19 @@ The following resources are supported by the quota system.
| secrets | Total number of secrets | | secrets | Total number of secrets |
| persistentvolumeclaims | Total number of persistent volume claims | | persistentvolumeclaims | Total number of persistent volume claims |
If a third-party wants to track additional resources, it must follow the resource naming conventions prescribed If a third-party wants to track additional resources, it must follow the
by Kubernetes. This means the resource must have a fully-qualified name (i.e. mycompany.org/shinynewresource) resource naming conventions prescribed by Kubernetes. This means the resource
must have a fully-qualified name (i.e. mycompany.org/shinynewresource)
## Resource Requirements: Requests vs Limits ## Resource Requirements: Requests vs Limits
If a resource supports the ability to distinguish between a request and a limit for a resource, If a resource supports the ability to distinguish between a request and a limit
the quota tracking system will only cost the request value against the quota usage. If a resource for a resource, the quota tracking system will only cost the request value
is tracked by quota, and no request value is provided, the associated entity is rejected as part of admission. against the quota usage. If a resource is tracked by quota, and no request value
is provided, the associated entity is rejected as part of admission.
For an example, consider the following scenarios relative to tracking quota on CPU: For an example, consider the following scenarios relative to tracking quota on
CPU:
| Pod | Container | Request CPU | Limit CPU | Result | | Pod | Container | Request CPU | Limit CPU | Result |
| --- | --------- | ----------- | --------- | ------ | | --- | --------- | ----------- | --------- | ------ |
...@@ -134,13 +138,14 @@ For an example, consider the following scenarios relative to tracking quota on C ...@@ -134,13 +138,14 @@ For an example, consider the following scenarios relative to tracking quota on C
| Y | C2 | none | 500m | The quota usage is incremented 500m since request will default to limit | | Y | C2 | none | 500m | The quota usage is incremented 500m since request will default to limit |
| Z | C3 | none | none | The pod is rejected since it does not enumerate a request. | | Z | C3 | none | none | The pod is rejected since it does not enumerate a request. |
The rationale for accounting for the requested amount of a resource versus the limit is the belief The rationale for accounting for the requested amount of a resource versus the
that a user should only be charged for what they are scheduled against in the cluster. In addition, limit is the belief that a user should only be charged for what they are
attempting to track usage against actual usage, where request < actual < limit, is considered highly scheduled against in the cluster. In addition, attempting to track usage against
volatile. actual usage, where request < actual < limit, is considered highly volatile.
As a consequence of this decision, the user is able to spread its usage of a resource across multiple tiers As a consequence of this decision, the user is able to spread its usage of a
of service. Let's demonstrate this via an example with a 4 cpu quota. resource across multiple tiers of service. Let's demonstrate this via an
example with a 4 cpu quota.
The quota may be allocated as follows: The quota may be allocated as follows:
...@@ -150,48 +155,62 @@ The quota may be allocated as follows: ...@@ -150,48 +155,62 @@ The quota may be allocated as follows:
| Y | C2 | 2 | 2 | Guaranteed | 2 | | Y | C2 | 2 | 2 | Guaranteed | 2 |
| Z | C3 | 1 | 3 | Burstable | 1 | | Z | C3 | 1 | 3 | Burstable | 1 |
It is possible that the pods may consume 9 cpu over a given time period depending on the nodes available cpu It is possible that the pods may consume 9 cpu over a given time period
that held pod X and Z, but since we scheduled X and Z relative to the request, we only track the requesting depending on the nodes available cpu that held pod X and Z, but since we
value against their allocated quota. If one wants to restrict the ratio between the request and limit, scheduled X and Z relative to the request, we only track the requesting value
it is encouraged that the user define a **LimitRange** with **LimitRequestRatio** to control burst out behavior. against their allocated quota. If one wants to restrict the ratio between the
This would in effect, let an administrator keep the difference between request and limit more in line with request and limit, it is encouraged that the user define a **LimitRange** with
**LimitRequestRatio** to control burst out behavior. This would in effect, let
an administrator keep the difference between request and limit more in line with
tracked usage if desired. tracked usage if desired.
## Status API ## Status API
A REST API endpoint to update the status section of the **ResourceQuota** is exposed. It requires an atomic compare-and-swap A REST API endpoint to update the status section of the **ResourceQuota** is
in order to keep resource usage tracking consistent. exposed. It requires an atomic compare-and-swap in order to keep resource usage
tracking consistent.
## Resource Quota Controller ## Resource Quota Controller
A resource quota controller monitors observed usage for tracked resources in the **Namespace**. A resource quota controller monitors observed usage for tracked resources in the
**Namespace**.
If there is observed difference between the current usage stats versus the current **ResourceQuota.Status**, the controller If there is observed difference between the current usage stats versus the
posts an update of the currently observed usage metrics to the **ResourceQuota** via the /status endpoint. current **ResourceQuota.Status**, the controller posts an update of the
currently observed usage metrics to the **ResourceQuota** via the /status
endpoint.
The resource quota controller is the only component capable of monitoring and recording usage updates after a DELETE operation The resource quota controller is the only component capable of monitoring and
since admission control is incapable of guaranteeing a DELETE request actually succeeded. recording usage updates after a DELETE operation since admission control is
incapable of guaranteeing a DELETE request actually succeeded.
## AdmissionControl plugin: ResourceQuota ## AdmissionControl plugin: ResourceQuota
The **ResourceQuota** plug-in introspects all incoming admission requests. The **ResourceQuota** plug-in introspects all incoming admission requests.
To enable the plug-in and support for ResourceQuota, the kube-apiserver must be configured as follows: To enable the plug-in and support for ResourceQuota, the kube-apiserver must be
configured as follows:
``` ```
$ kube-apiserver --admission-control=ResourceQuota $ kube-apiserver --admission-control=ResourceQuota
``` ```
It makes decisions by evaluating the incoming object against all defined **ResourceQuota.Status.Hard** resource limits in the request It makes decisions by evaluating the incoming object against all defined
namespace. If acceptance of the resource would cause the total usage of a named resource to exceed its hard limit, the request is denied. **ResourceQuota.Status.Hard** resource limits in the request namespace. If
acceptance of the resource would cause the total usage of a named resource to
exceed its hard limit, the request is denied.
If the incoming request does not cause the total usage to exceed any of the enumerated hard resource limits, the plug-in will post a If the incoming request does not cause the total usage to exceed any of the
**ResourceQuota.Status** document to the server to atomically update the observed usage based on the previously read enumerated hard resource limits, the plug-in will post a
**ResourceQuota.ResourceVersion**. This keeps incremental usage atomically consistent, but does introduce a bottleneck (intentionally) **ResourceQuota.Status** document to the server to atomically update the
into the system. observed usage based on the previously read **ResourceQuota.ResourceVersion**.
This keeps incremental usage atomically consistent, but does introduce a
bottleneck (intentionally) into the system.
To optimize system performance, it is encouraged that all resource quotas are tracked on the same **ResourceQuota** document in a **Namespace**. As a result, its encouraged to impose a cap on the total number of individual quotas that are tracked in the **Namespace** To optimize system performance, it is encouraged that all resource quotas are
to 1 in the **ResourceQuota** document. tracked on the same **ResourceQuota** document in a **Namespace**. As a result,
it is encouraged to impose a cap on the total number of individual quotas that
are tracked in the **Namespace** to 1 in the **ResourceQuota** document.
## kubectl ## kubectl
...@@ -199,7 +218,7 @@ kubectl is modified to support the **ResourceQuota** resource. ...@@ -199,7 +218,7 @@ kubectl is modified to support the **ResourceQuota** resource.
`kubectl describe` provides a human-readable output of quota. `kubectl describe` provides a human-readable output of quota.
For example, For example:
```console ```console
$ kubectl create -f docs/admin/resourcequota/namespace.yaml $ kubectl create -f docs/admin/resourcequota/namespace.yaml
......
docs/design/architecture.md
View file @ 6bdc0bfd
...@@ -34,49 +34,84 @@ Documentation for other releases can be found at ...@@ -34,49 +34,84 @@ Documentation for other releases can be found at
# Kubernetes architecture # Kubernetes architecture
A running Kubernetes cluster contains node agents (`kubelet`) and master components (APIs, scheduler, etc), on top of a distributed storage solution. This diagram shows our desired eventual state, though we're still working on a few things, like making `kubelet` itself (all our components, really) run within containers, and making the scheduler 100% pluggable. A running Kubernetes cluster contains node agents (`kubelet`) and master
components (APIs, scheduler, etc), on top of a distributed storage solution.
This diagram shows our desired eventual state, though we're still working on a
few things, like making `kubelet` itself (all our components, really) run within
containers, and making the scheduler 100% pluggable.
![Architecture Diagram](architecture.png?raw=true "Architecture overview") ![Architecture Diagram](architecture.png?raw=true "Architecture overview")
## The Kubernetes Node ## The Kubernetes Node
When looking at the architecture of the system, we'll break it down to services that run on the worker node and services that compose the cluster-level control plane. When looking at the architecture of the system, we'll break it down to services
that run on the worker node and services that compose the cluster-level control
plane.
The Kubernetes node has the services necessary to run application containers and be managed from the master systems. The Kubernetes node has the services necessary to run application containers and
be managed from the master systems.
Each node runs Docker, of course. Docker takes care of the details of downloading images and running containers. Each node runs Docker, of course. Docker takes care of the details of
downloading images and running containers.
### `kubelet` ### `kubelet`
The `kubelet` manages [pods](../user-guide/pods.md) and their containers, their images, their volumes, etc. The `kubelet` manages [pods](../user-guide/pods.md) and their containers, their
images, their volumes, etc.
### `kube-proxy` ### `kube-proxy`
Each node also runs a simple network proxy and load balancer (see the [services FAQ](https://github.com/kubernetes/kubernetes/wiki/Services-FAQ) for more details). This reflects `services` (see [the services doc](../user-guide/services.md) for more details) as defined in the Kubernetes API on each node and can do simple TCP and UDP stream forwarding (round robin) across a set of backends. Each node also runs a simple network proxy and load balancer (see the
[services FAQ](https://github.com/kubernetes/kubernetes/wiki/Services-FAQ) for
more details). This reflects `services` (see
[the services doc](../user-guide/services.md) for more details) as defined in
the Kubernetes API on each node and can do simple TCP and UDP stream forwarding
(round robin) across a set of backends.
Service endpoints are currently found via [DNS](../admin/dns.md) or through environment variables (both [Docker-links-compatible](https://docs.docker.com/userguide/dockerlinks/) and Kubernetes `{FOO}_SERVICE_HOST` and `{FOO}_SERVICE_PORT` variables are supported). These variables resolve to ports managed by the service proxy. Service endpoints are currently found via [DNS](../admin/dns.md) or through
environment variables (both
[Docker-links-compatible](https://docs.docker.com/userguide/dockerlinks/) and
Kubernetes `{FOO}_SERVICE_HOST` and `{FOO}_SERVICE_PORT` variables are
supported). These variables resolve to ports managed by the service proxy.
## The Kubernetes Control Plane ## The Kubernetes Control Plane
The Kubernetes control plane is split into a set of components. Currently they all run on a single _master_ node, but that is expected to change soon in order to support high-availability clusters. These components work together to provide a unified view of the cluster. The Kubernetes control plane is split into a set of components. Currently they
all run on a single _master_ node, but that is expected to change soon in order
to support high-availability clusters. These components work together to provide
a unified view of the cluster.
### `etcd` ### `etcd`
All persistent master state is stored in an instance of `etcd`. This provides a great way to store configuration data reliably. With `watch` support, coordinating components can be notified very quickly of changes. All persistent master state is stored in an instance of `etcd`. This provides a
great way to store configuration data reliably. With `watch` support,
coordinating components can be notified very quickly of changes.
### Kubernetes API Server ### Kubernetes API Server
The apiserver serves up the [Kubernetes API](../api.md). It is intended to be a CRUD-y server, with most/all business logic implemented in separate components or in plug-ins. It mainly processes REST operations, validates them, and updates the corresponding objects in `etcd` (and eventually other stores). The apiserver serves up the [Kubernetes API](../api.md). It is intended to be a
CRUD-y server, with most/all business logic implemented in separate components
or in plug-ins. It mainly processes REST operations, validates them, and updates
the corresponding objects in `etcd` (and eventually other stores).
### Scheduler ### Scheduler
The scheduler binds unscheduled pods to nodes via the `/binding` API. The scheduler is pluggable, and we expect to support multiple cluster schedulers and even user-provided schedulers in the future. The scheduler binds unscheduled pods to nodes via the `/binding` API. The
scheduler is pluggable, and we expect to support multiple cluster schedulers and
even user-provided schedulers in the future.
### Kubernetes Controller Manager Server ### Kubernetes Controller Manager Server
All other cluster-level functions are currently performed by the Controller Manager. For instance, `Endpoints` objects are created and updated by the endpoints controller, and nodes are discovered, managed, and monitored by the node controller. These could eventually be split into separate components to make them independently pluggable. All other cluster-level functions are currently performed by the Controller
Manager. For instance, `Endpoints` objects are created and updated by the
endpoints controller, and nodes are discovered, managed, and monitored by the
node controller. These could eventually be split into separate components to
make them independently pluggable.
The [`replicationcontroller`](../user-guide/replication-controller.md) is a mechanism that is layered on top of the simple [`pod`](../user-guide/pods.md) API. We eventually plan to port it to a generic plug-in mechanism, once one is implemented. The [`replicationcontroller`](../user-guide/replication-controller.md) is a
mechanism that is layered on top of the simple [`pod`](../user-guide/pods.md)
API. We eventually plan to port it to a generic plug-in mechanism, once one is
implemented.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/clustering/README.md
View file @ 6bdc0bfd
...@@ -33,7 +33,8 @@ Documentation for other releases can be found at ...@@ -33,7 +33,8 @@ Documentation for other releases can be found at
<!-- END MUNGE: UNVERSIONED_WARNING --> <!-- END MUNGE: UNVERSIONED_WARNING -->
This directory contains diagrams for the clustering design doc. This directory contains diagrams for the clustering design doc.
This depends on the `seqdiag` [utility](http://blockdiag.com/en/seqdiag/index.html). Assuming you have a non-borked python install, this should be installable with This depends on the `seqdiag` [utility](http://blockdiag.com/en/seqdiag/index.html).
Assuming you have a non-borked python install, this should be installable with:
```sh ```sh
pip install seqdiag pip install seqdiag
...@@ -43,7 +44,8 @@ Just call `make` to regenerate the diagrams. ...@@ -43,7 +44,8 @@ Just call `make` to regenerate the diagrams.
## Building with Docker ## Building with Docker
If you are on a Mac or your pip install is messed up, you can easily build with docker. If you are on a Mac or your pip install is messed up, you can easily build with
docker:
```sh ```sh
make docker make docker
...@@ -51,13 +53,18 @@ make docker ...@@ -51,13 +53,18 @@ make docker
The first run will be slow but things should be fast after that. The first run will be slow but things should be fast after that.
To clean up the docker containers that are created (and other cruft that is left around) you can run `make docker-clean`. To clean up the docker containers that are created (and other cruft that is left
around) you can run `make docker-clean`.
If you are using boot2docker and get warnings about clock skew (or if things aren't building for some reason) then you can fix that up with `make fix-clock-skew`. If you are using boot2docker and get warnings about clock skew (or if things
aren't building for some reason) then you can fix that up with
`make fix-clock-skew`.
## Automatically rebuild on file changes ## Automatically rebuild on file changes
If you have the fswatch utility installed, you can have it monitor the file system and automatically rebuild when files have changed. Just do a `make watch`. If you have the fswatch utility installed, you can have it monitor the file
system and automatically rebuild when files have changed. Just do a
`make watch`.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/command_execution_port_forwarding.md
View file @ 6bdc0bfd
...@@ -36,14 +36,13 @@ Documentation for other releases can be found at ...@@ -36,14 +36,13 @@ Documentation for other releases can be found at
## Abstract ## Abstract
This describes an approach for providing support for: This document describes how to use Kubernetes to execute commands in containers,
with stdin/stdout/stderr streams attached and how to implement port forwarding
- executing commands in containers, with stdin/stdout/stderr streams attached to the containers.
- port forwarding to containers
## Background ## Background
There are several related issues/PRs: See the following related issues/PRs:
- [Support attach](http://issue.k8s.io/1521) - [Support attach](http://issue.k8s.io/1521)
- [Real container ssh](http://issue.k8s.io/1513) - [Real container ssh](http://issue.k8s.io/1513)
...@@ -77,34 +76,39 @@ won't be able to work with this mechanism, unless adapters can be written. ...@@ -77,34 +76,39 @@ won't be able to work with this mechanism, unless adapters can be written.
## Constraints and Assumptions ## Constraints and Assumptions
- SSH support is not currently in scope - SSH support is not currently in scope.
- CGroup confinement is ultimately desired, but implementing that support is not currently in scope - CGroup confinement is ultimately desired, but implementing that support is not
- SELinux confinement is ultimately desired, but implementing that support is not currently in scope currently in scope.
- SELinux confinement is ultimately desired, but implementing that support is
not currently in scope.
## Use Cases ## Use Cases
- As a user of a Kubernetes cluster, I want to run arbitrary commands in a container, attaching my local stdin/stdout/stderr to the container - A user of a Kubernetes cluster wants to run arbitrary commands in a
- As a user of a Kubernetes cluster, I want to be able to connect to local ports on my computer and have them forwarded to ports in the container container with local stdin/stdout/stderr attached to the container.
- A user of a Kubernetes cluster wants to connect to local ports on his computer
and have them forwarded to ports in a container.
## Process Flow ## Process Flow
### Remote Command Execution Flow ### Remote Command Execution Flow
1. The client connects to the Kubernetes Master to initiate a remote command execution 1. The client connects to the Kubernetes Master to initiate a remote command
request execution request.
2. The Master proxies the request to the Kubelet where the container lives 2. The Master proxies the request to the Kubelet where the container lives.
3. The Kubelet executes nsenter + the requested command and streams stdin/stdout/stderr back and forth between the client and the container 3. The Kubelet executes nsenter + the requested command and streams
stdin/stdout/stderr back and forth between the client and the container.
### Port Forwarding Flow ### Port Forwarding Flow
1. The client connects to the Kubernetes Master to initiate a remote command execution 1. The client connects to the Kubernetes Master to initiate a remote command
request execution request.
2. The Master proxies the request to the Kubelet where the container lives 2. The Master proxies the request to the Kubelet where the container lives.
3. The client listens on each specified local port, awaiting local connections 3. The client listens on each specified local port, awaiting local connections.
4. The client connects to one of the local listening ports 4. The client connects to one of the local listening ports.
4. The client notifies the Kubelet of the new connection 4. The client notifies the Kubelet of the new connection.
5. The Kubelet executes nsenter + socat and streams data back and forth between the client and the port in the container 5. The Kubelet executes nsenter + socat and streams data back and forth between
the client and the port in the container.
## Design Considerations ## Design Considerations
...@@ -177,7 +181,10 @@ functionality. We need to make sure that users are not allowed to execute ...@@ -177,7 +181,10 @@ functionality. We need to make sure that users are not allowed to execute
remote commands or do port forwarding to containers they aren't allowed to remote commands or do port forwarding to containers they aren't allowed to
access. access.
Additional work is required to ensure that multiple command execution or port forwarding connections from different clients are not able to see each other's data. This can most likely be achieved via SELinux labeling and unique process contexts. Additional work is required to ensure that multiple command execution or port
forwarding connections from different clients are not able to see each other's
data. This can most likely be achieved via SELinux labeling and unique process
contexts.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/configmap.md
View file @ 6bdc0bfd
...@@ -36,8 +36,8 @@ Documentation for other releases can be found at ...@@ -36,8 +36,8 @@ Documentation for other releases can be found at
## Abstract ## Abstract
The `ConfigMap` API resource stores data used for the configuration of applications deployed on The `ConfigMap` API resource stores data used for the configuration of
Kubernetes. applications deployed on Kubernetes.
The main focus of this resource is to: The main focus of this resource is to:
...@@ -47,71 +47,74 @@ The main focus of this resource is to: ...@@ -47,71 +47,74 @@ The main focus of this resource is to:
## Motivation ## Motivation
A `Secret`-like API resource is needed to store configuration data that pods can consume. A `Secret`-like API resource is needed to store configuration data that pods can
consume.
Goals of this design: Goals of this design:
1. Describe a `ConfigMap` API resource 1. Describe a `ConfigMap` API resource.
2. Describe the semantics of consuming `ConfigMap` as environment variables 2. Describe the semantics of consuming `ConfigMap` as environment variables.
3. Describe the semantics of consuming `ConfigMap` as files in a volume 3. Describe the semantics of consuming `ConfigMap` as files in a volume.
## Use Cases ## Use Cases
1. As a user, I want to be able to consume configuration data as environment variables 1. As a user, I want to be able to consume configuration data as environment
2. As a user, I want to be able to consume configuration data as files in a volume variables.
3. As a user, I want my view of configuration data in files to be eventually consistent with changes 2. As a user, I want to be able to consume configuration data as files in a
to the data volume.
3. As a user, I want my view of configuration data in files to be eventually
consistent with changes to the data.
### Consuming `ConfigMap` as Environment Variables ### Consuming `ConfigMap` as Environment Variables
Many programs read their configuration from environment variables. `ConfigMap` should be possible A series of events for consuming `ConfigMap` as environment variables:
to consume in environment variables. The rough series of events for consuming `ConfigMap` this way
is:
1. A `ConfigMap` object is created 1. Create a `ConfigMap` object.
2. A pod that consumes the configuration data via environment variables is created 2. Create a pod to consume the configuration data via environment variables.
3. The pod is scheduled onto a node 3. The pod is scheduled onto a node.
4. The kubelet retrieves the `ConfigMap` resource(s) referenced by the pod and starts the container 4. The Kubelet retrieves the `ConfigMap` resource(s) referenced by the pod and
processes with the appropriate data in environment variables starts the container processes with the appropriate configuration data from
environment variables.
### Consuming `ConfigMap` in Volumes ### Consuming `ConfigMap` in Volumes
Many programs read their configuration from configuration files. `ConfigMap` should be possible A series of events for consuming `ConfigMap` as configuration files in a volume:
to consume in a volume. The rough series of events for consuming `ConfigMap` this way
is:
1. A `ConfigMap` object is created 1. Create a `ConfigMap` object.
2. A new pod using the `ConfigMap` via the volume plugin is created 2. Create a new pod using the `ConfigMap` via a volume plugin.
3. The pod is scheduled onto a node 3. The pod is scheduled onto a node.
4. The Kubelet creates an instance of the volume plugin and calls its `Setup()` method 4. The Kubelet creates an instance of the volume plugin and calls its `Setup()`
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod and projects method.
the appropriate data into the volume 5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod
and projects the appropriate configuration data into the volume.
### Consuming `ConfigMap` Updates ### Consuming `ConfigMap` Updates
Any long-running system has configuration that is mutated over time. Changes made to configuration Any long-running system has configuration that is mutated over time. Changes
data must be made visible to pods consuming data in volumes so that they can respond to those made to configuration data must be made visible to pods consuming data in
changes. volumes so that they can respond to those changes.
The `resourceVersion` of the `ConfigMap` object will be updated by the API server every time the The `resourceVersion` of the `ConfigMap` object will be updated by the API
object is modified. After an update, modifications will be made visible to the consumer container: server every time the object is modified. After an update, modifications will be
made visible to the consumer container:
1. A `ConfigMap` object is created 1. Create a `ConfigMap` object.
2. A new pod using the `ConfigMap` via the volume plugin is created 2. Create a new pod using the `ConfigMap` via the volume plugin.
3. The pod is scheduled onto a node 3. The pod is scheduled onto a node.
4. During the sync loop, the Kubelet creates an instance of the volume plugin and calls its 4. During the sync loop, the Kubelet creates an instance of the volume plugin
`Setup()` method and calls its `Setup()` method.
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod and projects 5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod
the appropriate data into the volume and projects the appropriate data into the volume.
6. The `ConfigMap` referenced by the pod is updated 6. The `ConfigMap` referenced by the pod is updated.
7. During the next iteration of the `syncLoop`, the Kubelet creates an instance of the volume plugin 7. During the next iteration of the `syncLoop`, the Kubelet creates an instance
and calls its `Setup()` method of the volume plugin and calls its `Setup()` method.
8. The volume plugin projects the updated data into the volume atomically 8. The volume plugin projects the updated data into the volume atomically.
It is the consuming pod's responsibility to make use of the updated data once it is made visible. It is the consuming pod's responsibility to make use of the updated data once it
is made visible.
Because environment variables cannot be updated without restarting a container, configuration data Because environment variables cannot be updated without restarting a container,
consumed in environment variables will not be updated. configuration data consumed in environment variables will not be updated.
### Advantages ### Advantages
...@@ -133,8 +136,8 @@ type ConfigMap struct { ...@@ -133,8 +136,8 @@ type ConfigMap struct {
TypeMeta `json:",inline"` TypeMeta `json:",inline"`
ObjectMeta `json:"metadata,omitempty"` ObjectMeta `json:"metadata,omitempty"`
// Data contains the configuration data. Each key must be a valid DNS_SUBDOMAIN or leading // Data contains the configuration data. Each key must be a valid
// dot followed by valid DNS_SUBDOMAIN. // DNS_SUBDOMAIN or leading dot followed by valid DNS_SUBDOMAIN.
Data map[string]string `json:"data,omitempty"` Data map[string]string `json:"data,omitempty"`
} }
...@@ -146,7 +149,8 @@ type ConfigMapList struct { ...@@ -146,7 +149,8 @@ type ConfigMapList struct {
} }
``` ```
A `Registry` implementation for `ConfigMap` will be added to `pkg/registry/configmap`. A `Registry` implementation for `ConfigMap` will be added to
`pkg/registry/configmap`.
### Environment Variables ### Environment Variables
...@@ -174,8 +178,8 @@ type ConfigMapSelector struct { ...@@ -174,8 +178,8 @@ type ConfigMapSelector struct {
### Volume Source ### Volume Source
A new `ConfigMapVolumeSource` type of volume source containing the `ConfigMap` object will be A new `ConfigMapVolumeSource` type of volume source containing the `ConfigMap`
added to the `VolumeSource` struct in the API: object will be added to the `VolumeSource` struct in the API:
```go ```go
package api package api
...@@ -209,13 +213,14 @@ type KeyToPath struct { ...@@ -209,13 +213,14 @@ type KeyToPath struct {
} }
``` ```
**Note:** The update logic used in the downward API volume plug-in will be extracted and re-used in **Note:** The update logic used in the downward API volume plug-in will be
the volume plug-in for `ConfigMap`. extracted and re-used in the volume plug-in for `ConfigMap`.
### Changes to Secret ### Changes to Secret
We will update the Secret volume plugin to have a similar API to the new ConfigMap volume plugin. We will update the Secret volume plugin to have a similar API to the new
The secret volume plugin will also begin updating secret content in the volume when secrets change. `ConfigMap` volume plugin. The secret volume plugin will also begin updating
secret content in the volume when secrets change.
## Examples ## Examples
...@@ -281,7 +286,8 @@ spec: ...@@ -281,7 +286,8 @@ spec:
#### Consuming `ConfigMap` as Volumes #### Consuming `ConfigMap` as Volumes
`redis-volume-config` is intended to be used as a volume containing a config file: `redis-volume-config` is intended to be used as a volume containing a config
file:
```yaml ```yaml
apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1
...@@ -320,8 +326,8 @@ spec: ...@@ -320,8 +326,8 @@ spec:
## Future Improvements ## Future Improvements
In the future, we may add the ability to specify an init-container that can watch the volume In the future, we may add the ability to specify an init-container that can
contents for updates and respond to changes when they occur. watch the volume contents for updates and respond to changes when they occur.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/configmap.md?pixel)]() [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/configmap.md?pixel)]()
......
docs/design/control-plane-resilience.md
View file @ 6bdc0bfd
...@@ -54,7 +54,7 @@ ideas. ...@@ -54,7 +54,7 @@ ideas.
* **High availability:** continuing to be available and work correctly * **High availability:** continuing to be available and work correctly
even if some components are down or uncontactable. This typically even if some components are down or uncontactable. This typically
involves multiple replicas of critical services, and a reliable way involves multiple replicas of critical services, and a reliable way
to find available replicas. Note that it's possible (but not to find available replicas. Note that it's possible (but not
desirable) to have high desirable) to have high
availability properties (e.g. multiple replicas) in the absence of availability properties (e.g. multiple replicas) in the absence of
self-healing properties (e.g. if a replica fails, nothing replaces self-healing properties (e.g. if a replica fails, nothing replaces
...@@ -109,11 +109,11 @@ ideas. ...@@ -109,11 +109,11 @@ ideas.
## Relative Priorities ## Relative Priorities
1. **(Possibly manual) recovery from catastrophic failures:** having a Kubernetes cluster, and all 1. **(Possibly manual) recovery from catastrophic failures:** having a
applications running inside it, disappear forever perhaps is the worst Kubernetes cluster, and all applications running inside it, disappear forever
possible failure mode. So it is critical that we be able to perhaps is the worst possible failure mode. So it is critical that we be able to
recover the applications running inside a cluster from such recover the applications running inside a cluster from such failures in some
failures in some well-bounded time period. well-bounded time period.
1. In theory a cluster can be recovered by replaying all API calls 1. In theory a cluster can be recovered by replaying all API calls
that have ever been executed against it, in order, but most that have ever been executed against it, in order, but most
often that state has been lost, and/or is scattered across often that state has been lost, and/or is scattered across
...@@ -121,12 +121,12 @@ ideas. ...@@ -121,12 +121,12 @@ ideas.
probably infeasible. probably infeasible.
1. In theory a cluster can also be recovered to some relatively 1. In theory a cluster can also be recovered to some relatively
recent non-corrupt backup/snapshot of the disk(s) backing the recent non-corrupt backup/snapshot of the disk(s) backing the
etcd cluster state. But we have no default consistent etcd cluster state. But we have no default consistent
backup/snapshot, verification or restoration process. And we backup/snapshot, verification or restoration process. And we
don't routinely test restoration, so even if we did routinely don't routinely test restoration, so even if we did routinely
perform and verify backups, we have no hard evidence that we perform and verify backups, we have no hard evidence that we
can in practise effectively recover from catastrophic cluster can in practise effectively recover from catastrophic cluster
failure or data corruption by restoring from these backups. So failure or data corruption by restoring from these backups. So
there's more work to be done here. there's more work to be done here.
1. **Self-healing:** Most major cloud providers provide the ability to 1. **Self-healing:** Most major cloud providers provide the ability to
easily and automatically replace failed virtual machines within a easily and automatically replace failed virtual machines within a
...@@ -144,7 +144,6 @@ ideas. ...@@ -144,7 +144,6 @@ ideas.
addition](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member) addition](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member)
or [backup and or [backup and
recovery](https://github.com/coreos/etcd/blob/master/Documentation/admin_guide.md#disaster-recovery)). recovery](https://github.com/coreos/etcd/blob/master/Documentation/admin_guide.md#disaster-recovery)).
1. and boot disks are either: 1. and boot disks are either:
1. truely persistent (i.e. remote persistent disks), or 1. truely persistent (i.e. remote persistent disks), or
1. reconstructible (e.g. using boot-from-snapshot, 1. reconstructible (e.g. using boot-from-snapshot,
...@@ -157,7 +156,7 @@ ideas. ...@@ -157,7 +156,7 @@ ideas.
quorum members). In environments where cloud-assisted automatic quorum members). In environments where cloud-assisted automatic
self-healing might be infeasible (e.g. on-premise bare-metal self-healing might be infeasible (e.g. on-premise bare-metal
deployments), it also gives cluster administrators more time to deployments), it also gives cluster administrators more time to
respond (e.g. replace/repair failed machines) without incurring respond (e.g. replace/repair failed machines) without incurring
system downtime. system downtime.
## Design and Status (as of December 2015) ## Design and Status (as of December 2015)
...@@ -174,7 +173,7 @@ ideas. ...@@ -174,7 +173,7 @@ ideas.
Multiple stateless, self-hosted, self-healing API servers behind a HA Multiple stateless, self-hosted, self-healing API servers behind a HA
load balancer, built out by the default "kube-up" automation on GCE, load balancer, built out by the default "kube-up" automation on GCE,
AWS and basic bare metal (BBM). Note that the single-host approach of AWS and basic bare metal (BBM). Note that the single-host approach of
hving etcd listen only on localhost to ensure that onyl API server can hving etcd listen only on localhost to ensure that onyl API server can
connect to it will no longer work, so alternative security will be connect to it will no longer work, so alternative security will be
needed in the regard (either using firewall rules, SSL certs, or needed in the regard (either using firewall rules, SSL certs, or
...@@ -189,13 +188,13 @@ design doc. ...@@ -189,13 +188,13 @@ design doc.
<td> <td>
No scripted self-healing or HA on GCE, AWS or basic bare metal No scripted self-healing or HA on GCE, AWS or basic bare metal
currently exists in the OSS distro. To be clear, "no self healing" currently exists in the OSS distro. To be clear, "no self healing"
means that even if multiple e.g. API servers are provisioned for HA means that even if multiple e.g. API servers are provisioned for HA
purposes, if they fail, nothing replaces them, so eventually the purposes, if they fail, nothing replaces them, so eventually the
system will fail. Self-healing and HA can be set up system will fail. Self-healing and HA can be set up
manually by following documented instructions, but this is not manually by following documented instructions, but this is not
currently an automated process, and it is not tested as part of currently an automated process, and it is not tested as part of
continuous integration. So it's probably safest to assume that it continuous integration. So it's probably safest to assume that it
doesn't actually work in practise. doesn't actually work in practise.
</td> </td>
...@@ -205,8 +204,8 @@ doesn't actually work in practise. ...@@ -205,8 +204,8 @@ doesn't actually work in practise.
<td> <td>
Multiple self-hosted, self healing warm standby stateless controller Multiple self-hosted, self healing warm standby stateless controller
managers and schedulers with leader election and automatic failover of API server managers and schedulers with leader election and automatic failover of API
clients, automatically installed by default "kube-up" automation. server clients, automatically installed by default "kube-up" automation.
</td> </td>
<td>As above.</td> <td>As above.</td>
...@@ -218,47 +217,49 @@ clients, automatically installed by default "kube-up" automation. ...@@ -218,47 +217,49 @@ clients, automatically installed by default "kube-up" automation.
Multiple (3-5) etcd quorum members behind a load balancer with session Multiple (3-5) etcd quorum members behind a load balancer with session
affinity (to prevent clients from being bounced from one to another). affinity (to prevent clients from being bounced from one to another).
Regarding self-healing, if a node running etcd goes down, it is always necessary to do three Regarding self-healing, if a node running etcd goes down, it is always necessary
things: to do three things:
<ol> <ol>
<li>allocate a new node (not necessary if running etcd as a pod, in <li>allocate a new node (not necessary if running etcd as a pod, in
which case specific measures are required to prevent user pods from which case specific measures are required to prevent user pods from
interfering with system pods, for example using node selectors as interfering with system pods, for example using node selectors as
described in <A HREF=") described in <A HREF="),
<li>start an etcd replica on that new node, <li>start an etcd replica on that new node, and
<li>have the new replica recover the etcd state. <li>have the new replica recover the etcd state.
</ol> </ol>
In the case of local disk (which fails in concert with the machine), the etcd In the case of local disk (which fails in concert with the machine), the etcd
state must be recovered from the other replicas. This is called <A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member">dynamic member state must be recovered from the other replicas. This is called
addition</A>. <A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member">
In the case of remote persistent disk, the etcd state can be recovered dynamic member addition</A>.
by attaching the remote persistent disk to the replacement node, thus
the state is recoverable even if all other replicas are down. In the case of remote persistent disk, the etcd state can be recovered by
attaching the remote persistent disk to the replacement node, thus the state is
recoverable even if all other replicas are down.
There are also significant performance differences between local disks and remote There are also significant performance differences between local disks and remote
persistent disks. For example, the <A HREF="https://cloud.google.com/compute/docs/disks/#comparison_of_disk_types">sustained throughput persistent disks. For example, the
local disks in GCE is approximatley 20x that of remote disks</A>. <A HREF="https://cloud.google.com/compute/docs/disks/#comparison_of_disk_types">
sustained throughput local disks in GCE is approximatley 20x that of remote
Hence we suggest that self-healing be provided by remotely mounted persistent disks in disks</A>.
non-performance critical, single-zone cloud deployments. For
performance critical installations, faster local SSD's should be used, Hence we suggest that self-healing be provided by remotely mounted persistent
in which case remounting on node failure is not an option, so disks in non-performance critical, single-zone cloud deployments. For
<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md ">etcd runtime configuration</A> performance critical installations, faster local SSD's should be used, in which
should be used to replace the failed machine. Similarly, for case remounting on node failure is not an option, so
cross-zone self-healing, cloud persistent disks are zonal, so <A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md ">
automatic etcd runtime configuration</A> should be used to replace the failed machine.
<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md">runtime configuration</A> Similarly, for cross-zone self-healing, cloud persistent disks are zonal, so
is required. Similarly, basic bare metal deployments cannot generally automatic <A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md">
rely on runtime configuration</A> is required. Similarly, basic bare metal deployments
remote persistent disks, so the same approach applies there. cannot generally rely on remote persistent disks, so the same approach applies
there.
</td> </td>
<td> <td>
<A HREF="http://kubernetes.io/v1.1/docs/admin/high-availability.html"> <A HREF="http://kubernetes.io/v1.1/docs/admin/high-availability.html">
Somewhat vague instructions exist</A> Somewhat vague instructions exist</A> on how to set some of this up manually in
on how to set some of this up manually in a self-hosted a self-hosted configuration. But automatic bootstrapping and self-healing is not
configuration. But automatic bootstrapping and self-healing is not described (and is not implemented for the non-PD cases). This all still needs to
described (and is not implemented for the non-PD cases). This all be automated and continuously tested.
still needs to be automated and continuously tested.
</td> </td>
</tr> </tr>
</table> </table>
......
docs/design/extending-api.md
View file @ 6bdc0bfd
...@@ -34,59 +34,62 @@ Documentation for other releases can be found at ...@@ -34,59 +34,62 @@ Documentation for other releases can be found at
# Adding custom resources to the Kubernetes API server # Adding custom resources to the Kubernetes API server
This document describes the design for implementing the storage of custom API types in the Kubernetes API Server. This document describes the design for implementing the storage of custom API
types in the Kubernetes API Server.
## Resource Model ## Resource Model
### The ThirdPartyResource ### The ThirdPartyResource
The `ThirdPartyResource` resource describes the multiple versions of a custom resource that the user wants to add The `ThirdPartyResource` resource describes the multiple versions of a custom
to the Kubernetes API. `ThirdPartyResource` is a non-namespaced resource; attempting to place it in a namespace resource that the user wants to add to the Kubernetes API. `ThirdPartyResource`
will return an error. is a non-namespaced resource; attempting to place it in a namespace will return
an error.
Each `ThirdPartyResource` resource has the following: Each `ThirdPartyResource` resource has the following:
* Standard Kubernetes object metadata. * Standard Kubernetes object metadata.
* ResourceKind - The kind of the resources described by this third party resource. * ResourceKind - The kind of the resources described by this third party
resource.
* Description - A free text description of the resource. * Description - A free text description of the resource.
* APIGroup - An API group that this resource should be placed into. * APIGroup - An API group that this resource should be placed into.
* Versions - One or more `Version` objects. * Versions - One or more `Version` objects.
### The `Version` Object ### The `Version` Object
The `Version` object describes a single concrete version of a custom resource. The `Version` object currently The `Version` object describes a single concrete version of a custom resource.
only specifies: The `Version` object currently only specifies:
* The `Name` of the version. * The `Name` of the version.
* The `APIGroup` this version should belong to. * The `APIGroup` this version should belong to.
## Expectations about third party objects ## Expectations about third party objects
Every object that is added to a third-party Kubernetes object store is expected to contain Kubernetes Every object that is added to a third-party Kubernetes object store is expected
compatible [object metadata](../devel/api-conventions.md#metadata). This requirement enables the to contain Kubernetes compatible [object metadata](../devel/api-conventions.md#metadata).
Kubernetes API server to provide the following features: This requirement enables the Kubernetes API server to provide the following
* Filtering lists of objects via label queries features:
* `resourceVersion`-based optimistic concurrency via compare-and-swap * Filtering lists of objects via label queries.
* Versioned storage * `resourceVersion`-based optimistic concurrency via compare-and-swap.
* Event recording * Versioned storage.
* Integration with basic `kubectl` command line tooling * Event recording.
* Watch for resource changes * Integration with basic `kubectl` command line tooling.
* Watch for resource changes.
The `Kind` for an instance of a third-party object (e.g. CronTab) below is expected to be
programmatically convertible to the name of the resource using The `Kind` for an instance of a third-party object (e.g. CronTab) below is
the following conversion. Kinds are expected to be of the form `<CamelCaseKind>`, and the expected to be programmatically convertible to the name of the resource using
`APIVersion` for the object is expected to be `<api-group>/<api-version>`. To the following conversion. Kinds are expected to be of the form
prevent collisions, it's expected that you'll use a fully qualified domain `<CamelCaseKind>`, and the `APIVersion` for the object is expected to be
name for the API group, e.g. `example.com`. `<api-group>/<api-version>`. To prevent collisions, it's expected that you'll
use a fully qualified domain name for the API group, e.g. `example.com`.
For example `stable.example.com/v1` For example `stable.example.com/v1`
'CamelCaseKind' is the specific type name. 'CamelCaseKind' is the specific type name.
To convert this into the `metadata.name` for the `ThirdPartyResource` resource instance, To convert this into the `metadata.name` for the `ThirdPartyResource` resource
the `<domain-name>` is copied verbatim, the `CamelCaseKind` is instance, the `<domain-name>` is copied verbatim, the `CamelCaseKind` is then
then converted converted using '-' instead of capitalization ('camel-case'), with the first
using '-' instead of capitalization ('camel-case'), with the first character being assumed to be character being assumed to be capitalized. In pseudo code:
capitalized. In pseudo code:
```go ```go
var result string var result string
...@@ -98,17 +101,20 @@ for ix := range kindName { ...@@ -98,17 +101,20 @@ for ix := range kindName {
} }
``` ```
As a concrete example, the resource named `camel-case-kind.example.com` defines resources of Kind `CamelCaseKind`, in As a concrete example, the resource named `camel-case-kind.example.com` defines
the APIGroup with the prefix `example.com/...`. resources of Kind `CamelCaseKind`, in the APIGroup with the prefix
`example.com/...`.
The reason for this is to enable rapid lookup of a `ThirdPartyResource` object given the kind information. The reason for this is to enable rapid lookup of a `ThirdPartyResource` object
This is also the reason why `ThirdPartyResource` is not namespaced. given the kind information. This is also the reason why `ThirdPartyResource` is
not namespaced.
## Usage ## Usage
When a user creates a new `ThirdPartyResource`, the Kubernetes API Server reacts by creating a new, namespaced When a user creates a new `ThirdPartyResource`, the Kubernetes API Server reacts
RESTful resource path. For now, non-namespaced objects are not supported. As with existing built-in objects, by creating a new, namespaced RESTful resource path. For now, non-namespaced
deleting a namespace deletes all third party resources in that namespace. objects are not supported. As with existing built-in objects, deleting a
namespace deletes all third party resources in that namespace.
For example, if a user creates: For example, if a user creates:
...@@ -143,14 +149,15 @@ Now that this schema has been created, a user can `POST`: ...@@ -143,14 +149,15 @@ Now that this schema has been created, a user can `POST`:
to: `/apis/stable.example.com/v1/namespaces/default/crontabs` to: `/apis/stable.example.com/v1/namespaces/default/crontabs`
and the corresponding data will be stored into etcd by the APIServer, so that when the user issues: and the corresponding data will be stored into etcd by the APIServer, so that
when the user issues:
``` ```
GET /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object` GET /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object`
``` ```
And when they do that, they will get back the same data, but with additional Kubernetes metadata And when they do that, they will get back the same data, but with additional
(e.g. `resourceVersion`, `createdTimestamp`) filled in. Kubernetes metadata (e.g. `resourceVersion`, `createdTimestamp`) filled in.
Likewise, to list all resources, a user can issue: Likewise, to list all resources, a user can issue:
...@@ -178,29 +185,35 @@ and get back: ...@@ -178,29 +185,35 @@ and get back:
} }
``` ```
Because all objects are expected to contain standard Kubernetes metadata fields, these Because all objects are expected to contain standard Kubernetes metadata fields,
list operations can also use label queries to filter requests down to specific subsets. these list operations can also use label queries to filter requests down to
specific subsets.
Likewise, clients can use watch endpoints to watch for changes to stored objects.
Likewise, clients can use watch endpoints to watch for changes to stored
objects.
## Storage ## Storage
In order to store custom user data in a versioned fashion inside of etcd, we need to also introduce a In order to store custom user data in a versioned fashion inside of etcd, we
`Codec`-compatible object for persistent storage in etcd. This object is `ThirdPartyResourceData` and it contains: need to also introduce a `Codec`-compatible object for persistent storage in
* Standard API Metadata etcd. This object is `ThirdPartyResourceData` and it contains:
* Standard API Metadata.
* `Data`: The raw JSON data for this custom object. * `Data`: The raw JSON data for this custom object.
### Storage key specification ### Storage key specification
Each custom object stored by the API server needs a custom key in storage, this is described below: Each custom object stored by the API server needs a custom key in storage, this
is described below:
#### Definitions #### Definitions
* `resource-namespace`: the namespace of the particular resource that is being stored * `resource-namespace`: the namespace of the particular resource that is
being stored
* `resource-name`: the name of the particular resource being stored * `resource-name`: the name of the particular resource being stored
* `third-party-resource-namespace`: the namespace of the `ThirdPartyResource` resource that represents the type for the specific instance being stored * `third-party-resource-namespace`: the namespace of the `ThirdPartyResource`
* `third-party-resource-name`: the name of the `ThirdPartyResource` resource that represents the type for the specific instance being stored resource that represents the type for the specific instance being stored
* `third-party-resource-name`: the name of the `ThirdPartyResource` resource
that represents the type for the specific instance being stored
#### Key #### Key
......
docs/design/federation-phase-1.md
View file @ 6bdc0bfd
...@@ -71,7 +71,8 @@ unified view. ...@@ -71,7 +71,8 @@ unified view.
Here are the functionality requirements derived from above use cases: Here are the functionality requirements derived from above use cases:
+ Clients of the federation control plane API server can register and deregister clusters. + Clients of the federation control plane API server can register and deregister
clusters.
+ Workloads should be spread to different clusters according to the + Workloads should be spread to different clusters according to the
workload distribution policy. workload distribution policy.
+ Pods are able to discover and connect to services hosted in other + Pods are able to discover and connect to services hosted in other
...@@ -90,7 +91,7 @@ Here are the functionality requirements derived from above use cases: ...@@ -90,7 +91,7 @@ Here are the functionality requirements derived from above use cases:
It’s difficult to have a perfect design with one click that implements It’s difficult to have a perfect design with one click that implements
all the above requirements. Therefore we will go with an iterative all the above requirements. Therefore we will go with an iterative
approach to design and build the system. This document describes the approach to design and build the system. This document describes the
phase one of the whole work. In phase one we will cover only the phase one of the whole work. In phase one we will cover only the
following objectives: following objectives:
+ Define the basic building blocks and API objects of control plane + Define the basic building blocks and API objects of control plane
...@@ -130,9 +131,9 @@ description of each module contained in above diagram. ...@@ -130,9 +131,9 @@ description of each module contained in above diagram.
The API Server in the Ubernetes control plane works just like the API The API Server in the Ubernetes control plane works just like the API
Server in K8S. It talks to a distributed key-value store to persist, Server in K8S. It talks to a distributed key-value store to persist,
retrieve and watch API objects. This store is completely distinct retrieve and watch API objects. This store is completely distinct
from the kubernetes key-value stores (etcd) in the underlying from the kubernetes key-value stores (etcd) in the underlying
kubernetes clusters. We still use `etcd` as the distributed kubernetes clusters. We still use `etcd` as the distributed
storage so customers don’t need to learn and manage a different storage so customers don’t need to learn and manage a different
storage system, although it is envisaged that other storage systems storage system, although it is envisaged that other storage systems
(consol, zookeeper) will probably be developedand supported over (consol, zookeeper) will probably be developedand supported over
...@@ -141,16 +142,16 @@ time. ...@@ -141,16 +142,16 @@ time.
## Ubernetes Scheduler ## Ubernetes Scheduler
The Ubernetes Scheduler schedules resources onto the underlying The Ubernetes Scheduler schedules resources onto the underlying
Kubernetes clusters. For example it watches for unscheduled Ubernetes Kubernetes clusters. For example it watches for unscheduled Ubernetes
replication controllers (those that have not yet been scheduled onto replication controllers (those that have not yet been scheduled onto
underlying Kubernetes clusters) and performs the global scheduling underlying Kubernetes clusters) and performs the global scheduling
work. For each unscheduled replication controller, it calls policy work. For each unscheduled replication controller, it calls policy
engine to decide how to spit workloads among clusters. It creates a engine to decide how to spit workloads among clusters. It creates a
Kubernetes Replication Controller on one ore more underlying cluster, Kubernetes Replication Controller on one ore more underlying cluster,
and post them back to `etcd` storage. and post them back to `etcd` storage.
One sublety worth noting here is that the scheduling decision is One sublety worth noting here is that the scheduling decision is arrived at by
arrived at by combining the application-specific request from the user (which might combining the application-specific request from the user (which might
include, for example, placement constraints), and the global policy specified include, for example, placement constraints), and the global policy specified
by the federation administrator (for example, "prefer on-premise by the federation administrator (for example, "prefer on-premise
clusters over AWS clusters" or "spread load equally across clusters"). clusters over AWS clusters" or "spread load equally across clusters").
...@@ -165,9 +166,9 @@ performs the following two kinds of work: ...@@ -165,9 +166,9 @@ performs the following two kinds of work:
corresponding API objects on the underlying K8S clusters. corresponding API objects on the underlying K8S clusters.
1. It periodically retrieves the available resources metrics from the 1. It periodically retrieves the available resources metrics from the
underlying K8S cluster, and updates them as object status of the underlying K8S cluster, and updates them as object status of the
`cluster` API object. An alternative design might be to run a pod `cluster` API object. An alternative design might be to run a pod
in each underlying cluster that reports metrics for that cluster to in each underlying cluster that reports metrics for that cluster to
the Ubernetes control plane. Which approach is better remains an the Ubernetes control plane. Which approach is better remains an
open topic of discussion. open topic of discussion.
## Ubernetes Service Controller ## Ubernetes Service Controller
...@@ -187,7 +188,7 @@ Cluster is a new first-class API object introduced in this design. For ...@@ -187,7 +188,7 @@ Cluster is a new first-class API object introduced in this design. For
each registered K8S cluster there will be such an API resource in each registered K8S cluster there will be such an API resource in
control plane. The way clients register or deregister a cluster is to control plane. The way clients register or deregister a cluster is to
send corresponding REST requests to following URL: send corresponding REST requests to following URL:
`/api/{$version}/clusters`. Because control plane is behaving like a `/api/{$version}/clusters`. Because control plane is behaving like a
regular K8S client to the underlying clusters, the spec of a cluster regular K8S client to the underlying clusters, the spec of a cluster
object contains necessary properties like K8S cluster address and object contains necessary properties like K8S cluster address and
credentials. The status of a cluster API object will contain credentials. The status of a cluster API object will contain
...@@ -294,7 +295,7 @@ $version.clusterStatus ...@@ -294,7 +295,7 @@ $version.clusterStatus
**For simplicity we didn’t introduce a separate “cluster metrics” API **For simplicity we didn’t introduce a separate “cluster metrics” API
object here**. The cluster resource metrics are stored in cluster object here**. The cluster resource metrics are stored in cluster
status section, just like what we did to nodes in K8S. In phase one it status section, just like what we did to nodes in K8S. In phase one it
only contains available CPU resources and memory resources. The only contains available CPU resources and memory resources. The
cluster controller will periodically poll the underlying cluster API cluster controller will periodically poll the underlying cluster API
Server to get cluster capability. In phase one it gets the metrics by Server to get cluster capability. In phase one it gets the metrics by
simply aggregating metrics from all nodes. In future we will improve simply aggregating metrics from all nodes. In future we will improve
...@@ -315,7 +316,7 @@ Below is the state transition diagram. ...@@ -315,7 +316,7 @@ Below is the state transition diagram.
## Replication Controller ## Replication Controller
A global workload submitted to control plane is represented as an A global workload submitted to control plane is represented as an
Ubernetes replication controller. When a replication controller Ubernetes replication controller. When a replication controller
is submitted to control plane, clients need a way to express its is submitted to control plane, clients need a way to express its
requirements or preferences on clusters. Depending on different use requirements or preferences on clusters. Depending on different use
cases it may be complex. For example: cases it may be complex. For example:
...@@ -327,7 +328,7 @@ cases it may be complex. For example: ...@@ -327,7 +328,7 @@ cases it may be complex. For example:
(use case: workload ) (use case: workload )
+ Seventy percent of this workload should be scheduled to cluster Foo, + Seventy percent of this workload should be scheduled to cluster Foo,
and thirty percent should be scheduled to cluster Bar (use case: and thirty percent should be scheduled to cluster Bar (use case:
vendor lock-in avoidance). In phase one, we only introduce a vendor lock-in avoidance). In phase one, we only introduce a
_clusterSelector_ field to filter acceptable clusters. In default _clusterSelector_ field to filter acceptable clusters. In default
case there is no such selector and it means any cluster is case there is no such selector and it means any cluster is
acceptable. acceptable.
...@@ -376,7 +377,7 @@ clusters. How to handle this will be addressed after phase one. ...@@ -376,7 +377,7 @@ clusters. How to handle this will be addressed after phase one.
The Service API object exposed by Ubernetes is similar to service The Service API object exposed by Ubernetes is similar to service
objects on Kubernetes. It defines the access to a group of pods. The objects on Kubernetes. It defines the access to a group of pods. The
Ubernetes service controller will create corresponding Kubernetes Ubernetes service controller will create corresponding Kubernetes
service objects on underlying clusters. These are detailed in a service objects on underlying clusters. These are detailed in a
separate design document: [Federated Services](federated-services.md). separate design document: [Federated Services](federated-services.md).
## Pod ## Pod
...@@ -389,7 +390,8 @@ order to keep the Ubernetes API compatible with the Kubernetes API. ...@@ -389,7 +390,8 @@ order to keep the Ubernetes API compatible with the Kubernetes API.
## Scheduling ## Scheduling
The below diagram shows how workloads are scheduled on the Ubernetes control plane: The below diagram shows how workloads are scheduled on the Ubernetes control\
plane:
1. A replication controller is created by the client. 1. A replication controller is created by the client.
1. APIServer persists it into the storage. 1. APIServer persists it into the storage.
...@@ -425,8 +427,8 @@ proposed solutions like resource reservation mechanisms. ...@@ -425,8 +427,8 @@ proposed solutions like resource reservation mechanisms.
This part has been included in the section “Federated Service” of This part has been included in the section “Federated Service” of
document document
“[Ubernetes Cross-cluster Load Balancing and Service Discovery Requirements and System Design](federated-services.md))”. Please “[Ubernetes Cross-cluster Load Balancing and Service Discovery Requirements and System Design](federated-services.md))”.
refer to that document for details. Please refer to that document for details.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/identifiers.md
View file @ 6bdc0bfd
...@@ -34,95 +34,111 @@ Documentation for other releases can be found at ...@@ -34,95 +34,111 @@ Documentation for other releases can be found at
# Identifiers and Names in Kubernetes # Identifiers and Names in Kubernetes
A summarization of the goals and recommendations for identifiers in Kubernetes. Described in [GitHub issue #199](http://issue.k8s.io/199). A summarization of the goals and recommendations for identifiers in Kubernetes.
Described in GitHub issue [#199](http://issue.k8s.io/199).
## Definitions ## Definitions
UID `UID`: A non-empty, opaque, system-generated value guaranteed to be unique in time
: A non-empty, opaque, system-generated value guaranteed to be unique in time and space; intended to distinguish between historical occurrences of similar entities. and space; intended to distinguish between historical occurrences of similar
entities.
Name `Name`: A non-empty string guaranteed to be unique within a given scope at a
: A non-empty string guaranteed to be unique within a given scope at a particular time; used in resource URLs; provided by clients at creation time and encouraged to be human friendly; intended to facilitate creation idempotence and space-uniqueness of singleton objects, distinguish distinct entities, and reference particular entities across operations. particular time; used in resource URLs; provided by clients at creation time and
encouraged to be human friendly; intended to facilitate creation idempotence and
space-uniqueness of singleton objects, distinguish distinct entities, and
reference particular entities across operations.
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) label (DNS_LABEL) [rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) `label` (DNS_LABEL):
: An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters, with the '-' character allowed anywhere except the first or last character, suitable for use as a hostname or segment in a domain name An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters,
with the '-' character allowed anywhere except the first or last character,
suitable for use as a hostname or segment in a domain name.
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) subdomain (DNS_SUBDOMAIN) [rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) `subdomain` (DNS_SUBDOMAIN):
: One or more lowercase rfc1035/rfc1123 labels separated by '.' with a maximum length of 253 characters One or more lowercase rfc1035/rfc1123 labels separated by '.' with a maximum
length of 253 characters.
[rfc4122](http://www.ietf.org/rfc/rfc4122.txt) universally unique identifier (UUID) [rfc4122](http://www.ietf.org/rfc/rfc4122.txt) `universally unique identifier` (UUID):
: A 128 bit generated value that is extremely unlikely to collide across time and space and requires no central coordination A 128 bit generated value that is extremely unlikely to collide across time and
space and requires no central coordination.
[rfc6335](https://tools.ietf.org/rfc/rfc6335.txt) port name (IANA_SVC_NAME) [rfc6335](https://tools.ietf.org/rfc/rfc6335.txt) `port name` (IANA_SVC_NAME):
: An alphanumeric (a-z, and 0-9) string, with a maximum length of 15 characters, with the '-' character allowed anywhere except the first or the last character or adjacent to another '-' character, it must contain at least a (a-z) character An alphanumeric (a-z, and 0-9) string, with a maximum length of 15 characters,
with the '-' character allowed anywhere except the first or the last character
or adjacent to another '-' character, it must contain at least a (a-z)
character.
## Objectives for names and UIDs ## Objectives for names and UIDs
1. Uniquely identify (via a UID) an object across space and time 1. Uniquely identify (via a UID) an object across space and time.
2. Uniquely name (via a name) an object across space.
2. Uniquely name (via a name) an object across space 3. Provide human-friendly names in API operations and/or configuration files.
4. Allow idempotent creation of API resources (#148) and enforcement of
3. Provide human-friendly names in API operations and/or configuration files space-uniqueness of singleton objects.
5. Allow DNS names to be automatically generated for some objects.
4. Allow idempotent creation of API resources (#148) and enforcement of space-uniqueness of singleton objects
5. Allow DNS names to be automatically generated for some objects
## General design ## General design
1. When an object is created via an API, a Name string (a DNS_SUBDOMAIN) must be specified. Name must be non-empty and unique within the apiserver. This enables idempotent and space-unique creation operations. Parts of the system (e.g. replication controller) may join strings (e.g. a base name and a random suffix) to create a unique Name. For situations where generating a name is impractical, some or all objects may support a param to auto-generate a name. Generating random names will defeat idempotency. 1. When an object is created via an API, a Name string (a DNS_SUBDOMAIN) must
be specified. Name must be non-empty and unique within the apiserver. This
enables idempotent and space-unique creation operations. Parts of the system
(e.g. replication controller) may join strings (e.g. a base name and a random
suffix) to create a unique Name. For situations where generating a name is
impractical, some or all objects may support a param to auto-generate a name.
Generating random names will defeat idempotency.
* Examples: "guestbook.user", "backend-x4eb1" * Examples: "guestbook.user", "backend-x4eb1"
2. When an object is created via an API, a Namespace string (a DNS_SUBDOMAIN?
2. When an object is created via an API, a Namespace string (a DNS_SUBDOMAIN? format TBD via #1114) may be specified. Depending on the API receiver, namespaces might be validated (e.g. apiserver might ensure that the namespace actually exists). If a namespace is not specified, one will be assigned by the API receiver. This assignment policy might vary across API receivers (e.g. apiserver might have a default, kubelet might generate something semi-random). format TBD via #1114) may be specified. Depending on the API receiver,
namespaces might be validated (e.g. apiserver might ensure that the namespace
actually exists). If a namespace is not specified, one will be assigned by the
API receiver. This assignment policy might vary across API receivers (e.g.
apiserver might have a default, kubelet might generate something semi-random).
* Example: "api.k8s.example.com" * Example: "api.k8s.example.com"
3. Upon acceptance of an object via an API, the object is assigned a UID
3. Upon acceptance of an object via an API, the object is assigned a UID (a UUID). UID must be non-empty and unique across space and time. (a UUID). UID must be non-empty and unique across space and time.
* Example: "01234567-89ab-cdef-0123-456789abcdef" * Example: "01234567-89ab-cdef-0123-456789abcdef"
## Case study: Scheduling a pod ## Case study: Scheduling a pod
Pods can be placed onto a particular node in a number of ways. This case Pods can be placed onto a particular node in a number of ways. This case study
study demonstrates how the above design can be applied to satisfy the demonstrates how the above design can be applied to satisfy the objectives.
objectives.
### A pod scheduled by a user through the apiserver ### A pod scheduled by a user through the apiserver
1. A user submits a pod with Namespace="" and Name="guestbook" to the apiserver. 1. A user submits a pod with Namespace="" and Name="guestbook" to the apiserver.
2. The apiserver validates the input. 2. The apiserver validates the input.
1. A default Namespace is assigned. 1. A default Namespace is assigned.
2. The pod name must be space-unique within the Namespace. 2. The pod name must be space-unique within the Namespace.
3. Each container within the pod has a name which must be space-unique within the pod. 3. Each container within the pod has a name which must be space-unique within
the pod.
3. The pod is accepted. 3. The pod is accepted.
1. A new UID is assigned. 1. A new UID is assigned.
4. The pod is bound to a node. 4. The pod is bound to a node.
1. The kubelet on the node is passed the pod's UID, Namespace, and Name. 1. The kubelet on the node is passed the pod's UID, Namespace, and Name.
5. Kubelet validates the input. 5. Kubelet validates the input.
6. Kubelet runs the pod. 6. Kubelet runs the pod.
1. Each container is started up with enough metadata to distinguish the pod from whence it came. 1. Each container is started up with enough metadata to distinguish the pod
2. Each attempt to run a container is assigned a UID (a string) that is unique across time. from whence it came.
* This may correspond to Docker's container ID. 2. Each attempt to run a container is assigned a UID (a string) that is
unique across time. * This may correspond to Docker's container ID.
### A pod placed by a config file on the node ### A pod placed by a config file on the node
1. A config file is stored on the node, containing a pod with UID="", Namespace="", and Name="cadvisor". 1. A config file is stored on the node, containing a pod with UID="",
Namespace="", and Name="cadvisor".
2. Kubelet validates the input. 2. Kubelet validates the input.
1. Since UID is not provided, kubelet generates one. 1. Since UID is not provided, kubelet generates one.
2. Since Namespace is not provided, kubelet generates one. 2. Since Namespace is not provided, kubelet generates one.
1. The generated namespace should be deterministic and cluster-unique for the source, such as a hash of the hostname and file path. 1. The generated namespace should be deterministic and cluster-unique for
the source, such as a hash of the hostname and file path.
* E.g. Namespace="file-f4231812554558a718a01ca942782d81" * E.g. Namespace="file-f4231812554558a718a01ca942782d81"
3. Kubelet runs the pod. 3. Kubelet runs the pod.
1. Each container is started up with enough metadata to distinguish the pod from whence it came. 1. Each container is started up with enough metadata to distinguish the pod
2. Each attempt to run a container is assigned a UID (a string) that is unique across time. from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is
unique across time.
1. This may correspond to Docker's container ID. 1. This may correspond to Docker's container ID.
......
docs/design/metadata-policy.md
View file @ 6bdc0bfd
...@@ -38,21 +38,24 @@ Documentation for other releases can be found at ...@@ -38,21 +38,24 @@ Documentation for other releases can be found at
This document describes a new API resource, `MetadataPolicy`, that configures an This document describes a new API resource, `MetadataPolicy`, that configures an
admission controller to take one or more actions based on an object's metadata. admission controller to take one or more actions based on an object's metadata.
Initially the metadata fields that the predicates can examine are labels and annotations, Initially the metadata fields that the predicates can examine are labels and
and the actions are to add one or more labels and/or annotations, or to reject creation/update annotations, and the actions are to add one or more labels and/or annotations,
of the object. In the future other actions might be supported, such as applying an initializer. or to reject creation/update of the object. In the future other actions might be
supported, such as applying an initializer.
The first use of `MetadataPolicy` will be to decide which scheduler should schedule a pod
in a [multi-scheduler](../proposals/multiple-schedulers.md) Kubernetes system. In particular, the The first use of `MetadataPolicy` will be to decide which scheduler should
policy will add the scheduler name annotation to a pod based on an annotation that schedule a pod in a [multi-scheduler](../proposals/multiple-schedulers.md)
is already on the pod that indicates the QoS of the pod. Kubernetes system. In particular, the policy will add the scheduler name
(That annotation was presumably set by a simpler admission controller that annotation to a pod based on an annotation that is already on the pod that
uses code, rather than configuration, to map the resource requests and limits of a pod indicates the QoS of the pod. (That annotation was presumably set by a simpler
to QoS, and attaches the corresponding annotation.) admission controller that uses code, rather than configuration, to map the
resource requests and limits of a pod to QoS, and attaches the corresponding
We anticipate a number of other uses for `MetadataPolicy`, such as defaulting for annotation.)
labels and annotations, prohibiting/requiring particular labels or annotations, or
choosing a scheduling policy within a scheduler. We do not discuss them in this doc. We anticipate a number of other uses for `MetadataPolicy`, such as defaulting
for labels and annotations, prohibiting/requiring particular labels or
annotations, or choosing a scheduling policy within a scheduler. We do not
discuss them in this doc.
## API ## API
...@@ -126,7 +129,8 @@ type MetadataPolicyList struct { ...@@ -126,7 +129,8 @@ type MetadataPolicyList struct {
## Implementation plan ## Implementation plan
1. Create `MetadataPolicy` API resource 1. Create `MetadataPolicy` API resource
1. Create admission controller that implements policies defined in `MetadataPolicy` 1. Create admission controller that implements policies defined in
`MetadataPolicy`
1. Create admission controller that sets annotation 1. Create admission controller that sets annotation
`scheduler.alpha.kubernetes.io/qos: <QoS>` `scheduler.alpha.kubernetes.io/qos: <QoS>`
(where `QOS` is one of `Guaranteed, Burstable, BestEffort`) (where `QOS` is one of `Guaranteed, Burstable, BestEffort`)
...@@ -134,30 +138,32 @@ based on pod's resource request and limit. ...@@ -134,30 +138,32 @@ based on pod's resource request and limit.
## Future work ## Future work
Longer-term we will have QoS be set on create and update by the registry, similar to `Pending` phase today, Longer-term we will have QoS be set on create and update by the registry,
instead of having an admission controller (that runs before the one that takes `MetadataPolicy` as input) similar to `Pending` phase today, instead of having an admission controller
do it. (that runs before the one that takes `MetadataPolicy` as input) do it.
We plan to eventually move from having an admission controller We plan to eventually move from having an admission controller set the scheduler
set the scheduler name as a pod annotation, to using the initializer concept. In particular, the name as a pod annotation, to using the initializer concept. In particular, the
scheduler will be an initializer, and the admission controller that decides which scheduler to use scheduler will be an initializer, and the admission controller that decides
will add the scheduler's name to the list of initializers for the pod (presumably the scheduler which scheduler to use will add the scheduler's name to the list of initializers
will be the last initializer to run on each pod). for the pod (presumably the scheduler will be the last initializer to run on
The admission controller would still be configured using the `MetadataPolicy` described here, only the each pod). The admission controller would still be configured using the
mechanism the admission controller uses to record its decision of which scheduler to use would change. `MetadataPolicy` described here, only the mechanism the admission controller
uses to record its decision of which scheduler to use would change.
## Related issues ## Related issues
The main issue for multiple schedulers is #11793. There was also a lot of discussion The main issue for multiple schedulers is #11793. There was also a lot of
in PRs #17197 and #17865. discussion in PRs #17197 and #17865.
We could use the approach described here to choose a scheduling We could use the approach described here to choose a scheduling policy within a
policy within a single scheduler, as opposed to choosing a scheduler, a desire mentioned in #9920. single scheduler, as opposed to choosing a scheduler, a desire mentioned in
Issue #17097 describes a scenario unrelated to scheduler-choosing where `MetadataPolicy` could be used.
Issue #17324 proposes to create a generalized API for matching
"claims" to "service classes"; matching a pod to a scheduler would be one use for such an API.
# 9920. Issue #17097 describes a scenario unrelated to scheduler-choosing where
`MetadataPolicy` could be used. Issue #17324 proposes to create a generalized
API for matching "claims" to "service classes"; matching a pod to a scheduler
would be one use for such an API.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/networking.md
View file @ 6bdc0bfd
...@@ -44,9 +44,9 @@ There are 4 distinct networking problems to solve: ...@@ -44,9 +44,9 @@ There are 4 distinct networking problems to solve:
## Model and motivation ## Model and motivation
Kubernetes deviates from the default Docker networking model (though as of Kubernetes deviates from the default Docker networking model (though as of
Docker 1.8 their network plugins are getting closer). The goal is for each pod Docker 1.8 their network plugins are getting closer). The goal is for each pod
to have an IP in a flat shared networking namespace that has full communication to have an IP in a flat shared networking namespace that has full communication
with other physical computers and containers across the network. IP-per-pod with other physical computers and containers across the network. IP-per-pod
creates a clean, backward-compatible model where pods can be treated much like creates a clean, backward-compatible model where pods can be treated much like
VMs or physical hosts from the perspectives of port allocation, networking, VMs or physical hosts from the perspectives of port allocation, networking,
naming, service discovery, load balancing, application configuration, and naming, service discovery, load balancing, application configuration, and
...@@ -71,15 +71,15 @@ among other problems. ...@@ -71,15 +71,15 @@ among other problems.
All containers within a pod behave as if they are on the same host with regard All containers within a pod behave as if they are on the same host with regard
to networking. They can all reach each other’s ports on localhost. This offers to networking. They can all reach each other’s ports on localhost. This offers
simplicity (static ports know a priori), security (ports bound to localhost simplicity (static ports know a priori), security (ports bound to localhost
are visible within the pod but never outside it), and performance. This also are visible within the pod but never outside it), and performance. This also
reduces friction for applications moving from the world of uncontainerized apps reduces friction for applications moving from the world of uncontainerized apps
on physical or virtual hosts. People running application stacks together on on physical or virtual hosts. People running application stacks together on
the same host have already figured out how to make ports not conflict and have the same host have already figured out how to make ports not conflict and have
arranged for clients to find them. arranged for clients to find them.
The approach does reduce isolation between containers within a pod &mdash; The approach does reduce isolation between containers within a pod &mdash;
ports could conflict, and there can be no container-private ports, but these ports could conflict, and there can be no container-private ports, but these
seem to be relatively minor issues with plausible future workarounds. Besides, seem to be relatively minor issues with plausible future workarounds. Besides,
the premise of pods is that containers within a pod share some resources the premise of pods is that containers within a pod share some resources
(volumes, cpu, ram, etc.) and therefore expect and tolerate reduced isolation. (volumes, cpu, ram, etc.) and therefore expect and tolerate reduced isolation.
Additionally, the user can control what containers belong to the same pod Additionally, the user can control what containers belong to the same pod
...@@ -88,7 +88,7 @@ whereas, in general, they don't control what pods land together on a host. ...@@ -88,7 +88,7 @@ whereas, in general, they don't control what pods land together on a host.
## Pod to pod ## Pod to pod
Because every pod gets a "real" (not machine-private) IP address, pods can Because every pod gets a "real" (not machine-private) IP address, pods can
communicate without proxies or translations. The pod can use well-known port communicate without proxies or translations. The pod can use well-known port
numbers and can avoid the use of higher-level service discovery systems like numbers and can avoid the use of higher-level service discovery systems like
DNS-SD, Consul, or Etcd. DNS-SD, Consul, or Etcd.
...@@ -98,7 +98,7 @@ each pod has its own IP address that other pods can know. By making IP addresses ...@@ -98,7 +98,7 @@ each pod has its own IP address that other pods can know. By making IP addresses
and ports the same both inside and outside the pods, we create a NAT-less, flat and ports the same both inside and outside the pods, we create a NAT-less, flat
address space. Running "ip addr show" should work as expected. This would enable address space. Running "ip addr show" should work as expected. This would enable
all existing naming/discovery mechanisms to work out of the box, including all existing naming/discovery mechanisms to work out of the box, including
self-registration mechanisms and applications that distribute IP addresses. We self-registration mechanisms and applications that distribute IP addresses. We
should be optimizing for inter-pod network communication. Within a pod, should be optimizing for inter-pod network communication. Within a pod,
containers are more likely to use communication through volumes (e.g., tmpfs) or containers are more likely to use communication through volumes (e.g., tmpfs) or
IPC. IPC.
...@@ -141,7 +141,7 @@ gcloud compute routes add "${NODE_NAMES[$i]}" \ ...@@ -141,7 +141,7 @@ gcloud compute routes add "${NODE_NAMES[$i]}" \
--next-hop-instance-zone "${ZONE}" & --next-hop-instance-zone "${ZONE}" &
``` ```
GCE itself does not know anything about these IPs, though. This means that when GCE itself does not know anything about these IPs, though. This means that when
a pod tries to egress beyond GCE's project the packets must be SNAT'ed a pod tries to egress beyond GCE's project the packets must be SNAT'ed
(masqueraded) to the VM's IP, which GCE recognizes and allows. (masqueraded) to the VM's IP, which GCE recognizes and allows.
...@@ -161,26 +161,26 @@ to serve the purpose outside of GCE. ...@@ -161,26 +161,26 @@ to serve the purpose outside of GCE.
## Pod to service ## Pod to service
The [service](../user-guide/services.md) abstraction provides a way to group pods under a The [service](../user-guide/services.md) abstraction provides a way to group pods under a
common access policy (e.g. load-balanced). The implementation of this creates a common access policy (e.g. load-balanced). The implementation of this creates a
virtual IP which clients can access and which is transparently proxied to the virtual IP which clients can access and which is transparently proxied to the
pods in a Service. Each node runs a kube-proxy process which programs pods in a Service. Each node runs a kube-proxy process which programs
`iptables` rules to trap access to service IPs and redirect them to the correct `iptables` rules to trap access to service IPs and redirect them to the correct
backends. This provides a highly-available load-balancing solution with low backends. This provides a highly-available load-balancing solution with low
performance overhead by balancing client traffic from a node on that same node. performance overhead by balancing client traffic from a node on that same node.
## External to internal ## External to internal
So far the discussion has been about how to access a pod or service from within So far the discussion has been about how to access a pod or service from within
the cluster. Accessing a pod from outside the cluster is a bit more tricky. We the cluster. Accessing a pod from outside the cluster is a bit more tricky. We
want to offer highly-available, high-performance load balancing to target want to offer highly-available, high-performance load balancing to target
Kubernetes Services. Most public cloud providers are simply not flexible enough Kubernetes Services. Most public cloud providers are simply not flexible enough
yet. yet.
The way this is generally implemented is to set up external load balancers (e.g. The way this is generally implemented is to set up external load balancers (e.g.
GCE's ForwardingRules or AWS's ELB) which target all nodes in a cluster. When GCE's ForwardingRules or AWS's ELB) which target all nodes in a cluster. When
traffic arrives at a node it is recognized as being part of a particular Service traffic arrives at a node it is recognized as being part of a particular Service
and routed to an appropriate backend Pod. This does mean that some traffic will and routed to an appropriate backend Pod. This does mean that some traffic will
get double-bounced on the network. Once cloud providers have better offerings get double-bounced on the network. Once cloud providers have better offerings
we can take advantage of those. we can take advantage of those.
## Challenges and future work ## Challenges and future work
...@@ -207,7 +207,13 @@ External IP assignment would also simplify DNS support (see below). ...@@ -207,7 +207,13 @@ External IP assignment would also simplify DNS support (see below).
### IPv6 ### IPv6
IPv6 would be a nice option, also, but we can't depend on it yet. Docker support is in progress: [Docker issue #2974](https://github.com/dotcloud/docker/issues/2974), [Docker issue #6923](https://github.com/dotcloud/docker/issues/6923), [Docker issue #6975](https://github.com/dotcloud/docker/issues/6975). Additionally, direct ipv6 assignment to instances doesn't appear to be supported by major cloud providers (e.g., AWS EC2, GCE) yet. We'd happily take pull requests from people running Kubernetes on bare metal, though. :-) IPv6 would be a nice option, also, but we can't depend on it yet. Docker support
is in progress: [Docker issue #2974](https://github.com/dotcloud/docker/issues/2974),
[Docker issue #6923](https://github.com/dotcloud/docker/issues/6923),
[Docker issue #6975](https://github.com/dotcloud/docker/issues/6975).
Additionally, direct ipv6 assignment to instances doesn't appear to be supported
by major cloud providers (e.g., AWS EC2, GCE) yet. We'd happily take pull
requests from people running Kubernetes on bare metal, though. :-)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/principles.md
View file @ 6bdc0bfd
...@@ -43,26 +43,57 @@ See also the [API conventions](../devel/api-conventions.md). ...@@ -43,26 +43,57 @@ See also the [API conventions](../devel/api-conventions.md).
* All APIs should be declarative. * All APIs should be declarative.
* API objects should be complementary and composable, not opaque wrappers. * API objects should be complementary and composable, not opaque wrappers.
* The control plane should be transparent -- there are no hidden internal APIs. * The control plane should be transparent -- there are no hidden internal APIs.
* The cost of API operations should be proportional to the number of objects intentionally operated upon. Therefore, common filtered lookups must be indexed. Beware of patterns of multiple API calls that would incur quadratic behavior. * The cost of API operations should be proportional to the number of objects
* Object status must be 100% reconstructable by observation. Any history kept must be just an optimization and not required for correct operation. intentionally operated upon. Therefore, common filtered lookups must be indexed.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add them. If you must have them, don't enforce them atomically in master components, that is contention-prone and doesn't provide a recovery path in the case of a bug allowing the invariant to be violated. Instead, provide a series of checks to reduce the probability of a violation, and make every component involved able to recover from an invariant violation. Beware of patterns of multiple API calls that would incur quadratic behavior.
* Low-level APIs should be designed for control by higher-level systems. Higher-level APIs should be intent-oriented (think SLOs) rather than implementation-oriented (think control knobs). * Object status must be 100% reconstructable by observation. Any history kept
must be just an optimization and not required for correct operation.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add
them. If you must have them, don't enforce them atomically in master components,
that is contention-prone and doesn't provide a recovery path in the case of a
bug allowing the invariant to be violated. Instead, provide a series of checks
to reduce the probability of a violation, and make every component involved able
to recover from an invariant violation.
* Low-level APIs should be designed for control by higher-level systems.
Higher-level APIs should be intent-oriented (think SLOs) rather than
implementation-oriented (think control knobs).
## Control logic ## Control logic
* Functionality must be *level-based*, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Edge-triggered behavior must be just an optimization. * Functionality must be *level-based*, meaning the system must operate correctly
* Assume an open world: continually verify assumptions and gracefully adapt to external events and/or actors. Example: we allow users to kill pods under control of a replication controller; it just replaces them. given the desired state and the current/observed state, regardless of how many
* Do not define comprehensive state machines for objects with behaviors associated with state transitions and/or "assumed" states that cannot be ascertained by observation. intermediate state updates may have been missed. Edge-triggered behavior must be
* Don't assume a component's decisions will not be overridden or rejected, nor for the component to always understand why. For example, etcd may reject writes. Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry, but back off and/or make alternative decisions. just an optimization.
* Components should be self-healing. For example, if you must keep some state (e.g., cache) the content needs to be periodically refreshed, so that if an item does get erroneously stored or a deletion event is missed etc, it will be soon fixed, ideally on timescales that are shorter than what will attract attention from humans. * Assume an open world: continually verify assumptions and gracefully adapt to
* Component behavior should degrade gracefully. Prioritize actions so that the most important activities can continue to function even when overloaded and/or in states of partial failure. external events and/or actors. Example: we allow users to kill pods under
control of a replication controller; it just replaces them.
* Do not define comprehensive state machines for objects with behaviors
associated with state transitions and/or "assumed" states that cannot be
ascertained by observation.
* Don't assume a component's decisions will not be overridden or rejected, nor
for the component to always understand why. For example, etcd may reject writes.
Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry,
but back off and/or make alternative decisions.
* Components should be self-healing. For example, if you must keep some state
(e.g., cache) the content needs to be periodically refreshed, so that if an item
does get erroneously stored or a deletion event is missed etc, it will be soon
fixed, ideally on timescales that are shorter than what will attract attention
from humans.
* Component behavior should degrade gracefully. Prioritize actions so that the
most important activities can continue to function even when overloaded and/or
in states of partial failure.
## Architecture ## Architecture
* Only the apiserver should communicate with etcd/store, and not other components (scheduler, kubelet, etc.). * Only the apiserver should communicate with etcd/store, and not other
components (scheduler, kubelet, etc.).
* Compromising a single node shouldn't compromise the cluster. * Compromising a single node shouldn't compromise the cluster.
* Components should continue to do what they were last told in the absence of new instructions (e.g., due to network partition or component outage). * Components should continue to do what they were last told in the absence of
* All components should keep all relevant state in memory all the time. The apiserver should write through to etcd/store, other components should write through to the apiserver, and they should watch for updates made by other clients. new instructions (e.g., due to network partition or component outage).
* All components should keep all relevant state in memory all the time. The
apiserver should write through to etcd/store, other components should write
through to the apiserver, and they should watch for updates made by other
clients.
* Watch is preferred over polling. * Watch is preferred over polling.
## Extensibility ## Extensibility
...@@ -72,13 +103,23 @@ TODO: pluggability ...@@ -72,13 +103,23 @@ TODO: pluggability
## Bootstrapping ## Bootstrapping
* [Self-hosting](http://issue.k8s.io/246) of all components is a goal. * [Self-hosting](http://issue.k8s.io/246) of all components is a goal.
* Minimize the number of dependencies, particularly those required for steady-state operation. * Minimize the number of dependencies, particularly those required for
steady-state operation.
* Stratify the dependencies that remain via principled layering. * Stratify the dependencies that remain via principled layering.
* Break any circular dependencies by converting hard dependencies to soft dependencies. * Break any circular dependencies by converting hard dependencies to soft
* Also accept that data from other components from another source, such as local files, which can then be manually populated at bootstrap time and then continuously updated once those other components are available. dependencies.
* Also accept that data from other components from another source, such as
local files, which can then be manually populated at bootstrap time and then
continuously updated once those other components are available.
* State should be rediscoverable and/or reconstructable. * State should be rediscoverable and/or reconstructable.
* Make it easy to run temporary, bootstrap instances of all components in order to create the runtime state needed to run the components in the steady state; use a lock (master election for distributed components, file lock for local components like Kubelet) to coordinate handoff. We call this technique "pivoting". * Make it easy to run temporary, bootstrap instances of all components in
* Have a solution to restart dead components. For distributed components, replication works well. For local components such as Kubelet, a process manager or even a simple shell loop works. order to create the runtime state needed to run the components in the steady
state; use a lock (master election for distributed components, file lock for
local components like Kubelet) to coordinate handoff. We call this technique
"pivoting".
* Have a solution to restart dead components. For distributed components,
replication works well. For local components such as Kubelet, a process manager
or even a simple shell loop works.
## Availability ## Availability
......
docs/design/scheduler_extender.md
View file @ 6bdc0bfd
...@@ -34,11 +34,26 @@ Documentation for other releases can be found at ...@@ -34,11 +34,26 @@ Documentation for other releases can be found at
# Scheduler extender # Scheduler extender
There are three ways to add new scheduling rules (predicates and priority functions) to Kubernetes: (1) by adding these rules to the scheduler and recompiling (described here: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler.md), (2) implementing your own scheduler process that runs instead of, or alongside of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender" process that the standard Kubernetes scheduler calls out to as a final pass when making scheduling decisions. There are three ways to add new scheduling rules (predicates and priority
functions) to Kubernetes: (1) by adding these rules to the scheduler and
This document describes the third approach. This approach is needed for use cases where scheduling decisions need to be made on resources not directly managed by the standard Kubernetes scheduler. The extender helps make scheduling decisions based on such resources. (Note that the three approaches are not mutually exclusive.) recompiling (described here:
https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler.md),
When scheduling a pod, the extender allows an external process to filter and prioritize nodes. Two separate http/https calls are issued to the extender, one for "filter" and one for "prioritize" actions. To use the extender, you must create a scheduler policy configuration file. The configuration specifies how to reach the extender, whether to use http or https and the timeout. (2) implementing your own scheduler process that runs instead of, or alongside
of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender"
process that the standard Kubernetes scheduler calls out to as a final pass when
making scheduling decisions.
This document describes the third approach. This approach is needed for use
cases where scheduling decisions need to be made on resources not directly
managed by the standard Kubernetes scheduler. The extender helps make scheduling
decisions based on such resources. (Note that the three approaches are not
mutually exclusive.)
When scheduling a pod, the extender allows an external process to filter and
prioritize nodes. Two separate http/https calls are issued to the extender, one
for "filter" and one for "prioritize" actions. To use the extender, you must
create a scheduler policy configuration file. The configuration specifies how to
reach the extender, whether to use http or https and the timeout.
```go ```go
// Holds the parameters used to communicate with the extender. If a verb is unspecified/empty, // Holds the parameters used to communicate with the extender. If a verb is unspecified/empty,
...@@ -94,7 +109,10 @@ A sample scheduler policy file with extender configuration: ...@@ -94,7 +109,10 @@ A sample scheduler policy file with extender configuration:
} }
``` ```
Arguments passed to the FilterVerb endpoint on the extender are the set of nodes filtered through the k8s predicates and the pod. Arguments passed to the PrioritizeVerb endpoint on the extender are the set of nodes filtered through the k8s predicates and extender predicates and the pod. Arguments passed to the FilterVerb endpoint on the extender are the set of nodes
filtered through the k8s predicates and the pod. Arguments passed to the
PrioritizeVerb endpoint on the extender are the set of nodes filtered through
the k8s predicates and extender predicates and the pod.
```go ```go
// ExtenderArgs represents the arguments needed by the extender to filter/prioritize // ExtenderArgs represents the arguments needed by the extender to filter/prioritize
...@@ -107,9 +125,12 @@ type ExtenderArgs struct { ...@@ -107,9 +125,12 @@ type ExtenderArgs struct {
} }
``` ```
The "filter" call returns a list of nodes (api.NodeList). The "prioritize" call returns priorities for each node (schedulerapi.HostPriorityList). The "filter" call returns a list of nodes (api.NodeList). The "prioritize" call
returns priorities for each node (schedulerapi.HostPriorityList).
The "filter" call may prune the set of nodes based on its predicates. Scores returned by the "prioritize" call are added to the k8s scores (computed through its priority functions) and used for final host selection. The "filter" call may prune the set of nodes based on its predicates. Scores
returned by the "prioritize" call are added to the k8s scores (computed through
its priority functions) and used for final host selection.
Multiple extenders can be configured in the scheduler policy. Multiple extenders can be configured in the scheduler policy.
......
docs/design/simple-rolling-update.md
View file @ 6bdc0bfd
...@@ -34,32 +34,47 @@ Documentation for other releases can be found at ...@@ -34,32 +34,47 @@ Documentation for other releases can be found at
## Simple rolling update ## Simple rolling update
This is a lightweight design document for simple [rolling update](../user-guide/kubectl/kubectl_rolling-update.md) in `kubectl`. This is a lightweight design document for simple
[rolling update](../user-guide/kubectl/kubectl_rolling-update.md) in `kubectl`.
Complete execution flow can be found [here](#execution-details). See the [example of rolling update](../user-guide/update-demo/) for more information. Complete execution flow can be found [here](#execution-details). See the
[example of rolling update](../user-guide/update-demo/) for more information.
### Lightweight rollout ### Lightweight rollout
Assume that we have a current replication controller named `foo` and it is running image `image:v1` Assume that we have a current replication controller named `foo` and it is
running image `image:v1`
`kubectl rolling-update foo [foo-v2] --image=myimage:v2` `kubectl rolling-update foo [foo-v2] --image=myimage:v2`
If the user doesn't specify a name for the 'next' replication controller, then the 'next' replication controller is renamed to If the user doesn't specify a name for the 'next' replication controller, then
the 'next' replication controller is renamed to
the name of the original replication controller. the name of the original replication controller.
Obviously there is a race here, where if you kill the client between delete foo, and creating the new version of 'foo' you might be surprised about what is there, but I think that's ok. Obviously there is a race here, where if you kill the client between delete foo,
See [Recovery](#recovery) below and creating the new version of 'foo' you might be surprised about what is
there, but I think that's ok. See [Recovery](#recovery) below
If the user does specify a name for the 'next' replication controller, then the 'next' replication controller is retained with its existing name, If the user does specify a name for the 'next' replication controller, then the
and the old 'foo' replication controller is deleted. For the purposes of the rollout, we add a unique-ifying label `kubernetes.io/deployment` to both the `foo` and `foo-next` replication controllers. 'next' replication controller is retained with its existing name, and the old
The value of that label is the hash of the complete JSON representation of the`foo-next` or`foo` replication controller. The name of this label can be overridden by the user with the `--deployment-label-key` flag. 'foo' replication controller is deleted. For the purposes of the rollout, we add
a unique-ifying label `kubernetes.io/deployment` to both the `foo` and
`foo-next` replication controllers. The value of that label is the hash of the
complete JSON representation of the`foo-next` or`foo` replication controller.
The name of this label can be overridden by the user with the
`--deployment-label-key` flag.
#### Recovery #### Recovery
If a rollout fails or is terminated in the middle, it is important that the user be able to resume the roll out. If a rollout fails or is terminated in the middle, it is important that the user
To facilitate recovery in the case of a crash of the updating process itself, we add the following annotations to each replication controller in the `kubernetes.io/` annotation namespace: be able to resume the roll out. To facilitate recovery in the case of a crash of
* `desired-replicas` The desired number of replicas for this replication controller (either N or zero) the updating process itself, we add the following annotations to each
* `update-partner` A pointer to the replication controller resource that is the other half of this update (syntax `<name>` the namespace is assumed to be identical to the namespace of this replication controller.) replication controller in the `kubernetes.io/` annotation namespace:
* `desired-replicas` The desired number of replicas for this replication
controller (either N or zero)
* `update-partner` A pointer to the replication controller resource that is
the other half of this update (syntax `<name>` the namespace is assumed to be
identical to the namespace of this replication controller.)
Recovery is achieved by issuing the same command again: Recovery is achieved by issuing the same command again:
...@@ -67,9 +82,12 @@ Recovery is achieved by issuing the same command again: ...@@ -67,9 +82,12 @@ Recovery is achieved by issuing the same command again:
kubectl rolling-update foo [foo-v2] --image=myimage:v2 kubectl rolling-update foo [foo-v2] --image=myimage:v2
``` ```
Whenever the rolling update command executes, the kubectl client looks for replication controllers called `foo` and `foo-next`, if they exist, an attempt is Whenever the rolling update command executes, the kubectl client looks for
made to roll `foo` to `foo-next`. If `foo-next` does not exist, then it is created, and the rollout is a new rollout. If `foo` doesn't exist, then replication controllers called `foo` and `foo-next`, if they exist, an attempt
it is assumed that the rollout is nearly completed, and `foo-next` is renamed to `foo`. Details of the execution flow are given below. is made to roll `foo` to `foo-next`. If `foo-next` does not exist, then it is
created, and the rollout is a new rollout. If `foo` doesn't exist, then it is
assumed that the rollout is nearly completed, and `foo-next` is renamed to
`foo`. Details of the execution flow are given below.
### Aborting a rollout ### Aborting a rollout
...@@ -82,22 +100,28 @@ This is really just semantic sugar for: ...@@ -82,22 +100,28 @@ This is really just semantic sugar for:
`kubectl rolling-update foo-v2 foo` `kubectl rolling-update foo-v2 foo`
With the added detail that it moves the `desired-replicas` annotation from `foo-v2` to `foo` With the added detail that it moves the `desired-replicas` annotation from
`foo-v2` to `foo`
### Execution Details ### Execution Details
For the purposes of this example, assume that we are rolling from `foo` to `foo-next` where the only change is an image update from `v1` to `v2` For the purposes of this example, assume that we are rolling from `foo` to
`foo-next` where the only change is an image update from `v1` to `v2`
If the user doesn't specify a `foo-next` name, then it is either discovered from the `update-partner` annotation on `foo`. If that annotation doesn't exist, If the user doesn't specify a `foo-next` name, then it is either discovered from
then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-next-controller-JSON>` the `update-partner` annotation on `foo`. If that annotation doesn't exist,
then `foo-next` is synthesized using the pattern
`<controller-name>-<hash-of-next-controller-JSON>`
#### Initialization #### Initialization
* If `foo` and `foo-next` do not exist: * If `foo` and `foo-next` do not exist:
* Exit, and indicate an error to the user, that the specified controller doesn't exist. * Exit, and indicate an error to the user, that the specified controller
doesn't exist.
* If `foo` exists, but `foo-next` does not: * If `foo` exists, but `foo-next` does not:
* Create `foo-next` populate it with the `v2` image, set `desired-replicas` to `foo.Spec.Replicas` * Create `foo-next` populate it with the `v2` image, set
`desired-replicas` to `foo.Spec.Replicas`
* Goto Rollout * Goto Rollout
* If `foo-next` exists, but `foo` does not: * If `foo-next` exists, but `foo` does not:
* Assume that we are in the rename phase. * Assume that we are in the rename phase.
...@@ -105,7 +129,8 @@ then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-nex ...@@ -105,7 +129,8 @@ then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-nex
* If both `foo` and `foo-next` exist: * If both `foo` and `foo-next` exist:
* Assume that we are in a partial rollout * Assume that we are in a partial rollout
* If `foo-next` is missing the `desired-replicas` annotation * If `foo-next` is missing the `desired-replicas` annotation
* Populate the `desired-replicas` annotation to `foo-next` using the current size of `foo` * Populate the `desired-replicas` annotation to `foo-next` using the
current size of `foo`
* Goto Rollout * Goto Rollout
#### Rollout #### Rollout
...@@ -125,11 +150,13 @@ then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-nex ...@@ -125,11 +150,13 @@ then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-nex
#### Abort #### Abort
* If `foo-next` doesn't exist * If `foo-next` doesn't exist
* Exit and indicate to the user that they may want to simply do a new rollout with the old version * Exit and indicate to the user that they may want to simply do a new
rollout with the old version
* If `foo` doesn't exist * If `foo` doesn't exist
* Exit and indicate not found to the user * Exit and indicate not found to the user
* Otherwise, `foo-next` and `foo` both exist * Otherwise, `foo-next` and `foo` both exist
* Set `desired-replicas` annotation on `foo` to match the annotation on `foo-next` * Set `desired-replicas` annotation on `foo` to match the annotation on
`foo-next`
* Goto Rollout with `foo` and `foo-next` trading places. * Goto Rollout with `foo` and `foo-next` trading places.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment