To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run
To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run
on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not
on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not
be considered). Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions.
be considered). Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions.
Call the number of regions to be in `R`.
Call the number of regions to be in `R`.
Second, decide how many clusters should be able to be unavailable at the same time, while still being available. Call
Second, decide how many clusters should be able to be unavailable at the same time, while still being available. Call
the number that can be unavailable `U`. If you are not sure, then 1 is a fine choice.
the number that can be unavailable `U`. If you are not sure, then 1 is a fine choice.
If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then
If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then
you need `R + U` clusters. If it is not (e.g you want to ensure low latency for all users in the event of a
you need `R + U` clusters. If it is not (e.g you want to ensure low latency for all users in the event of a
cluster failure), then you need to have `R * U` clusters (`U` in each of `R` regions). In any case, try to put each cluster in a different zone.
cluster failure), then you need to have `R * U` clusters (`U` in each of `R` regions). In any case, try to put each cluster in a different zone.
@@ -55,7 +55,7 @@ What constitutes a compatible change and how to change the API are detailed by t
...
@@ -55,7 +55,7 @@ What constitutes a compatible change and how to change the API are detailed by t
## API versioning
## API versioning
To make it easier to eliminate fields or restructure resource representations, Kubernetes supports multiple API versions, each at a different API path prefix, such as `/api/v1beta3`. These are simply different interfaces to read and/or modify the same underlying resources. In general, all API resources are accessible via all API versions, though there may be some cases in the future where that is not true.
To make it easier to eliminate fields or restructure resource representations, Kubernetes supports multiple API versions, each at a different API path prefix, such as `/api/v1beta3`. These are simply different interfaces to read and/or modify the same underlying resources. In general, all API resources are accessible via all API versions, though there may be some cases in the future where that is not true.
We chose to version at the API level rather than at the resource or field level to ensure that the API presents a clear, consistent view of system resources and behavior, and to enable controlling access to end-of-lifed and/or experimental APIs.
We chose to version at the API level rather than at the resource or field level to ensure that the API presents a clear, consistent view of system resources and behavior, and to enable controlling access to end-of-lifed and/or experimental APIs.
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
...
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
# Kubernetes Design Overview
# Kubernetes Design Overview
Kubernetes is a system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
Kubernetes is a system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
Kubernetes establishes robust declarative primitives for maintaining the desired state requested by the user. We see these primitives as the main value added by Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers require active controllers, not just imperative orchestration.
Kubernetes establishes robust declarative primitives for maintaining the desired state requested by the user. We see these primitives as the main value added by Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers require active controllers, not just imperative orchestration.
A `PersistentVolume` (PV) is a storage resource provisioned by an administrator. It is analogous to a node. See [Persistent Volume Guide](../user-guide/persistent-volumes/) for how to use it.
A `PersistentVolume` (PV) is a storage resource provisioned by an administrator. It is analogous to a node. See [Persistent Volume Guide](../user-guide/persistent-volumes/) for how to use it.
A `PersistentVolumeClaim` (PVC) is a user's request for a persistent volume to use in a pod. It is analogous to a pod.
A `PersistentVolumeClaim` (PVC) is a user's request for a persistent volume to use in a pod. It is analogous to a pod.
One new system component:
One new system component:
`PersistentVolumeClaimBinder` is a singleton running in master that watches all PersistentVolumeClaims in the system and binds them to the closest matching available PersistentVolume. The volume manager watches the API for newly created volumes to manage.
`PersistentVolumeClaimBinder` is a singleton running in master that watches all PersistentVolumeClaims in the system and binds them to the closest matching available PersistentVolume. The volume manager watches the API for newly created volumes to manage.
One new volume:
One new volume:
...
@@ -69,7 +69,7 @@ Cluster administrators use the API to manage *PersistentVolumes*. A custom stor
...
@@ -69,7 +69,7 @@ Cluster administrators use the API to manage *PersistentVolumes*. A custom stor
PVs are system objects and, thus, have no namespace.
PVs are system objects and, thus, have no namespace.
Many means of dynamic provisioning will be eventually be implemented for various storage types.
Many means of dynamic provisioning will be eventually be implemented for various storage types.
##### PersistentVolume API
##### PersistentVolume API
...
@@ -116,7 +116,7 @@ TBD
...
@@ -116,7 +116,7 @@ TBD
#### Events
#### Events
The implementation of persistent storage will not require events to communicate to the user the state of their claim. The CLI for bound claims contains a reference to the backing persistent volume. This is always present in the API and CLI, making an event to communicate the same unnecessary.
The implementation of persistent storage will not require events to communicate to the user the state of their claim. The CLI for bound claims contains a reference to the backing persistent volume. This is always present in the API and CLI, making an event to communicate the same unnecessary.
Events that communicate the state of a mounted volume are left to the volume plugins.
Events that communicate the state of a mounted volume are left to the volume plugins.
...
@@ -232,9 +232,9 @@ When a claim holder is finished with their data, they can delete their claim.
...
@@ -232,9 +232,9 @@ When a claim holder is finished with their data, they can delete their claim.
$ kubectl delete pvc myclaim-1
$ kubectl delete pvc myclaim-1
```
```
The ```PersistentVolumeClaimBinder``` will reconcile this by removing the claim reference from the PV and change the PVs status to 'Released'.
The ```PersistentVolumeClaimBinder``` will reconcile this by removing the claim reference from the PV and change the PVs status to 'Released'.
Admins can script the recycling of released volumes. Future dynamic provisioners will understand how a volume should be recycled.
Admins can script the recycling of released volumes. Future dynamic provisioners will understand how a volume should be recycled.
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
...
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
# Design Principles
# Design Principles
Principles to follow when extending Kubernetes.
Principles to follow when extending Kubernetes.
## API
## API
...
@@ -44,14 +44,14 @@ See also the [API conventions](../devel/api-conventions.md).
...
@@ -44,14 +44,14 @@ See also the [API conventions](../devel/api-conventions.md).
* The control plane should be transparent -- there are no hidden internal APIs.
* The control plane should be transparent -- there are no hidden internal APIs.
* The cost of API operations should be proportional to the number of objects intentionally operated upon. Therefore, common filtered lookups must be indexed. Beware of patterns of multiple API calls that would incur quadratic behavior.
* The cost of API operations should be proportional to the number of objects intentionally operated upon. Therefore, common filtered lookups must be indexed. Beware of patterns of multiple API calls that would incur quadratic behavior.
* Object status must be 100% reconstructable by observation. Any history kept must be just an optimization and not required for correct operation.
* Object status must be 100% reconstructable by observation. Any history kept must be just an optimization and not required for correct operation.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add them. If you must have them, don't enforce them atomically in master components, that is contention-prone and doesn't provide a recovery path in the case of a bug allowing the invariant to be violated. Instead, provide a series of checks to reduce the probability of a violation, and make every component involved able to recover from an invariant violation.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add them. If you must have them, don't enforce them atomically in master components, that is contention-prone and doesn't provide a recovery path in the case of a bug allowing the invariant to be violated. Instead, provide a series of checks to reduce the probability of a violation, and make every component involved able to recover from an invariant violation.
* Low-level APIs should be designed for control by higher-level systems. Higher-level APIs should be intent-oriented (think SLOs) rather than implementation-oriented (think control knobs).
* Low-level APIs should be designed for control by higher-level systems. Higher-level APIs should be intent-oriented (think SLOs) rather than implementation-oriented (think control knobs).
## Control logic
## Control logic
* Functionality must be *level-based*, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Edge-triggered behavior must be just an optimization.
* Functionality must be *level-based*, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Edge-triggered behavior must be just an optimization.
* Assume an open world: continually verify assumptions and gracefully adapt to external events and/or actors. Example: we allow users to kill pods under control of a replication controller; it just replaces them.
* Assume an open world: continually verify assumptions and gracefully adapt to external events and/or actors. Example: we allow users to kill pods under control of a replication controller; it just replaces them.
* Do not define comprehensive state machines for objects with behaviors associated with state transitions and/or "assumed" states that cannot be ascertained by observation.
* Do not define comprehensive state machines for objects with behaviors associated with state transitions and/or "assumed" states that cannot be ascertained by observation.
* Don't assume a component's decisions will not be overridden or rejected, nor for the component to always understand why. For example, etcd may reject writes. Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry, but back off and/or make alternative decisions.
* Don't assume a component's decisions will not be overridden or rejected, nor for the component to always understand why. For example, etcd may reject writes. Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry, but back off and/or make alternative decisions.
* Components should be self-healing. For example, if you must keep some state (e.g., cache) the content needs to be periodically refreshed, so that if an item does get erroneously stored or a deletion event is missed etc, it will be soon fixed, ideally on timescales that are shorter than what will attract attention from humans.
* Components should be self-healing. For example, if you must keep some state (e.g., cache) the content needs to be periodically refreshed, so that if an item does get erroneously stored or a deletion event is missed etc, it will be soon fixed, ideally on timescales that are shorter than what will attract attention from humans.
* Component behavior should degrade gracefully. Prioritize actions so that the most important activities can continue to function even when overloaded and/or in states of partial failure.
* Component behavior should degrade gracefully. Prioritize actions so that the most important activities can continue to function even when overloaded and/or in states of partial failure.
...
@@ -61,7 +61,7 @@ See also the [API conventions](../devel/api-conventions.md).
...
@@ -61,7 +61,7 @@ See also the [API conventions](../devel/api-conventions.md).
* Only the apiserver should communicate with etcd/store, and not other components (scheduler, kubelet, etc.).
* Only the apiserver should communicate with etcd/store, and not other components (scheduler, kubelet, etc.).
* Compromising a single node shouldn't compromise the cluster.
* Compromising a single node shouldn't compromise the cluster.
* Components should continue to do what they were last told in the absence of new instructions (e.g., due to network partition or component outage).
* Components should continue to do what they were last told in the absence of new instructions (e.g., due to network partition or component outage).
* All components should keep all relevant state in memory all the time. The apiserver should write through to etcd/store, other components should write through to the apiserver, and they should watch for updates made by other clients.
* All components should keep all relevant state in memory all the time. The apiserver should write through to etcd/store, other components should write through to the apiserver, and they should watch for updates made by other clients.
A Kubernetes _resource_ is something that can be requested by, allocated to, or consumed by a pod or container. Examples include memory (RAM), CPU, disk-time, and network bandwidth.
A Kubernetes _resource_ is something that can be requested by, allocated to, or consumed by a pod or container. Examples include memory (RAM), CPU, disk-time, and network bandwidth.
Once resources on a node have been allocated to one pod, they should not be allocated to another until that pod is removed or exits. This means that Kubernetes schedulers should ensure that the sum of the resources allocated (requested and granted) to its pods never exceeds the usable capacity of the node. Testing whether a pod will fit on a node is called _feasibility checking_.
Once resources on a node have been allocated to one pod, they should not be allocated to another until that pod is removed or exits. This means that Kubernetes schedulers should ensure that the sum of the resources allocated (requested and granted) to its pods never exceeds the usable capacity of the node. Testing whether a pod will fit on a node is called _feasibility checking_.
Note that the resource model currently prohibits over-committing resources; we will want to relax that restriction later.
Note that the resource model currently prohibits over-committing resources; we will want to relax that restriction later.
...
@@ -70,7 +70,7 @@ For future reference, note that some resources, such as CPU and network bandwidt
...
@@ -70,7 +70,7 @@ For future reference, note that some resources, such as CPU and network bandwidt
### Resource quantities
### Resource quantities
Initially, all Kubernetes resource types are _quantitative_, and have an associated _unit_ for quantities of the associated resource (e.g., bytes for memory, bytes per seconds for bandwidth, instances for software licences). The units will always be a resource type's natural base units (e.g., bytes, not MB), to avoid confusion between binary and decimal multipliers and the underlying unit multiplier (e.g., is memory measured in MiB, MB, or GB?).
Initially, all Kubernetes resource types are _quantitative_, and have an associated _unit_ for quantities of the associated resource (e.g., bytes for memory, bytes per seconds for bandwidth, instances for software licences). The units will always be a resource type's natural base units (e.g., bytes, not MB), to avoid confusion between binary and decimal multipliers and the underlying unit multiplier (e.g., is memory measured in MiB, MB, or GB?).
Resource quantities can be added and subtracted: for example, a node has a fixed quantity of each resource type that can be allocated to pods/containers; once such an allocation has been made, the allocated resources cannot be made available to other pods/containers without over-committing the resources.
Resource quantities can be added and subtracted: for example, a node has a fixed quantity of each resource type that can be allocated to pods/containers; once such an allocation has been made, the allocated resources cannot be made available to other pods/containers without over-committing the resources.
...
@@ -110,7 +110,7 @@ resourceCapacitySpec: [
...
@@ -110,7 +110,7 @@ resourceCapacitySpec: [
```
```
Where:
Where:
* _total_: the total allocatable resources of a node. Initially, the resources at a given scope will bound the resources of the sum of inner scopes.
* _total_: the total allocatable resources of a node. Initially, the resources at a given scope will bound the resources of the sum of inner scopes.
#### Notes
#### Notes
...
@@ -194,7 +194,7 @@ The following are planned future extensions to the resource model, included here
...
@@ -194,7 +194,7 @@ The following are planned future extensions to the resource model, included here
Because resource usage and related metrics change continuously, need to be tracked over time (i.e., historically), can be characterized in a variety of ways, and are fairly voluminous, we will not include usage in core API objects, such as [Pods](../user-guide/pods.md) and Nodes, but will provide separate APIs for accessing and managing that data. See the Appendix for possible representations of usage data, but the representation we'll use is TBD.
Because resource usage and related metrics change continuously, need to be tracked over time (i.e., historically), can be characterized in a variety of ways, and are fairly voluminous, we will not include usage in core API objects, such as [Pods](../user-guide/pods.md) and Nodes, but will provide separate APIs for accessing and managing that data. See the Appendix for possible representations of usage data, but the representation we'll use is TBD.
Singleton values for observed and predicted future usage will rapidly prove inadequate, so we will support the following structure for extended usage information:
Singleton values for observed and predicted future usage will rapidly prove inadequate, so we will support the following structure for extended usage information:
```yaml
```yaml
resourceStatus:[
resourceStatus:[
...
@@ -223,7 +223,7 @@ where a `<CPU-info>` or `<memory-info>` structure looks like this:
...
@@ -223,7 +223,7 @@ where a `<CPU-info>` or `<memory-info>` structure looks like this:
```
```
All parts of this structure are optional, although we strongly encourage including quantities for 50, 90, 95, 99, 99.5, and 99.9 percentiles. _[In practice, it will be important to include additional info such as the length of the time window over which the averages are calculated, the confidence level, and information-quality metrics such as the number of dropped or discarded data points.]_
All parts of this structure are optional, although we strongly encourage including quantities for 50, 90, 95, 99, 99.5, and 99.9 percentiles. _[In practice, it will be important to include additional info such as the length of the time window over which the averages are calculated, the confidence level, and information-quality metrics such as the number of dropped or discarded data points.]_
@@ -34,7 +34,7 @@ Documentation for other releases can be found at
...
@@ -34,7 +34,7 @@ Documentation for other releases can be found at
## Abstract
## Abstract
A proposal for the distribution of [secrets](../user-guide/secrets.md)(passwords, keys, etc) to the Kubelet and to
A proposal for the distribution of [secrets](../user-guide/secrets.md)(passwords, keys, etc) to the Kubelet and to
containers inside Kubernetes using a custom [volume](../user-guide/volumes.md#secrets) type. See the [secrets example](../user-guide/secrets/) for more information.
containers inside Kubernetes using a custom [volume](../user-guide/volumes.md#secrets) type. See the [secrets example](../user-guide/secrets/) for more information.
## Motivation
## Motivation
...
@@ -117,7 +117,7 @@ which consumes this type of secret, the Kubelet may take a number of actions:
...
@@ -117,7 +117,7 @@ which consumes this type of secret, the Kubelet may take a number of actions:
1. Expose the secret in a `.kubernetes_auth` file in a well-known location in the container's
1. Expose the secret in a `.kubernetes_auth` file in a well-known location in the container's
file system
file system
2. Configure that node's `kube-proxy` to decorate HTTP requests from that pod to the
2. Configure that node's `kube-proxy` to decorate HTTP requests from that pod to the
`kubernetes-master` service with the auth token, e. g. by adding a header to the request
`kubernetes-master` service with the auth token, e. g. by adding a header to the request
(see the [LOAS Daemon](https://github.com/GoogleCloudPlatform/kubernetes/issues/2209) proposal)
(see the [LOAS Daemon](https://github.com/GoogleCloudPlatform/kubernetes/issues/2209) proposal)
...
@@ -146,7 +146,7 @@ We should consider what the best way to allow this is; there are a few different
...
@@ -146,7 +146,7 @@ We should consider what the best way to allow this is; there are a few different
export MY_SECRET_ENV=MY_SECRET_VALUE
export MY_SECRET_ENV=MY_SECRET_VALUE
The user could `source` the file at `/etc/secrets/my-secret` prior to executing the command for
The user could `source` the file at `/etc/secrets/my-secret` prior to executing the command for
the image either inline in the command or in an init script,
the image either inline in the command or in an init script,
2. Give secrets an attribute that allows users to express the intent that the platform should
2. Give secrets an attribute that allows users to express the intent that the platform should
generate the above syntax in the file used to present a secret. The user could consume these
generate the above syntax in the file used to present a secret. The user could consume these
@@ -48,55 +48,55 @@ The problem of securing containers in Kubernetes has come up [before](https://gi
...
@@ -48,55 +48,55 @@ The problem of securing containers in Kubernetes has come up [before](https://gi
### Container isolation
### Container isolation
In order to improve container isolation from host and other containers running on the host, containers should only be
In order to improve container isolation from host and other containers running on the host, containers should only be
granted the access they need to perform their work. To this end it should be possible to take advantage of Docker
granted the access they need to perform their work. To this end it should be possible to take advantage of Docker
features such as the ability to [add or remove capabilities](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration) and [assign MCS labels](https://docs.docker.com/reference/run/#security-configuration)
features such as the ability to [add or remove capabilities](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration) and [assign MCS labels](https://docs.docker.com/reference/run/#security-configuration)
to the container process.
to the container process.
Support for user namespaces has recently been [merged](https://github.com/docker/libcontainer/pull/304) into Docker's libcontainer project and should soon surface in Docker itself. It will make it possible to assign a range of unprivileged uids and gids from the host to each container, improving the isolation between host and container and between containers.
Support for user namespaces has recently been [merged](https://github.com/docker/libcontainer/pull/304) into Docker's libcontainer project and should soon surface in Docker itself. It will make it possible to assign a range of unprivileged uids and gids from the host to each container, improving the isolation between host and container and between containers.
### External integration with shared storage
### External integration with shared storage
In order to support external integration with shared storage, processes running in a Kubernetes cluster
In order to support external integration with shared storage, processes running in a Kubernetes cluster
should be able to be uniquely identified by their Unix UID, such that a chain of ownership can be established.
should be able to be uniquely identified by their Unix UID, such that a chain of ownership can be established.
Processes in pods will need to have consistent UID/GID/SELinux category labels in order to access shared disks.
Processes in pods will need to have consistent UID/GID/SELinux category labels in order to access shared disks.
## Constraints and Assumptions
## Constraints and Assumptions
* It is out of the scope of this document to prescribe a specific set
* It is out of the scope of this document to prescribe a specific set
of constraints to isolate containers from their host. Different use cases need different
of constraints to isolate containers from their host. Different use cases need different
settings.
settings.
* The concept of a security context should not be tied to a particular security mechanism or platform
* The concept of a security context should not be tied to a particular security mechanism or platform
(ie. SELinux, AppArmor)
(ie. SELinux, AppArmor)
* Applying a different security context to a scope (namespace or pod) requires a solution such as the one proposed for
* Applying a different security context to a scope (namespace or pod) requires a solution such as the one proposed for
[service accounts](service_accounts.md).
[service accounts](service_accounts.md).
## Use Cases
## Use Cases
In order of increasing complexity, following are example use cases that would
In order of increasing complexity, following are example use cases that would
be addressed with security contexts:
be addressed with security contexts:
1. Kubernetes is used to run a single cloud application. In order to protect
1. Kubernetes is used to run a single cloud application. In order to protect
nodes from containers:
nodes from containers:
* All containers run as a single non-root user
* All containers run as a single non-root user
* Privileged containers are disabled
* Privileged containers are disabled
* All containers run with a particular MCS label
* All containers run with a particular MCS label
* Kernel capabilities like CHOWN and MKNOD are removed from containers
* Kernel capabilities like CHOWN and MKNOD are removed from containers
2. Just like case #1, except that I have more than one application running on
2. Just like case #1, except that I have more than one application running on
the Kubernetes cluster.
the Kubernetes cluster.
* Each application is run in its own namespace to avoid name collisions
* Each application is run in its own namespace to avoid name collisions
* For each application a different uid and MCS label is used
* For each application a different uid and MCS label is used
3. Kubernetes is used as the base for a PAAS with
3. Kubernetes is used as the base for a PAAS with
multiple projects, each project represented by a namespace.
multiple projects, each project represented by a namespace.
* Each namespace is associated with a range of uids/gids on the node that
* Each namespace is associated with a range of uids/gids on the node that
are mapped to uids/gids on containers using linux user namespaces.
are mapped to uids/gids on containers using linux user namespaces.
* Certain pods in each namespace have special privileges to perform system
* Certain pods in each namespace have special privileges to perform system
actions such as talking back to the server for deployment, run docker
actions such as talking back to the server for deployment, run docker
builds, etc.
builds, etc.
* External NFS storage is assigned to each namespace and permissions set
* External NFS storage is assigned to each namespace and permissions set
using the range of uids/gids assigned to that namespace.
using the range of uids/gids assigned to that namespace.
## Proposed Design
## Proposed Design
...
@@ -109,7 +109,7 @@ to mutate Docker API calls in order to apply the security context.
...
@@ -109,7 +109,7 @@ to mutate Docker API calls in order to apply the security context.
It is recommended that this design be implemented in two phases:
It is recommended that this design be implemented in two phases:
1. Implement the security context provider extension point in the Kubelet
1. Implement the security context provider extension point in the Kubelet
so that a default security context can be applied on container run and creation.
so that a default security context can be applied on container run and creation.
2. Implement a security context structure that is part of a service account. The
2. Implement a security context structure that is part of a service account. The
default context provider can then be used to apply a security context based
default context provider can then be used to apply a security context based
...
@@ -137,7 +137,7 @@ type SecurityContextProvider interface {
...
@@ -137,7 +137,7 @@ type SecurityContextProvider interface {
}
}
```
```
If the value of the SecurityContextProvider field on the Kubelet is nil, the kubelet will create and run the container as it does today.
If the value of the SecurityContextProvider field on the Kubelet is nil, the kubelet will create and run the container as it does today.
@@ -173,11 +173,11 @@ Objects that contain both spec and status should not contain additional top-leve
...
@@ -173,11 +173,11 @@ Objects that contain both spec and status should not contain additional top-leve
##### Typical status properties
##### Typical status properties
***phase**: The phase is a simple, high-level summary of the phase of the lifecycle of an object. The phase should progress monotonically. Typical phase values are `Pending` (not yet fully physically realized), `Running` or `Active` (fully realized and active, but not necessarily operating correctly), and `Terminated` (no longer active), but may vary slightly for different types of objects. New phase values should not be added to existing objects in the future. Like other status fields, it must be possible to ascertain the lifecycle phase by observation. Additional details regarding the current phase may be contained in other fields.
***phase**: The phase is a simple, high-level summary of the phase of the lifecycle of an object. The phase should progress monotonically. Typical phase values are `Pending` (not yet fully physically realized), `Running` or `Active` (fully realized and active, but not necessarily operating correctly), and `Terminated` (no longer active), but may vary slightly for different types of objects. New phase values should not be added to existing objects in the future. Like other status fields, it must be possible to ascertain the lifecycle phase by observation. Additional details regarding the current phase may be contained in other fields.
***conditions**: Conditions represent orthogonal observations of an object's current state. Objects may report multiple conditions, and new types of conditions may be added in the future. Condition status values may be `True`, `False`, or `Unknown`. Unlike the phase, conditions are not expected to be monotonic -- their values may change back and forth. A typical condition type is `Ready`, which indicates the object was believed to be fully operational at the time it was last probed. Conditions may carry additional information, such as the last probe time or last transition time.
***conditions**: Conditions represent orthogonal observations of an object's current state. Objects may report multiple conditions, and new types of conditions may be added in the future. Condition status values may be `True`, `False`, or `Unknown`. Unlike the phase, conditions are not expected to be monotonic -- their values may change back and forth. A typical condition type is `Ready`, which indicates the object was believed to be fully operational at the time it was last probed. Conditions may carry additional information, such as the last probe time or last transition time.
TODO(@vishh): Reason and Message.
TODO(@vishh): Reason and Message.
Phases and conditions are observations and not, themselves, state machines, nor do we define comprehensive state machines for objects with behaviors associated with state transitions. The system is level-based and should assume an Open World. Additionally, new observations and details about these observations may be added over time.
Phases and conditions are observations and not, themselves, state machines, nor do we define comprehensive state machines for objects with behaviors associated with state transitions. The system is level-based and should assume an Open World. Additionally, new observations and details about these observations may be added over time.
In order to preserve extensibility, in the future, we intend to explicitly convey properties that users and components care about rather than requiring those properties to be inferred from observations.
In order to preserve extensibility, in the future, we intend to explicitly convey properties that users and components care about rather than requiring those properties to be inferred from observations.
...
@@ -376,7 +376,7 @@ Late-initializers should only make the following types of modifications:
...
@@ -376,7 +376,7 @@ Late-initializers should only make the following types of modifications:
- Adding keys to maps
- Adding keys to maps
- Adding values to arrays which have mergeable semantics (`patchStrategy:"merge"` attribute in
- Adding values to arrays which have mergeable semantics (`patchStrategy:"merge"` attribute in
the type definition).
the type definition).
These conventions:
These conventions:
1. allow a user (with sufficient privilege) to override any system-default behaviors by setting
1. allow a user (with sufficient privilege) to override any system-default behaviors by setting
the fields that would otherwise have been defaulted.
the fields that would otherwise have been defaulted.
@@ -61,7 +61,7 @@ Maintainers will do merges of appropriately reviewed-and-approved changes during
...
@@ -61,7 +61,7 @@ Maintainers will do merges of appropriately reviewed-and-approved changes during
There may be discussion an even approvals granted outside of the above hours, but merges will generally be deferred.
There may be discussion an even approvals granted outside of the above hours, but merges will generally be deferred.
If a PR is considered complex or controversial, the merge of that PR should be delayed to give all interested parties in all timezones the opportunity to provide feedback. Concretely, this means that such PRs should be held for 24
If a PR is considered complex or controversial, the merge of that PR should be delayed to give all interested parties in all timezones the opportunity to provide feedback. Concretely, this means that such PRs should be held for 24
hours before merging. Of course "complex" and "controversial" are left to the judgment of the people involved, but we trust that part of being a committer is the judgment required to evaluate such things honestly, and not be
hours before merging. Of course "complex" and "controversial" are left to the judgment of the people involved, but we trust that part of being a committer is the judgment required to evaluate such things honestly, and not be
motivated by your desire (or your cube-mate's desire) to get their code merged. Also see "Holds" below, any reviewer can issue a "hold" to indicate that the PR is in fact complicated or complex and deserves further review.
motivated by your desire (or your cube-mate's desire) to get their code merged. Also see "Holds" below, any reviewer can issue a "hold" to indicate that the PR is in fact complicated or complex and deserves further review.
@@ -44,7 +44,7 @@ The purpose of filtering the nodes is to filter out the nodes that do not meet c
...
@@ -44,7 +44,7 @@ The purpose of filtering the nodes is to filter out the nodes that do not meet c
-`PodFitsPorts`: Check if any HostPort required by the Pod is already occupied on the node.
-`PodFitsPorts`: Check if any HostPort required by the Pod is already occupied on the node.
-`PodFitsHost`: Filter out all nodes except the one specified in the PodSpec's NodeName field.
-`PodFitsHost`: Filter out all nodes except the one specified in the PodSpec's NodeName field.
-`PodSelectorMatches`: Check if the labels of the node match the labels specified in the Pod's `nodeSelector` field ([Here](../user-guide/node-selection/) is an example of how to use `nodeSelector` field).
-`PodSelectorMatches`: Check if the labels of the node match the labels specified in the Pod's `nodeSelector` field ([Here](../user-guide/node-selection/) is an example of how to use `nodeSelector` field).
-`CheckNodeLabelPresence`: Check if all the specified labels exist on a node or not, regardless of the value.
-`CheckNodeLabelPresence`: Check if all the specified labels exist on a node or not, regardless of the value.
The details of the above predicates can be found in [plugin/pkg/scheduler/algorithm/predicates/predicates.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/predicates/predicates.go). All predicates mentioned above can be used in combination to perform a sophisticated filtering policy. Kubernetes uses some, but not all, of these predicates by default. You can see which ones are used by default in [plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go).
The details of the above predicates can be found in [plugin/pkg/scheduler/algorithm/predicates/predicates.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/predicates/predicates.go). All predicates mentioned above can be used in combination to perform a sophisticated filtering policy. Kubernetes uses some, but not all, of these predicates by default. You can see which ones are used by default in [plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go).
...
@@ -53,7 +53,7 @@ The details of the above predicates can be found in [plugin/pkg/scheduler/algori
...
@@ -53,7 +53,7 @@ The details of the above predicates can be found in [plugin/pkg/scheduler/algori
The filtered nodes are considered suitable to host the Pod, and it is often that there are more than one nodes remaining. Kubernetes prioritizes the remaining nodes to find the "best" one for the Pod. The prioritization is performed by a set of priority functions. For each remaining node, a priority function gives a score which scales from 0-10 with 10 representing for "most preferred" and 0 for "least preferred". Each priority function is weighted by a positive number and the final score of each node is calculated by adding up all the weighted scores. For example, suppose there are two priority functions, `priorityFunc1` and `priorityFunc2` with weighting factors `weight1` and `weight2` respectively, the final score of some NodeA is:
The filtered nodes are considered suitable to host the Pod, and it is often that there are more than one nodes remaining. Kubernetes prioritizes the remaining nodes to find the "best" one for the Pod. The prioritization is performed by a set of priority functions. For each remaining node, a priority function gives a score which scales from 0-10 with 10 representing for "most preferred" and 0 for "least preferred". Each priority function is weighted by a positive number and the final score of each node is calculated by adding up all the weighted scores. For example, suppose there are two priority functions, `priorityFunc1` and `priorityFunc2` with weighting factors `weight1` and `weight2` respectively, the final score of some NodeA is:
After the scores of all nodes are calculated, the node with highest score is chosen as the host of the Pod. If there are more than one nodes with equal highest scores, a random one among them is chosen.
After the scores of all nodes are calculated, the node with highest score is chosen as the host of the Pod. If there are more than one nodes with equal highest scores, a random one among them is chosen.
Currently, Kubernetes scheduler provides some practical priority functions, including:
Currently, Kubernetes scheduler provides some practical priority functions, including:
This configuration file will now boot from local drive but have the option to PXE image CoreOS.
This configuration file will now boot from local drive but have the option to PXE image CoreOS.
## DHCP configuration
## DHCP configuration
This section covers configuring the DHCP server to hand out our new images. In this case we are assuming that there are other servers that will boot alongside other images.
This section covers configuring the DHCP server to hand out our new images. In this case we are assuming that there are other servers that will boot alongside other images.
...
@@ -186,7 +186,7 @@ This section covers configuring the DHCP server to hand out our new images. In t
...
@@ -186,7 +186,7 @@ This section covers configuring the DHCP server to hand out our new images. In t
next-server 10.20.30.242;
next-server 10.20.30.242;
option broadcast-address 10.20.30.255;
option broadcast-address 10.20.30.255;
filename "<other default image>";
filename "<other default image>";
...
...
# http://www.syslinux.org/wiki/index.php/PXELINUX
# http://www.syslinux.org/wiki/index.php/PXELINUX
host core_os_master {
host core_os_master {
...
@@ -194,7 +194,7 @@ This section covers configuring the DHCP server to hand out our new images. In t
...
@@ -194,7 +194,7 @@ This section covers configuring the DHCP server to hand out our new images. In t
option routers 10.20.30.1;
option routers 10.20.30.1;
fixed-address 10.20.30.40;
fixed-address 10.20.30.40;
option domain-name-servers 10.20.30.242;
option domain-name-servers 10.20.30.242;
filename "/pxelinux.0";
filename "/pxelinux.0";
}
}
host core_os_slave {
host core_os_slave {
hardware ethernet d0:00:67:13:0d:01;
hardware ethernet d0:00:67:13:0d:01;
...
@@ -217,7 +217,7 @@ We will be specifying the node configuration later in the guide.
...
@@ -217,7 +217,7 @@ We will be specifying the node configuration later in the guide.
## Kubernetes
## Kubernetes
To deploy our configuration we need to create an `etcd` master. To do so we want to pxe CoreOS with a specific cloud-config.yml. There are two options we have here.
To deploy our configuration we need to create an `etcd` master. To do so we want to pxe CoreOS with a specific cloud-config.yml. There are two options we have here.
1. Is to template the cloud config file and programmatically create new static configs for different cluster setups.
1. Is to template the cloud config file and programmatically create new static configs for different cluster setups.
2. Have a service discovery protocol running in our stack to do auto discovery.
2. Have a service discovery protocol running in our stack to do auto discovery.
...
@@ -427,7 +427,7 @@ On the PXE server make and fill in the variables `vi /var/www/html/coreos/pxe-cl
...
@@ -427,7 +427,7 @@ On the PXE server make and fill in the variables `vi /var/www/html/coreos/pxe-cl
--logtostderr=true
--logtostderr=true
Restart=always
Restart=always
RestartSec=10
RestartSec=10
- name: kube-controller-manager.service
- name: kube-controller-manager.service
command: start
command: start
content: |
content: |
[Unit]
[Unit]
...
@@ -535,7 +535,7 @@ On the PXE server make and fill in the variables `vi /var/www/html/coreos/pxe-cl
...
@@ -535,7 +535,7 @@ On the PXE server make and fill in the variables `vi /var/www/html/coreos/pxe-cl
command: start
command: start
content: |
content: |
[Unit]
[Unit]
After=network-online.target
After=network-online.target
Wants=network-online.target
Wants=network-online.target
Description=flannel is an etcd backed overlay network for containers
Description=flannel is an etcd backed overlay network for containers
* Install Kubernetes on all hosts - fed-{master,node}. This will also pull in docker. Also install etcd on fed-master. This guide has been tested with kubernetes-0.18 and beyond.
* Install Kubernetes on all hosts - fed-{master,node}. This will also pull in docker. Also install etcd on fed-master. This guide has been tested with kubernetes-0.18 and beyond.
* The [--enablerepo=update-testing](https://fedoraproject.org/wiki/QA:Updates_Testing) directive in the yum command below will ensure that the most recent Kubernetes version that is scheduled for pre-release will be installed. This should be a more recent version than the Fedora "stable" release for Kubernetes that you would get without adding the directive.
* The [--enablerepo=update-testing](https://fedoraproject.org/wiki/QA:Updates_Testing) directive in the yum command below will ensure that the most recent Kubernetes version that is scheduled for pre-release will be installed. This should be a more recent version than the Fedora "stable" release for Kubernetes that you would get without adding the directive.
* If you want the very latest Kubernetes release [you can download and yum install the RPM directly from Fedora Koji](http://koji.fedoraproject.org/koji/packageinfo?packageID=19202) instead of using the yum install command below.
* If you want the very latest Kubernetes release [you can download and yum install the RPM directly from Fedora Koji](http://koji.fedoraproject.org/koji/packageinfo?packageID=19202) instead of using the yum install command below.
The Docker daemon must be started and told to use the already configured cbr0 instead of using the usual docker0, as well as disabling ip-masquerading and modification of the ip-tables.
The Docker daemon must be started and told to use the already configured cbr0 instead of using the usual docker0, as well as disabling ip-masquerading and modification of the ip-tables.
1.) Edit the ubuntu-15.04 docker.service for systemd at: `/lib/systemd/system/docker.service`
1.) Edit the ubuntu-15.04 docker.service for systemd at: `/lib/systemd/system/docker.service`
@@ -49,7 +49,7 @@ This document describes how to deploy Kubernetes on ubuntu nodes, including 1 Ku
...
@@ -49,7 +49,7 @@ This document describes how to deploy Kubernetes on ubuntu nodes, including 1 Ku
## Prerequisites
## Prerequisites
*1 The nodes have installed docker version 1.2+ and bridge-utils to manipulate linux bridge*
*1 The nodes have installed docker version 1.2+ and bridge-utils to manipulate linux bridge*
*2 All machines can communicate with each other, no need to connect Internet (should use private docker registry in this case)*
*2 All machines can communicate with each other, no need to connect Internet (should use private docker registry in this case)*
...
@@ -57,7 +57,7 @@ This document describes how to deploy Kubernetes on ubuntu nodes, including 1 Ku
...
@@ -57,7 +57,7 @@ This document describes how to deploy Kubernetes on ubuntu nodes, including 1 Ku
*4 Dependencies of this guide: etcd-2.0.12, flannel-0.4.0, k8s-1.0.1, but it may work with higher versions*
*4 Dependencies of this guide: etcd-2.0.12, flannel-0.4.0, k8s-1.0.1, but it may work with higher versions*
*5 All the remote servers can be ssh logged in without a password by using key authentication*
*5 All the remote servers can be ssh logged in without a password by using key authentication*
### Starting a Cluster
### Starting a Cluster
...
@@ -80,7 +80,7 @@ Please make sure that there are `kube-apiserver`, `kube-controller-manager`, `ku
...
@@ -80,7 +80,7 @@ Please make sure that there are `kube-apiserver`, `kube-controller-manager`, `ku
An example cluster is listed as below:
An example cluster is listed as below:
| IP Address|Role |
| IP Address|Role |
|---------|------|
|---------|------|
|10.10.103.223| node |
|10.10.103.223| node |
|10.10.103.162| node |
|10.10.103.162| node |
...
@@ -112,13 +112,13 @@ The `SERVICE_CLUSTER_IP_RANGE` variable defines the Kubernetes service IP range.
...
@@ -112,13 +112,13 @@ The `SERVICE_CLUSTER_IP_RANGE` variable defines the Kubernetes service IP range.
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
The `FLANNEL_NET` variable defines the IP range used for flannel overlay network, should not conflict with above `SERVICE_CLUSTER_IP_RANGE`.
The `FLANNEL_NET` variable defines the IP range used for flannel overlay network, should not conflict with above `SERVICE_CLUSTER_IP_RANGE`.
After all the above variables being set correctly, we can use following command in cluster/ directory to bring up the whole cluster.
After all the above variables being set correctly, we can use following command in cluster/ directory to bring up the whole cluster.
`$ KUBERNETES_PROVIDER=ubuntu ./kube-up.sh`
`$ KUBERNETES_PROVIDER=ubuntu ./kube-up.sh`
The scripts automatically scp binaries and config files to all the machines and start the k8s service on them. The only thing you need to do is to type the sudo password when promoted. The current machine name is shown below, so you will not type in the wrong password.
The scripts automatically scp binaries and config files to all the machines and start the k8s service on them. The only thing you need to do is to type the sudo password when promoted. The current machine name is shown below, so you will not type in the wrong password.
...
@@ -135,9 +135,9 @@ If all things goes right, you will see the below message from console
...
@@ -135,9 +135,9 @@ If all things goes right, you will see the below message from console
**All done !**
**All done !**
You can also use `kubectl` command to see if the newly created k8s is working correctly. The `kubectl` binary is under the `cluster/ubuntu/binaries` directory. You can move it into your PATH. Then you can use the below command smoothly.
You can also use `kubectl` command to see if the newly created k8s is working correctly. The `kubectl` binary is under the `cluster/ubuntu/binaries` directory. You can move it into your PATH. Then you can use the below command smoothly.
For example, use `$ kubectl get nodes` to see if all your nodes are in ready status. It may take some time for the nodes ready to use like below.
For example, use `$ kubectl get nodes` to see if all your nodes are in ready status. It may take some time for the nodes ready to use like below.
```console
```console
NAME LABELS STATUS
NAME LABELS STATUS
...
@@ -192,19 +192,19 @@ We are working on these features which we'd like to let everybody know:
...
@@ -192,19 +192,19 @@ We are working on these features which we'd like to let everybody know:
#### Trouble Shooting
#### Trouble Shooting
Generally, what this approach did is quite simple:
Generally, what this approach did is quite simple:
1. Download and copy binaries and configuration files to proper directories on every node
1. Download and copy binaries and configuration files to proper directories on every node
2. Configure `etcd` using IPs based on input from user
2. Configure `etcd` using IPs based on input from user
3. Create and start flannel network
3. Create and start flannel network
So, if you see a problem, **check etcd configuration first**
So, if you see a problem, **check etcd configuration first**
Please try:
Please try:
1. Check `/var/log/upstart/etcd.log` for suspicious etcd log
1. Check `/var/log/upstart/etcd.log` for suspicious etcd log
2. Check `/etc/default/etcd`, as we do not have much input validation, a right config should be like:
2. Check `/etc/default/etcd`, as we do not have much input validation, a right config should be like:
@@ -89,11 +89,11 @@ loosely speaking, a cluster can be thought of as running in a single
...
@@ -89,11 +89,11 @@ loosely speaking, a cluster can be thought of as running in a single
data center, or cloud provider availability zone, a more precise
data center, or cloud provider availability zone, a more precise
definition is that each cluster provides:
definition is that each cluster provides:
1. a single Kubernetes API entry point,
1. a single Kubernetes API entry point,
1. a consistent, cluster-wide resource naming scheme
1. a consistent, cluster-wide resource naming scheme
1. a scheduling/container placement domain
1. a scheduling/container placement domain
1. a service network routing domain
1. a service network routing domain
1. (in future) an authentication and authorization model.
1. (in future) an authentication and authorization model.
1. ....
1. ....
The above in turn imply the need for a relatively performant, reliable
The above in turn imply the need for a relatively performant, reliable
...
@@ -220,7 +220,7 @@ the multi-cloud provider implementation should just work for a single
...
@@ -220,7 +220,7 @@ the multi-cloud provider implementation should just work for a single
cloud provider). Propose high-level design catering for both, with
cloud provider). Propose high-level design catering for both, with
initial implementation targeting single cloud provider only.
initial implementation targeting single cloud provider only.
**Clarifying questions:**
**Clarifying questions:**
**How does global external service discovery work?** In the steady
**How does global external service discovery work?** In the steady
state, which external clients connect to which clusters? GeoDNS or
state, which external clients connect to which clusters? GeoDNS or
similar? What is the tolerable failover latency if a cluster goes
similar? What is the tolerable failover latency if a cluster goes
...
@@ -266,8 +266,8 @@ Doing nothing (i.e. forcing users to choose between 1 and 2 on their
...
@@ -266,8 +266,8 @@ Doing nothing (i.e. forcing users to choose between 1 and 2 on their
own) is probably an OK starting point. Kubernetes autoscaling can get
own) is probably an OK starting point. Kubernetes autoscaling can get
us to 3 at some later date.
us to 3 at some later date.
Up to this point, this use case ("Unavailability Zones") seems materially different from all the others above. It does not require dynamic cross-cluster service migration (we assume that the service is already running in more than one cluster when the failure occurs). Nor does it necessarily involve cross-cluster service discovery or location affinity. As a result, I propose that we address this use case somewhat independently of the others (although I strongly suspect that it will become substantially easier once we've solved the others).
Up to this point, this use case ("Unavailability Zones") seems materially different from all the others above. It does not require dynamic cross-cluster service migration (we assume that the service is already running in more than one cluster when the failure occurs). Nor does it necessarily involve cross-cluster service discovery or location affinity. As a result, I propose that we address this use case somewhat independently of the others (although I strongly suspect that it will become substantially easier once we've solved the others).
All of the above (regarding "Unavailibility Zones") refers primarily
All of the above (regarding "Unavailibility Zones") refers primarily
to already-running user-facing services, and minimizing the impact on
to already-running user-facing services, and minimizing the impact on
end users of those services becoming unavailable in a given cluster.
end users of those services becoming unavailable in a given cluster.
...
@@ -322,7 +322,7 @@ location affinity:
...
@@ -322,7 +322,7 @@ location affinity:
(other than the source of YouTube videos, which is assumed to be
(other than the source of YouTube videos, which is assumed to be
equally remote from all clusters in this example). Each pod can be
equally remote from all clusters in this example). Each pod can be
scheduled independently, in any cluster, and moved at any time.
scheduled independently, in any cluster, and moved at any time.
1.**"Preferentially Coupled"**: Somewhere between Coupled and Decoupled. These applications prefer to have all of their pods located in the same cluster (e.g. for failure correlation, network latency or bandwidth cost reasons), but can tolerate being partitioned for "short" periods of time (for example while migrating the application from one cluster to another). Most small to medium sized LAMP stacks with not-very-strict latency goals probably fall into this category (provided that they use sane service discovery and reconnect-on-fail, which they need to do anyway to run effectively, even in a single Kubernetes cluster).
1.**"Preferentially Coupled"**: Somewhere between Coupled and Decoupled. These applications prefer to have all of their pods located in the same cluster (e.g. for failure correlation, network latency or bandwidth cost reasons), but can tolerate being partitioned for "short" periods of time (for example while migrating the application from one cluster to another). Most small to medium sized LAMP stacks with not-very-strict latency goals probably fall into this category (provided that they use sane service discovery and reconnect-on-fail, which they need to do anyway to run effectively, even in a single Kubernetes cluster).
And then there's what I'll call _absolute_ location affinity. Some
And then there's what I'll call _absolute_ location affinity. Some
applications are required to run in bounded geographical or network
applications are required to run in bounded geographical or network
...
@@ -341,7 +341,7 @@ of our users are in Western Europe, U.S. West Coast" etc).
...
@@ -341,7 +341,7 @@ of our users are in Western Europe, U.S. West Coast" etc).
## Cross-cluster service discovery
## Cross-cluster service discovery
I propose having pods use standard discovery methods used by external clients of Kubernetes applications (i.e. DNS). DNS might resolve to a public endpoint in the local or a remote cluster. Other than Strictly Coupled applications, software should be largely oblivious of which of the two occurs.
I propose having pods use standard discovery methods used by external clients of Kubernetes applications (i.e. DNS). DNS might resolve to a public endpoint in the local or a remote cluster. Other than Strictly Coupled applications, software should be largely oblivious of which of the two occurs.
_Aside:_ How do we avoid "tromboning" through an external VIP when DNS
_Aside:_ How do we avoid "tromboning" through an external VIP when DNS
resolves to a public IP on the local cluster? Strictly speaking this
resolves to a public IP on the local cluster? Strictly speaking this
would be an optimization, and probably only matters to high bandwidth,
would be an optimization, and probably only matters to high bandwidth,
...
@@ -384,15 +384,15 @@ such events include:
...
@@ -384,15 +384,15 @@ such events include:
1. A change of scheduling policy ("we no longer use cloud provider X").
1. A change of scheduling policy ("we no longer use cloud provider X").
1. A change of resource pricing ("cloud provider Y dropped their prices - lets migrate there").
1. A change of resource pricing ("cloud provider Y dropped their prices - lets migrate there").
Strictly Decoupled applications can be trivially moved, in part or in whole, one pod at a time, to one or more clusters.
Strictly Decoupled applications can be trivially moved, in part or in whole, one pod at a time, to one or more clusters.
For Preferentially Decoupled applications, the federation system must first locate a single cluster with sufficient capacity to accommodate the entire application, then reserve that capacity, and incrementally move the application, one (or more) resources at a time, over to the new cluster, within some bounded time period (and possibly within a predefined "maintenance" window).
For Preferentially Decoupled applications, the federation system must first locate a single cluster with sufficient capacity to accommodate the entire application, then reserve that capacity, and incrementally move the application, one (or more) resources at a time, over to the new cluster, within some bounded time period (and possibly within a predefined "maintenance" window).
Strictly Coupled applications (with the exception of those deemed
Strictly Coupled applications (with the exception of those deemed
completely immovable) require the federation system to:
completely immovable) require the federation system to:
1. start up an entire replica application in the destination cluster
1. start up an entire replica application in the destination cluster
1. copy persistent data to the new application instance
1. copy persistent data to the new application instance
1. switch traffic across
1. switch traffic across
1. tear down the original application instance
1. tear down the original application instance
It is proposed that support for automated migration of Strictly Coupled applications be
It is proposed that support for automated migration of Strictly Coupled applications be
deferred to a later date.
deferred to a later date.
...
@@ -422,11 +422,11 @@ TBD: All very hand-wavey still, but some initial thoughts to get the conversatio
...
@@ -422,11 +422,11 @@ TBD: All very hand-wavey still, but some initial thoughts to get the conversatio
## Ubernetes API
## Ubernetes API
This looks a lot like the existing Kubernetes API but is explicitly multi-cluster.
This looks a lot like the existing Kubernetes API but is explicitly multi-cluster.
+ Clusters become first class objects, which can be registered, listed, described, deregistered etc via the API.
+ Clusters become first class objects, which can be registered, listed, described, deregistered etc via the API.
+ Compute resources can be explicitly requested in specific clusters, or automatically scheduled to the "best" cluster by Ubernetes (by a pluggable Policy Engine).
+ Compute resources can be explicitly requested in specific clusters, or automatically scheduled to the "best" cluster by Ubernetes (by a pluggable Policy Engine).
+ There is a federated equivalent of a replication controller type, which is multicluster-aware, and delegates to cluster-specific replication controllers as required (e.g. a federated RC for n replicas might simply spawn multiple replication controllers in different clusters to do the hard work).
+ There is a federated equivalent of a replication controller type, which is multicluster-aware, and delegates to cluster-specific replication controllers as required (e.g. a federated RC for n replicas might simply spawn multiple replication controllers in different clusters to do the hard work).
+ These federated replication controllers (and in fact all the
+ These federated replication controllers (and in fact all the
services comprising the Ubernetes Control Plane) have to run
services comprising the Ubernetes Control Plane) have to run
somewhere. For high availability Ubernetes deployments, these
somewhere. For high availability Ubernetes deployments, these
@@ -33,21 +33,21 @@ Documentation for other releases can be found at
...
@@ -33,21 +33,21 @@ Documentation for other releases can be found at
# Security
# Security
If you believe you have discovered a vulnerability or a have a security incident to report, please follow the steps below. This applies to Kubernetes releases v1.0 or later.
If you believe you have discovered a vulnerability or a have a security incident to report, please follow the steps below. This applies to Kubernetes releases v1.0 or later.
To watch for security and major API announcements, please join our [kubernetes-announce](https://groups.google.com/forum/#!forum/kubernetes-announce) group.
To watch for security and major API announcements, please join our [kubernetes-announce](https://groups.google.com/forum/#!forum/kubernetes-announce) group.
## Reporting a security issue
## Reporting a security issue
To report an issue, please:
To report an issue, please:
- Submit a bug report [here](http://goo.gl/vulnz).
- Submit a bug report [here](http://goo.gl/vulnz).
- Select “I want to report a technical security bug in a Google product (SQLi, XSS, etc.).”
- Select “I want to report a technical security bug in a Google product (SQLi, XSS, etc.).”
- Select “Other” as the Application Type.
- Select “Other” as the Application Type.
- Under reproduction steps, please additionally include
- Under reproduction steps, please additionally include
- the words "Kubernetes Security issue"
- the words "Kubernetes Security issue"
- Description of the issue
- Description of the issue
- Kubernetes release (e.g. output of `kubectl version` command, which includes server version.)
- Kubernetes release (e.g. output of `kubectl version` command, which includes server version.)
- Environment setup (e.g. which "Getting Started Guide" you followed, if any; what node operating system used; what service or software creates your virtual machines, if any)
- Environment setup (e.g. which "Getting Started Guide" you followed, if any; what node operating system used; what service or software creates your virtual machines, if any)
An online submission will have the fastest response; however, if you prefer email, please send mail to security@google.com. If you feel the need, please use the [PGP public key](https://services.google.com/corporate/publickey.txt) to encrypt communications.
An online submission will have the fastest response; however, if you prefer email, please send mail to security@google.com. If you feel the need, please use the [PGP public key](https://services.google.com/corporate/publickey.txt) to encrypt communications.
@@ -150,7 +150,7 @@ There are [client libraries](../devel/client-libraries.md) for accessing the API
...
@@ -150,7 +150,7 @@ There are [client libraries](../devel/client-libraries.md) for accessing the API
from several languages. The Kubernetes project-supported
from several languages. The Kubernetes project-supported
[Go](http://releases.k8s.io/HEAD/pkg/client/)
[Go](http://releases.k8s.io/HEAD/pkg/client/)
client library can use the same [kubeconfig file](kubeconfig-file.md)
client library can use the same [kubeconfig file](kubeconfig-file.md)
as the kubectl CLI does to locate and authenticate to the apiserver.
as the kubectl CLI does to locate and authenticate to the apiserver.
See documentation for other libraries for how they authenticate.
See documentation for other libraries for how they authenticate.
...
@@ -241,7 +241,7 @@ at `https://104.197.5.247/api/v1/proxy/namespaces/kube-system/services/elasticse
...
@@ -241,7 +241,7 @@ at `https://104.197.5.247/api/v1/proxy/namespaces/kube-system/services/elasticse
#### Manually constructing apiserver proxy URLs
#### Manually constructing apiserver proxy URLs
As mentioned above, you use the `kubectl cluster-info` command to retrieve the service's proxy URL. To create proxy URLs that include service endpoints, suffixes, and parameters, you simply append to the service's proxy URL:
As mentioned above, you use the `kubectl cluster-info` command to retrieve the service's proxy URL. To create proxy URLs that include service endpoints, suffixes, and parameters, you simply append to the service's proxy URL:
@@ -87,7 +87,7 @@ there are insufficient resources of one type or another that prevent scheduling.
...
@@ -87,7 +87,7 @@ there are insufficient resources of one type or another that prevent scheduling.
your pod. Reasons include:
your pod. Reasons include:
***You don't have enough resources**: You may have exhausted the supply of CPU or Memory in your cluster, in this case
***You don't have enough resources**: You may have exhausted the supply of CPU or Memory in your cluster, in this case
you need to delete Pods, adjust resource requests, or add new nodes to your cluster. See [Compute Resources document](compute-resources.md#my-pods-are-pending-with-event-message-failedscheduling) for more information.
you need to delete Pods, adjust resource requests, or add new nodes to your cluster. See [Compute Resources document](compute-resources.md#my-pods-are-pending-with-event-message-failedscheduling) for more information.
***You are using `hostPort`**: When you bind a Pod to a `hostPort` there are a limited number of places that pod can be
***You are using `hostPort`**: When you bind a Pod to a `hostPort` there are a limited number of places that pod can be
scheduled. In most cases, `hostPort` is unnecessary, try using a Service object to expose your Pod. If you do require
scheduled. In most cases, `hostPort` is unnecessary, try using a Service object to expose your Pod. If you do require
...
@@ -100,7 +100,7 @@ If a Pod is stuck in the `Waiting` state, then it has been scheduled to a worker
...
@@ -100,7 +100,7 @@ If a Pod is stuck in the `Waiting` state, then it has been scheduled to a worker
Again, the information from `kubectl describe ...` should be informative. The most common cause of `Waiting` pods is a failure to pull the image. There are three things to check:
Again, the information from `kubectl describe ...` should be informative. The most common cause of `Waiting` pods is a failure to pull the image. There are three things to check:
* Make sure that you have the name of the image correct
* Make sure that you have the name of the image correct
* Have you pushed the image to the repository?
* Have you pushed the image to the repository?
* Run a manual `docker pull <image>` on your machine to see if the image can be pulled.
* Run a manual `docker pull <image>` on your machine to see if the image can be pulled.
#### My pod is crashing or otherwise unhealthy
#### My pod is crashing or otherwise unhealthy
...
@@ -139,7 +139,7 @@ feature request on GitHub describing your use case and why these tools are insuf
...
@@ -139,7 +139,7 @@ feature request on GitHub describing your use case and why these tools are insuf
### Debugging Replication Controllers
### Debugging Replication Controllers
Replication controllers are fairly straightforward. They can either create Pods or they can't. If they can't
Replication controllers are fairly straightforward. They can either create Pods or they can't. If they can't
create pods, then please refer to the [instructions above](#debugging-pods) to debug your pods.
create pods, then please refer to the [instructions above](#debugging-pods) to debug your pods.
You can also use `kubectl describe rc ${CONTROLLER_NAME}` to introspect events related to the replication
You can also use `kubectl describe rc ${CONTROLLER_NAME}` to introspect events related to the replication
controller.
controller.
...
@@ -199,11 +199,11 @@ check:
...
@@ -199,11 +199,11 @@ check:
* Can you connect to your pods directly? Get the IP address for the Pod, and try to connect directly to that IP
* Can you connect to your pods directly? Get the IP address for the Pod, and try to connect directly to that IP
* Is your application serving on the port that you configured? Kubernetes doesn't do port remapping, so if your application serves on 8080, the `containerPort` field needs to be 8080.
* Is your application serving on the port that you configured? Kubernetes doesn't do port remapping, so if your application serves on 8080, the `containerPort` field needs to be 8080.
#### More information
#### More information
If none of the above solves your problem, follow the instructions in [Debugging Service document](debugging-services.md) to make sure that your `Service` is running, has `Endpoints`, and your `Pods` are actually serving; you have DNS working, iptables rules installed, and kube-proxy does not seem to be misbehaving.
If none of the above solves your problem, follow the instructions in [Debugging Service document](debugging-services.md) to make sure that your `Service` is running, has `Endpoints`, and your `Pods` are actually serving; you have DNS working, iptables rules installed, and kube-proxy does not seem to be misbehaving.
You may also visit [troubleshooting document](../troubleshooting.md) for more information.
You may also visit [troubleshooting document](../troubleshooting.md) for more information.
If a container exceeds its memory limit, it may be terminated. If it is restartable, it will be
If a container exceeds its memory limit, it may be terminated. If it is restartable, it will be
restarted by kubelet, as will any other type of runtime failure.
restarted by kubelet, as will any other type of runtime failure.
A container may or may not be allowed to exceed its CPU limit for extended periods of time.
A container may or may not be allowed to exceed its CPU limit for extended periods of time.
However, it will not be killed for excessive CPU usage.
However, it will not be killed for excessive CPU usage.
...
@@ -178,7 +178,7 @@ The [resource quota](../admin/resource-quota.md) feature can be configured
...
@@ -178,7 +178,7 @@ The [resource quota](../admin/resource-quota.md) feature can be configured
to limit the total amount of resources that can be consumed. If used in conjunction
to limit the total amount of resources that can be consumed. If used in conjunction
with namespaces, it can prevent one team from hogging all the resources.
with namespaces, it can prevent one team from hogging all the resources.
### My container is terminated
### My container is terminated
Your container may be terminated because it's resource-starved. To check if a container is being killed because it is hitting a resource limit, call `kubectl describe pod`
Your container may be terminated because it's resource-starved. To check if a container is being killed because it is hitting a resource limit, call `kubectl describe pod`
@@ -35,7 +35,7 @@ Documentation for other releases can be found at
...
@@ -35,7 +35,7 @@ Documentation for other releases can be found at
This document is meant to highlight and consolidate in one place configuration best practices that are introduced throughout the user-guide and getting-started documentation and examples. This is a living document so if you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR.
This document is meant to highlight and consolidate in one place configuration best practices that are introduced throughout the user-guide and getting-started documentation and examples. This is a living document so if you think of something that is not on this list but might be useful to others, please don't hesitate to file an issue or submit a PR.
1. When writing configuration, use the latest stable API version (currently v1).
1. When writing configuration, use the latest stable API version (currently v1).
1. Configuration should be stored in version control before being pushed to the cluster. This allows configuration to be quickly rolled back if needed and will aid with cluster re-creation and restoration if the worst were to happen.
1. Configuration should be stored in version control before being pushed to the cluster. This allows configuration to be quickly rolled back if needed and will aid with cluster re-creation and restoration if the worst were to happen.
1. Use YAML rather than JSON. They can be used interchangeably in almost all scenarios but YAML tends to be more user-friendly for config.
1. Use YAML rather than JSON. They can be used interchangeably in almost all scenarios but YAML tends to be more user-friendly for config.
1. Group related objects together in a single file. This is often better than separate files.
1. Group related objects together in a single file. This is often better than separate files.
@@ -73,7 +73,7 @@ spec: # specification of the pod’s contents
...
@@ -73,7 +73,7 @@ spec: # specification of the pod’s contents
The value of `metadata.name`, `hello-world`, will be the name of the pod resource created, and must be unique within the cluster, whereas `containers[0].name` is just a nickname for the container within that pod. `image` is the name of the Docker image, which Kubernetes expects to be able to pull from a registry, the [Docker Hub](https://registry.hub.docker.com/) by default.
The value of `metadata.name`, `hello-world`, will be the name of the pod resource created, and must be unique within the cluster, whereas `containers[0].name` is just a nickname for the container within that pod. `image` is the name of the Docker image, which Kubernetes expects to be able to pull from a registry, the [Docker Hub](https://registry.hub.docker.com/) by default.
`restartPolicy: Never` indicates that we just want to run the container once and then terminate the pod.
`restartPolicy: Never` indicates that we just want to run the container once and then terminate the pod.
The [`command`](containers.md#containers-and-commands) overrides the Docker container’s `Entrypoint`. Command arguments (corresponding to Docker’s `Cmd`) may be specified using `args`, as follows:
The [`command`](containers.md#containers-and-commands) overrides the Docker container’s `Entrypoint`. Command arguments (corresponding to Docker’s `Cmd`) may be specified using `args`, as follows:
...
@@ -142,7 +142,7 @@ However, a shell isn’t necessary just to expand environment variables. Kuberne
...
@@ -142,7 +142,7 @@ However, a shell isn’t necessary just to expand environment variables. Kuberne
## Viewing pod status
## Viewing pod status
You can see the pod you created (actually all of your cluster's pods) using the `get` command.
You can see the pod you created (actually all of your cluster's pods) using the `get` command.
@@ -52,10 +52,10 @@ Documentation for other releases can be found at
...
@@ -52,10 +52,10 @@ Documentation for other releases can be found at
## Overview
## Overview
This document describes the environment for Kubelet managed containers on a Kubernetes node (kNode). In contrast to the Kubernetes cluster API, which provides an API for creating and managing containers, the Kubernetes container environment provides the container access to information about what else is going on in the cluster.
This document describes the environment for Kubelet managed containers on a Kubernetes node (kNode). In contrast to the Kubernetes cluster API, which provides an API for creating and managing containers, the Kubernetes container environment provides the container access to information about what else is going on in the cluster.
This cluster information makes it possible to build applications that are *cluster aware*.
This cluster information makes it possible to build applications that are *cluster aware*.
Additionally, the Kubernetes container environment defines a series of hooks that are surfaced to optional hook handlers defined as part of individual containers. Container hooks are somewhat analogous to operating system signals in a traditional process model. However these hooks are designed to make it easier to build reliable, scalable cloud applications in the Kubernetes cluster. Containers that participate in this cluster lifecycle become *cluster native*.
Additionally, the Kubernetes container environment defines a series of hooks that are surfaced to optional hook handlers defined as part of individual containers. Container hooks are somewhat analogous to operating system signals in a traditional process model. However these hooks are designed to make it easier to build reliable, scalable cloud applications in the Kubernetes cluster. Containers that participate in this cluster lifecycle become *cluster native*.
Another important part of the container environment is the file system that is available to the container. In Kubernetes, the filesystem is a combination of an [image](images.md) and one or more [volumes](volumes.md).
Another important part of the container environment is the file system that is available to the container. In Kubernetes, the filesystem is a combination of an [image](images.md) and one or more [volumes](volumes.md).
...
@@ -89,7 +89,7 @@ Services have dedicated IP address, and are also surfaced to the container via D
...
@@ -89,7 +89,7 @@ Services have dedicated IP address, and are also surfaced to the container via D
*NB*: Container hooks are under active development, we anticipate adding additional hooks as the Kubernetes container management system evolves.*
*NB*: Container hooks are under active development, we anticipate adding additional hooks as the Kubernetes container management system evolves.*
Container hooks provide information to the container about events in its management lifecycle. For example, immediately after a container is started, it receives a *PostStart* hook. These hooks are broadcast *into* the container with information about the life-cycle of the container. They are different from the events provided by Docker and other systems which are *output* from the container. Output events provide a log of what has already happened. Input hooks provide real-time notification about things that are happening, but no historical log.
Container hooks provide information to the container about events in its management lifecycle. For example, immediately after a container is started, it receives a *PostStart* hook. These hooks are broadcast *into* the container with information about the life-cycle of the container. They are different from the events provided by Docker and other systems which are *output* from the container. Output events provide a log of what has already happened. Input hooks provide real-time notification about things that are happening, but no historical log.
@@ -138,7 +138,7 @@ Lastly, you see a log of recent events related to your Pod. The system compresse
...
@@ -138,7 +138,7 @@ Lastly, you see a log of recent events related to your Pod. The system compresse
## Example: debugging Pending Pods
## Example: debugging Pending Pods
A common scenario that you can detect using events is when you’ve created a Pod that won’t fit on any node. For example, the Pod might request more resources than are free on any node, or it might specify a label selector that doesn’t match any nodes. Let’s say we created the previous Replication Controller with 5 replicas (instead of 2) and requesting 600 millicores instead of 500, on a four-node cluster where each (virtual) machine has 1 CPU. In that case one of the Pods will not be able to schedule. (Note that because of the cluster addon pods such as fluentd, skydns, etc., that run on each node, if we requested 1000 millicores then none of the Pods would be able to schedule.)
A common scenario that you can detect using events is when you’ve created a Pod that won’t fit on any node. For example, the Pod might request more resources than are free on any node, or it might specify a label selector that doesn’t match any nodes. Let’s say we created the previous Replication Controller with 5 replicas (instead of 2) and requesting 600 millicores instead of 500, on a four-node cluster where each (virtual) machine has 1 CPU. In that case one of the Pods will not be able to schedule. (Note that because of the cluster addon pods such as fluentd, skydns, etc., that run on each node, if we requested 1000 millicores then none of the Pods would be able to schedule.)
The examples we’ve used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another.
The examples we’ve used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another.
For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](../../examples/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels:
For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](../../examples/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels:
At some point, you’ll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios.
At some point, you’ll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above. `kubectl` supports several update operations, each of which is applicable to different scenarios.
To update a service without an outage, `kubectl` supports what is called [“rolling update”](kubectl/kubectl_rolling-update.md), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](../design/simple-rolling-update.md) and the [example of rolling update](update-demo/) for more information.
To update a service without an outage, `kubectl` supports what is called [“rolling update”](kubectl/kubectl_rolling-update.md), which updates one pod at a time, rather than taking down the entire service at the same time. See the [rolling update design document](../design/simple-rolling-update.md) and the [example of rolling update](update-demo/) for more information.
Let’s say you were running version 1.7.9 of nginx:
Let’s say you were running version 1.7.9 of nginx:
@@ -59,7 +59,7 @@ The Kubelet acts as a bridge between the Kubernetes master and the nodes. It man
...
@@ -59,7 +59,7 @@ The Kubelet acts as a bridge between the Kubernetes master and the nodes. It man
### InfluxDB and Grafana
### InfluxDB and Grafana
A Grafana setup with InfluxDB is a very popular combination for monitoring in the open source world. InfluxDB exposes an easy to use API to write and fetch time series data. Heapster is setup to use this storage backend by default on most Kubernetes clusters. A detailed setup guide can be found [here](https://github.com/GoogleCloudPlatform/heapster/blob/master/docs/influxdb.md). InfluxDB and Grafana run in Pods. The pod exposes itself as a Kubernetes service which is how Heapster discovers it.
A Grafana setup with InfluxDB is a very popular combination for monitoring in the open source world. InfluxDB exposes an easy to use API to write and fetch time series data. Heapster is setup to use this storage backend by default on most Kubernetes clusters. A detailed setup guide can be found [here](https://github.com/GoogleCloudPlatform/heapster/blob/master/docs/influxdb.md). InfluxDB and Grafana run in Pods. The pod exposes itself as a Kubernetes service which is how Heapster discovers it.
The Grafana container serves Grafana’s UI which provides an easy to configure dashboard interface. The default dashboard for Kubernetes contains an example dashboard that monitors resource usage of the cluster and the pods inside of it. This dashboard can easily be customized and expanded. Take a look at the storage schema for InfluxDB [here](https://github.com/GoogleCloudPlatform/heapster/blob/master/docs/storage-schema.md#metrics).
The Grafana container serves Grafana’s UI which provides an easy to configure dashboard interface. The default dashboard for Kubernetes contains an example dashboard that monitors resource usage of the cluster and the pods inside of it. This dashboard can easily be customized and expanded. Take a look at the storage schema for InfluxDB [here](https://github.com/GoogleCloudPlatform/heapster/blob/master/docs/storage-schema.md#metrics).
...
@@ -88,7 +88,7 @@ Here is a snapshot of the a Google Cloud Monitoring dashboard showing cluster-wi
...
@@ -88,7 +88,7 @@ Here is a snapshot of the a Google Cloud Monitoring dashboard showing cluster-wi
Now that you’ve learned a bit about Heapster, feel free to try it out on your own clusters! The [Heapster repository](https://github.com/GoogleCloudPlatform/heapster) is available on GitHub. It contains detailed instructions to setup Heapster and its storage backends. Heapster runs by default on most Kubernetes clusters, so you may already have it! Feedback is always welcome. Please let us know if you run into any issues. Heapster and Kubernetes developers hang out in the [#google-containers](http://webchat.freenode.net/?channels=google-containers) IRC channel on freenode.net. You can also reach us on the [google-containers Google Groups mailing list](https://groups.google.com/forum/#!forum/google-containers).
Now that you’ve learned a bit about Heapster, feel free to try it out on your own clusters! The [Heapster repository](https://github.com/GoogleCloudPlatform/heapster) is available on GitHub. It contains detailed instructions to setup Heapster and its storage backends. Heapster runs by default on most Kubernetes clusters, so you may already have it! Feedback is always welcome. Please let us know if you run into any issues. Heapster and Kubernetes developers hang out in the [#google-containers](http://webchat.freenode.net/?channels=google-containers) IRC channel on freenode.net. You can also reach us on the [google-containers Google Groups mailing list](https://groups.google.com/forum/#!forum/google-containers).
***
***
*Authors: Vishnu Kannan and Victor Marmol, Google Software Engineers.*
*Authors: Vishnu Kannan and Victor Marmol, Google Software Engineers.*
*This article was originally posted in [Kubernetes blog](http://blog.kubernetes.io/2015/05/resource-usage-monitoring-kubernetes.html).*
*This article was originally posted in [Kubernetes blog](http://blog.kubernetes.io/2015/05/resource-usage-monitoring-kubernetes.html).*
@@ -35,7 +35,7 @@ Documentation for other releases can be found at
...
@@ -35,7 +35,7 @@ Documentation for other releases can be found at
Kubernetes is an open-source system for managing containerized applications across multiple hosts in a cluster. Kubernetes is intended to make deploying containerized/microservice-based applications easy but powerful.
Kubernetes is an open-source system for managing containerized applications across multiple hosts in a cluster. Kubernetes is intended to make deploying containerized/microservice-based applications easy but powerful.
Kubernetes provides mechanisms for application deployment, scheduling, updating, maintenance, and scaling. A key feature of Kubernetes is that it actively manages the containers to ensure that the state of the cluster continually matches the user's intentions. An operations user should be able to launch a micro-service, letting the scheduler find the right placement. We also want to improve the tools and experience for how users can roll-out applications through patterns like canary deployments.
Kubernetes provides mechanisms for application deployment, scheduling, updating, maintenance, and scaling. A key feature of Kubernetes is that it actively manages the containers to ensure that the state of the cluster continually matches the user's intentions. An operations user should be able to launch a micro-service, letting the scheduler find the right placement. We also want to improve the tools and experience for how users can roll-out applications through patterns like canary deployments.
Kubernetes supports [Docker](http://www.docker.io) and [Rocket](https://coreos.com/blog/rocket/) containers, and other container image formats and container runtimes will be supported in the future.
Kubernetes supports [Docker](http://www.docker.io) and [Rocket](https://coreos.com/blog/rocket/) containers, and other container image formats and container runtimes will be supported in the future.
...
@@ -45,7 +45,7 @@ In Kubernetes, all containers run inside [pods](pods.md). A pod can host a singl
...
@@ -45,7 +45,7 @@ In Kubernetes, all containers run inside [pods](pods.md). A pod can host a singl
Users can create and manage pods themselves, but Kubernetes drastically simplifies system management by allowing users to delegate two common pod-related activities: deploying multiple pod replicas based on the same pod configuration, and creating replacement pods when a pod or its machine fails. The Kubernetes API object that manages these behaviors is called a [replication controller](replication-controller.md). It defines a pod in terms of a template, that the system then instantiates as some number of pods (specified by the user). The replicated set of pods might constitute an entire application, a micro-service, or one layer in a multi-tier application. Once the pods are created, the system continually monitors their health and that of the machines they are running on; if a pod fails due to a software problem or machine failure, the replication controller automatically creates a new pod on a healthy machine, to maintain the set of pods at the desired replication level. Multiple pods from the same or different applications can share the same machine. Note that a replication controller is needed even in the case of a single non-replicated pod if the user wants it to be re-created when it or its machine fails.
Users can create and manage pods themselves, but Kubernetes drastically simplifies system management by allowing users to delegate two common pod-related activities: deploying multiple pod replicas based on the same pod configuration, and creating replacement pods when a pod or its machine fails. The Kubernetes API object that manages these behaviors is called a [replication controller](replication-controller.md). It defines a pod in terms of a template, that the system then instantiates as some number of pods (specified by the user). The replicated set of pods might constitute an entire application, a micro-service, or one layer in a multi-tier application. Once the pods are created, the system continually monitors their health and that of the machines they are running on; if a pod fails due to a software problem or machine failure, the replication controller automatically creates a new pod on a healthy machine, to maintain the set of pods at the desired replication level. Multiple pods from the same or different applications can share the same machine. Note that a replication controller is needed even in the case of a single non-replicated pod if the user wants it to be re-created when it or its machine fails.
Frequently it is useful to refer to a set of pods, for example to limit the set of pods on which a mutating operation should be performed, or that should be queried for status. As a general mechanism, users can attach to most Kubernetes API objects arbitrary key-value pairs called [labels](labels.md), and then use a set of label selectors (key-value queries over labels) to constrain the target of API operations. Each resource also has a map of string keys and values that can be used by external tooling to store and retrieve arbitrary metadata about this object, called [annotations](annotations.md).
Frequently it is useful to refer to a set of pods, for example to limit the set of pods on which a mutating operation should be performed, or that should be queried for status. As a general mechanism, users can attach to most Kubernetes API objects arbitrary key-value pairs called [labels](labels.md), and then use a set of label selectors (key-value queries over labels) to constrain the target of API operations. Each resource also has a map of string keys and values that can be used by external tooling to store and retrieve arbitrary metadata about this object, called [annotations](annotations.md).
Kubernetes supports a unique [networking model](../admin/networking.md). Kubernetes encourages a flat address space and does not dynamically allocate ports, instead allowing users to select whichever ports are convenient for them. To achieve this, it allocates an IP address for each pod.
Kubernetes supports a unique [networking model](../admin/networking.md). Kubernetes encourages a flat address space and does not dynamically allocate ports, instead allowing users to select whichever ports are convenient for them. To achieve this, it allocates an IP address for each pod.
@@ -65,7 +65,7 @@ Managing storage is a distinct problem from managing compute. The `PersistentVol
...
@@ -65,7 +65,7 @@ Managing storage is a distinct problem from managing compute. The `PersistentVol
A `PersistentVolume` (PV) is a piece of networked storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
A `PersistentVolume` (PV) is a piece of networked storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
A `PersistentVolumeClaim` (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g, can be mounted once read/write or many times read-only).
A `PersistentVolumeClaim` (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g, can be mounted once read/write or many times read-only).
Please see the [detailed walkthrough with working examples](persistent-volumes/).
Please see the [detailed walkthrough with working examples](persistent-volumes/).
...
@@ -113,7 +113,7 @@ A `PersistentVolume's` reclaim policy tells the cluster what to do with the volu
...
@@ -113,7 +113,7 @@ A `PersistentVolume's` reclaim policy tells the cluster what to do with the volu
## Persistent Volumes
## Persistent Volumes
Each PV contains a spec and status, which is the specification and status of the volume.
Each PV contains a spec and status, which is the specification and status of the volume.
@@ -93,22 +93,22 @@ That approach would provide co-location, but would not provide most of the benef
...
@@ -93,22 +93,22 @@ That approach would provide co-location, but would not provide most of the benef
## Durability of pods (or lack thereof)
## Durability of pods (or lack thereof)
Pods aren't intended to be treated as durable [pets](https://blog.engineyard.com/2014/pets-vs-cattle). They won't survive scheduling failures, node failures, or other evictions, such as due to lack of resources, or in the case of node maintenance.
Pods aren't intended to be treated as durable [pets](https://blog.engineyard.com/2014/pets-vs-cattle). They won't survive scheduling failures, node failures, or other evictions, such as due to lack of resources, or in the case of node maintenance.
In general, users shouldn't need to create pods directly. They should almost always use controllers (e.g., [replication controller](replication-controller.md)), even for singletons. Controllers provide self-healing with a cluster scope, as well as replication and rollout management.
In general, users shouldn't need to create pods directly. They should almost always use controllers (e.g., [replication controller](replication-controller.md)), even for singletons. Controllers provide self-healing with a cluster scope, as well as replication and rollout management.
The use of collective APIs as the primary user-facing primitive is relatively common among cluster scheduling systems, including [Borg](https://research.google.com/pubs/pub43438.html), [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html), [Aurora](http://aurora.apache.org/documentation/latest/configuration-reference/#job-schema), and [Tupperware](http://www.slideshare.net/Docker/aravindnarayanan-facebook140613153626phpapp02-37588997).
The use of collective APIs as the primary user-facing primitive is relatively common among cluster scheduling systems, including [Borg](https://research.google.com/pubs/pub43438.html), [Marathon](https://mesosphere.github.io/marathon/docs/rest-api.html), [Aurora](http://aurora.apache.org/documentation/latest/configuration-reference/#job-schema), and [Tupperware](http://www.slideshare.net/Docker/aravindnarayanan-facebook140613153626phpapp02-37588997).
Pod is exposed as a primitive in order to facilitate:
Pod is exposed as a primitive in order to facilitate:
* scheduler and controller pluggability
* scheduler and controller pluggability
* support for pod-level operations without the need to "proxy" them via controller APIs
* support for pod-level operations without the need to "proxy" them via controller APIs
* decoupling of pod lifetime from controller lifetime, such as for bootstrapping
* decoupling of pod lifetime from controller lifetime, such as for bootstrapping
* decoupling of controllers and services — the endpoint controller just watches pods
* decoupling of controllers and services — the endpoint controller just watches pods
* clean composition of Kubelet-level functionality with cluster-level functionality — Kubelet is effectively the "pod controller"
* clean composition of Kubelet-level functionality with cluster-level functionality — Kubelet is effectively the "pod controller"
* high-availability applications, which will expect pods to be replaced in advance of their termination and certainly in advance of deletion, such as in the case of planned evictions, image prefetching, or live pod migration [#3949](https://github.com/GoogleCloudPlatform/kubernetes/issues/3949)
* high-availability applications, which will expect pods to be replaced in advance of their termination and certainly in advance of deletion, such as in the case of planned evictions, image prefetching, or live pod migration [#3949](https://github.com/GoogleCloudPlatform/kubernetes/issues/3949)
The current best practice for pets is to create a replication controller with `replicas` equal to `1` and a corresponding service. If you find this cumbersome, please comment on [issue #260](https://github.com/GoogleCloudPlatform/kubernetes/issues/260).
The current best practice for pets is to create a replication controller with `replicas` equal to `1` and a corresponding service. If you find this cumbersome, please comment on [issue #260](https://github.com/GoogleCloudPlatform/kubernetes/issues/260).
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
...
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
# Kubernetes User Guide: Managing Applications: Prerequisites
# Kubernetes User Guide: Managing Applications: Prerequisites
To deploy and manage applications on Kubernetes, you’ll use the Kubernetes command-line tool, [kubectl](kubectl/kubectl.md). It lets you inspect your cluster resources, create, delete, and update components, and much more. You will use it to look at your new cluster and bring up example apps.
To deploy and manage applications on Kubernetes, you’ll use the Kubernetes command-line tool, [kubectl](kubectl/kubectl.md). It lets you inspect your cluster resources, create, delete, and update components, and much more. You will use it to look at your new cluster and bring up example apps.
@@ -90,7 +90,7 @@ In addition to the local disk storage provided by `emptyDir`, Kubernetes support
...
@@ -90,7 +90,7 @@ In addition to the local disk storage provided by `emptyDir`, Kubernetes support
## Distributing credentials
## Distributing credentials
Many applications need credentials, such as passwords, OAuth tokens, and TLS keys, to authenticate with other applications, databases, and services. Storing these credentials in container images or environment variables is less than ideal, since the credentials can then be copied by anyone with access to the image, pod/container specification, host file system, or host Docker daemon.
Many applications need credentials, such as passwords, OAuth tokens, and TLS keys, to authenticate with other applications, databases, and services. Storing these credentials in container images or environment variables is less than ideal, since the credentials can then be copied by anyone with access to the image, pod/container specification, host file system, or host Docker daemon.
Kubernetes provides a mechanism, called [*secrets*](secrets.md), that facilitates delivery of sensitive credentials to applications. A `Secret` is a simple resource containing a map of data. For instance, a simple secret with a username and password might look as follows:
Kubernetes provides a mechanism, called [*secrets*](secrets.md), that facilitates delivery of sensitive credentials to applications. A `Secret` is a simple resource containing a map of data. For instance, a simple secret with a username and password might look as follows:
...
@@ -245,7 +245,7 @@ More examples can be found in our [blog article](http://blog.kubernetes.io/2015/
...
@@ -245,7 +245,7 @@ More examples can be found in our [blog article](http://blog.kubernetes.io/2015/
## Resource management
## Resource management
Kubernetes’s scheduler will place applications only where they have adequate CPU and memory, but it can only do so if it knows how much [resources they require](compute-resources.md). The consequence of specifying too little CPU is that the containers could be starved of CPU if too many other containers were scheduled onto the same node. Similarly, containers could die unpredictably due to running out of memory if no memory were requested, which can be especially likely for large-memory applications.
Kubernetes’s scheduler will place applications only where they have adequate CPU and memory, but it can only do so if it knows how much [resources they require](compute-resources.md). The consequence of specifying too little CPU is that the containers could be starved of CPU if too many other containers were scheduled onto the same node. Similarly, containers could die unpredictably due to running out of memory if no memory were requested, which can be especially likely for large-memory applications.
If no resource requirements are specified, a nominal amount of resources is assumed. (This default is applied via a [LimitRange](limitrange/) for the default [Namespace](namespaces.md). It can be viewed with `kubectl describe limitrange limits`.) You may explicitly specify the amount of resources required as follows:
If no resource requirements are specified, a nominal amount of resources is assumed. (This default is applied via a [LimitRange](limitrange/) for the default [Namespace](namespaces.md). It can be viewed with `kubectl describe limitrange limits`.) You may explicitly specify the amount of resources required as follows:
...
@@ -318,7 +318,7 @@ For more details (e.g., how to specify command-based probes), see the [example i
...
@@ -318,7 +318,7 @@ For more details (e.g., how to specify command-based probes), see the [example i
Of course, nodes and applications may fail at any time, but many applications benefit from clean shutdown, such as to complete in-flight requests, when the termination of the application is deliberate. To support such cases, Kubernetes supports two kinds of notifications:
Of course, nodes and applications may fail at any time, but many applications benefit from clean shutdown, such as to complete in-flight requests, when the termination of the application is deliberate. To support such cases, Kubernetes supports two kinds of notifications:
Kubernetes will send SIGTERM to applications, which can be handled in order to effect graceful termination. SIGKILL is sent 10 seconds later if the application does not terminate sooner.
Kubernetes will send SIGTERM to applications, which can be handled in order to effect graceful termination. SIGKILL is sent 10 seconds later if the application does not terminate sooner.
Kubernetes supports the (optional) specification of a [*pre-stop lifecycle hook*](container-environment.md#container-hooks), which will execute prior to sending SIGTERM.
Kubernetes supports the (optional) specification of a [*pre-stop lifecycle hook*](container-environment.md#container-hooks), which will execute prior to sending SIGTERM.
The specification of a pre-stop hook is similar to that of probes, but without the timing-related parameters. For example:
The specification of a pre-stop hook is similar to that of probes, but without the timing-related parameters. For example:
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
...
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
# Secrets example
# Secrets example
Following this example, you will create a [secret](../secrets.md) and a [pod](../pods.md) that consumes that secret in a [volume](../volumes.md). See [Secrets design document](../../design/secrets.md) for more information.
Following this example, you will create a [secret](../secrets.md) and a [pod](../pods.md) that consumes that secret in a [volume](../volumes.md). See [Secrets design document](../../design/secrets.md) for more information.
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
...
@@ -33,7 +33,7 @@ Documentation for other releases can be found at
# Kubernetes User Interface
# Kubernetes User Interface
Kubernetes has a web-based user interface that displays the current cluster state graphically.
Kubernetes has a web-based user interface that displays the current cluster state graphically.
## Accessing the UI
## Accessing the UI
...
@@ -50,34 +50,34 @@ Normally, this should be taken care of automatically by the [`kube-addons.sh`](h
...
@@ -50,34 +50,34 @@ Normally, this should be taken care of automatically by the [`kube-addons.sh`](h
## Using the UI
## Using the UI
The Kubernetes UI can be used to introspect your current cluster, such as checking how resources are used, or looking at error messages. You cannot, however, use the UI to modify your cluster.
The Kubernetes UI can be used to introspect your current cluster, such as checking how resources are used, or looking at error messages. You cannot, however, use the UI to modify your cluster.
### Node Resource Usage
### Node Resource Usage
After accessing Kubernetes UI, you'll see a homepage dynamically listing out all nodes in your current cluster, with related information including internal IP addresses, CPU usage, memory usage, and file systems usage.
After accessing Kubernetes UI, you'll see a homepage dynamically listing out all nodes in your current cluster, with related information including internal IP addresses, CPU usage, memory usage, and file systems usage.


### Dashboard Views
### Dashboard Views
Click on the "Views" button in the top-right of the page to see other views available, which include: Explore, Pods, Nodes, Replication Controllers, Services, and Events.
Click on the "Views" button in the top-right of the page to see other views available, which include: Explore, Pods, Nodes, Replication Controllers, Services, and Events.
#### Explore View
#### Explore View
The "Explore" view allows your to see the pods, replication controllers, and services in current cluster easily.
The "Explore" view allows your to see the pods, replication controllers, and services in current cluster easily.


The "Group by" dropdown list allows you to group these resources by a number of factors, such as type, name, host, etc.
The "Group by" dropdown list allows you to group these resources by a number of factors, such as type, name, host, etc.


You can also create filters by clicking on the down triangle of any listed resource instances and choose which filters you want to add.
You can also create filters by clicking on the down triangle of any listed resource instances and choose which filters you want to add.
Other views (Pods, Nodes, Replication Controllers, Services, and Events) simply list information about each type of resource. You can also click on any instance for more details.
Other views (Pods, Nodes, Replication Controllers, Services, and Events) simply list information about each type of resource. You can also click on any instance for more details.


## More Information
## More Information
For more information, see the [Kubernetes UI development document](http://releases.k8s.io/HEAD/www/README.md) in the www directory.
For more information, see the [Kubernetes UI development document](http://releases.k8s.io/HEAD/www/README.md) in the www directory.
This example demonstrates the usage of Kubernetes to perform a [rolling update](../kubectl/kubectl_rolling-update.md) on a running group of [pods](../../../docs/user-guide/pods.md). See [here](../managing-deployments.md#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](../../design/simple-rolling-update.md) for more information.
This example demonstrates the usage of Kubernetes to perform a [rolling update](../kubectl/kubectl_rolling-update.md) on a running group of [pods](../../../docs/user-guide/pods.md). See [here](../managing-deployments.md#updating-your-application-without-a-service-outage) to understand why you need a rolling update. Also check [rolling update design document](../../design/simple-rolling-update.md) for more information.
@@ -31,11 +31,11 @@ Documentation for other releases can be found at
...
@@ -31,11 +31,11 @@ Documentation for other releases can be found at
<!-- END MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# What is Kubernetes?
# What is Kubernetes?
Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts.
Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts.
With Kubernetes, you are able to quickly and efficiently respond to customer demand:
With Kubernetes, you are able to quickly and efficiently respond to customer demand:
- Scale your applications on the fly.
- Scale your applications on the fly.
- Seamlessly roll out new features.
- Seamlessly roll out new features.
...
@@ -54,27 +54,27 @@ The Kubernetes project was started by Google in 2014. Kubernetes builds upon a [
...
@@ -54,27 +54,27 @@ The Kubernetes project was started by Google in 2014. Kubernetes builds upon a [
<hr>
<hr>
Looking for reasons why you should be using [containers](http://aucouranton.com/2014/06/13/linux-containers-parallels-lxc-openvz-docker-and-more/)?
Looking for reasons why you should be using [containers](http://aucouranton.com/2014/06/13/linux-containers-parallels-lxc-openvz-docker-and-more/)?
Here are some key points:
Here are some key points:
***Application-centric management**:
***Application-centric management**:
Raises the level of abstraction from running an OS on virtual hardware to running an application on an OS using logical resources. This provides the simplicity of PaaS with the flexibility of IaaS and enables you to run much more than just [12-factor apps](http://12factor.net/).
Raises the level of abstraction from running an OS on virtual hardware to running an application on an OS using logical resources. This provides the simplicity of PaaS with the flexibility of IaaS and enables you to run much more than just [12-factor apps](http://12factor.net/).
***Dev and Ops separation of concerns**:
***Dev and Ops separation of concerns**:
Provides separatation of build and deployment; therefore, decoupling applications from infrastructure.
Provides separatation of build and deployment; therefore, decoupling applications from infrastructure.
***Agile application creation and deployment**:
***Agile application creation and deployment**:
Increased ease and efficiency of container image creation compared to VM image use.
Increased ease and efficiency of container image creation compared to VM image use.
***Continuous development, integration, and deployment**:
***Continuous development, integration, and deployment**:
Provides for reliable and frequent container image build and deployment with quick and easy rollbacks (due to image immutability).
Provides for reliable and frequent container image build and deployment with quick and easy rollbacks (due to image immutability).
Applications are broken into smaller, independent pieces and can be deployed and managed dynamically -- not a fat monolithic stack running on one big single-purpose machine.
Applications are broken into smaller, independent pieces and can be deployed and managed dynamically -- not a fat monolithic stack running on one big single-purpose machine.
***Environmental consistency across development, testing, and production**:
***Environmental consistency across development, testing, and production**:
Runs the same on a laptop as it does in the cloud.
Runs the same on a laptop as it does in the cloud.
***Cloud and OS distribution portability**:
***Cloud and OS distribution portability**:
Runs on Ubuntu, RHEL, on-prem, or Google Container Engine, which makes sense for all environments: build, test, and production.
Runs on Ubuntu, RHEL, on-prem, or Google Container Engine, which makes sense for all environments: build, test, and production.
@@ -35,7 +35,7 @@ Documentation for other releases can be found at
...
@@ -35,7 +35,7 @@ Documentation for other releases can be found at
## Introduction
## Introduction
Celery is an asynchronous task queue based on distributed message passing. It is used to create execution units (i.e. tasks) which are then executed on one or more worker nodes, either synchronously or asynchronously.
Celery is an asynchronous task queue based on distributed message passing. It is used to create execution units (i.e. tasks) which are then executed on one or more worker nodes, either synchronously or asynchronously.
Celery is implemented in Python.
Celery is implemented in Python.
...
@@ -249,7 +249,7 @@ On GCE this can be done with:
...
@@ -249,7 +249,7 @@ On GCE this can be done with:
```
```
Please remember to delete the rule after you are done with the example (on GCE: `$ gcloud compute firewall-rules delete kubernetes-minion-5555`)
Please remember to delete the rule after you are done with the example (on GCE: `$ gcloud compute firewall-rules delete kubernetes-minion-5555`)
To bring up the pods, run this command `$ kubectl create -f examples/celery-rabbitmq/flower-controller.yaml`. This controller is defined as so:
To bring up the pods, run this command `$ kubectl create -f examples/celery-rabbitmq/flower-controller.yaml`. This controller is defined as so:
<!-- BEGIN MUNGE: EXAMPLE flower-controller.yaml -->
<!-- BEGIN MUNGE: EXAMPLE flower-controller.yaml -->
@@ -59,7 +59,7 @@ Here is a snippet of [glusterfs-endpoints.json](glusterfs-endpoints.json),
...
@@ -59,7 +59,7 @@ Here is a snippet of [glusterfs-endpoints.json](glusterfs-endpoints.json),
```
```
The "IP" field should be filled with the address of a node in the Glusterfs server cluster. In this example, it is fine to give any valid value (from 1 to 65535) to the "port" field.
The "IP" field should be filled with the address of a node in the Glusterfs server cluster. In this example, it is fine to give any valid value (from 1 to 65535) to the "port" field.
Create the endpoints,
Create the endpoints,
...
@@ -90,11 +90,11 @@ The following *volume* spec in [glusterfs-pod.json](glusterfs-pod.json) illustra
...
@@ -90,11 +90,11 @@ The following *volume* spec in [glusterfs-pod.json](glusterfs-pod.json) illustra
}
}
```
```
The parameters are explained as the followings.
The parameters are explained as the followings.
-**endpoints** is endpoints name that represents a Gluster cluster configuration. *kubelet* is optimized to avoid mount storm, it will randomly pick one from the endpoints to mount. If this host is unresponsive, the next Gluster host in the endpoints is automatically selected.
-**endpoints** is endpoints name that represents a Gluster cluster configuration. *kubelet* is optimized to avoid mount storm, it will randomly pick one from the endpoints to mount. If this host is unresponsive, the next Gluster host in the endpoints is automatically selected.
-**path** is the Glusterfs volume name.
-**path** is the Glusterfs volume name.
-**readOnly** is the boolean that sets the mountpoint readOnly or readWrite.
-**readOnly** is the boolean that sets the mountpoint readOnly or readWrite.
Create a pod that has a container using Glusterfs volume,
Create a pod that has a container using Glusterfs volume,
@@ -37,16 +37,16 @@ This example shows how to build a simple multi-tier web application using Kubern
...
@@ -37,16 +37,16 @@ This example shows how to build a simple multi-tier web application using Kubern
If you are running a cluster in Google Container Engine (GKE), instead see the [Guestbook Example for Google Container Engine](https://cloud.google.com/container-engine/docs/tutorials/guestbook).
If you are running a cluster in Google Container Engine (GKE), instead see the [Guestbook Example for Google Container Engine](https://cloud.google.com/container-engine/docs/tutorials/guestbook).
##### Table of Contents
##### Table of Contents
*[Step Zero: Prerequisites](#step-zero)
*[Step Zero: Prerequisites](#step-zero)
*[Step One: Create the Redis master pod](#step-one)
*[Step One: Create the Redis master pod](#step-one)
*[Step Two: Create the Redis master service](#step-two)
*[Step Two: Create the Redis master service](#step-two)
*[Step Three: Create the Redis slave pods](#step-three)
*[Step Three: Create the Redis slave pods](#step-three)
*[Step Four: Create the Redis slave service](#step-four)
*[Step Four: Create the Redis slave service](#step-four)
*[Step Five: Create the guestbook pods](#step-five)
*[Step Five: Create the guestbook pods](#step-five)
*[Step Six: Create the guestbook service](#step-six)
*[Step Six: Create the guestbook service](#step-six)
<nop>2. To verify that the redis-master-controller is up, list all the replication controllers in the cluster with the `kubectl get rc` command:
<nop>2. To verify that the redis-master-controller is up, list all the replication controllers in the cluster with the `kubectl get rc` command:
...
@@ -102,7 +102,7 @@ Use the `examples/guestbook-go/redis-master-controller.json` file to create a [r
...
@@ -102,7 +102,7 @@ Use the `examples/guestbook-go/redis-master-controller.json` file to create a [r
### Step Two: Create the Redis master service <a id="step-two"></a>
### Step Two: Create the Redis master service <a id="step-two"></a>
A Kubernetes '[service](../../docs/user-guide/services.md)' is a named load balancer that proxies traffic to one or more containers. The services in a Kubernetes cluster are discoverable inside other containers via environment variables or DNS.
A Kubernetes '[service](../../docs/user-guide/services.md)' is a named load balancer that proxies traffic to one or more containers. The services in a Kubernetes cluster are discoverable inside other containers via environment variables or DNS.
Services find the containers to load balance based on pod labels. The pod that you created in Step One has the label `app=redis` and `role=master`. The selector field of the service determines which pods will receive the traffic sent to the service.
Services find the containers to load balance based on pod labels. The pod that you created in Step One has the label `app=redis` and `role=master`. The selector field of the service determines which pods will receive the traffic sent to the service.
...
@@ -179,7 +179,7 @@ Just like the master, we want to have a service to proxy connections to the read
...
@@ -179,7 +179,7 @@ Just like the master, we want to have a service to proxy connections to the read
services/redis-slave
services/redis-slave
```
```
<nop>2. To verify that the redis-slave service is up, list all the services in the cluster with the `kubectl get services` command:
<nop>2. To verify that the redis-slave service is up, list all the services in the cluster with the `kubectl get services` command:
```console
```console
$ kubectl get services
$ kubectl get services
...
@@ -189,7 +189,7 @@ Just like the master, we want to have a service to proxy connections to the read
...
@@ -189,7 +189,7 @@ Just like the master, we want to have a service to proxy connections to the read
...
...
```
```
Result: The service is created with labels `app=redis` and `role=slave` to identify that the pods are running the Redis slaves.
Result: The service is created with labels `app=redis` and `role=slave` to identify that the pods are running the Redis slaves.
Tip: It is helpful to set labels on your services themselves--as we've done here--to make it easy to locate them later.
Tip: It is helpful to set labels on your services themselves--as we've done here--to make it easy to locate them later.
...
@@ -264,7 +264,7 @@ You can now play with the guestbook that you just created by opening it in a bro
...
@@ -264,7 +264,7 @@ You can now play with the guestbook that you just created by opening it in a bro
If you are running Kubernetes locally, to view the guestbook, navigate to `http://localhost:3000` in your browser.
If you are running Kubernetes locally, to view the guestbook, navigate to `http://localhost:3000` in your browser.
***Remote Host:**
***Remote Host:**
1. To view the guestbook on a remote host, locate the external IP of the load balancer in the **IP** column of the `kubectl get services` output. In our example, the internal IP address is `10.0.217.218` and the external IP address is `146.148.81.8` (*Note: you might need to scroll to see the IP column*).
1. To view the guestbook on a remote host, locate the external IP of the load balancer in the **IP** column of the `kubectl get services` output. In our example, the internal IP address is `10.0.217.218` and the external IP address is `146.148.81.8` (*Note: you might need to scroll to see the IP column*).
2. Append port `3000` to the IP address (for example `http://146.148.81.8:3000`), and then navigate to that address in your browser.
2. Append port `3000` to the IP address (for example `http://146.148.81.8:3000`), and then navigate to that address in your browser.
In Kubernetes, the atomic unit of an application is a [_Pod_](../../docs/user-guide/pods.md). A Pod is one or more containers that _must_ be scheduled onto the same host. All containers in a pod share a network namespace, and may optionally share mounted volumes.
In Kubernetes, the atomic unit of an application is a [_Pod_](../../docs/user-guide/pods.md). A Pod is one or more containers that _must_ be scheduled onto the same host. All containers in a pod share a network namespace, and may optionally share mounted volumes.
In this case, we shall not run a single Hazelcast pod, because the discovery mechanism now relies on a service definition.
In this case, we shall not run a single Hazelcast pod, because the discovery mechanism now relies on a service definition.
@@ -38,8 +38,8 @@ Documentation for other releases can be found at
...
@@ -38,8 +38,8 @@ Documentation for other releases can be found at
If you use Fedora 21 on Kubernetes node, then first install iSCSI initiator on the node:
If you use Fedora 21 on Kubernetes node, then first install iSCSI initiator on the node:
# yum -y install iscsi-initiator-utils
# yum -y install iscsi-initiator-utils
then edit */etc/iscsi/initiatorname.iscsi* and */etc/iscsi/iscsid.conf* to match your iSCSI target configuration.
then edit */etc/iscsi/initiatorname.iscsi* and */etc/iscsi/iscsid.conf* to match your iSCSI target configuration.
I mostly followed these [instructions](http://www.server-world.info/en/note?os=Fedora_21&p=iscsi) to setup iSCSI target. and these [instructions](http://www.server-world.info/en/note?os=Fedora_21&p=iscsi&f=2) to setup iSCSI initiator.
I mostly followed these [instructions](http://www.server-world.info/en/note?os=Fedora_21&p=iscsi) to setup iSCSI target. and these [instructions](http://www.server-world.info/en/note?os=Fedora_21&p=iscsi&f=2) to setup iSCSI initiator.
...
@@ -50,7 +50,7 @@ GCE does not provide preconfigured Fedora 21 image, so I set up the iSCSI target
...
@@ -50,7 +50,7 @@ GCE does not provide preconfigured Fedora 21 image, so I set up the iSCSI target
## Step 2. Creating the pod with iSCSI persistent storage
## Step 2. Creating the pod with iSCSI persistent storage
Once you have installed iSCSI initiator and new Kubernetes, you can create a pod based on my example *iscsi.json*. In the pod JSON, you need to provide *targetPortal* (the iSCSI target's **IP** address and *port* if not the default port 3260), target's *iqn*, *lun*, and the type of the filesystem that has been created on the lun, and *readOnly* boolean.
Once you have installed iSCSI initiator and new Kubernetes, you can create a pod based on my example *iscsi.json*. In the pod JSON, you need to provide *targetPortal* (the iSCSI target's **IP** address and *port* if not the default port 3260), target's *iqn*, *lun*, and the type of the filesystem that has been created on the lun, and *readOnly* boolean.
**Note:** If you have followed the instructions in the links above you
**Note:** If you have followed the instructions in the links above you
may have partitioned the device, the iSCSI volume plugin does not
may have partitioned the device, the iSCSI volume plugin does not
@@ -43,9 +43,9 @@ This is a follow up to the [Guestbook Example](../guestbook/README.md)'s [Go imp
...
@@ -43,9 +43,9 @@ This is a follow up to the [Guestbook Example](../guestbook/README.md)'s [Go imp
This application will run a web server which returns REDIS records for a petstore application.
This application will run a web server which returns REDIS records for a petstore application.
It is meant to simulate and test high load on Kubernetes or any other docker based system.
It is meant to simulate and test high load on Kubernetes or any other docker based system.
If you are new to Kubernetes, and you haven't run guestbook yet,
If you are new to Kubernetes, and you haven't run guestbook yet,
you might want to stop here and go back and run guestbook app first.
you might want to stop here and go back and run guestbook app first.
The guestbook tutorial will teach you a lot about the basics of Kubernetes, and we've tried not to be redundant here.
The guestbook tutorial will teach you a lot about the basics of Kubernetes, and we've tried not to be redundant here.
...
@@ -61,15 +61,15 @@ This project depends on three docker images which you can build for yourself and
...
@@ -61,15 +61,15 @@ This project depends on three docker images which you can build for yourself and
in your dockerhub "dockerhub-name".
in your dockerhub "dockerhub-name".
Since these images are already published under other parties like redis, jayunit100, and so on,
Since these images are already published under other parties like redis, jayunit100, and so on,
so you don't need to build the images to run the app.
so you don't need to build the images to run the app.
If you do want to build the images, you will need to build and push the images in this repository.
If you do want to build the images, you will need to build and push the images in this repository.
For a list of those images, see the `build-and-push` shell script - it builds and pushes all the images for you, just
For a list of those images, see the `build-and-push` shell script - it builds and pushes all the images for you, just
modify the dockerhub user name in it accordingly.
modify the dockerhub user name in it accordingly.
## Get started with the WEBAPP
## Get started with the WEBAPP
The web app is written in Go, and borrowed from the original Guestbook example by brendan burns.
The web app is written in Go, and borrowed from the original Guestbook example by brendan burns.
...
@@ -87,13 +87,13 @@ If that is all working, you can finally run `k8petstore.sh` in any Kubernetes cl
...
@@ -87,13 +87,13 @@ If that is all working, you can finally run `k8petstore.sh` in any Kubernetes cl
The web front end provides users an interface for watching pet store transactions in real time as they occur.
The web front end provides users an interface for watching pet store transactions in real time as they occur.
To generate those transactions, you can use the bigpetstore data generator. Alternatively, you could just write a
To generate those transactions, you can use the bigpetstore data generator. Alternatively, you could just write a
shell script which calls "curl localhost:3000/k8petstore/rpush/blahblahblah" over and over again :). But thats not nearly
shell script which calls "curl localhost:3000/k8petstore/rpush/blahblahblah" over and over again :). But thats not nearly
as fun, and its not a good test of a real world scenario where payloads scale and have lots of information content.
as fun, and its not a good test of a real world scenario where payloads scale and have lots of information content.
Similarly, you can locally run and test the data generator code, which is Java based, you can pull it down directly from
Similarly, you can locally run and test the data generator code, which is Java based, you can pull it down directly from
apache bigtop.
apache bigtop.
...
@@ -101,13 +101,13 @@ Directions for that are here : https://github.com/apache/bigtop/tree/master/bigt
...
@@ -101,13 +101,13 @@ Directions for that are here : https://github.com/apache/bigtop/tree/master/bigt
You will likely want to checkout the branch 2b2392bf135e9f1256bd0b930f05ae5aef8bbdcb, which is the exact commit which the current k8petstore was tested on.
You will likely want to checkout the branch 2b2392bf135e9f1256bd0b930f05ae5aef8bbdcb, which is the exact commit which the current k8petstore was tested on.
## Now what?
## Now what?
Once you have done the above 3 steps, you have a working, from source, locally runnable version of the k8petstore app, now, we can try to run it in Kubernetes.
Once you have done the above 3 steps, you have a working, from source, locally runnable version of the k8petstore app, now, we can try to run it in Kubernetes.
## Hacking, testing, benchmarking
## Hacking, testing, benchmarking
Once the app is running, you can access the app in your browser, you should see a chart
Once the app is running, you can access the app in your browser, you should see a chart
and the k8petstore title page, as well as an indicator of transaction throughput, and so on.
and the k8petstore title page, as well as an indicator of transaction throughput, and so on.
...
@@ -117,7 +117,7 @@ You can modify the HTML pages, add new REST paths to the Go app, and so on.
...
@@ -117,7 +117,7 @@ You can modify the HTML pages, add new REST paths to the Go app, and so on.
Now that you are done hacking around on the app, you can run it in Kubernetes. To do this, you will want to rebuild the docker images (most likely, for the Go web-server app), but less likely for the other images which you are less likely to need to change. Then you will push those images to dockerhub.
Now that you are done hacking around on the app, you can run it in Kubernetes. To do this, you will want to rebuild the docker images (most likely, for the Go web-server app), but less likely for the other images which you are less likely to need to change. Then you will push those images to dockerhub.
Now, how to run the entire application in Kubernetes?
Now, how to run the entire application in Kubernetes?
To simplify running this application, we have a single file, k8petstore.sh, which writes out json files on to disk. This allows us to have dynamic parameters, without needing to worry about managing multiple json files.
To simplify running this application, we have a single file, k8petstore.sh, which writes out json files on to disk. This allows us to have dynamic parameters, without needing to worry about managing multiple json files.
...
@@ -127,13 +127,13 @@ So, to run this app in Kubernetes, simply run [The all in one k8petstore.sh shel
...
@@ -127,13 +127,13 @@ So, to run this app in Kubernetes, simply run [The all in one k8petstore.sh shel
Note that at the top of the script there are a few self explanatory parameters to set, among which the Public IPs parameter is where you can checkout the web ui (at $PUBLIC_IP:3000), which will show a plot and read outs of transaction throughput.
Note that at the top of the script there are a few self explanatory parameters to set, among which the Public IPs parameter is where you can checkout the web ui (at $PUBLIC_IP:3000), which will show a plot and read outs of transaction throughput.
In the mean time, because the public IP will be deprecated in Kubernetes v1, we provide other 2 scripts k8petstore-loadbalancer.sh and k8petstore-nodeport.sh. As the names suggest, they rely on LoadBalancer and NodePort respectively. More details can be found [here](../../docs/user-guide/services.md#external-services).
In the mean time, because the public IP will be deprecated in Kubernetes v1, we provide other 2 scripts k8petstore-loadbalancer.sh and k8petstore-nodeport.sh. As the names suggest, they rely on LoadBalancer and NodePort respectively. More details can be found [here](../../docs/user-guide/services.md#external-services).
## Future
## Future
In the future, we plan to add cassandra support. Redis is a fabulous in memory data store, but it is not meant for truly available and resilient storage.
In the future, we plan to add cassandra support. Redis is a fabulous in memory data store, but it is not meant for truly available and resilient storage.
Thus we plan to add another tier of queueing, which empties the REDIS transactions into a cassandra store which persists.
Thus we plan to add another tier of queueing, which empties the REDIS transactions into a cassandra store which persists.
@@ -36,16 +36,16 @@ Documentation for other releases can be found at
...
@@ -36,16 +36,16 @@ Documentation for other releases can be found at
Install Ceph on the Kubernetes host. For example, on Fedora 21
Install Ceph on the Kubernetes host. For example, on Fedora 21
# yum -y install ceph
# yum -y install ceph
If you don't have a Ceph cluster, you can set up a [containerized Ceph cluster](https://github.com/rootfs/docker-ceph)
If you don't have a Ceph cluster, you can set up a [containerized Ceph cluster](https://github.com/rootfs/docker-ceph)
Then get the keyring from the Ceph cluster and copy it to */etc/ceph/keyring*.
Then get the keyring from the Ceph cluster and copy it to */etc/ceph/keyring*.
Once you have installed Ceph and new Kubernetes, you can create a pod based on my examples [rbd.json](rbd.json)[rbd-with-secret.json](rbd-with-secret.json). In the pod JSON, you need to provide the following information.
Once you have installed Ceph and new Kubernetes, you can create a pod based on my examples [rbd.json](rbd.json)[rbd-with-secret.json](rbd-with-secret.json). In the pod JSON, you need to provide the following information.
-*monitors*: Ceph monitors.
-*monitors*: Ceph monitors.
-*pool*: The name of the RADOS pool, if not provided, default *rbd* pool is used.
-*pool*: The name of the RADOS pool, if not provided, default *rbd* pool is used.
-*image*: The image name that rbd has created.
-*image*: The image name that rbd has created.
-*user*: The RADOS user name. If not provided, default *admin* is used.
-*user*: The RADOS user name. If not provided, default *admin* is used.
-*keyring*: The path to the keyring file. If not provided, default */etc/ceph/keyring* is used.
-*keyring*: The path to the keyring file. If not provided, default */etc/ceph/keyring* is used.
-*secretName*: The name of the authentication secrets. If provided, *secretName* overrides *keyring*. Note, see below about how to create a secret.
-*secretName*: The name of the authentication secrets. If provided, *secretName* overrides *keyring*. Note, see below about how to create a secret.
...
@@ -58,7 +58,7 @@ If Ceph authentication secret is provided, the secret should be first be base64
...
@@ -58,7 +58,7 @@ If Ceph authentication secret is provided, the secret should be first be base64