Skip to content

K
k3s
Commit dd639c63 authored by Michelle Noorali's avatar Michelle Noorali
Browse files
Options

replace contents of docs/design with stubs

parent ba611194
Expand all Show whitespace changes
Inline Side-by-side
Showing with 21 additions and 3372 deletions
docs/design/README.md
View file @ dd639c63
# Kubernetes Design Overview This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/README.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/README.md)
Kubernetes is a system for managing containerized applications across multiple
hosts, providing basic mechanisms for deployment, maintenance, and scaling of
applications.
Kubernetes establishes robust declarative primitives for maintaining the desired
state requested by the user. We see these primitives as the main value added by
Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and
replicating containers require active controllers, not just imperative
orchestration.
Kubernetes is primarily targeted at applications composed of multiple
containers, such as elastic, distributed micro-services. It is also designed to
facilitate migration of non-containerized application stacks to Kubernetes. It
therefore includes abstractions for grouping containers in both loosely coupled
and tightly coupled formations, and provides ways for containers to find and
communicate with each other in relatively familiar ways.
Kubernetes enables users to ask a cluster to run a set of containers. The system
automatically chooses hosts to run those containers on. While Kubernetes's
scheduler is currently very simple, we expect it to grow in sophistication over
time. Scheduling is a policy-rich, topology-aware, workload-specific function
that significantly impacts availability, performance, and capacity. The
scheduler needs to take into account individual and collective resource
requirements, quality of service requirements, hardware/software/policy
constraints, affinity and anti-affinity specifications, data locality,
inter-workload interference, deadlines, and so on. Workload-specific
requirements will be exposed through the API as necessary.
Kubernetes is intended to run on a number of cloud providers, as well as on
physical hosts.
A single Kubernetes cluster is not intended to span multiple availability zones.
Instead, we recommend building a higher-level layer to replicate complete
deployments of highly available applications across multiple zones (see
[the multi-cluster doc](../admin/multi-cluster.md) and [cluster federation proposal](../proposals/federation.md)
for more details).
Finally, Kubernetes aspires to be an extensible, pluggable, building-block OSS
platform and toolkit. Therefore, architecturally, we want Kubernetes to be built
as a collection of pluggable components and layers, with the ability to use
alternative schedulers, controllers, storage systems, and distribution
mechanisms, and we're evolving its current code in that direction. Furthermore,
we want others to be able to extend Kubernetes functionality, such as with
higher-level PaaS functionality or multi-cluster layers, without modification of
core Kubernetes source. Therefore, its API isn't just (or even necessarily
mainly) targeted at end users, but at tool and extension developers. Its APIs
are intended to serve as the foundation for an open ecosystem of tools,
automation systems, and higher-level API layers. Consequently, there are no
"internal" inter-component APIs. All APIs are visible and available, including
the APIs used by the scheduler, the node controller, the replication-controller
manager, Kubelet's API, etc. There's no glass to break -- in order to handle
more complex use cases, one can just access the lower-level APIs in a fully
transparent, composable manner.
For more about the Kubernetes architecture, see [architecture](architecture.md).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/README.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/admission_control.md
View file @ dd639c63
# Kubernetes Proposal - Admission Control This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control.md)
**Related PR:**
| Topic | Link |
| ----- | ---- |
| Separate validation from RESTStorage | http://issue.k8s.io/2977 |
## Background
High level goals:
* Enable an easy-to-use mechanism to provide admission control to cluster.
* Enable a provider to support multiple admission control strategies or author
their own.
* Ensure any rejected request can propagate errors back to the caller with why
the request failed.
Authorization via policy is focused on answering if a user is authorized to
perform an action.
Admission Control is focused on if the system will accept an authorized action.
Kubernetes may choose to dismiss an authorized action based on any number of
admission control strategies.
This proposal documents the basic design, and describes how any number of
admission control plug-ins could be injected.
Implementation of specific admission control strategies are handled in separate
documents.
## kube-apiserver
The kube-apiserver takes the following OPTIONAL arguments to enable admission
control:
| Option | Behavior |
| ------ | -------- |
| admission-control | Comma-delimited, ordered list of admission control choices to invoke prior to modifying or deleting an object. |
| admission-control-config-file | File with admission control configuration parameters to boot-strap plug-in. |
An **AdmissionControl** plug-in is an implementation of the following interface:
```go
package admission
// Attributes is an interface used by a plug-in to make an admission decision
// on a individual request.
type Attributes interface {
GetNamespace() string
GetKind() string
GetOperation() string
GetObject() runtime.Object
}
// Interface is an abstract, pluggable interface for Admission Control decisions.
type Interface interface {
// Admit makes an admission decision based on the request attributes
// An error is returned if it denies the request.
Admit(a Attributes) (err error)
}
```
A **plug-in** must be compiled with the binary, and is registered as an
available option by providing a name, and implementation of admission.Interface.
```go
func init() {
admission.RegisterPlugin("AlwaysDeny", func(client client.Interface, config io.Reader) (admission.Interface, error) { return NewAlwaysDeny(), nil })
}
```
A **plug-in** must be added to the imports in [plugins.go](../../cmd/kube-apiserver/app/plugins.go)
```go
// Admission policies
_ "k8s.io/kubernetes/plugin/pkg/admission/admit"
_ "k8s.io/kubernetes/plugin/pkg/admission/alwayspullimages"
_ "k8s.io/kubernetes/plugin/pkg/admission/antiaffinity"
...
_ "<YOUR NEW PLUGIN>"
```
Invocation of admission control is handled by the **APIServer** and not
individual **RESTStorage** implementations.
This design assumes that **Issue 297** is adopted, and as a consequence, the
general framework of the APIServer request/response flow will ensure the
following:
1. Incoming request
2. Authenticate user
3. Authorize user
4. If operation=create|update|delete|connect, then admission.Admit(requestAttributes)
- invoke each admission.Interface object in sequence
5. Case on the operation:
- If operation=create|update, then validate(object) and persist
- If operation=delete, delete the object
- If operation=connect, exec
If at any step, there is an error, the request is canceled.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/admission_control.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/admission_control_limit_range.md
View file @ dd639c63
# Admission control plugin: LimitRanger This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_limit_range.md)
## Background
This document proposes a system for enforcing resource requirements constraints
as part of admission control.
## Use cases
1. Ability to enumerate resource requirement constraints per namespace
2. Ability to enumerate min/max resource constraints for a pod
3. Ability to enumerate min/max resource constraints for a container
4. Ability to specify default resource limits for a container
5. Ability to specify default resource requests for a container
6. Ability to enforce a ratio between request and limit for a resource.
7. Ability to enforce min/max storage requests for persistent volume claims
## Data Model
The **LimitRange** resource is scoped to a **Namespace**.
### Type
```go
// LimitType is a type of object that is limited
type LimitType string
const (
// Limit that applies to all pods in a namespace
LimitTypePod LimitType = "Pod"
// Limit that applies to all containers in a namespace
LimitTypeContainer LimitType = "Container"
)
// LimitRangeItem defines a min/max usage limit for any resource that matches
// on kind.
type LimitRangeItem struct {
// Type of resource that this limit applies to.
Type LimitType `json:"type,omitempty"`
// Max usage constraints on this kind by resource name.
Max ResourceList `json:"max,omitempty"`
// Min usage constraints on this kind by resource name.
Min ResourceList `json:"min,omitempty"`
// Default resource requirement limit value by resource name if resource limit
// is omitted.
Default ResourceList `json:"default,omitempty"`
// DefaultRequest is the default resource requirement request value by
// resource name if resource request is omitted.
DefaultRequest ResourceList `json:"defaultRequest,omitempty"`
// MaxLimitRequestRatio if specified, the named resource must have a request
// and limit that are both non-zero where limit divided by request is less
// than or equal to the enumerated value; this represents the max burst for
// the named resource.
MaxLimitRequestRatio ResourceList `json:"maxLimitRequestRatio,omitempty"`
}
// LimitRangeSpec defines a min/max usage limit for resources that match
// on kind.
type LimitRangeSpec struct {
// Limits is the list of LimitRangeItem objects that are enforced.
Limits []LimitRangeItem `json:"limits"`
}
// LimitRange sets resource usage limits for each kind of resource in a
// Namespace.
type LimitRange struct {
TypeMeta `json:",inline"`
// Standard object's metadata.
// More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
ObjectMeta `json:"metadata,omitempty"`
// Spec defines the limits enforced.
// More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status
Spec LimitRangeSpec `json:"spec,omitempty"`
}
// LimitRangeList is a list of LimitRange items.
type LimitRangeList struct {
TypeMeta `json:",inline"`
// Standard list metadata.
// More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds
ListMeta `json:"metadata,omitempty"`
// Items is a list of LimitRange objects.
// More info:
// http://releases.k8s.io/HEAD/docs/design/admission_control_limit_range.md
Items []LimitRange `json:"items"`
}
```
### Validation
Validation of a **LimitRange** enforces that for a given named resource the
following rules apply:
Min (if specified) <= DefaultRequest (if specified) <= Default (if specified)
<= Max (if specified)
### Default Value Behavior
The following default value behaviors are applied to a LimitRange for a given
named resource.
```
if LimitRangeItem.Default[resourceName] is undefined
if LimitRangeItem.Max[resourceName] is defined
LimitRangeItem.Default[resourceName] = LimitRangeItem.Max[resourceName]
```
```
if LimitRangeItem.DefaultRequest[resourceName] is undefined
if LimitRangeItem.Default[resourceName] is defined
LimitRangeItem.DefaultRequest[resourceName] = LimitRangeItem.Default[resourceName]
else if LimitRangeItem.Min[resourceName] is defined
LimitRangeItem.DefaultRequest[resourceName] = LimitRangeItem.Min[resourceName]
```
## AdmissionControl plugin: LimitRanger
The **LimitRanger** plug-in introspects all incoming pod requests and evaluates
the constraints defined on a LimitRange.
If a constraint is not specified for an enumerated resource, it is not enforced
or tracked.
To enable the plug-in and support for LimitRange, the kube-apiserver must be
configured as follows:
```console
$ kube-apiserver --admission-control=LimitRanger
```
### Enforcement of constraints
**Type: Container**
Supported Resources:
1. memory
2. cpu
Supported Constraints:
Per container, the following must hold true:
| Constraint | Behavior |
| ---------- | -------- |
| Min | Min <= Request (required) <= Limit (optional) |
| Max | Limit (required) <= Max |
| LimitRequestRatio | LimitRequestRatio <= ( Limit (required, non-zero) / Request (required, non-zero)) |
Supported Defaults:
1. Default - if the named resource has no enumerated value, the Limit is equal
to the Default
2. DefaultRequest - if the named resource has no enumerated value, the Request
is equal to the DefaultRequest
**Type: Pod**
Supported Resources:
1. memory
2. cpu
Supported Constraints:
Across all containers in pod, the following must hold true
| Constraint | Behavior |
| ---------- | -------- |
| Min | Min <= Request (required) <= Limit (optional) |
| Max | Limit (required) <= Max |
| LimitRequestRatio | LimitRequestRatio <= ( Limit (required, non-zero) / Request (non-zero) ) |
**Type: PersistentVolumeClaim**
Supported Resources:
1. storage
Supported Constraints:
Across all claims in a namespace, the following must hold true:
| Constraint | Behavior |
| ---------- | -------- |
| Min | Min >= Request (required) |
| Max | Max <= Request (required) |
Supported Defaults: None. Storage is a required field in `PersistentVolumeClaim`, so defaults are not applied at this time.
## Run-time configuration
The default ```LimitRange``` that is applied via Salt configuration will be
updated as follows:
```
apiVersion: "v1"
kind: "LimitRange"
metadata:
name: "limits"
namespace: default
spec:
limits:
- type: "Container"
defaultRequests:
cpu: "100m"
```
## Example
An example LimitRange configuration:
| Type | Resource | Min | Max | Default | DefaultRequest | LimitRequestRatio |
| ---- | -------- | --- | --- | ------- | -------------- | ----------------- |
| Container | cpu | .1 | 1 | 500m | 250m | 4 |
| Container | memory | 250Mi | 1Gi | 500Mi | 250Mi | |
Assuming an incoming container that specified no incoming resource requirements,
the following would happen.
1. The incoming container cpu would request 250m with a limit of 500m.
2. The incoming container memory would request 250Mi with a limit of 500Mi
3. If the container is later resized, it's cpu would be constrained to between
.1 and 1 and the ratio of limit to request could not exceed 4.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/admission_control_limit_range.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/architecture.md
View file @ dd639c63
# Kubernetes architecture This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md)
A running Kubernetes cluster contains node agents (`kubelet`) and master
components (APIs, scheduler, etc), on top of a distributed storage solution.
This diagram shows our desired eventual state, though we're still working on a
few things, like making `kubelet` itself (all our components, really) run within
containers, and making the scheduler 100% pluggable.
![Architecture Diagram](architecture.png?raw=true "Architecture overview")
## The Kubernetes Node
When looking at the architecture of the system, we'll break it down to services
that run on the worker node and services that compose the cluster-level control
plane.
The Kubernetes node has the services necessary to run application containers and
be managed from the master systems.
Each node runs Docker, of course. Docker takes care of the details of
downloading images and running containers.
### `kubelet`
The `kubelet` manages [pods](../user-guide/pods.md) and their containers, their
images, their volumes, etc.
### `kube-proxy`
Each node also runs a simple network proxy and load balancer (see the
[services FAQ](https://github.com/kubernetes/kubernetes/wiki/Services-FAQ) for
more details). This reflects `services` (see
[the services doc](../user-guide/services.md) for more details) as defined in
the Kubernetes API on each node and can do simple TCP and UDP stream forwarding
(round robin) across a set of backends.
Service endpoints are currently found via [DNS](../admin/dns.md) or through
environment variables (both
[Docker-links-compatible](https://docs.docker.com/userguide/dockerlinks/) and
Kubernetes `{FOO}_SERVICE_HOST` and `{FOO}_SERVICE_PORT` variables are
supported). These variables resolve to ports managed by the service proxy.
## The Kubernetes Control Plane
The Kubernetes control plane is split into a set of components. Currently they
all run on a single _master_ node, but that is expected to change soon in order
to support high-availability clusters. These components work together to provide
a unified view of the cluster.
### `etcd`
All persistent master state is stored in an instance of `etcd`. This provides a
great way to store configuration data reliably. With `watch` support,
coordinating components can be notified very quickly of changes.
### Kubernetes API Server
The apiserver serves up the [Kubernetes API](../api.md). It is intended to be a
CRUD-y server, with most/all business logic implemented in separate components
or in plug-ins. It mainly processes REST operations, validates them, and updates
the corresponding objects in `etcd` (and eventually other stores).
### Scheduler
The scheduler binds unscheduled pods to nodes via the `/binding` API. The
scheduler is pluggable, and we expect to support multiple cluster schedulers and
even user-provided schedulers in the future.
### Kubernetes Controller Manager Server
All other cluster-level functions are currently performed by the Controller
Manager. For instance, `Endpoints` objects are created and updated by the
endpoints controller, and nodes are discovered, managed, and monitored by the
node controller. These could eventually be split into separate components to
make them independently pluggable.
The [`replicationcontroller`](../user-guide/replication-controller.md) is a
mechanism that is layered on top of the simple [`pod`](../user-guide/pods.md)
API. We eventually plan to port it to a generic plug-in mechanism, once one is
implemented.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/architecture.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/clustering.md
View file @ dd639c63
# Clustering in Kubernetes This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/clustering.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/clustering.md)
## Overview
The term "clustering" refers to the process of having all members of the
Kubernetes cluster find and trust each other. There are multiple different ways
to achieve clustering with different security and usability profiles. This
document attempts to lay out the user experiences for clustering that Kubernetes
aims to address.
Once a cluster is established, the following is true:
1. **Master -> Node** The master needs to know which nodes can take work and
what their current status is wrt capacity.
1. **Location** The master knows the name and location of all of the nodes in
the cluster.
* For the purposes of this doc, location and name should be enough
information so that the master can open a TCP connection to the Node. Most
probably we will make this either an IP address or a DNS name. It is going to be
important to be consistent here (master must be able to reach kubelet on that
DNS name) so that we can verify certificates appropriately.
2. **Target AuthN** A way to securely talk to the kubelet on that node.
Currently we call out to the kubelet over HTTP. This should be over HTTPS and
the master should know what CA to trust for that node.
3. **Caller AuthN/Z** This would be the master verifying itself (and
permissions) when calling the node. Currently, this is only used to collect
statistics as authorization isn't critical. This may change in the future
though.
2. **Node -> Master** The nodes currently talk to the master to know which pods
have been assigned to them and to publish events.
1. **Location** The nodes must know where the master is at.
2. **Target AuthN** Since the master is assigning work to the nodes, it is
critical that they verify whom they are talking to.
3. **Caller AuthN/Z** The nodes publish events and so must be authenticated to
the master. Ideally this authentication is specific to each node so that
authorization can be narrowly scoped. The details of the work to run (including
things like environment variables) might be considered sensitive and should be
locked down also.
**Note:** While the description here refers to a singular Master, in the future
we should enable multiple Masters operating in an HA mode. While the "Master" is
currently the combination of the API Server, Scheduler and Controller Manager,
we will restrict ourselves to thinking about the main API and policy engine --
the API Server.
## Current Implementation
A central authority (generally the master) is responsible for determining the
set of machines which are members of the cluster. Calls to create and remove
worker nodes in the cluster are restricted to this single authority, and any
other requests to add or remove worker nodes are rejected. (1.i.)
Communication from the master to nodes is currently over HTTP and is not secured
or authenticated in any way. (1.ii, 1.iii.)
The location of the master is communicated out of band to the nodes. For GCE,
this is done via Salt. Other cluster instructions/scripts use other methods.
(2.i.)
Currently most communication from the node to the master is over HTTP. When it
is done over HTTPS there is currently no verification of the cert of the master
(2.ii.)
Currently, the node/kubelet is authenticated to the master via a token shared
across all nodes. This token is distributed out of band (using Salt for GCE) and
is optional. If it is not present then the kubelet is unable to publish events
to the master. (2.iii.)
Our current mix of out of band communication doesn't meet all of our needs from
a security point of view and is difficult to set up and configure.
## Proposed Solution
The proposed solution will provide a range of options for setting up and
maintaining a secure Kubernetes cluster. We want to both allow for centrally
controlled systems (leveraging pre-existing trust and configuration systems) or
more ad-hoc automagic systems that are incredibly easy to set up.
The building blocks of an easier solution:
* **Move to TLS** We will move to using TLS for all intra-cluster communication.
We will explicitly identify the trust chain (the set of trusted CAs) as opposed
to trusting the system CAs. We will also use client certificates for all AuthN.
* [optional] **API driven CA** Optionally, we will run a CA in the master that
will mint certificates for the nodes/kubelets. There will be pluggable policies
that will automatically approve certificate requests here as appropriate.
* **CA approval policy** This is a pluggable policy object that can
automatically approve CA signing requests. Stock policies will include
`always-reject`, `queue` and `insecure-always-approve`. With `queue` there would
be an API for evaluating and accepting/rejecting requests. Cloud providers could
implement a policy here that verifies other out of band information and
automatically approves/rejects based on other external factors.
* **Scoped Kubelet Accounts** These accounts are per-node and (optionally) give
a node permission to register itself.
* To start with, we'd have the kubelets generate a cert/account in the form of
`kubelet:<host>`. To start we would then hard code policy such that we give that
particular account appropriate permissions. Over time, we can make the policy
engine more generic.
* [optional] **Bootstrap API endpoint** This is a helper service hosted outside
of the Kubernetes cluster that helps with initial discovery of the master.
### Static Clustering
In this sequence diagram there is out of band admin entity that is creating all
certificates and distributing them. It is also making sure that the kubelets
know where to find the master. This provides for a lot of control but is more
difficult to set up as lots of information must be communicated outside of
Kubernetes.
![Static Sequence Diagram](clustering/static.png)
### Dynamic Clustering
This diagram shows dynamic clustering using the bootstrap API endpoint. This
endpoint is used to both find the location of the master and communicate the
root CA for the master.
This flow has the admin manually approving the kubelet signing requests. This is
the `queue` policy defined above. This manual intervention could be replaced by
code that can verify the signing requests via other means.
![Dynamic Sequence Diagram](clustering/dynamic.png)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/clustering.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/clustering/Dockerfile deleted 100644 → 0
View file @ ba611194
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM debian:jessie
RUN apt-get update
RUN apt-get -qy install python-seqdiag make curl
WORKDIR /diagrams
RUN curl -sLo DroidSansMono.ttf https://googlefontdirectory.googlecode.com/hg/apache/droidsansmono/DroidSansMono.ttf
ADD . /diagrams
CMD bash -c 'make >/dev/stderr && tar cf - *.png'
\ No newline at end of file
docs/design/clustering/Makefile deleted 100644 → 0
View file @ ba611194
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FONT := DroidSansMono.ttf
PNGS := $(patsubst %.seqdiag,%.png,$(wildcard *.seqdiag))
.PHONY: all
all: $(PNGS)
.PHONY: watch
watch:
fswatch *.seqdiag | xargs -n 1 sh -c "make || true"
$(FONT):
curl -sLo $@ https://googlefontdirectory.googlecode.com/hg/apache/droidsansmono/$(FONT)
%.png: %.seqdiag $(FONT)
seqdiag --no-transparency -a -f '$(FONT)' $<
# Build the stuff via a docker image
.PHONY: docker
docker:
docker build -t clustering-seqdiag .
docker run --rm clustering-seqdiag | tar xvf -
.PHONY: docker-clean
docker-clean:
docker rmi clustering-seqdiag || true
docker images -q --filter "dangling=true" | xargs docker rmi
docs/design/clustering/README.md
View file @ dd639c63
This directory contains diagrams for the clustering design doc. This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/clustering/README.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/clustering/README.md)
This depends on the `seqdiag` [utility](http://blockdiag.com/en/seqdiag/index.html).
Assuming you have a non-borked python install, this should be installable with:
```sh
pip install seqdiag
```
Just call `make` to regenerate the diagrams.
## Building with Docker
If you are on a Mac or your pip install is messed up, you can easily build with
docker:
```sh
make docker
```
The first run will be slow but things should be fast after that.
To clean up the docker containers that are created (and other cruft that is left
around) you can run `make docker-clean`.
## Automatically rebuild on file changes
If you have the fswatch utility installed, you can have it monitor the file
system and automatically rebuild when files have changed. Just do a
`make watch`.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/clustering/README.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/clustering/dynamic.seqdiag deleted 100644 → 0
View file @ ba611194
seqdiag {
activation = none;
user[label = "Admin User"];
bootstrap[label = "Bootstrap API\nEndpoint"];
master;
kubelet[stacked];
user -> bootstrap [label="createCluster", return="cluster ID"];
user <-- bootstrap [label="returns\n- bootstrap-cluster-uri"];
user ->> master [label="start\n- bootstrap-cluster-uri"];
master => bootstrap [label="setMaster\n- master-location\n- master-ca"];
user ->> kubelet [label="start\n- bootstrap-cluster-uri"];
kubelet => bootstrap [label="get-master", return="returns\n- master-location\n- master-ca"];
kubelet ->> master [label="signCert\n- unsigned-kubelet-cert", return="returns\n- kubelet-cert"];
user => master [label="getSignRequests"];
user => master [label="approveSignRequests"];
kubelet <<-- master [label="returns\n- kubelet-cert"];
kubelet => master [label="register\n- kubelet-location"]
}
docs/design/clustering/static.seqdiag deleted 100644 → 0
View file @ ba611194
seqdiag {
activation = none;
admin[label = "Manual Admin"];
ca[label = "Manual CA"]
master;
kubelet[stacked];
admin => ca [label="create\n- master-cert"];
admin ->> master [label="start\n- ca-root\n- master-cert"];
admin => ca [label="create\n- kubelet-cert"];
admin ->> kubelet [label="start\n- ca-root\n- kubelet-cert\n- master-location"];
kubelet => master [label="register\n- kubelet-location"];
}
docs/design/command_execution_port_forwarding.md
View file @ dd639c63
# Container Command Execution & Port Forwarding in Kubernetes This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/command_execution_port_forwarding.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/command_execution_port_forwarding.md)
## Abstract
This document describes how to use Kubernetes to execute commands in containers,
with stdin/stdout/stderr streams attached and how to implement port forwarding
to the containers.
## Background
See the following related issues/PRs:
- [Support attach](http://issue.k8s.io/1521)
- [Real container ssh](http://issue.k8s.io/1513)
- [Provide easy debug network access to services](http://issue.k8s.io/1863)
- [OpenShift container command execution proposal](https://github.com/openshift/origin/pull/576)
## Motivation
Users and administrators are accustomed to being able to access their systems
via SSH to run remote commands, get shell access, and do port forwarding.
Supporting SSH to containers in Kubernetes is a difficult task. You must
specify a "user" and a hostname to make an SSH connection, and `sshd` requires
real users (resolvable by NSS and PAM). Because a container belongs to a pod,
and the pod belongs to a namespace, you need to specify namespace/pod/container
to uniquely identify the target container. Unfortunately, a
namespace/pod/container is not a real user as far as SSH is concerned. Also,
most Linux systems limit user names to 32 characters, which is unlikely to be
large enough to contain namespace/pod/container. We could devise some scheme to
map each namespace/pod/container to a 32-character user name, adding entries to
`/etc/passwd` (or LDAP, etc.) and keeping those entries fully in sync all the
time. Alternatively, we could write custom NSS and PAM modules that allow the
host to resolve a namespace/pod/container to a user without needing to keep
files or LDAP in sync.
As an alternative to SSH, we are using a multiplexed streaming protocol that
runs on top of HTTP. There are no requirements about users being real users,
nor is there any limitation on user name length, as the protocol is under our
control. The only downside is that standard tooling that expects to use SSH
won't be able to work with this mechanism, unless adapters can be written.
## Constraints and Assumptions
- SSH support is not currently in scope.
- CGroup confinement is ultimately desired, but implementing that support is not
currently in scope.
- SELinux confinement is ultimately desired, but implementing that support is
not currently in scope.
## Use Cases
- A user of a Kubernetes cluster wants to run arbitrary commands in a
container with local stdin/stdout/stderr attached to the container.
- A user of a Kubernetes cluster wants to connect to local ports on his computer
and have them forwarded to ports in a container.
## Process Flow
### Remote Command Execution Flow
1. The client connects to the Kubernetes Master to initiate a remote command
execution request.
2. The Master proxies the request to the Kubelet where the container lives.
3. The Kubelet executes nsenter + the requested command and streams
stdin/stdout/stderr back and forth between the client and the container.
### Port Forwarding Flow
1. The client connects to the Kubernetes Master to initiate a remote command
execution request.
2. The Master proxies the request to the Kubelet where the container lives.
3. The client listens on each specified local port, awaiting local connections.
4. The client connects to one of the local listening ports.
4. The client notifies the Kubelet of the new connection.
5. The Kubelet executes nsenter + socat and streams data back and forth between
the client and the port in the container.
## Design Considerations
### Streaming Protocol
The current multiplexed streaming protocol used is SPDY. This is not the
long-term desire, however. As soon as there is viable support for HTTP/2 in Go,
we will switch to that.
### Master as First Level Proxy
Clients should not be allowed to communicate directly with the Kubelet for
security reasons. Therefore, the Master is currently the only suggested entry
point to be used for remote command execution and port forwarding. This is not
necessarily desirable, as it means that all remote command execution and port
forwarding traffic must travel through the Master, potentially impacting other
API requests.
In the future, it might make more sense to retrieve an authorization token from
the Master, and then use that token to initiate a remote command execution or
port forwarding request with a load balanced proxy service dedicated to this
functionality. This would keep the streaming traffic out of the Master.
### Kubelet as Backend Proxy
The kubelet is currently responsible for handling remote command execution and
port forwarding requests. Just like with the Master described above, this means
that all remote command execution and port forwarding streaming traffic must
travel through the Kubelet, which could result in a degraded ability to service
other requests.
In the future, it might make more sense to use a separate service on the node.
Alternatively, we could possibly inject a process into the container that only
listens for a single request, expose that process's listening port on the node,
and then issue a redirect to the client such that it would connect to the first
level proxy, which would then proxy directly to the injected process's exposed
port. This would minimize the amount of proxying that takes place.
### Scalability
There are at least 2 different ways to execute a command in a container:
`docker exec` and `nsenter`. While `docker exec` might seem like an easier and
more obvious choice, it has some drawbacks.
#### `docker exec`
We could expose `docker exec` (i.e. have Docker listen on an exposed TCP port
on the node), but this would require proxying from the edge and securing the
Docker API. `docker exec` calls go through the Docker daemon, meaning that all
stdin/stdout/stderr traffic is proxied through the Daemon, adding an extra hop.
Additionally, you can't isolate 1 malicious `docker exec` call from normal
usage, meaning an attacker could initiate a denial of service or other attack
and take down the Docker daemon, or the node itself.
We expect remote command execution and port forwarding requests to be long
running and/or high bandwidth operations, and routing all the streaming data
through the Docker daemon feels like a bottleneck we can avoid.
#### `nsenter`
The implementation currently uses `nsenter` to run commands in containers,
joining the appropriate container namespaces. `nsenter` runs directly on the
node and is not proxied through any single daemon process.
### Security
Authentication and authorization hasn't specifically been tested yet with this
functionality. We need to make sure that users are not allowed to execute
remote commands or do port forwarding to containers they aren't allowed to
access.
Additional work is required to ensure that multiple command execution or port
forwarding connections from different clients are not able to see each other's
data. This can most likely be achieved via SELinux labeling and unique process
contexts.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/command_execution_port_forwarding.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/configmap.md
View file @ dd639c63
# Generic Configuration Object This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/configmap.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/configmap.md)
## Abstract
The `ConfigMap` API resource stores data used for the configuration of
applications deployed on Kubernetes.
The main focus of this resource is to:
* Provide dynamic distribution of configuration data to deployed applications.
* Encapsulate configuration information and simplify `Kubernetes` deployments.
* Create a flexible configuration model for `Kubernetes`.
## Motivation
A `Secret`-like API resource is needed to store configuration data that pods can
consume.
Goals of this design:
1. Describe a `ConfigMap` API resource.
2. Describe the semantics of consuming `ConfigMap` as environment variables.
3. Describe the semantics of consuming `ConfigMap` as files in a volume.
## Use Cases
1. As a user, I want to be able to consume configuration data as environment
variables.
2. As a user, I want to be able to consume configuration data as files in a
volume.
3. As a user, I want my view of configuration data in files to be eventually
consistent with changes to the data.
### Consuming `ConfigMap` as Environment Variables
A series of events for consuming `ConfigMap` as environment variables:
1. Create a `ConfigMap` object.
2. Create a pod to consume the configuration data via environment variables.
3. The pod is scheduled onto a node.
4. The Kubelet retrieves the `ConfigMap` resource(s) referenced by the pod and
starts the container processes with the appropriate configuration data from
environment variables.
### Consuming `ConfigMap` in Volumes
A series of events for consuming `ConfigMap` as configuration files in a volume:
1. Create a `ConfigMap` object.
2. Create a new pod using the `ConfigMap` via a volume plugin.
3. The pod is scheduled onto a node.
4. The Kubelet creates an instance of the volume plugin and calls its `Setup()`
method.
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod
and projects the appropriate configuration data into the volume.
### Consuming `ConfigMap` Updates
Any long-running system has configuration that is mutated over time. Changes
made to configuration data must be made visible to pods consuming data in
volumes so that they can respond to those changes.
The `resourceVersion` of the `ConfigMap` object will be updated by the API
server every time the object is modified. After an update, modifications will be
made visible to the consumer container:
1. Create a `ConfigMap` object.
2. Create a new pod using the `ConfigMap` via the volume plugin.
3. The pod is scheduled onto a node.
4. During the sync loop, the Kubelet creates an instance of the volume plugin
and calls its `Setup()` method.
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod
and projects the appropriate data into the volume.
6. The `ConfigMap` referenced by the pod is updated.
7. During the next iteration of the `syncLoop`, the Kubelet creates an instance
of the volume plugin and calls its `Setup()` method.
8. The volume plugin projects the updated data into the volume atomically.
It is the consuming pod's responsibility to make use of the updated data once it
is made visible.
Because environment variables cannot be updated without restarting a container,
configuration data consumed in environment variables will not be updated.
### Advantages
* Easy to consume in pods; consumer-agnostic
* Configuration data is persistent and versioned
* Consumers of configuration data in volumes can respond to changes in the data
## Proposed Design
### API Resource
The `ConfigMap` resource will be added to the main API:
```go
package api
// ConfigMap holds configuration data for pods to consume.
type ConfigMap struct {
TypeMeta `json:",inline"`
ObjectMeta `json:"metadata,omitempty"`
// Data contains the configuration data. Each key must be a valid
// DNS_SUBDOMAIN or leading dot followed by valid DNS_SUBDOMAIN.
Data map[string]string `json:"data,omitempty"`
}
type ConfigMapList struct {
TypeMeta `json:",inline"`
ListMeta `json:"metadata,omitempty"`
Items []ConfigMap `json:"items"`
}
```
A `Registry` implementation for `ConfigMap` will be added to
`pkg/registry/configmap`.
### Environment Variables
The `EnvVarSource` will be extended with a new selector for `ConfigMap`:
```go
package api
// EnvVarSource represents a source for the value of an EnvVar.
type EnvVarSource struct {
// other fields omitted
// Selects a key of a ConfigMap.
ConfigMapKeyRef *ConfigMapKeySelector `json:"configMapKeyRef,omitempty"`
}
// Selects a key from a ConfigMap.
type ConfigMapKeySelector struct {
// The ConfigMap to select from.
LocalObjectReference `json:",inline"`
// The key to select.
Key string `json:"key"`
}
```
### Volume Source
A new `ConfigMapVolumeSource` type of volume source containing the `ConfigMap`
object will be added to the `VolumeSource` struct in the API:
```go
package api
type VolumeSource struct {
// other fields omitted
ConfigMap *ConfigMapVolumeSource `json:"configMap,omitempty"`
}
// Represents a volume that holds configuration data.
type ConfigMapVolumeSource struct {
LocalObjectReference `json:",inline"`
// A list of keys to project into the volume.
// If unspecified, each key-value pair in the Data field of the
// referenced ConfigMap will be projected into the volume as a file whose name
// is the key and content is the value.
// If specified, the listed keys will be project into the specified paths, and
// unlisted keys will not be present.
Items []KeyToPath `json:"items,omitempty"`
}
// Represents a mapping of a key to a relative path.
type KeyToPath struct {
// The name of the key to select
Key string `json:"key"`
// The relative path name of the file to be created.
// Must not be absolute or contain the '..' path. Must be utf-8 encoded.
// The first item of the relative path must not start with '..'
Path string `json:"path"`
}
```
**Note:** The update logic used in the downward API volume plug-in will be
extracted and re-used in the volume plug-in for `ConfigMap`.
### Changes to Secret
We will update the Secret volume plugin to have a similar API to the new
`ConfigMap` volume plugin. The secret volume plugin will also begin updating
secret content in the volume when secrets change.
## Examples
#### Consuming `ConfigMap` as Environment Variables
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: etcd-env-config
data:
number-of-members: "1"
initial-cluster-state: new
initial-cluster-token: DUMMY_ETCD_INITIAL_CLUSTER_TOKEN
discovery-token: DUMMY_ETCD_DISCOVERY_TOKEN
discovery-url: http://etcd-discovery:2379
etcdctl-peers: http://etcd:2379
```
This pod consumes the `ConfigMap` as environment variables:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: config-env-example
spec:
containers:
- name: etcd
image: openshift/etcd-20-centos7
ports:
- containerPort: 2379
protocol: TCP
- containerPort: 2380
protocol: TCP
env:
- name: ETCD_NUM_MEMBERS
valueFrom:
configMapKeyRef:
name: etcd-env-config
key: number-of-members
- name: ETCD_INITIAL_CLUSTER_STATE
valueFrom:
configMapKeyRef:
name: etcd-env-config
key: initial-cluster-state
- name: ETCD_DISCOVERY_TOKEN
valueFrom:
configMapKeyRef:
name: etcd-env-config
key: discovery-token
- name: ETCD_DISCOVERY_URL
valueFrom:
configMapKeyRef:
name: etcd-env-config
key: discovery-url
- name: ETCDCTL_PEERS
valueFrom:
configMapKeyRef:
name: etcd-env-config
key: etcdctl-peers
```
#### Consuming `ConfigMap` as Volumes
`redis-volume-config` is intended to be used as a volume containing a config
file:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-volume-config
data:
redis.conf: "pidfile /var/run/redis.pid\nport 6379\ntcp-backlog 511\ndatabases 1\ntimeout 0\n"
```
The following pod consumes the `redis-volume-config` in a volume:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: config-volume-example
spec:
containers:
- name: redis
image: kubernetes/redis
command: ["redis-server", "/mnt/config-map/etc/redis.conf"]
ports:
- containerPort: 6379
volumeMounts:
- name: config-map-volume
mountPath: /mnt/config-map
volumes:
- name: config-map-volume
configMap:
name: redis-volume-config
items:
- path: "etc/redis.conf"
key: redis.conf
```
## Future Improvements
In the future, we may add the ability to specify an init-container that can
watch the volume contents for updates and respond to changes when they occur.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/configmap.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/daemon.md
View file @ dd639c63
# DaemonSet in Kubernetes This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/daemon.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/daemon.md)
**Author**: Ananya Kumar (@AnanyaKumar)
**Status**: Implemented.
This document presents the design of the Kubernetes DaemonSet, describes use
cases, and gives an overview of the code.
## Motivation
Many users have requested for a way to run a daemon on every node in a
Kubernetes cluster, or on a certain set of nodes in a cluster. This is essential
for use cases such as building a sharded datastore, or running a logger on every
node. In comes the DaemonSet, a way to conveniently create and manage
daemon-like workloads in Kubernetes.
## Use Cases
The DaemonSet can be used for user-specified system services, cluster-level
applications with strong node ties, and Kubernetes node services. Below are
example use cases in each category.
### User-Specified System Services:
Logging: Some users want a way to collect statistics about nodes in a cluster
and send those logs to an external database. For example, system administrators
might want to know if their machines are performing as expected, if they need to
add more machines to the cluster, or if they should switch cloud providers. The
DaemonSet can be used to run a data collection service (for example fluentd) on
every node and send the data to a service like ElasticSearch for analysis.
### Cluster-Level Applications
Datastore: Users might want to implement a sharded datastore in their cluster. A
few nodes in the cluster, labeled ‘app=datastore’, might be responsible for
storing data shards, and pods running on these nodes might serve data. This
architecture requires a way to bind pods to specific nodes, so it cannot be
achieved using a Replication Controller. A DaemonSet is a convenient way to
implement such a datastore.
For other uses, see the related [feature request](https://issues.k8s.io/1518)
## Functionality
The DaemonSet supports standard API features:
- create
- The spec for DaemonSets has a pod template field.
- Using the pod’s nodeSelector field, DaemonSets can be restricted to operate
over nodes that have a certain label. For example, suppose that in a cluster
some nodes are labeled ‘app=database’. You can use a DaemonSet to launch a
datastore pod on exactly those nodes labeled ‘app=database’.
- Using the pod's nodeName field, DaemonSets can be restricted to operate on a
specified node.
- The PodTemplateSpec used by the DaemonSet is the same as the PodTemplateSpec
used by the Replication Controller.
- The initial implementation will not guarantee that DaemonSet pods are
created on nodes before other pods.
- The initial implementation of DaemonSet does not guarantee that DaemonSet
pods show up on nodes (for example because of resource limitations of the node),
but makes a best effort to launch DaemonSet pods (like Replication Controllers
do with pods). Subsequent revisions might ensure that DaemonSet pods show up on
nodes, preempting other pods if necessary.
- The DaemonSet controller adds an annotation:
```"kubernetes.io/created-by: \<json API object reference\>"```
- YAML example:
```YAML
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
app: datastore
name: datastore
spec:
template:
metadata:
labels:
app: datastore-shard
spec:
nodeSelector:
app: datastore-node
containers:
name: datastore-shard
image: kubernetes/sharded
ports:
- containerPort: 9042
name: main
```
- commands that get info:
- get (e.g. kubectl get daemonsets)
- describe
- Modifiers:
- delete (if --cascade=true, then first the client turns down all the pods
controlled by the DaemonSet (by setting the nodeSelector to a uuid pair that is
unlikely to be set on any node); then it deletes the DaemonSet; then it deletes
the pods)
- label
- annotate
- update operations like patch and replace (only allowed to selector and to
nodeSelector and nodeName of pod template)
- DaemonSets have labels, so you could, for example, list all DaemonSets
with certain labels (the same way you would for a Replication Controller).
In general, for all the supported features like get, describe, update, etc,
the DaemonSet works in a similar way to the Replication Controller. However,
note that the DaemonSet and the Replication Controller are different constructs.
### Persisting Pods
- Ordinary liveness probes specified in the pod template work to keep pods
created by a DaemonSet running.
- If a daemon pod is killed or stopped, the DaemonSet will create a new
replica of the daemon pod on the node.
### Cluster Mutations
- When a new node is added to the cluster, the DaemonSet controller starts
daemon pods on the node for DaemonSets whose pod template nodeSelectors match
the node’s labels.
- Suppose the user launches a DaemonSet that runs a logging daemon on all
nodes labeled “logger=fluentd”. If the user then adds the “logger=fluentd” label
to a node (that did not initially have the label), the logging daemon will
launch on the node. Additionally, if a user removes the label from a node, the
logging daemon on that node will be killed.
## Alternatives Considered
We considered several alternatives, that were deemed inferior to the approach of
creating a new DaemonSet abstraction.
One alternative is to include the daemon in the machine image. In this case it
would run outside of Kubernetes proper, and thus not be monitored, health
checked, usable as a service endpoint, easily upgradable, etc.
A related alternative is to package daemons as static pods. This would address
most of the problems described above, but they would still not be easily
upgradable, and more generally could not be managed through the API server
interface.
A third alternative is to generalize the Replication Controller. We would do
something like: if you set the `replicas` field of the ReplicationControllerSpec
to -1, then it means "run exactly one replica on every node matching the
nodeSelector in the pod template." The ReplicationController would pretend
`replicas` had been set to some large number -- larger than the largest number
of nodes ever expected in the cluster -- and would use some anti-affinity
mechanism to ensure that no more than one Pod from the ReplicationController
runs on any given node. There are two downsides to this approach. First,
there would always be a large number of Pending pods in the scheduler (these
will be scheduled onto new machines when they are added to the cluster). The
second downside is more philosophical: DaemonSet and the Replication Controller
are very different concepts. We believe that having small, targeted controllers
for distinct purposes makes Kubernetes easier to understand and use, compared to
having larger multi-functional controllers (see
["Convert ReplicationController to a plugin"](http://issues.k8s.io/3058) for
some discussion of this topic).
## Design
#### Client
- Add support for DaemonSet commands to kubectl and the client. Client code was
added to pkg/client/unversioned. The main files in Kubectl that were modified are
pkg/kubectl/describe.go and pkg/kubectl/stop.go, since for other calls like Get, Create,
and Update, the client simply forwards the request to the backend via the REST
API.
#### Apiserver
- Accept, parse, validate client commands
- REST API calls are handled in pkg/registry/daemonset
- In particular, the api server will add the object to etcd
- DaemonManager listens for updates to etcd (using Framework.informer)
- API objects for DaemonSet were created in expapi/v1/types.go and
expapi/v1/register.go
- Validation code is in expapi/validation
#### Daemon Manager
- Creates new DaemonSets when requested. Launches the corresponding daemon pod
on all nodes with labels matching the new DaemonSet’s selector.
- Listens for addition of new nodes to the cluster, by setting up a
framework.NewInformer that watches for the creation of Node API objects. When a
new node is added, the daemon manager will loop through each DaemonSet. If the
label of the node matches the selector of the DaemonSet, then the daemon manager
will create the corresponding daemon pod in the new node.
- The daemon manager creates a pod on a node by sending a command to the API
server, requesting for a pod to be bound to the node (the node will be specified
via its hostname.)
#### Kubelet
- Does not need to be modified, but health checking will occur for the daemon
pods and revive the pods if they are killed (we set the pod restartPolicy to
Always). We reject DaemonSet objects with pod templates that don’t have
restartPolicy set to Always.
## Open Issues
- Should work similarly to [Deployment](http://issues.k8s.io/1743).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/daemon.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/extending-api.md
View file @ dd639c63
# Adding custom resources to the Kubernetes API server This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md)
This document describes the design for implementing the storage of custom API
types in the Kubernetes API Server.
## Resource Model
### The ThirdPartyResource
The `ThirdPartyResource` resource describes the multiple versions of a custom
resource that the user wants to add to the Kubernetes API. `ThirdPartyResource`
is a non-namespaced resource; attempting to place it in a namespace will return
an error.
Each `ThirdPartyResource` resource has the following:
* Standard Kubernetes object metadata.
* ResourceKind - The kind of the resources described by this third party
resource.
* Description - A free text description of the resource.
* APIGroup - An API group that this resource should be placed into.
* Versions - One or more `Version` objects.
### The `Version` Object
The `Version` object describes a single concrete version of a custom resource.
The `Version` object currently only specifies:
* The `Name` of the version.
* The `APIGroup` this version should belong to.
## Expectations about third party objects
Every object that is added to a third-party Kubernetes object store is expected
to contain Kubernetes compatible [object metadata](../devel/api-conventions.md#metadata).
This requirement enables the Kubernetes API server to provide the following
features:
* Filtering lists of objects via label queries.
* `resourceVersion`-based optimistic concurrency via compare-and-swap.
* Versioned storage.
* Event recording.
* Integration with basic `kubectl` command line tooling.
* Watch for resource changes.
The `Kind` for an instance of a third-party object (e.g. CronTab) below is
expected to be programmatically convertible to the name of the resource using
the following conversion. Kinds are expected to be of the form
`<CamelCaseKind>`, and the `APIVersion` for the object is expected to be
`<api-group>/<api-version>`. To prevent collisions, it's expected that you'll
use a DNS name of at least three segments for the API group, e.g. `mygroup.example.com`.
For example `mygroup.example.com/v1`
'CamelCaseKind' is the specific type name.
To convert this into the `metadata.name` for the `ThirdPartyResource` resource
instance, the `<domain-name>` is copied verbatim, the `CamelCaseKind` is then
converted using '-' instead of capitalization ('camel-case'), with the first
character being assumed to be capitalized. In pseudo code:
```go
var result string
for ix := range kindName {
if isCapital(kindName[ix]) {
result = append(result, '-')
}
result = append(result, toLowerCase(kindName[ix])
}
```
As a concrete example, the resource named `camel-case-kind.mygroup.example.com` defines
resources of Kind `CamelCaseKind`, in the APIGroup with the prefix
`mygroup.example.com/...`.
The reason for this is to enable rapid lookup of a `ThirdPartyResource` object
given the kind information. This is also the reason why `ThirdPartyResource` is
not namespaced.
## Usage
When a user creates a new `ThirdPartyResource`, the Kubernetes API Server reacts
by creating a new, namespaced RESTful resource path. For now, non-namespaced
objects are not supported. As with existing built-in objects, deleting a
namespace deletes all third party resources in that namespace.
For example, if a user creates:
```yaml
metadata:
name: cron-tab.mygroup.example.com
apiVersion: extensions/v1beta1
kind: ThirdPartyResource
description: "A specification of a Pod to run on a cron style schedule"
versions:
- name: v1
- name: v2
```
Then the API server will program in the new RESTful resource path:
* `/apis/mygroup.example.com/v1/namespaces/<namespace>/crontabs/...`
**Note: This may take a while before RESTful resource path registration happen, please
always check this before you create resource instances.**
Now that this schema has been created, a user can `POST`:
```json
{
"metadata": {
"name": "my-new-cron-object"
},
"apiVersion": "mygroup.example.com/v1",
"kind": "CronTab",
"cronSpec": "* * * * /5",
"image": "my-awesome-cron-image"
}
```
to: `/apis/mygroup.example.com/v1/namespaces/default/crontabs`
and the corresponding data will be stored into etcd by the APIServer, so that
when the user issues:
```
GET /apis/mygroup.example.com/v1/namespaces/default/crontabs/my-new-cron-object`
```
And when they do that, they will get back the same data, but with additional
Kubernetes metadata (e.g. `resourceVersion`, `createdTimestamp`) filled in.
Likewise, to list all resources, a user can issue:
```
GET /apis/mygroup.example.com/v1/namespaces/default/crontabs
```
and get back:
```json
{
"apiVersion": "mygroup.example.com/v1",
"kind": "CronTabList",
"items": [
{
"metadata": {
"name": "my-new-cron-object"
},
"apiVersion": "mygroup.example.com/v1",
"kind": "CronTab",
"cronSpec": "* * * * /5",
"image": "my-awesome-cron-image"
}
]
}
```
Because all objects are expected to contain standard Kubernetes metadata fields,
these list operations can also use label queries to filter requests down to
specific subsets.
Likewise, clients can use watch endpoints to watch for changes to stored
objects.
## Storage
In order to store custom user data in a versioned fashion inside of etcd, we
need to also introduce a `Codec`-compatible object for persistent storage in
etcd. This object is `ThirdPartyResourceData` and it contains:
* Standard API Metadata.
* `Data`: The raw JSON data for this custom object.
### Storage key specification
Each custom object stored by the API server needs a custom key in storage, this
is described below:
#### Definitions
* `resource-namespace`: the namespace of the particular resource that is
being stored
* `resource-name`: the name of the particular resource being stored
* `third-party-resource-namespace`: the namespace of the `ThirdPartyResource`
resource that represents the type for the specific instance being stored
* `third-party-resource-name`: the name of the `ThirdPartyResource` resource
that represents the type for the specific instance being stored
#### Key
Given the definitions above, the key for a specific third-party object is:
```
${standard-k8s-prefix}/third-party-resources/${third-party-resource-namespace}/${third-party-resource-name}/${resource-namespace}/${resource-name}
```
Thus, listing a third-party resource can be achieved by listing the directory:
```
${standard-k8s-prefix}/third-party-resources/${third-party-resource-namespace}/${third-party-resource-name}/${resource-namespace}/
```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/extending-api.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/identifiers.md
View file @ dd639c63
# Identifiers and Names in Kubernetes This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/identifiers.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/identifiers.md)
A summarization of the goals and recommendations for identifiers in Kubernetes.
Described in GitHub issue [#199](http://issue.k8s.io/199).
## Definitions
`UID`: A non-empty, opaque, system-generated value guaranteed to be unique in time
and space; intended to distinguish between historical occurrences of similar
entities.
`Name`: A non-empty string guaranteed to be unique within a given scope at a
particular time; used in resource URLs; provided by clients at creation time and
encouraged to be human friendly; intended to facilitate creation idempotence and
space-uniqueness of singleton objects, distinguish distinct entities, and
reference particular entities across operations.
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) `label` (DNS_LABEL):
An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters,
with the '-' character allowed anywhere except the first or last character,
suitable for use as a hostname or segment in a domain name.
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) `subdomain` (DNS_SUBDOMAIN):
One or more lowercase rfc1035/rfc1123 labels separated by '.' with a maximum
length of 253 characters.
[rfc4122](http://www.ietf.org/rfc/rfc4122.txt) `universally unique identifier` (UUID):
A 128 bit generated value that is extremely unlikely to collide across time and
space and requires no central coordination.
[rfc6335](https://tools.ietf.org/rfc/rfc6335.txt) `port name` (IANA_SVC_NAME):
An alphanumeric (a-z, and 0-9) string, with a maximum length of 15 characters,
with the '-' character allowed anywhere except the first or the last character
or adjacent to another '-' character, it must contain at least a (a-z)
character.
## Objectives for names and UIDs
1. Uniquely identify (via a UID) an object across space and time.
2. Uniquely name (via a name) an object across space.
3. Provide human-friendly names in API operations and/or configuration files.
4. Allow idempotent creation of API resources (#148) and enforcement of
space-uniqueness of singleton objects.
5. Allow DNS names to be automatically generated for some objects.
## General design
1. When an object is created via an API, a Name string (a DNS_SUBDOMAIN) must
be specified. Name must be non-empty and unique within the apiserver. This
enables idempotent and space-unique creation operations. Parts of the system
(e.g. replication controller) may join strings (e.g. a base name and a random
suffix) to create a unique Name. For situations where generating a name is
impractical, some or all objects may support a param to auto-generate a name.
Generating random names will defeat idempotency.
* Examples: "guestbook.user", "backend-x4eb1"
2. When an object is created via an API, a Namespace string (a DNS_SUBDOMAIN?
format TBD via #1114) may be specified. Depending on the API receiver,
namespaces might be validated (e.g. apiserver might ensure that the namespace
actually exists). If a namespace is not specified, one will be assigned by the
API receiver. This assignment policy might vary across API receivers (e.g.
apiserver might have a default, kubelet might generate something semi-random).
* Example: "api.k8s.example.com"
3. Upon acceptance of an object via an API, the object is assigned a UID
(a UUID). UID must be non-empty and unique across space and time.
* Example: "01234567-89ab-cdef-0123-456789abcdef"
## Case study: Scheduling a pod
Pods can be placed onto a particular node in a number of ways. This case study
demonstrates how the above design can be applied to satisfy the objectives.
### A pod scheduled by a user through the apiserver
1. A user submits a pod with Namespace="" and Name="guestbook" to the apiserver.
2. The apiserver validates the input.
1. A default Namespace is assigned.
2. The pod name must be space-unique within the Namespace.
3. Each container within the pod has a name which must be space-unique within
the pod.
3. The pod is accepted.
1. A new UID is assigned.
4. The pod is bound to a node.
1. The kubelet on the node is passed the pod's UID, Namespace, and Name.
5. Kubelet validates the input.
6. Kubelet runs the pod.
1. Each container is started up with enough metadata to distinguish the pod
from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is
unique across time. * This may correspond to Docker's container ID.
### A pod placed by a config file on the node
1. A config file is stored on the node, containing a pod with UID="",
Namespace="", and Name="cadvisor".
2. Kubelet validates the input.
1. Since UID is not provided, kubelet generates one.
2. Since Namespace is not provided, kubelet generates one.
1. The generated namespace should be deterministic and cluster-unique for
the source, such as a hash of the hostname and file path.
* E.g. Namespace="file-f4231812554558a718a01ca942782d81"
3. Kubelet runs the pod.
1. Each container is started up with enough metadata to distinguish the pod
from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is
unique across time.
1. This may correspond to Docker's container ID.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/identifiers.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/metadata-policy.md
View file @ dd639c63
# MetadataPolicy and its use in choosing the scheduler in a multi-scheduler system This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/metadata-policy.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/metadata-policy.md)
## Introduction
This document describes a new API resource, `MetadataPolicy`, that configures an
admission controller to take one or more actions based on an object's metadata.
Initially the metadata fields that the predicates can examine are labels and
annotations, and the actions are to add one or more labels and/or annotations,
or to reject creation/update of the object. In the future other actions might be
supported, such as applying an initializer.
The first use of `MetadataPolicy` will be to decide which scheduler should
schedule a pod in a [multi-scheduler](../proposals/multiple-schedulers.md)
Kubernetes system. In particular, the policy will add the scheduler name
annotation to a pod based on an annotation that is already on the pod that
indicates the QoS of the pod. (That annotation was presumably set by a simpler
admission controller that uses code, rather than configuration, to map the
resource requests and limits of a pod to QoS, and attaches the corresponding
annotation.)
We anticipate a number of other uses for `MetadataPolicy`, such as defaulting
for labels and annotations, prohibiting/requiring particular labels or
annotations, or choosing a scheduling policy within a scheduler. We do not
discuss them in this doc.
## API
```go
// MetadataPolicySpec defines the configuration of the MetadataPolicy API resource.
// Every rule is applied, in an unspecified order, but if the action for any rule
// that matches is to reject the object, then the object is rejected without being mutated.
type MetadataPolicySpec struct {
Rules []MetadataPolicyRule `json:"rules,omitempty"`
}
// If the PolicyPredicate is met, then the PolicyAction is applied.
// Example rules:
// reject object if label with key X is present (i.e. require X)
// reject object if label with key X is not present (i.e. forbid X)
// add label X=Y if label with key X is not present (i.e. default X)
// add annotation A=B if object has annotation C=D or E=F
type MetadataPolicyRule struct {
PolicyPredicate PolicyPredicate `json:"policyPredicate"`
PolicyAction PolicyAction `json:policyAction"`
}
// All criteria must be met for the PolicyPredicate to be considered met.
type PolicyPredicate struct {
// Note that Namespace is not listed here because MetadataPolicy is per-Namespace.
LabelSelector *LabelSelector `json:"labelSelector,omitempty"`
AnnotationSelector *LabelSelector `json:"annotationSelector,omitempty"`
}
// Apply the indicated Labels and/or Annotations (if present), unless Reject is set
// to true, in which case reject the object without mutating it.
type PolicyAction struct {
// If true, the object will be rejected and not mutated.
Reject bool `json:"reject"`
// The labels to add or update, if any.
UpdatedLabels *map[string]string `json:"updatedLabels,omitempty"`
// The annotations to add or update, if any.
UpdatedAnnotations *map[string]string `json:"updatedAnnotations,omitempty"`
}
// MetadataPolicy describes the MetadataPolicy API resource, which is used for specifying
// policies that should be applied to objects based on the objects' metadata. All MetadataPolicy's
// are applied to all objects in the namespace; the order of evaluation is not guaranteed,
// but if any of the matching policies have an action of rejecting the object, then the object
// will be rejected without being mutated.
type MetadataPolicy struct {
unversioned.TypeMeta `json:",inline"`
// Standard object's metadata.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
ObjectMeta `json:"metadata,omitempty"`
// Spec defines the metadata policy that should be enforced.
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status
Spec MetadataPolicySpec `json:"spec,omitempty"`
}
// MetadataPolicyList is a list of MetadataPolicy items.
type MetadataPolicyList struct {
unversioned.TypeMeta `json:",inline"`
// Standard list metadata.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds
unversioned.ListMeta `json:"metadata,omitempty"`
// Items is a list of MetadataPolicy objects.
// More info: http://releases.k8s.io/HEAD/docs/design/admission_control_resource_quota.md#admissioncontrol-plugin-resourcequota
Items []MetadataPolicy `json:"items"`
}
```
## Implementation plan
1. Create `MetadataPolicy` API resource
1. Create admission controller that implements policies defined in
`MetadataPolicy`
1. Create admission controller that sets annotation
`scheduler.alpha.kubernetes.io/qos: <QoS>`
(where `QOS` is one of `Guaranteed, Burstable, BestEffort`)
based on pod's resource request and limit.
## Future work
Longer-term we will have QoS be set on create and update by the registry,
similar to `Pending` phase today, instead of having an admission controller
(that runs before the one that takes `MetadataPolicy` as input) do it.
We plan to eventually move from having an admission controller set the scheduler
name as a pod annotation, to using the initializer concept. In particular, the
scheduler will be an initializer, and the admission controller that decides
which scheduler to use will add the scheduler's name to the list of initializers
for the pod (presumably the scheduler will be the last initializer to run on
each pod). The admission controller would still be configured using the
`MetadataPolicy` described here, only the mechanism the admission controller
uses to record its decision of which scheduler to use would change.
## Related issues
The main issue for multiple schedulers is #11793. There was also a lot of
discussion in PRs #17197 and #17865.
We could use the approach described here to choose a scheduling policy within a
single scheduler, as opposed to choosing a scheduler, a desire mentioned in
# 9920. Issue #17097 describes a scenario unrelated to scheduler-choosing where
`MetadataPolicy` could be used. Issue #17324 proposes to create a generalized
API for matching "claims" to "service classes"; matching a pod to a scheduler
would be one use for such an API.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/metadata-policy.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/principles.md
View file @ dd639c63
# Design Principles This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/principles.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/principles.md)
Principles to follow when extending Kubernetes.
## API
See also the [API conventions](../devel/api-conventions.md).
* All APIs should be declarative.
* API objects should be complementary and composable, not opaque wrappers.
* The control plane should be transparent -- there are no hidden internal APIs.
* The cost of API operations should be proportional to the number of objects
intentionally operated upon. Therefore, common filtered lookups must be indexed.
Beware of patterns of multiple API calls that would incur quadratic behavior.
* Object status must be 100% reconstructable by observation. Any history kept
must be just an optimization and not required for correct operation.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add
them. If you must have them, don't enforce them atomically in master components,
that is contention-prone and doesn't provide a recovery path in the case of a
bug allowing the invariant to be violated. Instead, provide a series of checks
to reduce the probability of a violation, and make every component involved able
to recover from an invariant violation.
* Low-level APIs should be designed for control by higher-level systems.
Higher-level APIs should be intent-oriented (think SLOs) rather than
implementation-oriented (think control knobs).
## Control logic
* Functionality must be *level-based*, meaning the system must operate correctly
given the desired state and the current/observed state, regardless of how many
intermediate state updates may have been missed. Edge-triggered behavior must be
just an optimization.
* Assume an open world: continually verify assumptions and gracefully adapt to
external events and/or actors. Example: we allow users to kill pods under
control of a replication controller; it just replaces them.
* Do not define comprehensive state machines for objects with behaviors
associated with state transitions and/or "assumed" states that cannot be
ascertained by observation.
* Don't assume a component's decisions will not be overridden or rejected, nor
for the component to always understand why. For example, etcd may reject writes.
Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry,
but back off and/or make alternative decisions.
* Components should be self-healing. For example, if you must keep some state
(e.g., cache) the content needs to be periodically refreshed, so that if an item
does get erroneously stored or a deletion event is missed etc, it will be soon
fixed, ideally on timescales that are shorter than what will attract attention
from humans.
* Component behavior should degrade gracefully. Prioritize actions so that the
most important activities can continue to function even when overloaded and/or
in states of partial failure.
## Architecture
* Only the apiserver should communicate with etcd/store, and not other
components (scheduler, kubelet, etc.).
* Compromising a single node shouldn't compromise the cluster.
* Components should continue to do what they were last told in the absence of
new instructions (e.g., due to network partition or component outage).
* All components should keep all relevant state in memory all the time. The
apiserver should write through to etcd/store, other components should write
through to the apiserver, and they should watch for updates made by other
clients.
* Watch is preferred over polling.
## Extensibility
TODO: pluggability
## Bootstrapping
* [Self-hosting](http://issue.k8s.io/246) of all components is a goal.
* Minimize the number of dependencies, particularly those required for
steady-state operation.
* Stratify the dependencies that remain via principled layering.
* Break any circular dependencies by converting hard dependencies to soft
dependencies.
* Also accept that data from other components from another source, such as
local files, which can then be manually populated at bootstrap time and then
continuously updated once those other components are available.
* State should be rediscoverable and/or reconstructable.
* Make it easy to run temporary, bootstrap instances of all components in
order to create the runtime state needed to run the components in the steady
state; use a lock (master election for distributed components, file lock for
local components like Kubelet) to coordinate handoff. We call this technique
"pivoting".
* Have a solution to restart dead components. For distributed components,
replication works well. For local components such as Kubelet, a process manager
or even a simple shell loop works.
## Availability
TODO
## General principles
* [Eric Raymond's 17 UNIX rules](https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E2.80.99s_17_Unix_Rules)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/principles.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/scheduler_extender.md
View file @ dd639c63
# Scheduler extender This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduler_extender.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduler_extender.md)
There are three ways to add new scheduling rules (predicates and priority
functions) to Kubernetes: (1) by adding these rules to the scheduler and
recompiling (described here:
https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler.md),
(2) implementing your own scheduler process that runs instead of, or alongside
of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender"
process that the standard Kubernetes scheduler calls out to as a final pass when
making scheduling decisions.
This document describes the third approach. This approach is needed for use
cases where scheduling decisions need to be made on resources not directly
managed by the standard Kubernetes scheduler. The extender helps make scheduling
decisions based on such resources. (Note that the three approaches are not
mutually exclusive.)
When scheduling a pod, the extender allows an external process to filter and
prioritize nodes. Two separate http/https calls are issued to the extender, one
for "filter" and one for "prioritize" actions. To use the extender, you must
create a scheduler policy configuration file. The configuration specifies how to
reach the extender, whether to use http or https and the timeout.
```go
// Holds the parameters used to communicate with the extender. If a verb is unspecified/empty,
// it is assumed that the extender chose not to provide that extension.
type ExtenderConfig struct {
// URLPrefix at which the extender is available
URLPrefix string `json:"urlPrefix"`
// Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
FilterVerb string `json:"filterVerb,omitempty"`
// Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
PrioritizeVerb string `json:"prioritizeVerb,omitempty"`
// The numeric multiplier for the node scores that the prioritize call generates.
// The weight should be a positive integer
Weight int `json:"weight,omitempty"`
// EnableHttps specifies whether https should be used to communicate with the extender
EnableHttps bool `json:"enableHttps,omitempty"`
// TLSConfig specifies the transport layer security config
TLSConfig *client.TLSClientConfig `json:"tlsConfig,omitempty"`
// HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize
// timeout is ignored, k8s/other extenders priorities are used to select the node.
HTTPTimeout time.Duration `json:"httpTimeout,omitempty"`
}
```
A sample scheduler policy file with extender configuration:
```json
{
"predicates": [
{
"name": "HostName"
},
{
"name": "MatchNodeSelector"
},
{
"name": "PodFitsResources"
}
],
"priorities": [
{
"name": "LeastRequestedPriority",
"weight": 1
}
],
"extenders": [
{
"urlPrefix": "http://127.0.0.1:12345/api/scheduler",
"filterVerb": "filter",
"enableHttps": false
}
]
}
```
Arguments passed to the FilterVerb endpoint on the extender are the set of nodes
filtered through the k8s predicates and the pod. Arguments passed to the
PrioritizeVerb endpoint on the extender are the set of nodes filtered through
the k8s predicates and extender predicates and the pod.
```go
// ExtenderArgs represents the arguments needed by the extender to filter/prioritize
// nodes for a pod.
type ExtenderArgs struct {
// Pod being scheduled
Pod api.Pod `json:"pod"`
// List of candidate nodes where the pod can be scheduled
Nodes api.NodeList `json:"nodes"`
}
```
The "filter" call returns a list of nodes (schedulerapi.ExtenderFilterResult). The "prioritize" call
returns priorities for each node (schedulerapi.HostPriorityList).
The "filter" call may prune the set of nodes based on its predicates. Scores
returned by the "prioritize" call are added to the k8s scores (computed through
its priority functions) and used for final host selection.
Multiple extenders can be configured in the scheduler policy.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/scheduler_extender.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/security_context.md
View file @ dd639c63
# Security Contexts This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/security_context.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/security_context.md)
## Abstract
A security context is a set of constraints that are applied to a container in
order to achieve the following goals (from [security design](security.md)):
1. Ensure a clear isolation between container and the underlying host it runs
on
2. Limit the ability of the container to negatively impact the infrastructure
or other containers
## Background
The problem of securing containers in Kubernetes has come up
[before](http://issue.k8s.io/398) and the potential problems with container
security are [well known](http://opensource.com/business/14/7/docker-security-selinux).
Although it is not possible to completely isolate Docker containers from their
hosts, new features like [user namespaces](https://github.com/docker/libcontainer/pull/304)
make it possible to greatly reduce the attack surface.
## Motivation
### Container isolation
In order to improve container isolation from host and other containers running
on the host, containers should only be granted the access they need to perform
their work. To this end it should be possible to take advantage of Docker
features such as the ability to
[add or remove capabilities](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration)
and [assign MCS labels](https://docs.docker.com/reference/run/#security-configuration)
to the container process.
Support for user namespaces has recently been
[merged](https://github.com/docker/libcontainer/pull/304) into Docker's
libcontainer project and should soon surface in Docker itself. It will make it
possible to assign a range of unprivileged uids and gids from the host to each
container, improving the isolation between host and container and between
containers.
### External integration with shared storage
In order to support external integration with shared storage, processes running
in a Kubernetes cluster should be able to be uniquely identified by their Unix
UID, such that a chain of ownership can be established. Processes in pods will
need to have consistent UID/GID/SELinux category labels in order to access
shared disks.
## Constraints and Assumptions
* It is out of the scope of this document to prescribe a specific set of
constraints to isolate containers from their host. Different use cases need
different settings.
* The concept of a security context should not be tied to a particular security
mechanism or platform (i.e. SELinux, AppArmor)
* Applying a different security context to a scope (namespace or pod) requires
a solution such as the one proposed for [service accounts](service_accounts.md).
## Use Cases
In order of increasing complexity, following are example use cases that would
be addressed with security contexts:
1. Kubernetes is used to run a single cloud application. In order to protect
nodes from containers:
* All containers run as a single non-root user
* Privileged containers are disabled
* All containers run with a particular MCS label
* Kernel capabilities like CHOWN and MKNOD are removed from containers
2. Just like case #1, except that I have more than one application running on
the Kubernetes cluster.
* Each application is run in its own namespace to avoid name collisions
* For each application a different uid and MCS label is used
3. Kubernetes is used as the base for a PAAS with multiple projects, each
project represented by a namespace.
* Each namespace is associated with a range of uids/gids on the node that
are mapped to uids/gids on containers using linux user namespaces.
* Certain pods in each namespace have special privileges to perform system
actions such as talking back to the server for deployment, run docker builds,
etc.
* External NFS storage is assigned to each namespace and permissions set
using the range of uids/gids assigned to that namespace.
## Proposed Design
### Overview
A *security context* consists of a set of constraints that determine how a
container is secured before getting created and run. A security context resides
on the container and represents the runtime parameters that will be used to
create and run the container via container APIs. A *security context provider*
is passed to the Kubelet so it can have a chance to mutate Docker API calls in
order to apply the security context.
It is recommended that this design be implemented in two phases:
1. Implement the security context provider extension point in the Kubelet
so that a default security context can be applied on container run and creation.
2. Implement a security context structure that is part of a service account. The
default context provider can then be used to apply a security context based on
the service account associated with the pod.
### Security Context Provider
The Kubelet will have an interface that points to a `SecurityContextProvider`.
The `SecurityContextProvider` is invoked before creating and running a given
container:
```go
type SecurityContextProvider interface {
// ModifyContainerConfig is called before the Docker createContainer call.
// The security context provider can make changes to the Config with which
// the container is created.
// An error is returned if it's not possible to secure the container as
// requested with a security context.
ModifyContainerConfig(pod *api.Pod, container *api.Container, config *docker.Config)
// ModifyHostConfig is called before the Docker runContainer call.
// The security context provider can make changes to the HostConfig, affecting
// security options, whether the container is privileged, volume binds, etc.
// An error is returned if it's not possible to secure the container as requested
// with a security context.
ModifyHostConfig(pod *api.Pod, container *api.Container, hostConfig *docker.HostConfig)
}
```
If the value of the SecurityContextProvider field on the Kubelet is nil, the
kubelet will create and run the container as it does today.
### Security Context
A security context resides on the container and represents the runtime
parameters that will be used to create and run the container via container APIs.
Following is an example of an initial implementation:
```go
type Container struct {
... other fields omitted ...
// Optional: SecurityContext defines the security options the pod should be run with
SecurityContext *SecurityContext
}
// SecurityContext holds security configuration that will be applied to a container. SecurityContext
// contains duplication of some existing fields from the Container resource. These duplicate fields
// will be populated based on the Container configuration if they are not set. Defining them on
// both the Container AND the SecurityContext will result in an error.
type SecurityContext struct {
// Capabilities are the capabilities to add/drop when running the container
Capabilities *Capabilities
// Run the container in privileged mode
Privileged *bool
// SELinuxOptions are the labels to be applied to the container
// and volumes
SELinuxOptions *SELinuxOptions
// RunAsUser is the UID to run the entrypoint of the container process.
RunAsUser *int64
}
// SELinuxOptions are the labels to be applied to the container.
type SELinuxOptions struct {
// SELinux user label
User string
// SELinux role label
Role string
// SELinux type label
Type string
// SELinux level label.
Level string
}
```
### Admission
It is up to an admission plugin to determine if the security context is
acceptable or not. At the time of writing, the admission control plugin for
security contexts will only allow a context that has defined capabilities or
privileged. Contexts that attempt to define a UID or SELinux options will be
denied by default. In the future the admission plugin will base this decision
upon configurable policies that reside within the [service account](http://pr.k8s.io/2297).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/security_context.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/selector-generation.md
View file @ dd639c63
Design This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/selector-generation.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/selector-generation.md)
=============
# Goals
Make it really hard to accidentally create a job which has an overlapping
selector, while still making it possible to chose an arbitrary selector, and
without adding complex constraint solving to the APIserver.
# Use Cases
1. user can leave all label and selector fields blank and system will fill in
reasonable ones: non-overlappingness guaranteed.
2. user can put on the pod template some labels that are useful to the user,
without reasoning about non-overlappingness. System adds additional label to
assure not overlapping.
3. If user wants to reparent pods to new job (very rare case) and knows what
they are doing, they can completely disable this behavior and specify explicit
selector.
4. If a controller that makes jobs, like scheduled job, wants to use different
labels, such as the time and date of the run, it can do that.
5. If User reads v1beta1 documentation or reuses v1beta1 Job definitions and
just changes the API group, the user should not automatically be allowed to
specify a selector, since this is very rarely what people want to do and is
error prone.
6. If User downloads an existing job definition, e.g. with
`kubectl get jobs/old -o yaml` and tries to modify and post it, he should not
create an overlapping job.
7. If User downloads an existing job definition, e.g. with
`kubectl get jobs/old -o yaml` and tries to modify and post it, and he
accidentally copies the uniquifying label from the old one, then he should not
get an error from a label-key conflict, nor get erratic behavior.
8. If user reads swagger docs and sees the selector field, he should not be able
to set it without realizing the risks.
8. (Deferred requirement:) If user wants to specify a preferred name for the
non-overlappingness key, they can pick a name.
# Proposed changes
## API
`extensions/v1beta1 Job` remains the same. `batch/v1 Job` changes change as
follows.
Field `job.spec.manualSelector` is added. It controls whether selectors are
automatically generated. In automatic mode, user cannot make the mistake of
creating non-unique selectors. In manual mode, certain rare use cases are
supported.
Validation is not changed. A selector must be provided, and it must select the
pod template.
Defaulting changes. Defaulting happens in one of two modes:
### Automatic Mode
- User does not specify `job.spec.selector`.
- User is probably unaware of the `job.spec.manualSelector` field and does not
think about it.
- User optionally puts labels on pod template (optional). User does not think
about uniqueness, just labeling for user's own reasons.
- Defaulting logic sets `job.spec.selector` to
`matchLabels["controller-uid"]="$UIDOFJOB"`
- Defaulting logic appends 2 labels to the `.spec.template.metadata.labels`.
- The first label is controller-uid=$UIDOFJOB.
- The second label is "job-name=$NAMEOFJOB".
### Manual Mode
- User means User or Controller for the rest of this list.
- User does specify `job.spec.selector`.
- User does specify `job.spec.manualSelector=true`
- User puts a unique label or label(s) on pod template (required). User does
think carefully about uniqueness.
- No defaulting of pod labels or the selector happen.
### Rationale
UID is better than Name in that:
- it allows cross-namespace control someday if we need it.
- it is unique across all kinds. `controller-name=foo` does not ensure
uniqueness across Kinds `job` vs `replicaSet`. Even `job-name=foo` has a
problem: you might have a `batch.Job` and a `snazzyjob.io/types.Job` -- the
latter cannot use label `job-name=foo`, though there is a temptation to do so.
- it uniquely identifies the controller across time. This prevents the case
where, for example, someone deletes a job via the REST api or client
(where cascade=false), leaving pods around. We don't want those to be picked up
unintentionally. It also prevents the case where a user looks at an old job that
finished but is not deleted, and tries to select its pods, and gets the wrong
impression that it is still running.
Job name is more user friendly. It is self documenting
Commands like `kubectl get pods -l job-name=myjob` should do exactly what is
wanted 99.9% of the time. Automated control loops should still use the
controller-uid=label.
Using both gets the benefits of both, at the cost of some label verbosity.
The field is a `*bool`. Since false is expected to be much more common,
and since the feature is complex, it is better to leave it unspecified so that
users looking at a stored pod spec do not need to be aware of this field.
### Overriding Unique Labels
If user does specify `job.spec.selector` then the user must also specify
`job.spec.manualSelector`. This ensures the user knows that what he is doing is
not the normal thing to do.
To prevent users from copying the `job.spec.manualSelector` flag from existing
jobs, it will be optional and default to false, which means when you ask GET and
existing job back that didn't use this feature, you don't even see the
`job.spec.manualSelector` flag, so you are not tempted to wonder if you should
fiddle with it.
## Job Controller
No changes
## Kubectl
No required changes. Suggest moving SELECTOR to wide output of `kubectl get
jobs` since users do not write the selector.
## Docs
Remove examples that use selector and remove labels from pod templates.
Recommend `kubectl get jobs -l job-name=name` as the way to find pods of a job.
# Conversion
The following applies to Job, as well as to other types that adopt this pattern:
- Type `extensions/v1beta1` gets a field called `job.spec.autoSelector`.
- Both the internal type and the `batch/v1` type will get
`job.spec.manualSelector`.
- The fields `manualSelector` and `autoSelector` have opposite meanings.
- Each field defaults to false when unset, and so v1beta1 has a different
default than v1 and internal. This is intentional: we want new uses to default
to the less error-prone behavior, and we do not want to change the behavior of
v1beta1.
*Note*: since the internal default is changing, client library consumers that
create Jobs may need to add "job.spec.manualSelector=true" to keep working, or
switch to auto selectors.
Conversion is as follows:
- `extensions/__internal` to `extensions/v1beta1`: the value of
`__internal.Spec.ManualSelector` is defaulted to false if nil, negated,
defaulted to nil if false, and written `v1beta1.Spec.AutoSelector`.
- `extensions/v1beta1` to `extensions/__internal`: the value of
`v1beta1.SpecAutoSelector` is defaulted to false if nil, negated, defaulted to
nil if false, and written to `__internal.Spec.ManualSelector`.
This conversion gives the following properties.
1. Users that previously used v1beta1 do not start seeing a new field when they
get back objects.
2. Distinction between originally unset versus explicitly set to false is not
preserved (would have been nice to do so, but requires more complicated
solution).
3. Users who only created v1beta1 examples or v1 examples, will not ever see the
existence of either field.
4. Since v1beta1 are convertable to/from v1, the storage location (path in etcd)
does not need to change, allowing scriptable rollforward/rollback.
# Future Work
Follow this pattern for Deployments, ReplicaSet, DaemonSet when going to v1, if
it works well for job.
Docs will be edited to show examples without a `job.spec.selector`.
We probably want as much as possible the same behavior for Job and
ReplicationController.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/selector-generation.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/service_accounts.md
View file @ dd639c63
# Service Accounts This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service_accounts.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service_accounts.md)
## Motivation
Processes in Pods may need to call the Kubernetes API. For example:
- scheduler
- replication controller
- node controller
- a map-reduce type framework which has a controller that then tries to make a
dynamically determined number of workers and watch them
- continuous build and push system
- monitoring system
They also may interact with services other than the Kubernetes API, such as:
- an image repository, such as docker -- both when the images are pulled to
start the containers, and for writing images in the case of pods that generate
images.
- accessing other cloud services, such as blob storage, in the context of a
large, integrated, cloud offering (hosted or private).
- accessing files in an NFS volume attached to the pod
## Design Overview
A service account binds together several things:
- a *name*, understood by users, and perhaps by peripheral systems, for an
identity
- a *principal* that can be authenticated and [authorized](../admin/authorization.md)
- a [security context](security_context.md), which defines the Linux
Capabilities, User IDs, Groups IDs, and other capabilities and controls on
interaction with the file system and OS.
- a set of [secrets](secrets.md), which a container may use to access various
networked resources.
## Design Discussion
A new object Kind is added:
```go
type ServiceAccount struct {
TypeMeta `json:",inline" yaml:",inline"`
ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
username string
securityContext ObjectReference // (reference to a securityContext object)
secrets []ObjectReference // (references to secret objects
}
```
The name ServiceAccount is chosen because it is widely used already (e.g. by
Kerberos and LDAP) to refer to this type of account. Note that it has no
relation to Kubernetes Service objects.
The ServiceAccount object does not include any information that could not be
defined separately:
- username can be defined however users are defined.
- securityContext and secrets are only referenced and are created using the
REST API.
The purpose of the serviceAccount object is twofold:
- to bind usernames to securityContexts and secrets, so that the username can
be used to refer succinctly in contexts where explicitly naming securityContexts
and secrets would be inconvenient
- to provide an interface to simplify allocation of new securityContexts and
secrets.
These features are explained later.
### Names
From the standpoint of the Kubernetes API, a `user` is any principal which can
authenticate to Kubernetes API. This includes a human running `kubectl` on her
desktop and a container in a Pod on a Node making API calls.
There is already a notion of a username in Kubernetes, which is populated into a
request context after authentication. However, there is no API object
representing a user. While this may evolve, it is expected that in mature
installations, the canonical storage of user identifiers will be handled by a
system external to Kubernetes.
Kubernetes does not dictate how to divide up the space of user identifier
strings. User names can be simple Unix-style short usernames, (e.g. `alice`), or
may be qualified to allow for federated identity (`alice@example.com` vs.
`alice@example.org`.) Naming convention may distinguish service accounts from
user accounts (e.g. `alice@example.com` vs.
`build-service-account-a3b7f0@foo-namespace.service-accounts.example.com`), but
Kubernetes does not require this.
Kubernetes also does not require that there be a distinction between human and
Pod users. It will be possible to setup a cluster where Alice the human talks to
the Kubernetes API as username `alice` and starts pods that also talk to the API
as user `alice` and write files to NFS as user `alice`. But, this is not
recommended.
Instead, it is recommended that Pods and Humans have distinct identities, and
reference implementations will make this distinction.
The distinction is useful for a number of reasons:
- the requirements for humans and automated processes are different:
- Humans need a wide range of capabilities to do their daily activities.
Automated processes often have more narrowly-defined activities.
- Humans may better tolerate the exceptional conditions created by
expiration of a token. Remembering to handle this in a program is more annoying.
So, either long-lasting credentials or automated rotation of credentials is
needed.
- A Human typically keeps credentials on a machine that is not part of the
cluster and so not subject to automatic management. A VM with a
role/service-account can have its credentials automatically managed.
- the identity of a Pod cannot in general be mapped to a single human.
- If policy allows, it may be created by one human, and then updated by
another, and another, until its behavior cannot be attributed to a single human.
**TODO**: consider getting rid of separate serviceAccount object and just
rolling its parts into the SecurityContext or Pod Object.
The `secrets` field is a list of references to /secret objects that an process
started as that service account should have access to be able to assert that
role.
The secrets are not inline with the serviceAccount object. This way, most or
all users can have permission to `GET /serviceAccounts` so they can remind
themselves what serviceAccounts are available for use.
Nothing will prevent creation of a serviceAccount with two secrets of type
`SecretTypeKubernetesAuth`, or secrets of two different types. Kubelet and
client libraries will have some behavior, TBD, to handle the case of multiple
secrets of a given type (pick first or provide all and try each in order, etc).
When a serviceAccount and a matching secret exist, then a `User.Info` for the
serviceAccount and a `BearerToken` from the secret are added to the map of
tokens used by the authentication process in the apiserver, and similarly for
other types. (We might have some types that do not do anything on apiserver but
just get pushed to the kubelet.)
### Pods
The `PodSpec` is extended to have a `Pods.Spec.ServiceAccountUsername` field. If
this is unset, then a default value is chosen. If it is set, then the
corresponding value of `Pods.Spec.SecurityContext` is set by the Service Account
Finalizer (see below).
TBD: how policy limits which users can make pods with which service accounts.
### Authorization
Kubernetes API Authorization Policies refer to users. Pods created with a
`Pods.Spec.ServiceAccountUsername` typically get a `Secret` which allows them to
authenticate to the Kubernetes APIserver as a particular user. So any policy
that is desired can be applied to them.
A higher level workflow is needed to coordinate creation of serviceAccounts,
secrets and relevant policy objects. Users are free to extend Kubernetes to put
this business logic wherever is convenient for them, though the Service Account
Finalizer is one place where this can happen (see below).
### Kubelet
The kubelet will treat as "not ready to run" (needing a finalizer to act on it)
any Pod which has an empty SecurityContext.
The kubelet will set a default, restrictive, security context for any pods
created from non-Apiserver config sources (http, file).
Kubelet watches apiserver for secrets which are needed by pods bound to it.
**TODO**: how to only let kubelet see secrets it needs to know.
### The service account finalizer
There are several ways to use Pods with SecurityContexts and Secrets.
One way is to explicitly specify the securityContext and all secrets of a Pod
when the pod is initially created, like this:
**TODO**: example of pod with explicit refs.
Another way is with the *Service Account Finalizer*, a plugin process which is
optional, and which handles business logic around service accounts.
The Service Account Finalizer watches Pods, Namespaces, and ServiceAccount
definitions.
First, if it finds pods which have a `Pod.Spec.ServiceAccountUsername` but no
`Pod.Spec.SecurityContext` set, then it copies in the referenced securityContext
and secrets references for the corresponding `serviceAccount`.
Second, if ServiceAccount definitions change, it may take some actions.
**TODO**: decide what actions it takes when a serviceAccount definition changes.
Does it stop pods, or just allow someone to list ones that are out of spec? In
general, people may want to customize this?
Third, if a new namespace is created, it may create a new serviceAccount for
that namespace. This may include a new username (e.g.
`NAMESPACE-default-service-account@serviceaccounts.$CLUSTERID.kubernetes.io`),
a new securityContext, a newly generated secret to authenticate that
serviceAccount to the Kubernetes API, and default policies for that service
account.
**TODO**: more concrete example. What are typical default permissions for
default service account (e.g. readonly access to services in the same namespace
and read-write access to events in that namespace?)
Finally, it may provide an interface to automate creation of new
serviceAccounts. In that case, the user may want to GET serviceAccounts to see
what has been created.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/service_accounts.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/simple-rolling-update.md
View file @ dd639c63
## Simple rolling update This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/simple-rolling-update.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/simple-rolling-update.md)
This is a lightweight design document for simple
[rolling update](../user-guide/kubectl/kubectl_rolling-update.md) in `kubectl`.
Complete execution flow can be found [here](#execution-details). See the
[example of rolling update](../user-guide/update-demo/) for more information.
### Lightweight rollout
Assume that we have a current replication controller named `foo` and it is
running image `image:v1`
`kubectl rolling-update foo [foo-v2] --image=myimage:v2`
If the user doesn't specify a name for the 'next' replication controller, then
the 'next' replication controller is renamed to
the name of the original replication controller.
Obviously there is a race here, where if you kill the client between delete foo,
and creating the new version of 'foo' you might be surprised about what is
there, but I think that's ok. See [Recovery](#recovery) below
If the user does specify a name for the 'next' replication controller, then the
'next' replication controller is retained with its existing name, and the old
'foo' replication controller is deleted. For the purposes of the rollout, we add
a unique-ifying label `kubernetes.io/deployment` to both the `foo` and
`foo-next` replication controllers. The value of that label is the hash of the
complete JSON representation of the`foo-next` or`foo` replication controller.
The name of this label can be overridden by the user with the
`--deployment-label-key` flag.
#### Recovery
If a rollout fails or is terminated in the middle, it is important that the user
be able to resume the roll out. To facilitate recovery in the case of a crash of
the updating process itself, we add the following annotations to each
replication controller in the `kubernetes.io/` annotation namespace:
* `desired-replicas` The desired number of replicas for this replication
controller (either N or zero)
* `update-partner` A pointer to the replication controller resource that is
the other half of this update (syntax `<name>` the namespace is assumed to be
identical to the namespace of this replication controller.)
Recovery is achieved by issuing the same command again:
```sh
kubectl rolling-update foo [foo-v2] --image=myimage:v2
```
Whenever the rolling update command executes, the kubectl client looks for
replication controllers called `foo` and `foo-next`, if they exist, an attempt
is made to roll `foo` to `foo-next`. If `foo-next` does not exist, then it is
created, and the rollout is a new rollout. If `foo` doesn't exist, then it is
assumed that the rollout is nearly completed, and `foo-next` is renamed to
`foo`. Details of the execution flow are given below.
### Aborting a rollout
Abort is assumed to want to reverse a rollout in progress.
`kubectl rolling-update foo [foo-v2] --rollback`
This is really just semantic sugar for:
`kubectl rolling-update foo-v2 foo`
With the added detail that it moves the `desired-replicas` annotation from
`foo-v2` to `foo`
### Execution Details
For the purposes of this example, assume that we are rolling from `foo` to
`foo-next` where the only change is an image update from `v1` to `v2`
If the user doesn't specify a `foo-next` name, then it is either discovered from
the `update-partner` annotation on `foo`. If that annotation doesn't exist,
then `foo-next` is synthesized using the pattern
`<controller-name>-<hash-of-next-controller-JSON>`
#### Initialization
* If `foo` and `foo-next` do not exist:
* Exit, and indicate an error to the user, that the specified controller
doesn't exist.
* If `foo` exists, but `foo-next` does not:
* Create `foo-next` populate it with the `v2` image, set
`desired-replicas` to `foo.Spec.Replicas`
* Goto Rollout
* If `foo-next` exists, but `foo` does not:
* Assume that we are in the rename phase.
* Goto Rename
* If both `foo` and `foo-next` exist:
* Assume that we are in a partial rollout
* If `foo-next` is missing the `desired-replicas` annotation
* Populate the `desired-replicas` annotation to `foo-next` using the
current size of `foo`
* Goto Rollout
#### Rollout
* While size of `foo-next` < `desired-replicas` annotation on `foo-next`
* increase size of `foo-next`
* if size of `foo` > 0
decrease size of `foo`
* Goto Rename
#### Rename
* delete `foo`
* create `foo` that is identical to `foo-next`
* delete `foo-next`
#### Abort
* If `foo-next` doesn't exist
* Exit and indicate to the user that they may want to simply do a new
rollout with the old version
* If `foo` doesn't exist
* Exit and indicate not found to the user
* Otherwise, `foo-next` and `foo` both exist
* Set `desired-replicas` annotation on `foo` to match the annotation on
`foo-next`
* Goto Rollout with `foo` and `foo-next` trading places.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/simple-rolling-update.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/design/versioning.md
View file @ dd639c63
# Kubernetes API and Release Versioning This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md)
Reference: [Semantic Versioning](http://semver.org)
Legend:
* **Kube X.Y.Z** refers to the version (git tag) of Kubernetes that is released.
This versions all components: apiserver, kubelet, kubectl, etc. (**X** is the
major version, **Y** is the minor version, and **Z** is the patch version.)
* **API vX[betaY]** refers to the version of the HTTP API.
## Release versioning
### Minor version scheme and timeline
* Kube X.Y.0-alpha.W, W > 0 (Branch: master)
* Alpha releases are released roughly every two weeks directly from the master
branch.
* No cherrypick releases. If there is a critical bugfix, a new release from
master can be created ahead of schedule.
* Kube X.Y.Z-beta.W (Branch: release-X.Y)
* When master is feature-complete for Kube X.Y, we will cut the release-X.Y
branch 2 weeks prior to the desired X.Y.0 date and cherrypick only PRs essential
to X.Y.
* This cut will be marked as X.Y.0-beta.0, and master will be revved to X.Y+1.0-alpha.0.
* If we're not satisfied with X.Y.0-beta.0, we'll release other beta releases,
(X.Y.0-beta.W | W > 0) as necessary.
* Kube X.Y.0 (Branch: release-X.Y)
* Final release, cut from the release-X.Y branch cut two weeks prior.
* X.Y.1-beta.0 will be tagged at the same commit on the same branch.
* X.Y.0 occur 3 to 4 months after X.(Y-1).0.
* Kube X.Y.Z, Z > 0 (Branch: release-X.Y)
* [Patch releases](#patch-releases) are released as we cherrypick commits into
the release-X.Y branch, (which is at X.Y.Z-beta.W,) as needed.
* X.Y.Z is cut straight from the release-X.Y branch, and X.Y.Z+1-beta.0 is
tagged on the followup commit that updates pkg/version/base.go with the beta
version.
* Kube X.Y.Z, Z > 0 (Branch: release-X.Y.Z)
* These are special and different in that the X.Y.Z tag is branched to isolate
the emergency/critical fix from all other changes that have landed on the
release branch since the previous tag
* Cut release-X.Y.Z branch to hold the isolated patch release
* Tag release-X.Y.Z branch + fixes with X.Y.(Z+1)
* Branched [patch releases](#patch-releases) are rarely needed but used for
emergency/critical fixes to the latest release
* See [#19849](https://issues.k8s.io/19849) tracking the work that is needed
for this kind of release to be possible.
### Major version timeline
There is no mandated timeline for major versions. They only occur when we need
to start the clock on deprecating features. A given major version should be the
latest major version for at least one year from its original release date.
### CI and dev version scheme
* Continuous integration versions also exist, and are versioned off of alpha and
beta releases. X.Y.Z-alpha.W.C+aaaa is C commits after X.Y.Z-alpha.W, with an
additional +aaaa build suffix added; X.Y.Z-beta.W.C+bbbb is C commits after
X.Y.Z-beta.W, with an additional +bbbb build suffix added. Furthermore, builds
that are built off of a dirty build tree, (during development, with things in
the tree that are not checked it,) it will be appended with -dirty.
### Supported releases and component skew
We expect users to stay reasonably up-to-date with the versions of Kubernetes
they use in production, but understand that it may take time to upgrade,
especially for production-critical components.
We expect users to be running approximately the latest patch release of a given
minor release; we often include critical bug fixes in
[patch releases](#patch-release), and so encourage users to upgrade as soon as
possible.
Different components are expected to be compatible across different amounts of
skew, all relative to the master version. Nodes may lag masters components by
up to two minor versions but should be at a version no newer than the master; a
client should be skewed no more than one minor version from the master, but may
lead the master by up to one minor version. For example, a v1.3 master should
work with v1.1, v1.2, and v1.3 nodes, and should work with v1.2, v1.3, and v1.4
clients.
Furthermore, we expect to "support" three minor releases at a time. "Support"
means we expect users to be running that version in production, though we may
not port fixes back before the latest minor version. For example, when v1.3
comes out, v1.0 will no longer be supported: basically, that means that the
reasonable response to the question "my v1.0 cluster isn't working," is, "you
should probably upgrade it, (and probably should have some time ago)". With
minor releases happening approximately every three months, that means a minor
release is supported for approximately nine months.
This policy is in line with
[GKE's supported upgrades policy](https://cloud.google.com/container-engine/docs/clusters/upgrade).
## API versioning
### Release versions as related to API versions
Here is an example major release cycle:
* **Kube 1.0 should have API v1 without v1beta\* API versions**
* The last version of Kube before 1.0 (e.g. 0.14 or whatever it is) will have
the stable v1 API. This enables you to migrate all your objects off of the beta
API versions of the API and allows us to remove those beta API versions in Kube
1.0 with no effect. There will be tooling to help you detect and migrate any
v1beta\* data versions or calls to v1 before you do the upgrade.
* **Kube 1.x may have API v2beta***
* The first incarnation of a new (backwards-incompatible) API in HEAD is
v2beta1. By default this will be unregistered in apiserver, so it can change
freely. Once it is available by default in apiserver (which may not happen for
several minor releases), it cannot change ever again because we serialize
objects in versioned form, and we always need to be able to deserialize any
objects that are saved in etcd, even between alpha versions. If further changes
to v2beta1 need to be made, v2beta2 is created, and so on, in subsequent 1.x
versions.
* **Kube 1.y (where y is the last version of the 1.x series) must have final
API v2**
* Before Kube 2.0 is cut, API v2 must be released in 1.x. This enables two
things: (1) users can upgrade to API v2 when running Kube 1.x and then switch
over to Kube 2.x transparently, and (2) in the Kube 2.0 release itself we can
cleanup and remove all API v2beta\* versions because no one should have
v2beta\* objects left in their database. As mentioned above, tooling will exist
to make sure there are no calls or references to a given API version anywhere
inside someone's kube installation before someone upgrades.
* Kube 2.0 must include the v1 API, but Kube 3.0 must include the v2 API only.
It *may* include the v1 API as well if the burden is not high - this will be
determined on a per-major-version basis.
#### Rationale for API v2 being complete before v2.0's release
It may seem a bit strange to complete the v2 API before v2.0 is released,
but *adding* a v2 API is not a breaking change. *Removing* the v2beta\*
APIs *is* a breaking change, which is what necessitates the major version bump.
There are other ways to do this, but having the major release be the fresh start
of that release's API without the baggage of its beta versions seems most
intuitive out of the available options.
## Patch releases
Patch releases are intended for critical bug fixes to the latest minor version,
such as addressing security vulnerabilities, fixes to problems affecting a large
number of users, severe problems with no workaround, and blockers for products
based on Kubernetes.
They should not contain miscellaneous feature additions or improvements, and
especially no incompatibilities should be introduced between patch versions of
the same minor version (or even major version).
Dependencies, such as Docker or Etcd, should also not be changed unless
absolutely necessary, and also just to fix critical bugs (so, at most patch
version changes, not new major nor minor versions).
## Upgrades
* Users can upgrade from any Kube 1.x release to any other Kube 1.x release as a
rolling upgrade across their cluster. (Rolling upgrade means being able to
upgrade the master first, then one node at a time. See #4855 for details.)
* However, we do not recommend upgrading more than two minor releases at a
time (see [Supported releases](#supported-releases)), and do not recommend
running non-latest patch releases of a given minor release.
* No hard breaking changes over version boundaries.
* For example, if a user is at Kube 1.x, we may require them to upgrade to
Kube 1.x+y before upgrading to Kube 2.x. In others words, an upgrade across
major versions (e.g. Kube 1.x to Kube 2.x) should effectively be a no-op and as
graceful as an upgrade from Kube 1.x to Kube 1.x+1. But you can require someone
to go from 1.x to 1.x+y before they go to 2.x.
There is a separate question of how to track the capabilities of a kubelet to
facilitate rolling upgrades. That is not addressed here.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/versioning.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment