Commit 77e63c50 authored by Mike Brown's avatar Mike Brown

devel/ tree 80col wrap and other minor edits

parent 1642083e
...@@ -34,69 +34,76 @@ Documentation for other releases can be found at ...@@ -34,69 +34,76 @@ Documentation for other releases can be found at
# Measuring Node Performance # Measuring Node Performance
This document outlines the issues and pitfalls of measuring Node performance, as well as the tools This document outlines the issues and pitfalls of measuring Node performance, as
available. well as the tools available.
## Cluster Set-up ## Cluster Set-up
There are lots of factors which can affect node performance numbers, so care must be taken in There are lots of factors which can affect node performance numbers, so care
setting up the cluster to make the intended measurements. In addition to taking the following steps must be taken in setting up the cluster to make the intended measurements. In
into consideration, it is important to document precisely which setup was used. For example, addition to taking the following steps into consideration, it is important to
performance can vary wildly from commit-to-commit, so it is very important to **document which commit document precisely which setup was used. For example, performance can vary
wildly from commit-to-commit, so it is very important to **document which commit
or version** of Kubernetes was used, which Docker version was used, etc. or version** of Kubernetes was used, which Docker version was used, etc.
### Addon pods ### Addon pods
Be aware of which addon pods are running on which nodes. By default Kubernetes runs 8 addon pods, Be aware of which addon pods are running on which nodes. By default Kubernetes
plus another 2 per node (`fluentd-elasticsearch` and `kube-proxy`) in the `kube-system` runs 8 addon pods, plus another 2 per node (`fluentd-elasticsearch` and
namespace. The addon pods can be disabled for more consistent results, but doing so can also have `kube-proxy`) in the `kube-system` namespace. The addon pods can be disabled for
performance implications. more consistent results, but doing so can also have performance implications.
For example, Heapster polls each node regularly to collect stats data. Disabling Heapster will hide For example, Heapster polls each node regularly to collect stats data. Disabling
the performance cost of serving those stats in the Kubelet. Heapster will hide the performance cost of serving those stats in the Kubelet.
#### Disabling Add-ons #### Disabling Add-ons
Disabling addons is simple. Just ssh into the Kubernetes master and move the addon from Disabling addons is simple. Just ssh into the Kubernetes master and move the
`/etc/kubernetes/addons/` to a backup location. More details [here](../../cluster/addons/). addon from `/etc/kubernetes/addons/` to a backup location. More details
[here](../../cluster/addons/).
### Which / how many pods? ### Which / how many pods?
Performance will vary a lot between a node with 0 pods and a node with 100 pods. In many cases Performance will vary a lot between a node with 0 pods and a node with 100 pods.
you'll want to make measurements with several different amounts of pods. On a single node cluster In many cases you'll want to make measurements with several different amounts of
scaling a replication controller makes this easy, just make sure the system reaches a steady-state pods. On a single node cluster scaling a replication controller makes this easy,
before starting the measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100` just make sure the system reaches a steady-state before starting the
measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100`
In most cases pause pods will yield the most consistent measurements since the system will not be In most cases pause pods will yield the most consistent measurements since the
affected by pod load. However, in some special cases Kubernetes has been tuned to optimize pods that system will not be affected by pod load. However, in some special cases
are not doing anything, such as the cAdvisor housekeeping (stats gathering). In these cases, Kubernetes has been tuned to optimize pods that are not doing anything, such as
performing a very light task (such as a simple network ping) can make a difference. the cAdvisor housekeeping (stats gathering). In these cases, performing a very
light task (such as a simple network ping) can make a difference.
Finally, you should also consider which features yours pods should be using. For example, if you Finally, you should also consider which features yours pods should be using. For
want to measure performance with probing, you should obviously use pods with liveness or readiness example, if you want to measure performance with probing, you should obviously
probes configured. Likewise for volumes, number of containers, etc. use pods with liveness or readiness probes configured. Likewise for volumes,
number of containers, etc.
### Other Tips ### Other Tips
**Number of nodes** - On the one hand, it can be easier to manage logs, pods, environment etc. with **Number of nodes** - On the one hand, it can be easier to manage logs, pods,
a single node to worry about. On the other hand, having multiple nodes will let you gather more environment etc. with a single node to worry about. On the other hand, having
data in parallel for more robust sampling. multiple nodes will let you gather more data in parallel for more robust
sampling.
## E2E Performance Test ## E2E Performance Test
There is an end-to-end test for collecting overall resource usage of node components: There is an end-to-end test for collecting overall resource usage of node
[kubelet_perf.go](../../test/e2e/kubelet_perf.go). To components: [kubelet_perf.go](../../test/e2e/kubelet_perf.go). To
run the test, simply make sure you have an e2e cluster running (`go run hack/e2e.go -up`) and run the test, simply make sure you have an e2e cluster running (`go run
[set up](#cluster-set-up) correctly. hack/e2e.go -up`) and [set up](#cluster-set-up) correctly.
Run the test with `go run hack/e2e.go -v -test Run the test with `go run hack/e2e.go -v -test
--test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to customise the number of --test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to
pods or other parameters of the test (remember to rerun `make WHAT=test/e2e/e2e.test` after you do). customise the number of pods or other parameters of the test (remember to rerun
`make WHAT=test/e2e/e2e.test` after you do).
## Profiling ## Profiling
Kubelet installs the [go pprof handlers](https://golang.org/pkg/net/http/pprof/), which can be Kubelet installs the [go pprof handlers]
queried for CPU profiles: (https://golang.org/pkg/net/http/pprof/), which can be queried for CPU profiles:
```console ```console
$ kubectl proxy & $ kubectl proxy &
...@@ -109,13 +116,15 @@ $ go tool pprof -web $KUBELET_BIN $OUTPUT ...@@ -109,13 +116,15 @@ $ go tool pprof -web $KUBELET_BIN $OUTPUT
`pprof` can also provide heap usage, from the `/debug/pprof/heap` endpoint `pprof` can also provide heap usage, from the `/debug/pprof/heap` endpoint
(e.g. `http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/heap`). (e.g. `http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/heap`).
More information on go profiling can be found [here](http://blog.golang.org/profiling-go-programs). More information on go profiling can be found
[here](http://blog.golang.org/profiling-go-programs).
## Benchmarks ## Benchmarks
Before jumping through all the hoops to measure a live Kubernetes node in a real cluster, it is Before jumping through all the hoops to measure a live Kubernetes node in a real
worth considering whether the data you need can be gathered through a Benchmark test. Go provides a cluster, it is worth considering whether the data you need can be gathered
really simple benchmarking mechanism, just add a unit test of the form: through a Benchmark test. Go provides a really simple benchmarking mechanism,
just add a unit test of the form:
```go ```go
// In foo_test.go // In foo_test.go
......
...@@ -31,23 +31,43 @@ Documentation for other releases can be found at ...@@ -31,23 +31,43 @@ Documentation for other releases can be found at
<!-- END STRIP_FOR_RELEASE --> <!-- END STRIP_FOR_RELEASE -->
<!-- END MUNGE: UNVERSIONED_WARNING --> <!-- END MUNGE: UNVERSIONED_WARNING -->
Kubernetes On-Call Rotations
====================
Kubernetes "first responder" rotations ## Kubernetes On-Call Rotations
--------------------------------------
### Kubernetes "first responder" rotations
Kubernetes has generated a lot of public traffic: email, pull-requests, bugs, etc. So much traffic that it's becoming impossible to keep up with it all! This is a fantastic problem to have. In order to be sure that SOMEONE, but not EVERYONE on the team is paying attention to public traffic, we have instituted two "first responder" rotations, listed below. Please read this page before proceeding to the pages linked below, which are specific to each rotation.
Kubernetes has generated a lot of public traffic: email, pull-requests, bugs,
Please also read our [notes on OSS collaboration](collab.md), particularly the bits about hours. Specifically, each rotation is expected to be active primarily during work hours, less so off hours. etc. So much traffic that it's becoming impossible to keep up with it all! This
is a fantastic problem to have. In order to be sure that SOMEONE, but not
During regular workday work hours of your shift, your primary responsibility is to monitor the traffic sources specific to your rotation. You can check traffic in the evenings if you feel so inclined, but it is not expected to be as highly focused as work hours. For weekends, you should check traffic very occasionally (e.g. once or twice a day). Again, it is not expected to be as highly focused as workdays. It is assumed that over time, everyone will get weekday and weekend shifts, so the workload will balance out. EVERYONE on the team is paying attention to public traffic, we have instituted
two "first responder" rotations, listed below. Please read this page before
If you can not serve your shift, and you know this ahead of time, it is your responsibility to find someone to cover and to change the rotation. If you have an emergency, your responsibilities fall on the primary of the other rotation, who acts as your secondary. If you need help to cover all of the tasks, partners with oncall rotations (e.g., [Redhat](https://github.com/orgs/kubernetes/teams/rh-oncall)). proceeding to the pages linked below, which are specific to each rotation.
If you are not on duty you DO NOT need to do these things. You are free to focus on "real work". Please also read our [notes on OSS collaboration](collab.md), particularly the
bits about hours. Specifically, each rotation is expected to be active primarily
Note that Kubernetes will occasionally enter code slush/freeze, prior to milestones. When it does, there might be changes in the instructions (assigning milestones, for instance). during work hours, less so off hours.
During regular workday work hours of your shift, your primary responsibility is
to monitor the traffic sources specific to your rotation. You can check traffic
in the evenings if you feel so inclined, but it is not expected to be as highly
focused as work hours. For weekends, you should check traffic very occasionally
(e.g. once or twice a day). Again, it is not expected to be as highly focused as
workdays. It is assumed that over time, everyone will get weekday and weekend
shifts, so the workload will balance out.
If you can not serve your shift, and you know this ahead of time, it is your
responsibility to find someone to cover and to change the rotation. If you have
an emergency, your responsibilities fall on the primary of the other rotation,
who acts as your secondary. If you need help to cover all of the tasks, partners
with oncall rotations (e.g.,
[Redhat](https://github.com/orgs/kubernetes/teams/rh-oncall)).
If you are not on duty you DO NOT need to do these things. You are free to focus
on "real work".
Note that Kubernetes will occasionally enter code slush/freeze, prior to
milestones. When it does, there might be changes in the instructions (assigning
milestones, for instance).
* [Github and Build Cop Rotation](on-call-build-cop.md) * [Github and Build Cop Rotation](on-call-build-cop.md)
* [User Support Rotation](on-call-user-support.md) * [User Support Rotation](on-call-user-support.md)
......
...@@ -31,55 +31,90 @@ Documentation for other releases can be found at ...@@ -31,55 +31,90 @@ Documentation for other releases can be found at
<!-- END STRIP_FOR_RELEASE --> <!-- END STRIP_FOR_RELEASE -->
<!-- END MUNGE: UNVERSIONED_WARNING --> <!-- END MUNGE: UNVERSIONED_WARNING -->
Kubernetes "User Support" Rotation
==================================
Traffic sources and responsibilities ## Kubernetes "User Support" Rotation
------------------------------------
### Traffic sources and responsibilities
* [StackOverflow](http://stackoverflow.com/questions/tagged/kubernetes) and
[ServerFault](http://serverfault.com/questions/tagged/google-kubernetes):
Respond to any thread that has no responses and is more than 6 hours old (over
time we will lengthen this timeout to allow community responses). If you are not
equipped to respond, it is your job to redirect to someone who can.
* [StackOverflow](http://stackoverflow.com/questions/tagged/kubernetes) and [ServerFault](http://serverfault.com/questions/tagged/google-kubernetes): Respond to any thread that has no responses and is more than 6 hours old (over time we will lengthen this timeout to allow community responses). If you are not equipped to respond, it is your job to redirect to someone who can.
* [Query for unanswered Kubernetes StackOverflow questions](http://stackoverflow.com/search?q=%5Bkubernetes%5D+answers%3A0) * [Query for unanswered Kubernetes StackOverflow questions](http://stackoverflow.com/search?q=%5Bkubernetes%5D+answers%3A0)
* [Query for unanswered Kubernetes ServerFault questions](http://serverfault.com/questions/tagged/google-kubernetes?sort=unanswered&pageSize=15) * [Query for unanswered Kubernetes ServerFault questions](http://serverfault.com/questions/tagged/google-kubernetes?sort=unanswered&pageSize=15)
* Direct poorly formulated questions to [stackoverflow's tips about how to ask](http://stackoverflow.com/help/how-to-ask) * Direct poorly formulated questions to [stackoverflow's tips about how to ask](http://stackoverflow.com/help/how-to-ask)
* Direct off-topic questions to [stackoverflow's policy](http://stackoverflow.com/help/on-topic) * Direct off-topic questions to [stackoverflow's policy](http://stackoverflow.com/help/on-topic)
* [Slack](https://kubernetes.slack.com) ([registration](http://slack.k8s.io)): Your job is to be on Slack, watching for questions and answering or redirecting as needed. Also check out the [Slack Archive](http://kubernetes.slackarchive.io/).
* [Email/Groups](https://groups.google.com/forum/#!forum/google-containers): Respond to any thread that has no responses and is more than 6 hours old (over time we will lengthen this timeout to allow community responses). If you are not equipped to respond, it is your job to redirect to someone who can. * [Slack](https://kubernetes.slack.com) ([registration](http://slack.k8s.io)):
* [Legacy] [IRC](irc://irc.freenode.net/#google-containers) (irc.freenode.net #google-containers): watch IRC for questions and try to redirect users to Slack. Also check out the [IRC logs](https://botbot.me/freenode/google-containers/). Your job is to be on Slack, watching for questions and answering or redirecting
as needed. Also check out the [Slack Archive](http://kubernetes.slackarchive.io/).
* [Email/Groups](https://groups.google.com/forum/#!forum/google-containers):
Respond to any thread that has no responses and is more than 6 hours old (over
time we will lengthen this timeout to allow community responses). If you are not
equipped to respond, it is your job to redirect to someone who can.
* [Legacy] [IRC](irc://irc.freenode.net/#google-containers)
(irc.freenode.net #google-containers): watch IRC for questions and try to
redirect users to Slack. Also check out the
[IRC logs](https://botbot.me/freenode/google-containers/).
In general, try to direct support questions to: In general, try to direct support questions to:
1. Documentation, such as the [user guide](../user-guide/README.md) and [troubleshooting guide](../troubleshooting.md) 1. Documentation, such as the [user guide](../user-guide/README.md) and
[troubleshooting guide](../troubleshooting.md)
2. Stackoverflow 2. Stackoverflow
If you see questions on a forum other than Stackoverflow, try to redirect them to Stackoverflow. Example response: If you see questions on a forum other than Stackoverflow, try to redirect them
to Stackoverflow. Example response:
```code
Please re-post your question to [stackoverflow]
(http://stackoverflow.com/questions/tagged/kubernetes).
We are trying to consolidate the channels to which questions for help/support
are posted so that we can improve our efficiency in responding to your requests,
and to make it easier for you to find answers to frequently asked questions and
how to address common use cases.
Please re-post your question to [stackoverflow](http://stackoverflow.com/questions/tagged/kubernetes). We regularly see messages posted in multiple forums, with the full response
thread only in one place or, worse, spread across multiple forums. Also, the
large volume of support issues on github is making it difficult for us to use
issues to identify real bugs.
We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. The Kubernetes team scans stackoverflow on a regular basis, and will try to
ensure your questions don't go unanswered.
We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. Before posting a new question, please search stackoverflow for answers to
similar questions, and also familiarize yourself with:
The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. * [user guide](http://kubernetes.io/v1.1/)
* [troubleshooting guide](http://kubernetes.io/v1.1/docs/troubleshooting.html)
Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes.
* [the user guide](http://kubernetes.io/v1.1/)
* [the troubleshooting guide](http://kubernetes.io/v1.1/docs/troubleshooting.html)
Again, thanks for using Kubernetes. The Kubernetes Team
```
The Kubernetes Team If you answer a question (in any of the above forums) that you think might be
useful for someone else in the future, *please add it to one of the FAQs in the
wiki*:
If you answer a question (in any of the above forums) that you think might be useful for someone else in the future, *please add it to one of the FAQs in the wiki*:
* [User FAQ](https://github.com/kubernetes/kubernetes/wiki/User-FAQ) * [User FAQ](https://github.com/kubernetes/kubernetes/wiki/User-FAQ)
* [Developer FAQ](https://github.com/kubernetes/kubernetes/wiki/Developer-FAQ) * [Developer FAQ](https://github.com/kubernetes/kubernetes/wiki/Developer-FAQ)
* [Debugging FAQ](https://github.com/kubernetes/kubernetes/wiki/Debugging-FAQ). * [Debugging FAQ](https://github.com/kubernetes/kubernetes/wiki/Debugging-FAQ).
Getting it into the FAQ is more important than polish. Please indicate the date it was added, so people can judge the likelihood that it is out-of-date (and please correct any FAQ entries that you see contain out-of-date information). Getting it into the FAQ is more important than polish. Please indicate the date
it was added, so people can judge the likelihood that it is out-of-date (and
please correct any FAQ entries that you see contain out-of-date information).
Contact information ### Contact information
-------------------
[@k8s-support-oncall](https://github.com/k8s-support-oncall) will reach the current person on call. [@k8s-support-oncall](https://github.com/k8s-support-oncall) will reach the
current person on call.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment