Automatic merge from submit-queue

Use etcd3.0.10 in integration tests

Also, start testing etcd2->etcd3 migration as part of test-update-storage-object test.

[We are already using etcd 3.0.10 in unit tests. And we are still testing the current binary as part of all e2e tests, which I think is enough. I think we can start using 3.0.10 in integrations tests now.

Ref #20504

@lavalamp @timothysc

Automatic merge from submit-queue

update deployment and replicaset listers

Updates the deployment lister to avoid copies and updates the deployment controller to use shared informers.

Pushing WIP to see which tests are broken.

Automatic merge from submit-queue

Revert "Revert "move pod networking tests common""

Reverts #34011

And fix the problem causing `Granular Checks: Services [Slow] should update nodePort` tests to fail

Automatic merge from submit-queue

decouple workqueue metrics from prometheus

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:
We want to include the workqueue in client-go, but do not want to having to import Prometheus. This PR decouples the workqueue from prometheus.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

Partially address https://github.com/kubernetes/kubernetes/issues/33497
User requested for `workqueue` in client-go: https://github.com/kubernetes/client-go/issues/4#issuecomment-249444848

**Special notes for your reviewer**:

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
The implicit registration of Prometheus metrics for workqueue has been removed, and a plug-able interface was added. If you were using workqueue in your own binaries and want these metrics, add the following to your imports in the main package: "k8s.io/pkg/util/workqueue/prometheus".
```

Automatic merge from submit-queue

Speed up dockerized builds

This PR speeds up dockerized builds.  First, we make sure that we are as incremental as possible.  The bigger change is that now we use rsync to move sources into the container and get data back out.

To do yet:
* [x] Add a random password to rsync.  This is 128bit MD4, but it is better than nothing.
* [x] Lock down rsync to only come from the host.
* [x] Deal with remote docker engines -- this should be necessary for docker-machine on the mac.
* [x] Allow users to specify the port for the rsync daemon.  Perhaps randomize this or let docker pick an ephemeral port and detect the port?
* [x] Copy back generated files so that users can check them in.  This is done for `zz_generated.*` files generated by `make generated_files` 
  * [x] This should include generated proto files so that we can remove the hack-o-rama that is `hack/hack/update-*-dockerized.sh` 
* [x] Start "versioning" the build container and the data container so that the CI system doesn't have to be manually kicked.
* [x] Get some benchmarks to qualify how much faster.

This replaces #28518 and is related to #30600.

cc @thockin @spxtr @david-mcmahon @MHBauer 

Benchmarks by running `make clean ; sync ; time bash -xc 'time build/make-build-image.sh ; time sync ; time build/run.sh make ; time sync; time build/run.sh make'` on a GCE n1-standard-8 with PD-SSD.

| setup | build image | sync | first build | sync | second build | total |
|-------|-------------|----- |----------|------|--------------|------|
| baseline | 0m11.420s | 0m0.812s | 7m2.353s | 0m42.380s | 7m8.381s | 15m5.348s |
| this pr | 0m10.977s | 0m15.168s | 7m31.096s | 1m55.692s | 0m16.514s | 10m9.449s |

Automatic merge from submit-queue

Add support for vpshere cloud provider in kubeup

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:
vSphere cloud provider added in 1.3 was not configured when deploying via kubeup

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
Add support for vSphere Cloud Provider when deploying via kubeup on vSphere.
```

When deploying on vSphere using kube up add configuration
for vSphere cloud provider.

Automatic merge from submit-queue

CRI: Add serial and benchmark test suite.

For https://github.com/kubernetes/kubernetes/issues/31459.

The serial test result will be shown on test-grid.
The benchmark test result will be shown [node-perf-dash](http://node-perf-dash.k8s.io/#/builds)

This PR also changes the cri validation test to use the same gci image with node e2e instead of the canary image. The docker version is still 1.11.2.

@yujuhong @feiskyer @yifan-gu 
/cc @kubernetes/sig-node

Automatic merge from submit-queue

kubenet: make it more apparent that kubenet ignores PodCIDR updates

Don't suppress a warning that kubenet will ignore PodCIDR updates.

See https://github.com/kubernetes/kubernetes/issues/32900

Automatic merge from submit-queue

CRI: Add presubmit CRI validation test.

For #31459.

This PR adds a new suite for CRI presubmit validation which runs non-flaky, non-serial, non-slow test per-pr.

Except this PR, I'll also change the test-infra side. Ideally, after this is done, we should be be able to trigger CRI validation test per-pr with something like `@k8s-bot cri node e2e test this` and `@k8s-bot cri e2e test this`.

@yujuhong @feiskyer @yifan-gu @freehan 
/cc @kubernetes/sig-node

Automatic merge from submit-queue

Add option to set a service nodeport

**Release note**:
```release-note
Add kubectl --node-port option for specifying the service nodeport
```

This patch adds the option to set a nodeport when creating a NodePort
service. In case of a port allocation error due to a specified port
being out of the valid range, the error now includes the valid
range. If a `--node-port` value is not specified, it defaults to zero, in
which case the allocator will default to its current behavior of
assigning an available port.

This patch also adds a new helper function in `cmd/util/helpers.go` to
retrieve `Int32` cobra flags.

**Example**
```
# create a nodeport service with an invalid port
$ kubectl create service nodeport mynodeport --tcp=8080:7777 --node-port=1
The Service "mynodeport" is invalid: spec.ports[0].nodePort: Invalid
value: 1: provided port is not in the valid range. Valid ports range
from 30000-32767

# create a nodeport service with a valid port
$ kubectl create service nodeport mynodeport --tcp=8080:7777 --node-port=30000
service "mynodeport" created

# create a nodeport service with a port already in use
$ kubectl create service nodeport mynodeport --tcp=8080:7777 --node-port=30000
The Service "mynodeport" is invalid: spec.ports[0].nodePort: Invalid value: 3000: provided port is already allocated

$ kubectl describe service mynodeport
Name:                   mynodeport
Namespace:              default
Labels:                 app=mynodeport
Selector:               app=mynodeport
Type:                   NodePort
IP:                     172.30.81.254
Port:                   8080-7777       8080/TCP
NodePort:               8080-7777       30000/TCP
Endpoints:              <none>
Session Affinity:       None
No events.
```

@fabianofranz

Automatic merge from submit-queue

Set deserialization cache size based on target memory usage

**Special notes for your reviewer**:

This is the PR we talked about yesterday.

**Release note**:

```release-note
To reduce memory usage to reasonable levels in smaller clusters, kube-apiserver now sets the deserialization cache size based on the target memory usage.
```

Automatic merge from submit-queue

use service accounts as clients for controllers

Makes it possible for the controller-manager to use service accounts to run individual controllers.  To start, this only enables this feature if this particular controller manager has the power to create service account tokens.  Otherwise, the full-powered client is used instead.  This is a necessary step on the way to subdividing the authority of controllers.

@kubernetes/sig-auth 
@erictune I know you care about this
@ncdc fyi

Automatic merge from submit-queue

Proposal: SELinux enhancements

TLDR: Try to make SELinux support better by not requiring Kubelet directory to be labeled with an SELinux type usable from the container.

cc @kubernetes/sig-node @yifan-gu

Automatic merge from submit-queue

Daemonset in federated clientset

Only hack/update-codegen.sh was edited manually.

cc: @nikhiljindal @quinton-hoole @kubernetes/sig-cluster-federation

Automatic merge from submit-queue

kubectl: add watch flag in rollout status

Fixes https://github.com/kubernetes/kubernetes/issues/30627

@kubernetes/kubectl

Automatic merge from submit-queue

Add a retry loop to KubeletManagerEtcHosts flake

Add a retry loop to KubeletManagerEtcHosts flake
    
    Note: this still makes the test fail if a retry occurs, but
    will give us more information regarding whether or not the
    test flake could be occuring due to delay in mounting of
    /etc/hosts.

Automatic merge from submit-queue

Allow empty annotation values

Related downstream issue: https://github.com/openshift/origin/issues/11175

**Release note**:
```release-note
release-note-none
```

Annotations with empty values can be used, for example, in diagnostics
logging. This patch removes the client-side check for empty values in
an annotation key-value pair.

**Before**
```
$ kubectl annotate pod zookeeper-1 node-selector="" --overwrite
error: invalid annotation format: node-selector=
```

**After**
```
$ kubectl annotate pod zookeeper-1 node-selector="" --overwrite
pod "zookeper-1" annotated
```

```
$ kubectl get po/zookeeper-1 --template='{{.metadata.annotations}}'
map[... node-selector: test-label:test]
```

@fabianofranz @liggitt

Automatic merge from submit-queue

Fix #31085, include output checking in retry loop

Fix #31085.

`wget -qO-` might omit the error and `RunHostCmd()` won't be able to catch it sometimes. Verify stdout is not empty within the retry loop to fix. Test passed 1500+ times in a row.

@freehan

Note: this still makes the test fail if a retry occurs, but
will give us more information regarding whether or not the
test flake could be occuring due to delay in mounting of
/etc/hosts.

Annotations with empty values can be used, for example, in diagnostics
logging. This patch removes the client-side check for empty values in
an annotation key-value pair.

**Before**
```
$ kubectl annotate pod zookeeper-1 node-selector="" --overwrite
error: invalid annotation format: node-selector=
```

**After**
```
$ kubectl annotate pod zookeeper-1 node-selector="" --overwrite
pod "zookeper-1" annotated
```

```
$ kubectl get po/zookeeper-1 --template='{{.metadata.annotations}}'
map[... node-selector: test-label:test]
```

Automatic merge from submit-queue

Customizable genericapiserver handler chain

Rebase of https://github.com/kubernetes/kubernetes/pull/33662.  This is @sttts, but I rebased it for him since he's out this week.

https://github.com/kubernetes/kubernetes/issues/32900

Automatic merge from submit-queue

Add asgroups to audit log

@deads2k || @sttts while reworking origing to use upstream I've noticed we're missing this, ptal

Automatic merge from submit-queue

Allow setting short value in integration tests

@deads2k

Automatic merge from submit-queue

Switch kubectl to use watch.Until

Squashed version of https://github.com/kubernetes/kubernetes/pull/30235 

@ncdc PTAL

cc: @kubernetes/kubectl

Automatic merge from submit-queue

Fix call to Index call in listers

This will actually eliminated all logs like this:
```
can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
```

And it should also speed up different things (as so far we we basically not using indexes at all.

@kubernetes/sig-api-machinery