Automatic merge from submit-queue

Fix scheduler_perf test so that QPS is non-zero even if there is a scheduling "cold start" 

@gmarek  ... is something like this more realistic as an expectation ?  That is, wait till scheduling starts, then wait 1 second before any polling attempts....  i.e. 

- gaurantee uniform sleep before measure, by doing it first rather then last
- print the initial status so that its easier to debug in case of issues

#36532

Automatic merge from submit-queue (batch tested with PRs 38294, 37009, 36778, 38130, 37835)

Add a cloudprovider validator flag to kubeadm and update the DNS spec

Broken out from: https://github.com/kubernetes/kubernetes/pull/37568

This PR creates a flag for `cloud-provider` that validates the value before `RunInit()` is run, which makes it now act as a "real" flag
Then it removes the `k8s.io/kubernetes/pkg/cloudprovider` dependency, which makes the binary ~40MB smaller! That's _really_ worth it!

In the second commit, the DNS addon is updated to the latest version: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.base

@kubernetes/sig-cluster-lifecycle

Automatic merge from submit-queue (batch tested with PRs 38294, 37009, 36778, 38130, 37835)

Bump github.com/docker/go-units dependency

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**: Bump [github.com/docker/go-units](https://github.com/docker/go-units/) to include https://github.com/docker/go-units/pull/22 which improves the precision of human durations ("about an hour ago", "2 days ago", etc).

**Which issue this PR fixes**: Issue detected in OpenShift: https://github.com/openshift/origin/issues/11967

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 38294, 37009, 36778, 38130, 37835)

Only configure basic auth on gci if KUBE_USER and KUBE_PASSWORD are specified.

This should not change the existing flow when KUBE_USER/KUBE_PASSWORD are specified.
It makes not specifying those a valid option that means "don't turn on basic auth".
I only did it for cluster/gce/gci for now, but others should be somewhat similar.

Automatic merge from submit-queue (batch tested with PRs 38294, 37009, 36778, 38130, 37835)

fix permissions when using fsGroup

Currently, when an fsGroup is specified, the permissions of the defaultMode are not respected and all files created by the atomic writer have mode 777.  This is because in `SetVolumeOwnership()` the `filepath.Walk` includes the symlinks created by the atomic writer.  The symlinks have mode 777 when read from `info.Mode()`.  However, when the are chmod'ed later, the chmod applies to the file the symlink points to, not the symlink itself, resulting in the wrong mode for the underlying file.

This PR skips chmod/chown for symlinks in the walk since those operations are carried out on the underlying file which will be included elsewhere in the walk.

xref https://bugzilla.redhat.com/show_bug.cgi?id=1384458

@derekwaynecarr @pmorie

Automatic merge from submit-queue (batch tested with PRs 38294, 37009, 36778, 38130, 37835)

Re-use tested ratelimiter

The ratelimiter introduced in #35583 is not working correctly when called from multiple threads
This reverts to the tested ratelimiter we were previously using to unblock 1.5 (automated revert wasn't possible)
Ref #38273

reproducing test case:
```
func TestMultiThreadedBlocking(t *testing.T) {
	done := make(chan bool)

	// 100 QPS, burst of 100
	b := NewBucketWithRate(100, 100)

	go func() {
		defer close(done)
		fmt.Println(time.Now(), "Waiting for 1000 (should block for ~9-10 seconds)")
		b.Wait(1000)
		fmt.Println(time.Now(), "Got 1000")
	}()

	// give the request for 1000 plenty of time to take the tokens
	time.Sleep(2 * time.Second)
	fmt.Println(time.Now(), "Waiting for 1 (should wait until 1000 block is refilled)")
	b.Wait(1)
	fmt.Println(time.Now(), "Got 1 (should happen right after the wait for 1000 completes)")

	<-done
}

$ go test ./pkg/util/ratelimit/ -v -run TestMultiThreadedBlocking
=== RUN   TestMultiThreadedBlocking
2016-12-07 12:15:36.222133049 -0500 EST Waiting for 1000 (should block for ~9-10 seconds)
2016-12-07 12:15:38.222797752 -0500 EST Waiting for 1 (should wait until 1000 block is refilled)
2016-12-07 12:15:38.222897951 -0500 EST Got 1 (should happen right after the wait for 1000 completes)
2016-12-07 12:15:45.223125234 -0500 EST Got 1000
```

in contrast, the same test run against juju/ratelimit:
```
go test ./pkg/util/flowcontrol/ -v -run TestMultiThreadedBlocking
=== RUN   TestMultiThreadedBlocking
2016-12-07 12:32:56.796077782 -0500 EST Waiting for 1000 (should block for ~9-10 seconds)
2016-12-07 12:32:58.799159059 -0500 EST Waiting for 1 (should wait until 1000 block is refilled)
2016-12-07 12:33:05.801076002 -0500 EST Got 1000
2016-12-07 12:33:05.807510387 -0500 EST Got 1 (should happen right after the wait for 1000 completes)
--- PASS: TestMultiThreadedBlocking (9.01s)
```

Reverts changes in cebfc821

Automatic merge from submit-queue (batch tested with PRs 35101, 38215, 38092)

fix alias conflict of clusterrolebinding

create_configmap alias is "cm"

Automatic merge from submit-queue (batch tested with PRs 35101, 38215, 38092)

fix informer generation

Informer generation doesn't work for informers from a different clientset.  This updates the generator to generate the internal interfaces required to break the cycle.

@ncdc take a look at the last two commits.

Automatic merge from submit-queue (batch tested with PRs 35101, 38215, 38092)

auth duplicate detect

I think we should not allow people set duplicate tokens in token file or set duplicate usernames in password file. because the default action overwriting the old data may let people misunderstand.

Create a new cloud-provider flag in order to catch wrong args directly and don't import the cloudprovider package and save ~50% in binary size

Automatic merge from submit-queue

Disable kubernetes-discovery in local-up-cluster.sh

fix #38257

Fixes local-up-cluster until kubernetes-discovery flags are hooked up

Automatic merge from submit-queue

add default label to rbac bootstrap policy

allow people to retrieve information of bootstrap policy by label :

`kubectl get clusterroles -l key=value` 
`kubectl get clusterrolebindings -l key=value`

Automatic merge from submit-queue

update local-up-cluster to allow full authentication proxying

Adds group and header information in auth proxy authenticator options for `local-up-cluster.sh`.  Must have been missed in the rebase madness.

Automatic merge from submit-queue (batch tested with PRs 38282, 38281)

Fix skip logic in e2e framework

Fixes 37792

cc: @wojtek-t

Automatic merge from submit-queue

e2e: mark 3rd portforwarding test as flaky

Follow-up of https://github.com/kubernetes/kubernetes/pull/38194 and https://github.com/kubernetes/kubernetes/issues/27680#issuecomment-265392033.

cc @kubernetes/sig-testing

Automatic merge from submit-queue (batch tested with PRs 38276, 38193)

Collect controller-manager logs from kubemark

Automatic merge from submit-queue

Bump Cluster Autoscaler to 0.4.0

This is the  same binary as 0.4.0-beta1, that has been tested for the last couple weeks.

@saad-ali This should be cherry-picked to 1.5 release.

```release-note
Cluster Autoscaler in version 0.4.0
```

cc: @fgrzadkowski @piosz @jszczepkowski

Automatic merge from submit-queue (batch tested with PRs 38181, 38128, 36711)

etcd2: have prefix always prepended

The prefix issue is discussed in #36290.

This is fixing etcd2 behavior separately.

**release note**:
```
etcd2: have prefix always prepended
```

Automatic merge from submit-queue (batch tested with PRs 38181, 38128, 36711)

Adding correct secret type for Ceph RBD storageclass provisioner example

StorageClass now requires provider-specific secret types, adding them to the RBD provisioning docs.

Automatic merge from submit-queue (batch tested with PRs 38181, 38128, 36711)

Remove redundant conditional statement

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:

1. remove redundant conditional statement
2. replace  errors.New(fmt.Sprintf with fmt.Errorf 

Thanks.

**Special notes for your reviewer**:

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
```

Automatic merge from submit-queue

Cache additional information in schedulercache.NodeInfo to speedup scheduler

Ref #35117

Automatic merge from submit-queue

Enable OpenAPI spec validation

Spec validation was failing on Jenkins. I am enabling it in this PR to figure out if we can get it pass. No review is necessary until all test passes.

Automatic merge from submit-queue (batch tested with PRs 37693, 38085)

fix typo in scale

fix typo.

Automatic merge from submit-queue (batch tested with PRs 37693, 38085)

Pipe get options to storage

Ref #37473

Automatic merge from submit-queue (batch tested with PRs 38173, 38151, 38197, 38221)

Revert "Disable "overlapping deployment" test on GKE temporarily. #32947"

Reverts kubernetes/kubernetes#38157

Automatic merge from submit-queue (batch tested with PRs 38173, 38151, 38197, 38221)

test: wait for ready replica set before adopting

Reworked version of https://github.com/kubernetes/kubernetes/pull/36439 which was reverted in https://github.com/kubernetes/kubernetes/pull/38049. This PR doesn't use any of the new status API added in replica sets so it should cause no trouble with upgrade tests.

@kubernetes/deployment @smarterclayton

Automatic merge from submit-queue (batch tested with PRs 38173, 38151, 38197, 38221)

Include resource type headers in diff report

**What this PR does / why we need it**: when checking for leaked resources, we report a failure only if there were additions to the diff - we don't want resources that were deleted during the run to count as a failure.

The logic for this removes the resource type headers from the diff report, however, so it's harder to understand what leaked.

For example, in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features-release-1.5/196/, the test failure shows
```
Error: 4 leaked resources
+NAME                              REGION       IP_ADDRESS      IP_PROTOCOL  TARGET
+a66878dbaad1f11e695b342010af0002  us-central1  104.198.182.83  TCP          us-central1/targetPools/a66878dbaad1f11e695b342010af0002
+NAME                              REGION       SESSION_AFFINITY  BACKUP  HEALTH_CHECKS
+a66878dbaad1f11e695b342010af0002  us-central1                            a66878dbaad1f11e695b342010af0002
```

but the [actual diff](https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features-release-1.5/196/artifacts/gcp-resources-diff.txt) contains more information:
```
--- /workspace/_artifacts/gcp-resources-before.txt	2016-12-01 14:39:14.260049037 -0800
+++ /workspace/_artifacts/gcp-resources-after.txt	2016-12-01 14:39:14.260049037 -0800
@@ -34,8 +34 @@ [ addresses ]
-NAME                              REGION       ADDRESS          STATUS
-a03d0e020a6bc11e6855f42010af0001  us-central1  104.197.135.88   RESERVED
-a23f20a47a6a011e6829a42010af0004  us-central1  104.197.147.212  RESERVED
-a32291b51a5e611e6b73342010af0002  us-central1  104.198.171.178  RESERVED
-a3c93cf79a65811e6b7c242010af0001  us-central1  104.198.58.57    RESERVED
-a9ac5bd1ba62811e68d1142010af0004  us-central1  104.154.134.135  RESERVED
-adae4c1a7a69511e6b2df42010af0000  us-central1  104.198.182.83   RESERVED
-adb4a80caa60311e689a942010af0001  us-central1  104.154.116.195  RESERVED
+
@@ -64 +57,2 @@ [ forwarding-rules ]
-
+NAME                              REGION       IP_ADDRESS      IP_PROTOCOL  TARGET
+a66878dbaad1f11e695b342010af0002  us-central1  104.198.182.83  TCP          us-central1/targetPools/a66878dbaad1f11e695b342010af0002
@@ -68 +62,2 @@ [ target-pools ]
-
+NAME                              REGION       SESSION_AFFINITY  BACKUP  HEALTH_CHECKS
+a66878dbaad1f11e695b342010af0002  us-central1                            a66878dbaad1f11e695b342010af0002
```

This PR will include the `[ type header ]`s in the report if resources are leaked.

cc @fejta

Automatic merge from submit-queue

fix typo in ingress

fix typo.

Automatic merge from submit-queue

Fix pods/log OpenAPI/Swagger spec to return string instead of v1.Pod

log operation is part of pod and had the assumption of returning a v1.Pod. First I added the support for an operation to be able to customize its return object then add the support to pods/log operation to return an string as an object sample.

Fixes #37881
Cc @kubernetes/sig-api-machinery