Automatic merge from submit-queue (batch tested with PRs 40796, 40878, 36033, 40838, 41210)

StatefulSet hardening

**What this PR does / why we need it**:

This PR contains the following changes to StatefulSet. Only one change effects the semantics of how the controller operates (This is described in #38418), and this change only brings the controller into conformance with its documented behavior.

1. pcb and pcb controller are removed and their functionality is encapsulated in StatefulPodControlInterface. This class modules the design contoller.PodControlInterface and provides an abstraction to clientset.Interface which is useful for testing purposes.
2. IdentityMappers has been removed to clarify what properties of a Pod are mutated by the controller. All mutations are performed in the UpdateStatefulPod method of the StatefulPodControlInterface.
3. The statefulSetIterator and petQueue classes are removed. These classes sorted Pods by CreationTimestamp. This is brittle and not resilient to clock skew. The current control loop, which implements the same logic, is in stateful_set_control.go. The Pods are now sorted and considered by their ordinal indices, as is outlined in the documentation.
4. StatefulSetController now checks to see if the Pods matching a StatefulSet's Selector also match the Name of the StatefulSet. This will make the controller resilient to overlapping, and will be enhanced by the addition of ControllerRefs.
5. The total lines of production code have been reduced, and the total number of unit tests has been increased. All new code has 100% unit coverage giving the module 83% coverage. Tests for StatefulSetController have been added, but it is not practical to achieve greater coverage in unit testing for this code (the e2e tests for StatefulSet cover these areas).
6. Issue #38418 is fixed in that StaefulSet will ensure that all Pods that are predecessors of another Pod are Running and Ready prior to launching a new Pod. This removes the potential for deadlock when a Pod needs to be rescheduled while its predecessor is hung in Pending or Initializing.
7. All reference to pet have been removed from the code and comments.

**Which issue this PR fixes**
 fixes #38418,#36859
**Special notes for your reviewer**:

**Release note**:

```release-note
Fixes issue #38418 which, under circumstance, could cause StatefulSet to deadlock. 
Mediates issue #36859. StatefulSet only acts on Pods whose identity matches the StatefulSet, providing a partial mediation for overlapping controllers.
```

Automatic merge from submit-queue (batch tested with PRs 40796, 40878, 36033, 40838, 41210)

HPA v2 (API Changes)

**Release note**:
```release-note
Introduces an new alpha version of the Horizontal Pod Autoscaler including expanded support for specifying metrics.
```

Implements the API changes for kubernetes/features#117.

This implements #34754, which is the new design for the Horizontal Pod Autoscaler.  It includes improved support for custom metrics (and/or arbitrary metrics) as well as expanded support for resource metrics.  The new HPA object is introduces in the API group "autoscaling/v1alpha1".

Note that the improved custom metric support currently is limited to per pod metrics from Heapster -- attempting to use the new "object metrics" will simply result in an error.  This will change once #34586 is merged and implemented.

Automatic merge from submit-queue (batch tested with PRs 40796, 40878, 36033, 40838, 41210)

kubeadm: added tests for preflight checks

**What this PR does / why we need it**: There hadn't been much care to add more unit tests as more preflight checks were added. I added tests that increased coverage from ~9% to ~71%

Adding unit tests is a WIP from https://github.com/kubernetes/kubernetes/issues/34136

**Special notes for your reviewer**: /cc @luxas @pires 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 40796, 40878, 36033, 40838, 41210)

Implement TTL controller and use the ttl annotation attached to node in secret manager

For every secret attached to a pod as volume, Kubelet is trying to refresh it every sync period. Currently Kubelet has a ttl-cache of secrets of its pods and the ttl is set to 1 minute. That means that in large clusters we are targetting (5k nodes, 30pods/node), given that each pod has a secret associated with ServiceAccount from its namespaces, and with large enough number of namespaces (where on each node (almost) every pod is from a different namespace), that resource in ~30 GETs to refresh all secrets every minute from one node, which gives ~2500QPS for GET secrets to apiserver.

Apiserver cannot keep up with it very easily.

Desired solution would be to watch for secret changes, but because of security we don't want a node watching for all secrets, and it is not possible for now to watch only for secrets attached to pods from my node.

So as a temporary solution, we are introducing an annotation that would be a suggestion for kubelet for the TTL of secrets in the cache and a very simple controller that would be setting this annotation based on the cluster size (the large cluster is, the bigger ttl is). 
That workaround mean that only very local changes are needed in Kubelet, we are creating a well separated very simple controller, and once watching "my secrets" will be possible it will be easy to remove it and switch to that. And it will allow us to reach scalability goals.

@dchen1107 @thockin @liggitt

Automatic merge from submit-queue (batch tested with PRs 40917, 41181, 41123, 36592, 41183)

fix ca cert in kubeadm

[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"

Merge pull request #36592 from andrewsykim/36273-set-all-node-conditions-unknown-when-node-unreachable

Automatic merge from submit-queue (batch tested with PRs 40917, 41181, 41123, 36592, 41183)

Set all node conditions to Unknown when node is unreachable

**What this PR does / why we need it**:
Sets all node conditions to Unknown when node does not report status/unreachable

**Which issue this PR fixes** 
fixes https://github.com/kubernetes/kubernetes/issues/36273

Automatic merge from submit-queue (batch tested with PRs 40917, 41181, 41123, 36592, 41183)

replace update in setupmaster.go

follow the TODO

Automatic merge from submit-queue (batch tested with PRs 40917, 41181, 41123, 36592, 41183)

fix scheduler performance test script 

**What this PR does / why we need it**:

 

 `test-performance.sh` is  in  dir `kubernetes/test/integration/scheduler_perf`

the  dir `kubernetes/test/component/scheduler/perf`  does not exist

Thanks.

**Special notes for your reviewer**:

**Release note**:

```release-note
```

Automatic merge from submit-queue (batch tested with PRs 40917, 41181, 41123, 36592, 41183)

[Federation] Add override flags options to kubefed init

**What this PR does / why we need it**:
Allows modification of startup flags (of apiserver and controller manager) through kubefed 
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
https://github.com/kubernetes/kubernetes/issues/40398
**Special notes for your reviewer**:
I haven't removed the existing redundant flags now (for example --dns-zone-name) intentionally to avoid breaking any existing tests that might use them.
I guess that would be better done as a follow up PR.
@madhusudancs @marun @nikhiljindal 
**Release note**:

```
It is now possible for the user to modify any startup flag of federation-apiserver and federation-controller-manager when deployed through kubefed.
There are two new options introduced in kubefed:
--apiserver-arg-overrides and --controllermanager-arg-overrides
Any number of actual federation-apiserver or federation-controller-manager flags can be specified using these options.
Example:
kubefed init "-other options-" ----apiserver-arg-overrides "--flag1=value1,--flag2=value2"
```

Automatic merge from submit-queue (batch tested with PRs 41074, 41147, 40854, 41167, 40045)

kubeadm: adding integration tests for init

**What this PR does / why we need it**: integration tests for kubeadm init focused on valid and invalid discovery tokens

**Special notes for your reviewer**: /cc @luxas @pires
This was taken from Pires's work in https://github.com/kubernetes/kubernetes/pull/40008 . This is just the testing aspect of it

As it stands, these tests will never complete. The reason being is once it passes with a valid discovery token, it will wait until `[apiclient] Created API client, waiting for the control plane to become ready ` which is an infinite poll (https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/master/apiclient.go#L71). 

This is a WIP
**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41074, 41147, 40854, 41167, 40045)

Fix some funky funcs.

This is code cleanup.  Fix function declarations and remove stale comment.

Automatic merge from submit-queue (batch tested with PRs 41074, 41147, 40854, 41167, 40045)

Upgrade test for deployments

Upgrade test for Deployments. Should prevent issues like https://github.com/kubernetes/kubernetes/issues/40415 in the future.

@krousey @janetkuo @soltysh 

Haven't managed to run it locally... 
```
$ go run hack/e2e.go  --up --test --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\] --upgrade-target=ci/latest --upgrade-image=gci"
2017/02/02 11:43:22 e2e.go:946: Running: ./hack/e2e-internal/e2e-down.sh
2017/02/02 11:43:22 e2e.go:948: Step './hack/e2e-internal/e2e-down.sh' finished in 7.278236ms
2017/02/02 11:43:22 e2e.go:946: Running: ./hack/e2e-internal/e2e-up.sh
2017/02/02 11:43:22 e2e.go:948: Step './hack/e2e-internal/e2e-up.sh' finished in 5.286328ms
2017/02/02 11:43:22 e2e.go:946: Running: ./cluster/kubectl.sh version --match-server-version=false
2017/02/02 11:43:22 e2e.go:948: Step './cluster/kubectl.sh version --match-server-version=false' finished in 213.847259ms
2017/02/02 11:43:22 e2e.go:946: Running: ./hack/e2e-internal/e2e-status.sh
2017/02/02 11:43:22 e2e.go:948: Step './hack/e2e-internal/e2e-status.sh' finished in 103.253183ms
2017/02/02 11:43:22 e2e.go:230: Something went wrong: encountered 2 errors: [exit status 1 exit status 1]
exit status 1
```

@krousey any eta for when the upgrade framework will be integrated in the pr builder?

Automatic merge from submit-queue (batch tested with PRs 41074, 41147, 40854, 41167, 40045)

Add debug logging to eviction manager

**What this PR does / why we need it**:
This PR adds debug logging to eviction manager.

We need it to help users understand when/why eviction manager is/is not making decisions to support information gathering during support.

Automatic merge from submit-queue

Add staging/ to gopath in copy.sh

An alternative to https://github.com/kubernetes/kubernetes/pull/40731.

Please check the last 2 commits of this branch out for an example run: https://github.com/caesarxuchao/kubernetes/tree/add-stagingto-gopath. I added package "validationchao" to apimachinery, and let k8s.io/kubernetes/pkg/client/generated_clientset/ depend on it, then I run hack/update-staging-client-go.sh, and it seems to work fine.

@sttts @deads2k @lavalamp

Automatic merge from submit-queue (batch tested with PRs 41037, 40118, 40959, 41084, 41092)

CRI node e2e: add tests for docker 1.10

Automatic merge from submit-queue (batch tested with PRs 41037, 40118, 40959, 41084, 41092)

Switch CSR controller to use shared informer

Switch the CSR controller to use a shared informer. Originally part of #40097 but I'm splitting that up into multiple PRs.

I have added a test to try to ensure we don't mutate the cache. It could use some fleshing out for additional coverage but it gets the initial job done, I think.

cc @mikedanese @deads2k @liggitt @sttts @kubernetes/sig-scalability-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 41037, 40118, 40959, 41084, 41092)

Add OpenAPI extensions to openapi_gen (Definition Generator)

fixes #40818

Automatic merge from submit-queue (batch tested with PRs 41037, 40118, 40959, 41084, 41092)

Fix for detach volume when node is not present/ powered off

Fixes #33061 
When a vm is reported as no longer present in cloud provider and is deleted by node controller, there are no attempts to detach respective volumes. For example, if a VM is powered off or paused, and pods are migrated to other nodes. In the case of vSphere, the VM cannot be started again because the VM still holds mount points to volumes that are now mounted to other VMs.

In order to re-join this node again, you will have to manually detach these volumes from the powered off vm before starting it.

The current fix will make sure the mount points are deleted when the VM is powered off. Since all the mount points are deleted, the VM can be powered on again.

This is a workaround proposal only. I still don't see the kubernetes issuing a detach request to the vsphere cloud provider which should be the case. (Details in original issue #33061 )

@luomiao @kerneltime @pdhamdhere @jingxu97 @saad-ali

Automatic merge from submit-queue (batch tested with PRs 41037, 40118, 40959, 41084, 41092)

Bump up GLBC version from 0.9.0-beta to 0.9.1

Tests have been green, moving the beta to a release.

Automatic merge from submit-queue (batch tested with PRs 41121, 40048, 40502, 41136, 40759)

Shared informer custom resync periods

**What this PR does / why we need it**: add the ability for event handlers to specify their own distinct resync periods to a shared informer

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #26406 

**Special notes for your reviewer**:

This is part of my series of PRs to convert as much of the code base to using generated shared informers as possible. Assuming this merges, we'll also need #40385, and I'll probably close #40097 and replace it with a series of individual PRs (1 per controller change).

Assuming we get close to agreement on the implementation, I can look at adding more unit tests.

**Release note**:

```release-note
```

@smarterclayton @deads2k @liggitt @sttts @derekwaynecarr @pmorie @lavalamp @wojtek-t @gmarek @timothysc @jayunit100 @kubernetes/sig-api-machinery-pr-reviews @kubernetes/sig-scalability-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 41121, 40048, 40502, 41136, 40759)

add k8s.io/sample-apiserver to demonstrate how to build an aggregated API server

builds on https://github.com/kubernetes/kubernetes/pull/41093

This creates a sample API server is a separate staging repo to guarantee no cheating with `k8s.io/kubernetes` dependencies.  The sample is run during integration tests (simple tests on it so far) to ensure that it continues to run.

@sttts @kubernetes/sig-api-machinery-misc ptal
@pwittrock @pmorie @kris-nova an aggregated API server example that will stay up to date.

Automatic merge from submit-queue (batch tested with PRs 41121, 40048, 40502, 41136, 40759)

Match etcd process name exactly

A process named, e.g., etcd-operator, should not match



**What this PR does / why we need it**: Matches etcd process name exactly 

**Which issue this PR fixes**: fixes #40499 

**Special notes for your reviewer**: 👀 @lavalamp

Automatic merge from submit-queue (batch tested with PRs 41121, 40048, 40502, 41136, 40759)

Remove deprecated kubelet flags that look safe to remove

Removes:
```
--config
--auth-path
--resource-container
--system-container
```
which have all been marked deprecated since at least 1.4 and look safe to remove.

```release-note
The deprecated flags --config, --auth-path, --resource-container, and --system-container were removed.
```

Automatic merge from submit-queue (batch tested with PRs 41121, 40048, 40502, 41136, 40759)

[Federation] Wait after cleanup only if the cleanup succeeds.

It is a waste of time to wait for the resources to cleanup if the
cleanup fails.

cc @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 41145, 38771, 41003, 41089, 40365)

Add `kubectl attach` support for multiple types

To address this issue: https://github.com/kubernetes/kubernetes/issues/24857
the new `kubectl attach` will contain three scenarios depend on args:
1. `kubectl attach POD` :   if only one argument provided, we assume it's a pod name
2. `kubectl attach TYPE NAME` : if two arguments provided, we assume first one is resource we [supported](https://github.com/shiywang/kubernetes/blob/4770162fd3fd0d65ff95348c71683cf4c717141a/pkg/kubectl/cmd/util/factory_object_mapping.go#L285), the second resource's name.
3. `kubectl attach TYPE/NAME` : one argument provided and arg[0] must contain `/`, ditto

Is there any other scenarios I haven't consider in ?

for now the first scenario is compatible with changed before, also `make test` pass ✅ 
will write some unit test to test second and third scenario, if you guys think i'm doing the right way.
@pwittrock @kargakis @fabianofranz @ymqytw @AdoHe

Automatic merge from submit-queue (batch tested with PRs 41145, 38771, 41003, 41089, 40365)

Use privileged containers for statefulset e2e tests

Test containers need to run as spc_t in order to interact with the host
filesystem under /tmp, as the tests for StatefulSet are doing. Docker
will transition the container into this domain when running the container
as privileged.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

**Release note**:
```release-note
NONE
```

/cc @ncdc @soltysh @pmorie

Automatic merge from submit-queue (batch tested with PRs 41145, 38771, 41003, 41089, 40365)

Remove useless param from kubectl create rolebinding

The `force` param is not used in
`kubectl create rolebinding` & `kubectl create clusterrolebinding`
commands, removed it.

Automatic merge from submit-queue (batch tested with PRs 41145, 38771, 41003, 41089, 40365)

Fix typo in drain command

Automatic merge from submit-queue

Do not cleanup already deleted replica sets and add more logging around it

For https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/3569

@ncdc will make the output of the test cleaner

Automatic merge from submit-queue

Add janetkuo to approvers for controllers

Automatic merge from submit-queue

Add OWNERS file for GCE cloud provider

GCE cloud provider does not have OWNERS file and all PRs need to be approved by owner of pkg/cloudprovider, which is currently only @mikedanese. Adding more options would be helpful to speed up reviews.

Feel free to add/remove some names, this first version is just my qualified guess. It's hard to distinguish generic Kubernetes refactoring from real cloud provider work in git log.

```release-note
NONE
```

Revert "[Kubelet] Delay deletion of pod from the API server until volumes are deleted"

1. pcb and pcb controller are removed and their functionality is
encapsulated in StatefulPodControlInterface.
2. IdentityMappers has been removed to clarify what properties of a Pod are
mutated by the controller. All mutations are performed in the
UpdateStatefulPod method of the StatefulPodControlInterface.
3. The statefulSetIterator and petQueue classes are removed. These classes
sorted Pods by CreationTimestamp. This is brittle and not resilient to
clock skew. The current control loop, which implements the same logic,
is in stateful_set_control.go. The Pods are now sorted and considered by
their ordinal indices, as is outlined in the documentation.
4. StatefulSetController now checks to see if the Pods matching a
StatefulSet's Selector also match the Name of the StatefulSet. This will
make the controller resilient to overlapping, and will be enhanced by
the addition of ControllerRefs.