Automatic merge from submit-queue

Eviction Manager Enforces Allocatable Thresholds

This PR modifies the eviction manager to enforce node allocatable thresholds for memory as described in kubernetes/community#348.
This PR should be merged after #41234. 

cc @kubernetes/sig-node-pr-reviews @kubernetes/sig-node-feature-requests @vishh 

** Why is this a bug/regression**

Kubelet uses `oom_score_adj` to enforce QoS policies. But the `oom_score_adj` is based on overall memory requested, which means that a Burstable pod that requested a lot of memory can lead to OOM kills for Guaranteed pods, which violates QoS. Even worse, we have observed system daemons like kubelet or kube-proxy being killed by the OOM killer.
Without this PR, v1.6 will have node stability issues and regressions in an existing GA feature `out of Resource` handling.

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Apply custom defaults to init containers

Adds overridden defaults to init containers. They were not being defaulted the same way normal containers were.

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Fix 'not patched' kubectl error

fixes #42384

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Fix StatefulSet e2e flake

**What this PR does / why we need it**:
Fixes StatefulSet e2e flake by ensuring that the StatefulSet controller has observed the unreadiness of Pods prior to attempting to exercise scale functionality.
**Which issue this PR fixes** 
fixes #41889
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Dell EMC ScaleIO Volume Plugin

**What this PR does / why we need it**
This PR implements the Kubernetes volume plugin to allow pods to seamlessly access and use data stored on ScaleIO volumes.  [ScaleIO](https://www.emc.com/storage/scaleio/index.htm) is a software-based storage platform that creates a pool of distributed block storage using locally attached disks on every server.  The code for this PR supports persistent volumes using PVs, PVCs, and dynamic provisioning.

You can find examples of how to use and configure the ScaleIO Kubernetes volume plugin in [examples/volumes/scaleio/README.md](examples/volumes/scaleio/README.md).

**Special notes for your reviewer**:
To facilitate code review, commits for source code implementation are separated from other artifacts such as generated, docs, and vendored sources.

```release-note
ScaleIO Kubernetes Volume Plugin added enabling pods to seamlessly access and use data stored on ScaleIO volumes.
```

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Cast system uptime to time.Duration to fix cross build.

Fixes https://github.com/kubernetes/kubernetes/issues/42441.

Cast system uptime to `time.Duration` to avoid different behavior on different architectures.

@sjenning @ixdy @ncdc

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

Fix error printing objects from kubectl get -w

Fixes #42276

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

Add read permissions for statefulsets for kube-scheduler

https://github.com/kubernetes/kubernetes/issues/41708 added statefulset awareness to the scheduler. This adds the corresponding permission to the scheduler role.

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

enable cgroups tiers and node allocatable enforcement on pods by default.

```release-note
Pods are launched in a separate cgroup hierarchy than system services.
```
Depends on #41753

cc @derekwaynecarr

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

kubelet: enable qos-level memory limits

```release-note
Experimental support to reserve a pod's memory request from being utilized by pods in lower QoS tiers.
```

Enables the QoS-level memory cgroup limits described in https://github.com/kubernetes/community/pull/314

**Note: QoS level cgroups have to be enabled for any of this to take effect.**

Adds a new `--experimental-qos-reserved` flag that can be used to set the percentage of a resource to be reserved at the QoS level for pod resource requests.

For example, `--experimental-qos-reserved="memory=50%`, means that if a Guaranteed pod sets a memory request of 2Gi, the Burstable and BestEffort QoS memory cgroups will have their `memory.limit_in_bytes` set to `NodeAllocatable - (2Gi*50%)` to reserve 50% of the guaranteed pod's request from being used by the lower QoS tiers.

If a Burstable pod sets a request, its reserve will be deducted from the BestEffort memory limit.

The result is that:
- Guaranteed limit matches root cgroup at is not set by this code
- Burstable limit is `NodeAllocatable - Guaranteed reserve`
- BestEffort limit is `NodeAllocatable - Guaranteed reserve - Burstable reserve`

The only resource currently supported is `memory`; however, the code is generic enough that other resources can be added in the future.

@derekwaynecarr @vishh

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

Juju: Disable anonymous auth on kubelet

**What this PR does / why we need it**:

This disables anonymous authentication on kubelet when deployed via Juju.

I've also adjusted a few other TLS options for kubelet and kube-apiserver. The end result is that:
1. kube-apiserver can now authenticate with kubelet
2. kube-apiserver now verifies the integrity of kubelet

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:

https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/219

**Special notes for your reviewer**:

This is dependent on PR #41251, where the tactics changes are being merged in separately.

Some useful pages from the documentation:
* [apiserver -> kubelet](https://kubernetes.io/docs/admin/master-node-communication/#apiserver---kubelet)
* [Kubelet authentication/authorization](https://kubernetes.io/docs/admin/kubelet-authentication-authorization/)

**Release note**:

```release-note
Juju: Disable anonymous auth on kubelet
```

Automatic merge from submit-queue

Critial pod test uses allocatable instead of capacity

This solves #42239.

When this test was first introduced, pods could request up to the capacity of the node.
With the addition of allocatable introduced in #41234, this is no longer the case, and pods can only use up to allocatable.

This should be included in 1.6, as it is a bug related to a 1.6 feature.

cc @vish @yujuhong

This commits implements the Kubernetes volume plugin allowing pods to seamlessly access and use data stored on ScaleIO volumes.

Pods prior mutating the StatefulSet object to trigger sclaing.

Add ObervedVersion check

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Bump test timeouts to make secret tests work in large clusters

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

discovery restmapping should always prefer /v1

The core kube API, empty group, version==v1 should always be the most preferred group and resource from a rest mapper.  This special cases that.  All the others should be based on discovery order as we previously agreed.

@kubernetes/sig-cli-pr-reviews @kubernetes/sig-api-machinery-pr-reviews 
@enj

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

remove support for debian masters in GCE

Asked about this on the mailing list and no one objects.

@zmerlynn @roberthbailey 

```release-note
Remove support for debian masters in GCE kube-up.
```

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Server timeout returns an incorrect error

Not a valid Status object in JSON

Part of #42163

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Implement bulk polling of volumes

This implements Bulk volume polling using ideas presented by
justin in https://github.com/kubernetes/kubernetes/pull/39564

But it changes the implementation to use an interface
and doesn't affect other implementations.

cc @justinsb

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Make Downward API test table-driven

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

kubeadm: Add --cert-dir, --cert-altnames instead of --api-external-dns-names

**What this PR does / why we need it**:

 - For the beta kubeadm init UX, we need this change
 - Also adds the `kubeadm phase certs selfsign` command that makes the phase invokable independently

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

This PR depends on https://github.com/kubernetes/kubernetes/pull/41897

**Release note**:

```release-note
```
@dmmcquay @pires @jbeda @errordeveloper @mikedanese @deads2k @liggitt

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Updated scheduler name for multi-scheduler.

fixes #41859

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

controller: statefulsets respect observed generation

StatefulSets do not update ObservedGeneration even though the API field is in place. This means that clients can never be sure whether the StatefulSet controller has observed the latest spec of a StatefulSet.

@kubernetes/sig-apps-bugs

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Remove kube-dns ConfigMap modification code from federation-{up,down}.sh scripts

PR #39338 has merged now. This shouldn't be necessary. This unbreaks federation tests.

```release-note
NONE
```

Handle potential conflicts on service endpoint test deletion

Cleanup should ignore conflicts and notfound errors

e2e/framework: Deflake cleanup of RCs in service_util.

The previous Get/Update pattern with no retry on resource version mismatch
would flake with the following error:

"the object has been modified; please apply your changes to the latest
version and try again"

Automatic merge from submit-queue

Remove defaults from string flags

- The default is printed automatically
 - The string text did not match the actual default

**What this PR does / why we need it**:
Adjust the documentation for flags on `client-gen`.

**Special notes for your reviewer**:
Doc change. String text only.

**Release note**:
```release-note
NONE
```

Before:
```
client-gen  --help
Usage of ./client-gen:
      --build-tag string                       A Go build tag to use to identify files generated by this command. Should be unique. (default "ignore_autogenerated")
      --clientset-api-path string              the value of default API path.
  -n, --clientset-name string                  the name of the generated clientset package. (default "internalclientset")
      --clientset-only                         when set, client-gen only generates the clientset shell, without generating the individual typed clients
      --clientset-path string                  the generated clientset will be output to <clientset-path>/<clientset-name>. Default to "k8s.io/kubernetes/pkg/client/clientset_generated/" (default "k8s.io/kubernetes/pkg/client/clientset_generated/")
      --fake-clientset                         when set, client-gen will generate the fake clientset that can be used in tests (default true)
  -h, --go-header-file string                  File containing boilerplate header text. The string YEAR will be replaced with the current 4-digit year. (default "/Users/mhb/go/src/k8s.io/gengo/boilerplate/boilerplate.go.txt")
      --included-types-overrides stringSlice   list of group/version/type for which client should be generated. By default, client is generated for all types which have genclient=true in types.go. This overrides that. For each groupVersion in this list, only the types mentioned here will be included. The default check of genclient=true will be used for other group versions.
      --input stringSlice                      group/versions that client-gen will generate clients for. At most one version per group is allowed. Specified in the format "group1/version1,group2/version2...". Default to "api/,extensions/,autoscaling/,batch/,rbac/" (default [api/,authentication/,authorization/,autoscaling/,batch/,certificates/,extensions/,rbac/,storage/,apps/,policy/])
      --input-base string                      base path to look for the api group. Default to "k8s.io/kubernetes/pkg/apis" (default "k8s.io/kubernetes/pkg/apis")
  -i, --input-dirs stringSlice                 Comma-separated list of import paths to get input types from.
  -o, --output-base string                     Output base; defaults to $GOPATH/src/ or ./ if $GOPATH is not set. (default "/Users/mhb/go/src")
  -O, --output-file-base string                Base name (without .go suffix) for output files.
  -p, --output-package string                  Base package path.
  -t, --test                                   set this flag to generate the client code for the testdata
      --verify-only                            If true, only verify existing output, do not write anything.
```
After:
```
client-gen  --help
Usage of ./client-gen:
      --build-tag string                       A Go build tag to use to identify files generated by this command. Should be unique. (default "ignore_autogenerated")
      --clientset-api-path string              the value of default API path.
  -n, --clientset-name string                  the name of the generated clientset package. (default "internalclientset")
      --clientset-only                         when set, client-gen only generates the clientset shell, without generating the individual typed clients
      --clientset-path string                  the generated clientset will be output to <clientset-path>/<clientset-name>. (default "k8s.io/kubernetes/pkg/client/clientset_generated/")
      --fake-clientset                         when set, client-gen will generate the fake clientset that can be used in tests (default true)
  -h, --go-header-file string                  File containing boilerplate header text. The string YEAR will be replaced with the current 4-digit year. (default "/Users/mhb/go/src/k8s.io/gengo/boilerplate/boilerplate.go.txt")
      --included-types-overrides stringSlice   list of group/version/type for which client should be generated. By default, client is generated for all types which have genclient=true in types.go. This overrides that. For each groupVersion in this list, only the types mentioned here will be included. The default check of genclient=true will be used for other group versions.
      --input stringSlice                      group/versions that client-gen will generate clients for. At most one version per group is allowed. Specified in the format "group1/version1,group2/version2...". (default [api/,authentication/,authorization/,autoscaling/,batch/,certificates/,extensions/,rbac/,storage/,apps/,policy/])
      --input-base string                      base path to look for the api group. (default "k8s.io/kubernetes/pkg/apis")
  -i, --input-dirs stringSlice                 Comma-separated list of import paths to get input types from.
  -o, --output-base string                     Output base; defaults to $GOPATH/src/ or ./ if $GOPATH is not set. (default "/Users/mhb/go/src")
  -O, --output-file-base string                Base name (without .go suffix) for output files.
  -p, --output-package string                  Base package path.
  -t, --test                                   set this flag to generate the client code for the testdata
      --verify-only                            If true, only verify existing output, do not write anything.
```

This implements Bulk volume polling using ideas presented by
justin in https://github.com/kubernetes/kubernetes/pull/39564

But it changes the implementation to use an interface
and doesn't affect other implementations.

Automatic merge from submit-queue

Describers with pod templates should have consistent output

Added a test to verify it.

Fixes #38698

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

Move node problem detector test into node e2e.

Move current NPD e2e test into node e2e.

In fact, current NPD e2e test is only a functionality test for NPD. It creates test NPD pod, sets test configuration, generates test logs and verifies test result.
It doesn't actually test the NPD really deployed in the cluster.

So it doesn't actually need to run in cluster e2e. Running it in node e2e will:
1) Make it easier to run the test.
2) Make it more light weight to introduce this as a pre/post submit test in NPD repo in the future.

Except this, I'm working on a cluster e2e to run some basic functionality test and benchmark test against the real NPD deployed in the cluster. Will send the PR later.

/cc @dchen1107 @kubernetes/node-problem-detector-reviewers

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

make edit respect --save-config

New behavior:

edit:
1) the flag is set to true: it will create or update the last-applied-config annotation
2) the flag is set to false or is unspecified: NOP

replace:
1) If saveConfig is true, create or update the annotation.
2) If saveConfig is false and the local config file doesn't have the annotation, we save the annotation from the live object if there is one (Try to not surprise the users).
3) If saveConfig is false and the local config file has the annotation, we use the annotation in the config file.
4) Same behavior for force replacing

fixes #40626

```release-note
stop kubectl edit from updating the last-applied-configuration annotation when --save-config is unspecified or false.
```

@pwittrock @liggitt 

This is a bug fix that prevented `edit` from being respected by `apply`

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

Add options to kubefed telling it to generate HTTP Basic and/or token credentials for the Federated API server

fixes #41265.

**Release notes**:
```release-note
Adds two options to kubefed, `-apiserver-enable-basic-auth` and `-apiserver-enable-token-auth`, which generate an HTTP Basic username/password and a token respectively for the Federated API server.
```

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

RC/RS: Fully Respect ControllerRef

**What this PR does / why we need it**:

This is part of the completion of the [ControllerRef](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/controller-ref.md) proposal. It brings ReplicaSet and ReplicationController into full compliance with ControllerRef. See the individual commit messages for details.

**Which issue this PR fixes**:

Although RC/RS had partially implemented ControllerRef, they didn't use it to determine which controller to sync, or to update expectations. This could lead to instability or controllers getting stuck.

Ref: https://github.com/kubernetes/kubernetes/issues/24433

**Special notes for your reviewer**:

**Release note**:
```release-note
```
cc @erictune @kubernetes/sig-apps-pr-reviews

kubeadm: Add --cert-dir, --apiserver-cert-extra-sans, remove --api-external-dns-names and add the phase command for certs. Also use the CertificatesDir var everywhere instead of the HostPKIPath variable and fix some bugs in certs.go