Automatic merge from submit-queue (batch tested with PRs 42456, 42457, 42414, 42480, 42370)

controller: reduce log verbosity for deployments

Fixes https://github.com/kubernetes/kubernetes/issues/41187

Labeling as a bug fix since I think excessive logging should be considered as a bug.

@kubernetes/sig-apps-bugs

Automatic merge from submit-queue (batch tested with PRs 42456, 42457, 42414, 42480, 42370)

Enque DaemonSet sync when node taints changed

Fixes #42398

 @kargakis @janetkuo @mdshuai PTAL

Automatic merge from submit-queue (batch tested with PRs 42456, 42457, 42414, 42480, 42370)

node e2e: apparmor test should fail instead of panicking

This doesn't fix #42420, but at least stop the test from panicking.

Automatic merge from submit-queue (batch tested with PRs 42456, 42457, 42414, 42480, 42370)

Update npd in kubemark since #42201 is merged.

Revert https://github.com/kubernetes/kubernetes/pull/41716.

#42201 has been merged, and #41713 is fixed. Now we could retry update npd in kubemark.

/cc @shyamjvs @wojtek-t @dchen1107

Automatic merge from submit-queue (batch tested with PRs 42369, 42375, 42397, 42435, 42455)

Add alsologtostderr flag to hollow node

@yujuhong @wojtek-t that should solve the kubemark log issue.

Automatic merge from submit-queue (batch tested with PRs 42369, 42375, 42397, 42435, 42455)

[Bug Fix]: Avoid evicting more pods than necessary by adding Timestamps for fsstats and ignoring stale stats

Continuation of #33121.  Credit for most of this goes to @sjenning.  I added volume fs timestamps.

**why is this a bug** 
This PR attempts to fix part of https://github.com/kubernetes/kubernetes/issues/31362 which results in multiple pods getting evicted unnecessarily whenever the node runs into resource pressure. This PR reduces the chances of such disruptions by avoiding reacting to old/stale metrics.
Without this PR, kubernetes nodes under resource pressure will cause unnecessary disruptions to user workloads. 
This PR will also help deflake a node e2e test suite.

The eviction manager currently avoids evicting pods if metrics are old.  However, timestamp data is not available for filesystem data, and this causes lots of extra evictions.
See the [inode eviction test flakes](https://k8s-testgrid.appspot.com/google-node#kubelet-flaky-gce-e2e) for examples.
This should probably be treated as a bugfix, as it should help mitigate extra evictions.

cc: @kubernetes/sig-storage-pr-reviews  @kubernetes/sig-node-pr-reviews @vishh @derekwaynecarr @sjenning

Automatic merge from submit-queue (batch tested with PRs 42369, 42375, 42397, 42435, 42455)

Kubelet: return container runtime's version instead of CRI's one

**What this PR does / why we need it**:

With CRI enabled by default, kubelet reports the version of CRI instead of container runtime version. This PR fixes this problem.

**Which issue this PR fixes** 

Fixes #42396.

**Special notes for your reviewer**:

Should also cherry-pick to 1.6 branch.

**Release note**:

```release-note
NONE
```

cc @yujuhong  @kubernetes/sig-node-bugs

Automatic merge from submit-queue (batch tested with PRs 42369, 42375, 42397, 42435, 42455)

Fixing federation controllers to support controllers flag

Fixes https://github.com/kubernetes/kubernetes/issues/42374

cc @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue

Output of `kubectl get` is inconsistent for pods

Builds on top of fixes from #42283, only the last two commits are new. Reverts behavior of #39042 which was inconsistent and confusing.

Fixes #15853

Automatic merge from submit-queue

ignore base.go in client-verify

We need to cherry-pick it to 1.6 to fix #42290.

Automatic merge from submit-queue

Eviction Manager Enforces Allocatable Thresholds

This PR modifies the eviction manager to enforce node allocatable thresholds for memory as described in kubernetes/community#348.
This PR should be merged after #41234. 

cc @kubernetes/sig-node-pr-reviews @kubernetes/sig-node-feature-requests @vishh 

** Why is this a bug/regression**

Kubelet uses `oom_score_adj` to enforce QoS policies. But the `oom_score_adj` is based on overall memory requested, which means that a Burstable pod that requested a lot of memory can lead to OOM kills for Guaranteed pods, which violates QoS. Even worse, we have observed system daemons like kubelet or kube-proxy being killed by the OOM killer.
Without this PR, v1.6 will have node stability issues and regressions in an existing GA feature `out of Resource` handling.

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Apply custom defaults to init containers

Adds overridden defaults to init containers. They were not being defaulted the same way normal containers were.

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Fix 'not patched' kubectl error

fixes #42384

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Fix StatefulSet e2e flake

**What this PR does / why we need it**:
Fixes StatefulSet e2e flake by ensuring that the StatefulSet controller has observed the unreadiness of Pods prior to attempting to exercise scale functionality.
**Which issue this PR fixes** 
fixes #41889
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Dell EMC ScaleIO Volume Plugin

**What this PR does / why we need it**
This PR implements the Kubernetes volume plugin to allow pods to seamlessly access and use data stored on ScaleIO volumes.  [ScaleIO](https://www.emc.com/storage/scaleio/index.htm) is a software-based storage platform that creates a pool of distributed block storage using locally attached disks on every server.  The code for this PR supports persistent volumes using PVs, PVCs, and dynamic provisioning.

You can find examples of how to use and configure the ScaleIO Kubernetes volume plugin in [examples/volumes/scaleio/README.md](examples/volumes/scaleio/README.md).

**Special notes for your reviewer**:
To facilitate code review, commits for source code implementation are separated from other artifacts such as generated, docs, and vendored sources.

```release-note
ScaleIO Kubernetes Volume Plugin added enabling pods to seamlessly access and use data stored on ScaleIO volumes.
```

Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Cast system uptime to time.Duration to fix cross build.

Fixes https://github.com/kubernetes/kubernetes/issues/42441.

Cast system uptime to `time.Duration` to avoid different behavior on different architectures.

@sjenning @ixdy @ncdc

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

Fix error printing objects from kubectl get -w

Fixes #42276

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

Add read permissions for statefulsets for kube-scheduler

https://github.com/kubernetes/kubernetes/issues/41708 added statefulset awareness to the scheduler. This adds the corresponding permission to the scheduler role.

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

enable cgroups tiers and node allocatable enforcement on pods by default.

```release-note
Pods are launched in a separate cgroup hierarchy than system services.
```
Depends on #41753

cc @derekwaynecarr

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

kubelet: enable qos-level memory limits

```release-note
Experimental support to reserve a pod's memory request from being utilized by pods in lower QoS tiers.
```

Enables the QoS-level memory cgroup limits described in https://github.com/kubernetes/community/pull/314

**Note: QoS level cgroups have to be enabled for any of this to take effect.**

Adds a new `--experimental-qos-reserved` flag that can be used to set the percentage of a resource to be reserved at the QoS level for pod resource requests.

For example, `--experimental-qos-reserved="memory=50%`, means that if a Guaranteed pod sets a memory request of 2Gi, the Burstable and BestEffort QoS memory cgroups will have their `memory.limit_in_bytes` set to `NodeAllocatable - (2Gi*50%)` to reserve 50% of the guaranteed pod's request from being used by the lower QoS tiers.

If a Burstable pod sets a request, its reserve will be deducted from the BestEffort memory limit.

The result is that:
- Guaranteed limit matches root cgroup at is not set by this code
- Burstable limit is `NodeAllocatable - Guaranteed reserve`
- BestEffort limit is `NodeAllocatable - Guaranteed reserve - Burstable reserve`

The only resource currently supported is `memory`; however, the code is generic enough that other resources can be added in the future.

@derekwaynecarr @vishh

Automatic merge from submit-queue (batch tested with PRs 41919, 41149, 42350, 42351, 42285)

Juju: Disable anonymous auth on kubelet

**What this PR does / why we need it**:

This disables anonymous authentication on kubelet when deployed via Juju.

I've also adjusted a few other TLS options for kubelet and kube-apiserver. The end result is that:
1. kube-apiserver can now authenticate with kubelet
2. kube-apiserver now verifies the integrity of kubelet

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:

https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/219

**Special notes for your reviewer**:

This is dependent on PR #41251, where the tactics changes are being merged in separately.

Some useful pages from the documentation:
* [apiserver -> kubelet](https://kubernetes.io/docs/admin/master-node-communication/#apiserver---kubelet)
* [Kubelet authentication/authorization](https://kubernetes.io/docs/admin/kubelet-authentication-authorization/)

**Release note**:

```release-note
Juju: Disable anonymous auth on kubelet
```

Automatic merge from submit-queue

Critial pod test uses allocatable instead of capacity

This solves #42239.

When this test was first introduced, pods could request up to the capacity of the node.
With the addition of allocatable introduced in #41234, this is no longer the case, and pods can only use up to allocatable.

This should be included in 1.6, as it is a bug related to a 1.6 feature.

cc @vish @yujuhong

This commits implements the Kubernetes volume plugin allowing pods to seamlessly access and use data stored on ScaleIO volumes.

Pods prior mutating the StatefulSet object to trigger sclaing.

Add ObervedVersion check

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Bump test timeouts to make secret tests work in large clusters

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

discovery restmapping should always prefer /v1

The core kube API, empty group, version==v1 should always be the most preferred group and resource from a rest mapper.  This special cases that.  All the others should be based on discovery order as we previously agreed.

@kubernetes/sig-cli-pr-reviews @kubernetes/sig-api-machinery-pr-reviews 
@enj

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

remove support for debian masters in GCE

Asked about this on the mailing list and no one objects.

@zmerlynn @roberthbailey 

```release-note
Remove support for debian masters in GCE kube-up.
```

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Server timeout returns an incorrect error

Not a valid Status object in JSON

Part of #42163

Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Implement bulk polling of volumes

This implements Bulk volume polling using ideas presented by
justin in https://github.com/kubernetes/kubernetes/pull/39564

But it changes the implementation to use an interface
and doesn't affect other implementations.

cc @justinsb

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Make Downward API test table-driven

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

kubeadm: Add --cert-dir, --cert-altnames instead of --api-external-dns-names

**What this PR does / why we need it**:

 - For the beta kubeadm init UX, we need this change
 - Also adds the `kubeadm phase certs selfsign` command that makes the phase invokable independently

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

This PR depends on https://github.com/kubernetes/kubernetes/pull/41897

**Release note**:

```release-note
```
@dmmcquay @pires @jbeda @errordeveloper @mikedanese @deads2k @liggitt

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Updated scheduler name for multi-scheduler.

fixes #41859

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

controller: statefulsets respect observed generation

StatefulSets do not update ObservedGeneration even though the API field is in place. This means that clients can never be sure whether the StatefulSet controller has observed the latest spec of a StatefulSet.

@kubernetes/sig-apps-bugs

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Remove kube-dns ConfigMap modification code from federation-{up,down}.sh scripts

PR #39338 has merged now. This shouldn't be necessary. This unbreaks federation tests.

```release-note
NONE
```

Handle potential conflicts on service endpoint test deletion

Cleanup should ignore conflicts and notfound errors

e2e/framework: Deflake cleanup of RCs in service_util.

The previous Get/Update pattern with no retry on resource version mismatch
would flake with the following error:

"the object has been modified; please apply your changes to the latest
version and try again"