Automatic merge from submit-queue (batch tested with PRs 42053, 41282, 42056, 41663, 40927)

Fully remove hand-written listers and informers

Note: the first commit is from #41927. Adding do-not-merge for now as we'll want that to go in first, and then I'll rebase this on top.

Update statefulset controller to use a lister for PVCs instead of a client request. Also replace a unit test's dependency on legacylisters with the generated ones. cc @kargakis @kow3ns @foxish @kubernetes/sig-apps-pr-reviews 

Remove all references to pkg/controller/informers and pkg/client/legacylisters, and remove those packages.

@smarterclayton @deads2k this should be it!

cc @gmarek @wojtek-t @derekwaynecarr @kubernetes/sig-scalability-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 42053, 41282, 42056, 41663, 40927)

Enable Garbage collection by default for RS and RC

Fixes https://github.com/kubernetes/kubernetes/issues/40898

Automatic merge from submit-queue (batch tested with PRs 42053, 41282, 42056, 41663, 40927)

Enable strict client-go verification in CI

**Follow-up of https://github.com/kubernetes/kubernetes/pull/41987. Only the two last commits are relevant here.**

This turns on strict client-go verification, i.e. PRs with missing client-go updates will fail in CI.

It also updates the client-go once to have a baseline.

**This should be enabled just after the freeze in order to enforce consistency of client-go.**

Make kubectl taint e2e serial

Make kubectl taint e2e run [Serial] so it doesn't affect any other
tests.

Automatic merge from submit-queue

fix typo:evaluator

Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

make kubectl taint command respect effect NoExecute

**What this PR does / why we need it**:
Part of feature forgiveness implementation, make kubectl taint command respect effect NoExecute.

**Which issue this PR fixes**: 
Related Issue: #1574
Related PR: #39469

**Special notes for your reviewer**:

**Release note**:

```release-note
make kubectl taint command respect effect NoExecute
```

Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

Remove SandboxReceived event

This PR removes SandboxReceived event in sync pod.

> This event seems somewhat meaningless, and clouds the event records for a pod. Do we actually need it? Pulling and pod received on the node are very relevant, this seems much less so. Would suggest we either remove it, or turn it into a message that clearly indicates why it has value.

Refer https://github.com/kubernetes/kubernetes/commit/d65309399a228bd0c085f5e0d1ce93d558aebfaa#commitcomment-21052453.

cc @smarterclayton @yujuhong

Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

Export Prometheus metrics in kube-proxy

Some metrics are already collected inside of kube-proxy (see https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/proxy.go#L28). This exposes `/metrics` endpoint on healtz port.

ref #40736 #9125

cc @brancz @fabxc

Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

Add support for attacher/detacher interface in Flex volume

Add support for attacher/detacher interface in Flex volume
This change breaks backward compatibility and requires to be release noted.

```release-note
Flex volume plugin is updated to support attach/detach interfaces. It broke backward compatibility. Please update your drivers and implement the new callouts. 
```

Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

DaemonSet updates - take 2

It implements https://github.com/kubernetes/community/blob/master/contributors/design-proposals/daemonset-update.md
Feature kubernetes/features#124

```release-note
Implement the update feature for DaemonSet.
```

cc @kargakis @janetkuo @mikedanese

Use a PVC lister instead of a client when retrieving PVCs.

Replace unit test's use of legacylisters with the generated listers.

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

[Federation] Create configmap for the cluster kube-dns when cluster joins and remove when it unjoins

This PR implements the functionality as needed in https://github.com/kubernetes/kubernetes/issues/38400

cc @kubernetes/sig-cluster-federation @nikhiljindal @madhusudancs 

**Release note**:

```
kubefed join can now automatically create a configmap or add information to already existing one, storing this federation name vs zone name information in the joining cluster.
Further kubefed unjoin can remove this configmap or only this information from the configmap, if the unjoin cluster is registered with multiple federations.
The name of the configmap is kube-dns and the information in it is consumed by the in-cluster dns server.
```

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

add OWNER file to kubelet/network

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

Bump up dns-horizontal-autoscaler to 1.1.1

cluster-proportional-autoscaler 1.1.1 is releasing by kubernetes-incubator/cluster-proportional-autoscaler#26, also bump it up for dns-horizontal-autoscaler to introduce below features:
- Add PreventSinglePointFailure option in linear mode.
- Use protobufs for communication with apiserver.
- Support switching control mode on-the-fly.

Note:
The new entry `"preventSinglePointFailure":true` ensures kube-dns to have at least 2 replicas when there is more than one node. Mitigate the issue mentioned in #40063.

@bowei @thockin 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

New command for stand-alone GKE certificates controller

New stand-alone certificates controller for GKE. Rather than requiring the CA's private key on disk, this allows making external calls to GKE in order to sign cluster certificates.
    
**Which issue this PR fixes**: fixes #39761

**Release note**:

```release-note
New GKE certificates controller.
```

CC @mikedanese @jcbsmpsn

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

Juju: Fix shebangs in charm actions to use python3

**What this PR does / why we need it**:

This fixes the microbot and create-rbd-pv actions by reverting them back to python3. We accidentally switched them to python2 to match the boilerplate checker's expectations for python files.

It looks like hack/verify-boilerplate.sh does not check these since they don't have the .py extension, so we should be good with no changes there.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:

https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/212

**Special notes for your reviewer**:

**Release note**:

```release-note
Juju: Fix shebangs in charm actions to use python3
```

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

Update flag to --check-version-skew instead of --check_version_skew

https://github.com/kubernetes/test-infra/issues/2012

Also add a `--` to send the flags to kubetest without triggering a warning.

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

PV upgrade test

**What this PR does / why we need it**:
This PR adds a PV upgrade test to the new upgrade test framework.  Before, this test had to be done manually.  Currently the upgrade test framework only works on the GCE environment, so I plan to add support for other providers later.  In order to write the test, I had to modify and refactor some volume test util libraries.  I reran the impacted tests to make sure they still passed.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:
It's probably easier to review the two commits separately.  I split it up into the refactor changes, and the upgrade test changes.

**Release note**:

NONE


cc @saad-ali @krousey

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

Don't restrict etcd on self host installs b/c a clipped etcd can have

**What this PR does / why we need it**:
Remove cpu-clipping from self hosted etcd install from the masters.

**Special notes for your reviewer**:

**Release note**:

```
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

dockershim puts pause container in pod cgroup

**What this PR does / why we need it**:
The CRI was not launching the pause container in the pod level cgroup.  The non-CRI code path was.

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

add kubeadm completion command

reference to `kubectl completion`

```
#kubeadm <tab><tab>
alpha       completion  init        join        reset       version
```

It implements https://github.com/kubernetes/community/blob/master/contributors/design-proposals/daemonset-update.md
Feature https://github.com/kubernetes/features/issues/124

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

Simplify and fix hack/{verify,update}-staging-{client-go,godeps}.sh

- merge `hack/{verify,update}-staging-client-go.sh`
- pin godep with shared code
- remove godep-restore completely from the process and replace with a simple check that godeps are restored
- add safety check in `staging/copy.sh` that there is no lingering `k8s.io/apimachinery` in the GOPATH which would lead to inconsistent client-go builds (!)
- check that all these scripts only operate in a clean working dir.

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

Use existing nginx image in the node

**What this PR does / why we need it**:
 Fixes a test flake: `[k8s.io] Garbage collector should orphan pods created by rc if delete options say so`

**Which issue this PR fixes** 

fixes #35771

**Special notes for your reviewer**:

**Release note**:
```release-note NONE
```

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

federation: use generated listers

**What this PR does / why we need it**: switch federation code to use generated listers

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
```

cc @smarterclayton @derekwaynecarr @marun @gmarek @wojtek-t @deads2k @liggitt @sttts @timothysc @nikhiljindal @jianhuiz @kubernetes/sig-federation-pr-reviews @kubernetes/sig-scalability-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

Add apply set-last-applied subcommand 

implement part of https://github.com/kubernetes/community/pull/287, will rebase after https://github.com/kubernetes/kubernetes/pull/41699 got merged, EDIT: since bug output format has been confirmed, will update the behavior of output format soon  
cc @kubernetes/sig-cli-pr-reviews @AdoHe @pwittrock 

```release-note
Support kubectl apply set-last-applied command to update the applied-applied-configuration annotation
```

Automatic merge from submit-queue

daemon not deamon

Automatic merge from submit-queue

Use docker log rotation mechanism instead of logrotate

This is a solution for https://github.com/kubernetes/kubernetes/issues/38495.

Instead of rotating logs using logrotate tool, which is configured quite rigidly, this PR makes docker responsible for the rotation and makes it possible to configure docker logging parameters. It solves the following problems:

* Logging agent will stop loosing lines upon rotation
* Container's logs size will be more strictly constrained. Instead of checking the size hourly, size will be checked upon write, preventing https://github.com/kubernetes/kubernetes/issues/27754

It's still far from ideal, for example setting logging options per pod, as suggested in https://github.com/kubernetes/kubernetes/issues/15478 would be much more flexible, but latter approach requires deep changes, including changes in API, which may be in vain because of CRI and long-term vision for logging.

Changes include:

* Change in salt. It's possible to configure docker log parameters, using variables in pillar. They're exported from env variables on `gce`, but for different cloud provider they have to be exported first.
* Change in `configure-helper.sh` scripts for those os on `gce` that don't use salt + default values exposed via env variables

This change may be problematic for kubelet logs functionality with CRI enabled, that will be tackled in the follow-up PR, if confirmed.

CC @piosz @Random-Liu @yujuhong @dashpole @dchen1107 @vishh @kubernetes/sig-node-pr-reviews

```release-note
On GCI by default logrotate is disabled for application containers in favor of rotation mechanism provided by docker logging driver.
```

Automatic merge from submit-queue

Cleanup fluentd-gcp image, rebase on debian-base

**Why we need this PR**:

There are several problems with our current fluentd-gcp image:

- It pulls in lots of unused packages, which expose unnecessary risk and create noise in CVE scans (and scare customers). The most notable example is the fluent-ui, which pulls in rails.
- `curl | sh ` is not a good practice for a Dockerfile. First, the script is not checked in the same source control branch, so builds are not reproducible. Second, the actions it is taking are opaque. Third, in this case, using non-standard packages means they're harder to manage with CVE scans & upstream fixes.

**What is changed by this PR?**

- Rather than relying on td-agent (which includes fluent-ui), use standard upstream packages. This is largely based off the [official fluentd debian-based image](https://github.com/fluent/fluentd-docker-image/blob/master/v0.12/debian/Dockerfile).
- Rebases the image on debian-base (depends on https://github.com/kubernetes/kubernetes/pull/41915). We would like to move towards a single full-distro base image we can maintain. This change should be relatively minor.

As a result of these changes, the image size is reduced from 360.6 MB to 185.8 MB (nearly half). Many packages were removed, and the full diff (focus on the unversioned files) is listed here: https://github.com/timstclair/kubernetes/commit/3fb704f977875bb4cbb64687b2f6836fbe101cf6

**Which issue this PR fixes** https://github.com/kubernetes/kubernetes/issues/40248

**Special notes for your reviewer**:

This change both addresses security concerns, and is expected to greatly reduce the maintenance burden of the fluentd-gcp image. I'd *really* like to get this into 1.6, so please prioritize this review if possible.

I tested this by running the default e2e suite on a private e2e cluster using the new image. If there are other tests you'd like me to run, please let me know ASAP.

**Release note**:
```release-note
Cleanup fluentd-gcp image: rebase on debian-base, switch to upstream packages, remove fluent-ui & rails
```