all remaining scalers were replaced by GenericScaler exept JobScaler.
It is not clear whether JobScaler could use generic scaler or not.
For more details see the pull request.

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add comments for explaining the group/version fields

**What this PR does / why we need it**:
This PR adds comments for explaining group/version fields. This is my understanding of what @deads2k told me today. Please, correct me if I'm wrong or you see how it can be further improved.

I didn't find where it's documented, so I hope that this PR makes it easy for other new contributors addition of new APIs.

**Release note**:
```release-note
NONE
```

PTAL @deads2k 
CC @simo5

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Instrument the Azure API calls for Prometheus monitoring

**What this PR does / why we need it**:

Instruments the Azure API calls for Prometheus monitoring.

**Special notes for your reviewer**:

This is version 2 based on the wrapped clients.

**Release note**:

```release-note
Instrument the Azure cloud provider for Prometheus monitoring.
```

cc @feiskyer @andyzhangx @jdumars @khenidak

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

unstructured helpers: print path in error

Prints the path in the error message for `Unstructured` helper functions, giving more useful info.

Useful for https://github.com/kubernetes/kubernetes/pull/55168.

**Release note**:

```release-note
NONE
```

/assign sttts ash2k

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

use shared informers for TokenCleaner controller

**What this PR does / why we need it**:

fix TODO: Switch to shared informers

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove vmUUID check in VSphere cloud provider

This PR addresses issue mentioned in https://github.com/vmware/kubernetes/issues/419.

The fix is to remove the check for vvmUUID in entirety from 1.9 branch - https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go#L378 as it is not longer needed. The NodeInfo object by default provides this information.

This fix will this enable:
- Enable e2e test cases to be executed locally on development envt. (Mac in particular)
- Remove unnecessary vmUUID code in 1.9 branch.

@prashima @divyenpatel

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Rename func name according TODO

**What this PR does / why we need it**:

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
None
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

-Add scheduler optimization options, short circuit all predicates if …

…one predicate fails
Signed-off-by: Wang Guoliang <iamwgliang@gmail.com>

**What this PR does / why we need it**:
Short circuit all predicates if one predicate fails. 

I think we can add a switch to control it, maybe some scenes do not need to know all the causes of failure, but also can get a great performance improvement; if you need to fully understand the reasons for the failure, and accept the current performance requirements, can maintain the current logic. It should expose this switch to the user.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:

Fixes #56889 and #48186

**Special notes for your reviewer**:
@davidopp

**Release note**:

```
Allow scheduler set AlwaysCheckAllPredicates, short circuit all predicates if one predicate fails can greatly improve the scheduling performance.
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

cluster: move logging library to hack/

it's used once in cluster and used a bunch in hack/ and build/

see #49213
```release-note
NONE
```

it's used once in cluster and used a bunch in hack/ and build/

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

periodically flush ResponseWriter when using response compression

This PR fixes API Compression when following logs by ensuring the compression writer periodically flushes (at the end of each write).

This is a commit extracted from #51508 to allow testing of flushing while feature remains disabled / in alpha.

Fixes https://github.com/kubernetes/kubernetes/issues/54205

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 58246, 58247). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

cluster: remove support for cvm from gce kube-up

see #49213


```release-note
Remove deprecated ContainerVM support from GCE kube-up. 
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

cluster: remove kube-push

see #49213

```release-note
Remove deprecated kube-push.sh functionality. 
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

added defaults for --watch-cache-sizes description.

**What this PR does / why we need it**:
added defaults for --watch-cache-sizes description to make it clear what the defaults would be when not specified

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
#57105

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

[Deployments] Move some tests to use go sub-test

**What this PR does / why we need it**:

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 58192, 58231). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove salt support for providers that no longer exist

xref #49213

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 58192, 58231). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Added metrics for preemption

**What this PR does / why we need it**:
Metrics for preemption duration in scheduler.

**Special notes for your reviewer**:
xref:  https://github.com/kubernetes/kubernetes/issues/57471 
**Release note**:

```release-note
NONE
```
cc @bsalamat

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

fix error when run release.sh not in {KUBE_ROOT}

**What this PR does / why we need it**:
add `${KUBE_ROOT}/` in directory, otherwise run 
```shell
cd kubernetes/build  
./release.sh
```
will exit:
```shell
+++ [0112 18:18:48] Verifying Prerequisites....
cp: cannot stat ‘build/build-image/Dockerfile’: No such file or directory
!!! [0112 18:18:48] Call tree:
!!! [0112 18:18:48]  1: ./release.sh:35 kube::build::build_image(...)
!!! Error in ./../build/common.sh:454
  Error in ./../build/common.sh:454. '((i<3-1))' exited with status 1
Call stack:
  1: ./../build/common.sh:454 kube::build::build_image(...)
  2: ./release.sh:35 main(...)
Exiting with status 1
```

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 58216, 58193, 53033, 58219, 55921). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix device plugin endpoint UT

**What this PR does / why we need it**:
Fix some issues in device plugin endpoint UT.

**Which issue(s) this PR fixes**:
Fixes #55920

**Special notes for your reviewer**:

@jiayingz @RenaudWasTaken @lichuqiang PTAL.

/sig node

**Release note**:

```release-note
None
```

Automatic merge from submit-queue (batch tested with PRs 58216, 58193, 53033, 58219, 55921). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Use the bazel version check function from bazel-skylib

**What this PR does / why we need it**: fixes a bug in parsing the bazel version when a component has more than one digit, e.g. "0.10.0".

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
x-ref https://github.com/bazelbuild/bazel/issues/4425

**Special notes for your reviewer**: we also need to update rules_go, but that's a more invasive change, so I've saved that for another PR.

**Release note**:

```release-note
NONE
```

/assign @BenTheElder @mikedanese

Automatic merge from submit-queue (batch tested with PRs 58216, 58193, 53033, 58219, 55921). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix golint errors on test/e2e/e2e.go

**What this PR does / why we need it**:

When running golint on test/e2e/e2e.go, the following erros
were faced:

 $ golint e2e.go
 e2e.go:329:2: var metricsJson should be metricsJSON
 e2e.go:342:1: comment on exported function RunE2ETests should be of the form "RunE2ETests ..."

This PR fixes them.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

`NONE`

**Special notes for your reviewer**:

`NONE`

**Release note**:

`NONE`

Automatic merge from submit-queue (batch tested with PRs 58216, 58193, 53033, 58219, 55921). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Use GinkgoRecover to avoid panic.

See this in the test:
```
I0111 14:28:31.010] panic: 
I0111 14:28:31.010] Your test failed.
I0111 14:28:31.010] Ginkgo panics to prevent subsequent assertions from running.
I0111 14:28:31.011] Normally Ginkgo rescues this panic so you shouldn't see it.
I0111 14:28:31.011] 
I0111 14:28:31.011] But, if you make an assertion in a goroutine, Ginkgo can't capture the panic.
I0111 14:28:31.011] To circumvent this, you should call
I0111 14:28:31.011] 
I0111 14:28:31.012] 	defer GinkgoRecover()
I0111 14:28:31.012] 
I0111 14:28:31.012] at the top of the goroutine that caused this panic.
I0111 14:28:31.012] 
I0111 14:28:31.012] 
I0111 14:28:31.012] goroutine 1028 [running]:
I0111 14:28:31.013] k8s.io/kubernetes/vendor/github.com/onsi/ginkgo.Fail(0xc421098000, 0xb0, 0xc420da24c8, 0x1, 0x1)
I0111 14:28:31.013] 	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/ginkgo_dsl.go:255 +0xda
I0111 14:28:31.014] k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc4220bd700, 0x9e897e0, 0xa123640, 0x0, 0x0, 0x0, 0x0, 0xa123640)
I0111 14:28:31.014] 	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:69 +0x1ef
I0111 14:28:31.014] k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc4220bd700, 0x9e897e0, 0xa123640, 0x0, 0x0, 0x0, 0xc4220bd700)
I0111 14:28:31.015] 	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:43 +0xae
I0111 14:28:31.015] k8s.io/kubernetes/test/e2e_node.deletePodsSync.func1(0xc421485220, 0xc421321680, 0xc421517180)
I0111 14:28:31.015] 	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/resource_collector.go:382 +0x320
I0111 14:28:31.015] created by k8s.io/kubernetes/test/e2e_node.deletePodsSync
I0111 14:28:31.016] 	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/resource_collector.go:375 +0x9e
```

e.g.: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-node-e2e-serial/17/build-log.txt

**Release note**:

```release-note
none
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

generate-bindata.sh: make output cleanly by suppressing pushd/popd output

**What this PR does / why we need it**:
`generate-bindata.sh` script produce extra output that is caused by `pushd`/`popd` usage. To make script output clean and to simplify debugging process, I suggest to suppress pushd/popd output.

Before this PR:
```console
$ make generated_files 
+++ [0112 15:46:26] Building the toolchain targets:
	k8s.io/kubernetes/hack/cmd/teststale
	k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0112 15:46:27] Generating bindata:
	test/e2e/generated/gobindata_util.go
~/git/src/home/kubernetes ~/git/src/home/kubernetes/test/e2e/generated
~/git/src/home/kubernetes/test/e2e/generated
+++ [0112 15:46:27] Building go targets for linux/amd64:
	./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0112 15:46:34] Building the toolchain targets:
	k8s.io/kubernetes/hack/cmd/teststale
	k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0112 15:46:34] Generating bindata:
	test/e2e/generated/gobindata_util.go
~/git/src/home/kubernetes ~/git/src/home/kubernetes/test/e2e/generated
~/git/src/home/kubernetes/test/e2e/generated
+++ [0112 15:46:34] Building go targets for linux/amd64:
	./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0112 15:46:38] Building the toolchain targets:
	k8s.io/kubernetes/hack/cmd/teststale
	k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0112 15:46:38] Generating bindata:
	test/e2e/generated/gobindata_util.go
~/git/src/home/kubernetes ~/git/src/home/kubernetes/test/e2e/generated
~/git/src/home/kubernetes/test/e2e/generated
+++ [0112 15:46:39] Building go targets for linux/amd64:
	./vendor/k8s.io/code-generator/cmd/conversion-gen
```

After this PR:
```console
$ make generated_files 
+++ [0112 16:28:31] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0112 16:28:32] Generating bindata:
    test/e2e/generated/gobindata_util.go
+++ [0112 16:28:32] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0112 16:28:39] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0112 16:28:39] Generating bindata:
    test/e2e/generated/gobindata_util.go
+++ [0112 16:28:39] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0112 16:28:43] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0112 16:28:43] Generating bindata:
    test/e2e/generated/gobindata_util.go
+++ [0112 16:28:44] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen

```

**Release note**:
```release-note
NONE
```

CC @simo5

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

wires the generic scaler to kubectl

**What this PR does / why we need it**:
the changes introduced in this commit plumbs in the generic scaler to kubectl.
note that we don't change the behaviour of kubectl.
For example it won't scale new resources. That's the end goal.
The first step is to retrofit existing code to use the generic scaler.

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

The lbaas.opts.SubnetId should be set by subnet id.

Fix #58145
The getSubnetIDForLB() should return subnet id rather than net id.

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Improved readability for messages being logged

**What this PR does / why we need it**:
This improves the readability for messages seen by end-user. /cc @jwforres @bsalamat - For UX
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #57152

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Allow kubectl drain to continue w ds-managed pods with local storage

**Release note**:
```release-note
NONE
```

Prevents oadm drain from failing if it encounters DaemonSet-managed pods
that have local storage, when the option to ignore DaemonSet-managed
pods has been specified.

Will add a test

cc @kubernetes/sig-cli-misc @deads2k @fabianofranz @dustymabe

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

remove providers which have been deleted

**What this PR does / why we need it**:

xref #49213

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```NONE

```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

refactor customeresource handler

**What this PR does / why we need it**:

- fix data race bug
- fix lock usage bug.
- remove some redundant code

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #57882

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```