Automatic merge from submit-queue

Rename test dir to allude sig-apps ownership

@kubernetes/sig-apps-pr-reviews other sigs have already followed this pattern and it seems that this is what @kubernetes/sig-testing-misc wants too

Ref https://github.com/kubernetes/kubernetes/issues/49161

ptal

Signed-off-by: Michail Kargakis <mkargaki@redhat.com>

Automatic merge from submit-queue (batch tested with PRs 49409, 49352, 49266, 48418)

Use helper to init ClusterIP and NodePort in Create of service

**What this PR does / why we need it**:
Make service `Create` more readable and testable.

- use `initClusterIP` introduced in #46197 to init ClusterIP allocation in service `Create`
- add a new helper `initNodePort` to init NodePort allocation in service `Create`
- TBD: add test case for `initNodePort`. This will cover the NodePort allocation process in `Create`. If this PR makes sense, I will write a test case later.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #35354 (not directly. #35354 was fixed by #46197. The idea of this PR is from https://github.com/kubernetes/kubernetes/pull/46197#discussion_r120910077)

**Special notes for your reviewer**:
/cc @thockin @freehan 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 49409, 49352, 49266, 48418)

[e2e] Also verify content returned by kube-proxy healthz url

**What this PR does / why we need it**: Enhance kube-proxy url test. This helps to detect the port collision case --- node-problem-detector also serves /healthz to return 200 ok. Verify the content to confirm /healthz is served by kube-proxy.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: From #49263

**Special notes for your reviewer**:
/assign @freehan @nicksardo 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 49409, 49352, 49266, 48418)

Use the yet uninitialized informer factory

`informerFactory` has been started already but the `factory.equivalencePodCache` field has not been initialized yet (done below by `.Create()`). So when Service informer calls `onServiceAdd()`, it tries to access this field and crashes with npe.

The fix is to use `informerFactory2`.
It is a race between listeners called by informer and `.Create()`. Do we run unit/integration tests with race detector switched off?

```
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14c2c2f]

goroutine 11688 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x28f5f00, 0x61621c0)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
k8s.io/kubernetes/plugin/pkg/scheduler/core.(*EquivalenceCache).InvalidateCachedPredicateItemOfAllNodes(0x0, 0xc4202a9020)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/core/equivalence_cache.go:141 +0x5f
k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).onServiceAdd(0xc424ad5950, 0x2cf6fc0, 0xc4249c05a0)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:347 +0x53
k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).(k8s.io/kubernetes/plugin/pkg/scheduler/factory.onServiceAdd)-fm(0x2cf6fc0, 0xc4249c05a0)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:247 +0x3e
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(0xc423a47310, 0xc423a47320, 0xc423a47330, 0x2cf6fc0, 0xc4249c05a0)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd(0xc423938ce0, 0x2cf6fc0, 0xc4249c05a0)
	<autogenerated>:54 +0x73
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc424a698b0)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:544 +0x287
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*processorListener).(k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.run)-fm()
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:380 +0x2a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc424a87868, 0xc423a47370)
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f
created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x62
```

**Special notes for your reviewer**:
Found while working on #47045.

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 49409, 49352, 49266, 48418)

Fix duplication in scheduler.

**What this PR does / why we need it**:
This fixes duplicate code in scheduler.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
No issue.
**Special notes for your reviewer**:

**Release note**:

```release-note
```
@kubernetes/sig-scheduling-pr-reviews @bsalamat

Automatic merge from submit-queue (batch tested with PRs 49326, 49394, 49346, 49379, 49399)

Update to version gate CRDs to 1.7 and greater

**What this PR does / why we need it**:
Allows e2e's to be tested on earlier version do to version check.  

xref: #49313

**Release note**:
```
NONE
```

/cc @kubernetes/sig-api-machinery-bugs @kubernetes/sig-testing-bugs

Automatic merge from submit-queue (batch tested with PRs 49326, 49394, 49346, 49379, 49399)

Pass clientset's Interface to CreateScheduler.

**What this PR does / why we need it**:
CreateSchedueler takes clientset.Clientset param and passes it to NewConfigFactory which takes clientset.Interface. It'd be better for CreateSchedueler to take clientset.Interface to be in sync with NewConfigFactory and it would be easier to reuse CreateSchedueler in implementations which relies on interface rather than actual client implementation.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
No issue.
@kubernetes/sig-scheduling-pr-reviews @bsalamat

Automatic merge from submit-queue (batch tested with PRs 49326, 49394, 49346, 49379, 49399)

Make client-go authoritative

client-go now is authoritative except for `client-go/pkg/version`, which I think we can leave as is for a while.

Automatic merge from submit-queue (batch tested with PRs 49326, 49394, 49346, 49379, 49399)

more robust stat handling from ceph df output in the kubernetes-master charm create-rbd-pv action

**What this PR does / why we need it**: more robust stat handling from ceph df output in the kubernetes-master charm create-rbd-pv action

**Release note**:
```release-note
more robust stat handling from ceph df output in the kubernetes-master charm create-rbd-pv action
```

Automatic merge from submit-queue

add cronjobs to all

Categories were added to the discovery API, but the `kubectl` plumbing didn't make it.  We *did* make `kubectl all` gate on discovery information, so it can least be a superset.  

`cronjobs` are user resources, so I've added them to the list.

@kubernetes/sig-cli-misc 

```release-note
added cronjobs.batch to all, so kubectl get all returns them.
```

Automatic merge from submit-queue (batch tested with PRs 49420, 49296, 49299, 49371, 46514)

Refactoring taint functions to reduce sprawl

**What this PR does / why we need it**:

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #45060

**Special notes for your reviewer**:
@gmarek @timothysc @k82cn @jayunit100 - I moved some fn's to helpers and some to utils. LMK, if you are ok with this change.

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 49420, 49296, 49299, 49371, 46514)

Fix: PV metric is not namespaced

**What this PR does / why we need it**: The PV metric of juju deployments is not namespaced. This PR fixes this bug. 

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/348

**Special notes for your reviewer**:

**Release note**:

```NONE
```

Automatic merge from submit-queue (batch tested with PRs 49420, 49296, 49299, 49371, 46514)

Avoid looking up instance id until we need it

**What this PR does / why we need it**:

currently kube-controller-manager cannot run outside of a vm started
by openstack (with --cloud-provider=openstack params). We try to read
the instance id from the metadata provider or the config drive or the
file location only when we really need it. In the normal scenario, the
controller-manager uses the node name to get the instance id.
https://github.com/kubernetes/kubernetes/blob/41541910e1699975a8f9202a89b6865e45921194/pkg/volume/cinder/attacher.go#L149

The localInstanceID is currently used only in the test case, so let
us not read it until it is really needed.

So let's try to find the instance-id only when we need it.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 49420, 49296, 49299, 49371, 46514)

Update status to show failing services.

**What this PR does / why we need it**: Report on charm status any services that are not running.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/341

**Special notes for your reviewer**:

**Release note**:

```Report failing services in Juju deployed clusters.
```

Automatic merge from submit-queue (batch tested with PRs 49420, 49296, 49299, 49371, 46514)

Fix controller crash because of nil volume spec

For volumes that don't support bulk volume verification, a nil
volume spec can cause crash of controller.

Fixes #49418

**Release note**:
```release-note
Fixes #49418 where kube-controller-manager can panic on volume.CanSupport methods and enter a crash loop.
```

Automatic merge from submit-queue (batch tested with PRs 46210, 48607, 46874, 46598, 49240)

Remove types.generated.go generated for internal API types

We don't need generated codecs for internal types.

Automatic merge from submit-queue (batch tested with PRs 46210, 48607, 46874, 46598, 49240)

Make "kubectl version" json format output more readable.

**What this PR does / why we need it**:
##39858 adds a flag --output to `kubectl version`, but the json format output is displayed in one line. It's not so readable. This PR fixes it.

and

- adds a shorthand for `output`
- ~~refactors that: if `--short` is specified, `--output` will be ignored~~

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #43750

**Special notes for your reviewer**:
/cc @php-coder @alejandroEsc 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 46210, 48607, 46874, 46598, 49240)

Ipv6 support for ChooseHostInterface (part 3 of 3)

**What this PR does / why we need it**:
Provides IPv6 support for the ChooseHostInterface() and ChooseBindAddress() functions
in the apimachinery's util/net/interface.go file. It strives to maintain backward compatibility with existing use of IPv4, but allow IPv6 to be used when available.  This is part 3 of a set of PRs, and has been rebased on top of #46138, which is rebased on top of #46044.

This is part of an overall effort to provide IPv6 support in the code, by addressing places where IPv4 logic exists and making it IPv6 ready.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #44848

**Special notes for your reviewer**:
You can look at the other PRs for the commit description of what each of them does. I do see some unrelated test failures in those commits - appears to be flakes.

**Release note**:

```release-noteNONE
```

Automatic merge from submit-queue (batch tested with PRs 46210, 48607, 46874, 46598, 49240)

kubeadm: Make the hostPath volume mount code more secure

**What this PR does / why we need it**:

 - Refactors the hostpath volume mounting code for the Static Pods
 - Splits out the functionality that was in a big function to something testable
 - Unit test a lot
 - Adds support for mounting external etcd CA/cert/key files in an other path than `/etc/ssl/certs`. Before this you **had** to have your files in there or the apiserver would crashloop
 - Significantly improves comment coverage
 - Now only mounts the bare essentials instead of nearly everything. For example, don't mount full `/etc/kubernetes` when the only thing you need is `/etc/kubernetes/scheduler.conf`
 - Make everything but the etcd datadir read-only for components.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

Fixes: https://github.com/kubernetes/kubeadm/issues/341

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
cc @kubernetes/sig-cluster-lifecycle-pr-reviews

Automatic merge from submit-queue

Add AzureFile,FC,Flex,Flocker volume source to describe printer.

**What this PR does / why we need it**:
Display other volume sources in describe printer.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:
Also need CephFS. It will be added in #46124 

/cc @rootfs 

_ADD: all the volume source printers are sorted out of order. I'd like to sort them alphabetically in order to be more readable in another PR after this PR merged. WDYT?_ 

**Release note**:

```release-note
NONE
```

For volumes that don't support bulk volume verifiction, a nil
volume spec can cause crash of controller.

Automatic merge from submit-queue

fix race in proxy unit test

fixes #49413, follow up to https://github.com/kubernetes/kubernetes/pull/49353 to fix a race in the unit test:


```
go test -v k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver -run TestProxyUpgrade$
handler_proxy_test.go:328: invalid hostname + insecure: expected called=true, got false
testing.go:610: race detected during execution of test
```
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48746/pull-kubernetes-unit/42230/

Automatic merge from submit-queue

Add a new API version apps/v1beta2

xref: #49135
This PR adds a new API version `apps/v1beta2` which contains a copy (of types, conversions, and defaults) of `apps/v1beta1` StatefulSet, Deployment, and their subresources. Note that `apps/v1beta2` is still WIP and we will make breaking changes to it before releasing 1.8.

Moving core controllers (StatefulSet, Deployment, ReplicaSet, DaemonSet) to `apps/v1beta2` is the first step of moving them to `apps/v1` (GA). 

This PR is a starting point for DaemonSet and ReplicaSet to move from `/extensions` to `/apps` and for Deployment and StatefulSet to make some breaking changes (e.g. new defaults and/or remove deprecated fields).

```release-note
Add a new API version apps/v1beta2
```

Automatic merge from submit-queue

Use specified ServerName in aggregator TLS validation

Fixes #49354

The aggregator sets a ServerName in the proxier tlsConfig, but the code path handling websocket upgrade requests did not honor it, and instead tried to verify TLS using the dialed host

* Honors ServerName if already set in tls.Config
* Adds unit tests for upgrade functionality via the aggregator
* Fixes mutation of shared tlsConfig.ServerName in spdy roundtripper

```release-note
Websocket requests to aggregated APIs now perform TLS verification using the service DNS name instead of the backend server's IP address, consistent with non-websocket requests.
```

Automatic merge from submit-queue

Reduce hollow proxy mem/node

As likely expected, kubemark-scale failed to even start with n1-standard-8 nodes. Because 1/3rd of our hollow nodes didn't even get scheduled due to their requests:

```
I0720 17:45:08.139] Found only 3325 ready hollow-nodes while waiting for 5000.
I0720 17:45:20.435] 3326 hollow-nodes are reported as 'Running'
I0720 17:45:20.442] 1675 hollow-nodes are reported as NOT 'Running'
```

If we want to experiment with smaller nodes anyway, then this change is needed. Though we most likely will end up OOM'ing.

Explanation for new value:
We have 62.5 hollow-node / real-node
=> mem available per hollow node = 30GB / 62.5 = 480MB
minus 100MB (kubelet)
minus 20MB (npd) 
=> 360MB for proxy should be = 100MB + 5000*(mem/node)
=> 50KB mem/node (with some slight slack)

cc @kubernetes/sig-scalability-misc

Automatic merge from submit-queue

Add an integration test library and some integration tests for scheduler

**What this PR does / why we need it**:

1. Add an integration test library (utils.go) for scheduler testing.
2. Cleaned up some of the tests in scheduler_test.go with the new integration test library.
3. Add priority_test.go with a couple of examples on how to test scheduler priority function in integration tests.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

ref/ #48176

@kubernetes/sig-scheduling-pr-reviews 
@davidopp @k82cn @vikaschoudhary16