Automatic merge from submit-queue (batch tested with PRs 54488, 54838, 54964). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add support to for alternative container runtime in `kube-up.sh`

For https://github.com/kubernetes/features/issues/286.

This PR added 4 new environment variables in `kube-up.sh` to support alternative container runtime:
1) `KUBE_MASTER_EXTRA_METADATA` and `KUBE_NODE_EXTRA_METADATA`. Add extra metadata on master and node instance. With this we could specify different cloud-init for a different container runtime, and also add extra metadata for the new cloud-init, e.g. [master.yaml](https://github.com/Random-Liu/cri-containerd/blob/7d739662141cc137f8b1e82a9824b18be2e5df21/test/e2e/master.yaml)
2) `KUBE_CONTAINER_RUNTIME_ENDPOINT`. Specify different sock for different container runtime. It's only used when it's not empty.
3) `KUBE_LOAD_IMAGE_COMMAND`. Specify different load image command for different container runtime.

An example for cri-containerd:
```
export KUBE_MASTER_EXTRA_METADATA="user-data=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/e2e/master.yaml,cri-containerd-configure-sh=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/configure.sh"
export KUBE_NODE_EXTRA_METADATA="user-data=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/e2e/node.yaml,cri-containerd-configure-sh=${GOPATH}/src/github.com/kubernetes-incubator/cri-containerd/test/configure.sh"
export KUBE_CONTAINER_RUNTIME="remote"
export KUBE_CONTAINER_RUNTIME_ENDPOINT="/var/run/cri-containerd.sock"
export KUBE_LOAD_IMAGE_COMMAND="/home/cri-containerd/usr/local/bin/cri-containerd load"
export NETWORK_POLICY_PROVIDER="calico"
```
Signed-off-by: Lantao Liu <lantaol@google.com>

```release-note
none
```
/cc @yujuhong @dchen1107 @feiskyer @mikebrow @abhi @mrunalp @runcom 
/cc @kubernetes/sig-node-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 54488, 54838, 54964). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix calico network policy for opensource.

For https://github.com/kubernetes/features/issues/286

This PR:
1) Add `NON_MASTER_NODE_LABELS` env, and only apply calico node label on non-master nodes.
2) Set ip masq rules in cloud init, so that we don't need the ip masq agent. @dchen1107 @dnardo As we discussed.
3) Let master use `${NETWORK_PROVIDER}` instead of fixed cni, because we won't run calico node agent on master. The master network should be configured separately (kubenet by default).

With this PR, I could bring up a cluster with `NETWORK_POLICY_PROVIDER=calico` on GCE now.
```console
$ cluster/kubectl.sh get pods --all-namespaces
NAMESPACE     NAME                                                  READY     STATUS    RESTARTS   AGE
kube-system   calico-node-9bxbv                                     2/2       Running   0          13m
kube-system   calico-node-kjxtw                                     2/2       Running   0          13m
kube-system   calico-node-vertical-autoscaler-67fb4f45bd-hcjmw      1/1       Running   0          16m
kube-system   calico-node-xs2s2                                     2/2       Running   0          13m
kube-system   calico-typha-7c4d876ddf-d4dtx                         1/1       Running   0          15m
kube-system   calico-typha-horizontal-autoscaler-5f477cdc66-qwwph   1/1       Running   0          16m
kube-system   calico-typha-vertical-autoscaler-58f7d686f7-pn72s     1/1       Running   0          16m
kube-system   etcd-empty-dir-cleanup-e2e-test-lantaol-master        1/1       Running   0          16m
kube-system   etcd-server-e2e-test-lantaol-master                   1/1       Running   0          16m
kube-system   etcd-server-events-e2e-test-lantaol-master            1/1       Running   0          16m
kube-system   event-exporter-v0.1.7-9d4dbb69c-m76v5                 2/2       Running   0          16m
kube-system   fluentd-gcp-v2.0.10-25dmf                             2/2       Running   0          16m
kube-system   fluentd-gcp-v2.0.10-kgxsk                             2/2       Running   0          16m
kube-system   fluentd-gcp-v2.0.10-p75xg                             2/2       Running   0          16m
kube-system   fluentd-gcp-v2.0.10-xzh77                             2/2       Running   0          16m
kube-system   heapster-v1.5.0-beta.0-5cf4d9dff7-dmvm7               4/4       Running   0          13m
kube-system   kube-addon-manager-e2e-test-lantaol-master            1/1       Running   0          15m
kube-system   kube-apiserver-e2e-test-lantaol-master                1/1       Running   0          16m
kube-system   kube-controller-manager-e2e-test-lantaol-master       1/1       Running   0          16m
kube-system   kube-dns-79bdcb6c9f-2bpc8                             3/3       Running   0          15m
kube-system   kube-dns-79bdcb6c9f-gr686                             3/3       Running   0          16m
kube-system   kube-dns-autoscaler-996dcfc9d-pfs4s                   1/1       Running   0          16m
kube-system   kube-proxy-e2e-test-lantaol-minion-group-3khw         1/1       Running   0          16m
kube-system   kube-proxy-e2e-test-lantaol-minion-group-6878         1/1       Running   0          16m
kube-system   kube-proxy-e2e-test-lantaol-minion-group-j9rq         1/1       Running   0          16m
kube-system   kube-scheduler-e2e-test-lantaol-master                1/1       Running   0          16m
kube-system   kubernetes-dashboard-765c6f47bd-lsw5r                 1/1       Running   0          16m
kube-system   l7-default-backend-6d477bf555-x54zf                   1/1       Running   0          16m
kube-system   l7-lb-controller-v0.9.7-e2e-test-lantaol-master       1/1       Running   0          16m
kube-system   metrics-server-v0.2.0-9c4f8c48d-gkl79                 2/2       Running   0          13m
kube-system   monitoring-influxdb-grafana-v4-54df94856c-krkvb       2/2       Running   0          16m
kube-system   rescheduler-v0.3.1-e2e-test-lantaol-master            1/1       Running   0          16m
```

**Note that with this PR, master node will be using kubenet by default. And network policy will not apply on master node.**

**We need this to unblock `cri-containerd` integration with `kube-up.sh`.**
/cc @dchen1107 @dnardo Please take a look.
@kubernetes/sig-network-misc @kubernetes/sig-cluster-lifecycle-misc 
Signed-off-by: Lantao Liu <lantaol@google.com>

```release-note
None
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add admission handler for device resources allocation

**What this PR does / why we need it**:
Add admission handler for device resources allocation to fail fast during pod creation

**Which issue this PR fixes** 
fixes #51592

**Special notes for your reviewer**:
@jiayingz Sorry, there is something wrong with my branch in #51895. And I think the existing comments in the PR might be too long for others to view. So I closed it and opened the new one, as we have basically reach an agreement on the implement :)
I have covered the functionality and unit test part here, and would set about the e2e part ASAP

/cc @jiayingz @vishh @RenaudWasTaken 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

discovery client not depend on pkg/api/legacyscheme

Switch kubectl's cached discovery client to use kubect's scheme (pkg/kubectl/scheme) which only registered with external version types.

The encoding and decoding here uses only `k8s.io/apimachinery/pkg/apis/meta/v1.APIGroupList` and `k8s.io/apimachinery/pkg/apis/meta/v1.APIResourceList` which are not internal version. So it should be safe.

```release-note
NONE
```

/assign @monopole

Automatic merge from submit-queue (batch tested with PRs 55016, 50887). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

If command.Execute() return err, print to stdErr

The current kube-proxy not print error if (execute() failed) . not so good and not so friendly to user.
If print err to stdError, will show us why it failed.

Automatic merge from submit-queue (batch tested with PRs 55016, 50887). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Cleanup resouce test

- Make client-go testing support decoding to external version
- Cleanup bad dependency of `pkg/kubectl/resource` pkg

```release-note
NONE
```

/assign @monopole @caesarxuchao

Signed-off-by: Lantao Liu <lantaol@google.com>

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add a file store utility package in kubelet

More and more components checkpoints (i.e., persist their states) in
kubelet. Refurbish and move the implementation in dockershim to a
utility package to improve code reusability.

Automatic merge from submit-queue (batch tested with PRs 52367, 53363, 54989, 54872, 54643). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Lift embedded structure out of ManifestURLHeader field

Related: #53833

```release-note
It is now possible to set multiple manifest url headers via the Kubelet's --manifest-url-header flag. Multiple headers for the same key will be added in the order provided. The ManifestURLHeader field in KubeletConfiguration object (kubeletconfig/v1alpha1) is now a map[string][]string, which facilitates writing JSON and YAML files.
```

Automatic merge from submit-queue (batch tested with PRs 52367, 53363, 54989, 54872, 54643). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Support retainkeys strategy for new merge code

- Some prefactoring for retainkeys
- Add retainkeys strategy


```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 52367, 53363, 54989, 54872, 54643). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

should check and return err when visit failure

**What this PR does / why we need it**:

should check and return err when visit failure

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 52367, 53363, 54989, 54872, 54643). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

cmd: genutils: remove entry from .golint_failures

**What this PR does / why we need it**:

`golint` emits warnings

```
exported function OutDir should have comment or be unexported
strings should not be capitalized or end with punctuation or a newline
```

- Add documentation comment to exported function OutDir.
- Remove newline from error string.
- Remove `hack\.golint_failures` entry for `cmd/genutils`

**Special notes for your reviewer**:
   
Don't know which sig to use?

**Release note**:

```release-note
NONE
```

/kind cleanup

Automatic merge from submit-queue (batch tested with PRs 52367, 53363, 54989, 54872, 54643). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Basic GCE PodSecurityPolicy Config

**What this PR does / why we need it**:

This PR lays the foundation for enabling PodSecurityPolicy in GCE and other default deployments. The 3 commits are:

1. Add policies, roles & bindings for the default addons on GCE.
2. Enable the PSP admission controller & load the addon policies when the`ENABLE_POD_SECURITY_POLICY=true` environment variable is set.
3. Support the PodSecurityPolicy in the E2E environment & add PSP tests.

NOTES:

- ~~Depends on https://github.com/kubernetes/kubernetes/pull/52301 for privileged capabilities~~
- ~~Depends on https://github.com/kubernetes/kubernetes/pull/52849 for sane mutations~~
- ~~Depends on https://github.com/kubernetes/kubernetes/pull/53479 for aggregator tests to pass~~
- ~~Depends on https://github.com/kubernetes/kubernetes/pull/54175 for dedicated fluentd service~~ account
- This PR is a fork of https://github.com/kubernetes/kubernetes/pull/46064, credit to @Q-Lee

**Which issue this PR fixes**: #43538

**Release note**:
```release-note
Add support for PodSecurityPolicy on GCE: `ENABLE_POD_SECURITY_POLICY=true` enables the admission controller, and installs policies for default addons.
```

Automatic merge from submit-queue (batch tested with PRs 55004, 54957). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Update kube-openapi to use validation

**What this PR does / why we need it**: Moves openapi validation code to kube-openapi, so that we can move the rest of the code to apimachinery repository, so that later we can use it from both the client and the server.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #Nothing

**Special notes for your reviewer**:

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 55004, 54957). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Explicitly set route_localnet on nodes & masters.

Allow for loopback addresses to be used for routing, specifically to
enable metadata proxy on master nodes.



**What this PR does / why we need it**: Enables metadata proxy (#8867) to work on the master.  This is already being done on the nodes by kube-proxy, but this makes it explicit, and sets it on the master where kube-proxy doesn't run.  Thanks to @dnardo for figuring this out!

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: fixes #54736
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Allow for loopback addresses to be used for routing, specifically to
enable metadata proxy on master nodes.

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

admission: wire through validating-only admission

Based on https://github.com/kubernetes/kubernetes/pull/54232.

This is important and required for beta because it affects the shape of the webhook admission plugins that are going to be produced and is needed to make sure that our existing chain continues to properly verify and protect the API objects based on their final state after webhook admission mutators run.

We discussed this in the October 11 API machinery call with @erictune and @caesarxuchao and we agreed to do this as a requirement for beta. See this part of the recording: https://www.youtube.com/watch?v=mrgDPHyr4VY#t=325 .

Automatic merge from submit-queue (batch tested with PRs 54800, 53898, 54812, 54921, 53558). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix error for strategic merge patch of custom resources

Fixes #50037.

We need the go struct tags `patchMergeKey` and `patchStrategy` for fields that support a strategic merge patch. For native resources, we can easily figure out these tags since we know the fields.

Because custom resources are decoded as Unstructured and because we're missing the metadata about how to handle each field in a strategic merge patch, we can't find the go struct tags. Hence, we can't easily  do a strategic merge for custom resources.

So we should fail fast and return an error.

**Release note**:

```release-note
NONE
```

/cc @sttts @deads2k @ncdc

Automatic merge from submit-queue (batch tested with PRs 54800, 53898, 54812, 54921, 53558). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

rename metric reflector_xx_last_resource_version

**What this PR does / why we need it**:

mv reflector name from metric name to metric label

before:

```
reflector_k8s_io_kubernetes_pkg_client_informers_informers_generated_internalversion_factory_go:73_8664_last_resource_version{instance="104.154.20.21:443",job="kubernetes-apiservers"}
```

after

```
reflector_last_resource_version{instance="10.0.2.15:6443",job="kubernetes-apiservers",name="k8s_io_kubernetes_pkg_client_informers_informers_generated_internalversion_factory_go_73_8664"}
```

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #52121 

**Special notes for your reviewer**:

None

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 54800, 53898, 54812, 54921, 53558). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Allow override of cluster level (default, whitelist) tolerations by namespace level empty (default, whitelist) tolerations.

Currently In PodTolerationRestriction admission plugin, if namespace level default and whitelist of tolerations are nil or empty, they do not override cluster level default and whitelist tolerations. 

This PR fixes the plugin to not override cluster level tolerations only when namespace level toleration are nil. IOW, if namespace level toleration are empty, they override cluster level tolerations. To be more clear, if following annotations are set to empty, they override cluster level tolerations.
 
``` 
scheduler.alpha.kubernetes.io/defaultTolerations : ""
scheduler.alpha.kubernetes.io/tolerationsWhitelist: ""
```

This behavior is inline with PodNodeSelector admission plugin too.

@sjenning @derekwaynecarr 

**Release Note**:

```release-note
In PodTolerationRestriction admisson plugin, if namespace level tolerations are empty, now they override cluster level tolerations. 
```

Automatic merge from submit-queue (batch tested with PRs 54800, 53898, 54812, 54921, 53558). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

NewProxierWithNoProxyCIDR: fix handling IPv6 URLs

**What this PR does / why we need it**:
Current logic of splitting hostname from URL does not work if URL
is for IPv6 address and does not explicitly specify port number.
Example: "https://[2001:db8::1]/".

Use standard library function to get hostname out of URL string.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: 
**Special notes for your reviewer**:

**Release note**:
```release-note
- Fix handling of IPv6 URLs in NO_PROXY.
```

/area ipv6
/sig api-machinery

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

kubenet: yield lock while executing CNI plugin.

The CNI plugin can take up to 3 seconds to execute. CNI plugins can safely be
executed in parallel, so yield the lock to speed up pod creation.

This caused problems with the pod latency tests - previously, CNI plugins executed
in under 20ms. Now they must wait for DAD to finish and addresses to leave
tentative state.

Fixes: #54651

**What this PR does / why we need it**:
After upgrading CNI plugins to v0.6 in #51250, the pod latency tests began failing. This is because the plugins, in order to support IPv6, need to wait for DAD to finish. Because this
delay is while the kubenet lock is held, it significantly slows down the pod creation rate.

**Special notes for your reviewer**:
The CNI plugins also do locking for their critical paths, so it is safe to run them concurrently.

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Improve deleteOptions doc

**What this PR does / why we need it**:
This PR improves the doc of the 'PropagationPolicy' property of the DeleteOptions struct.
The said field is a string. It is very important a field for people who invoke a delete API.
For example, direct invocation of the deployment-delete API using the python client will have the replicaset and pods left behind. However, this field was not well documented. When people wanted to try different options, they have to dig into k8s source code to find out the acceptable values.

**Which issue this PR fixes**: fixes #

**Special notes for your reviewer**:

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 54787, 51940). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Useful helper functions for Unstructured

**Which issue this PR fixes**:
Fixes #40790

**Release note**:
```release-note
NONE
```
/kind feature
/sig api-machinery
/area client-libraries
/assign @sttts @liggitt

Automatic merge from submit-queue (batch tested with PRs 54787, 51940). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Migrate netwrok partition test to sig apps

**What this PR does / why we need it**:

Migrate network partition relevant e2e test to sig-app. 

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Ref Umbrella issue #49161

**Special notes for your reviewer**:

**Release note**:
```release-note
none
```

Automatic merge from submit-queue (batch tested with PRs 54895, 54449). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Update the get command to follow more conventions of commands

Pure code movement, builds on top of #54446 and only the last commit is new. Will make refactoring get easier.

Automatic merge from submit-queue (batch tested with PRs 54895, 54449). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix a bug checking DaemonSet pods are updated in e2e test

**What this PR does / why we need it**:

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #50586

**Special notes for your reviewer**: @kubernetes/sig-apps-bugs 

**Release note**:

```release-note
NONE
```