Automatic merge from submit-queue

Add a test that emulates large N node behavior against master

Automatic merge from submit-queue

Move .gitattributes annotation to the root, so GitHub will respect them.

This should fix the merge conflicts by letting GitHub use the simpler line-by-line algorithm for this file. Having .gitattributes in a sub-directory would work for local merging, but would show conflicts on the web UI.

Automatic merge from submit-queue

Expose etcd version.

Ref #20504

This PRs makes use of the following two env variables
- ETCD_IMAGE (if set) allows to use a custom etcd image.
- ETCD_VERSION (if set) allows you to use custom version of etcd. The main purpose of using it may be rollback of etcd v3 API, where we need 3.0.* image, but are rolling back to 2.3.7.

Automatic merge from submit-queue

bazel: fix and start verify ./vendor/BUILD

-dry-run was not verifying vendor because the walkVendor path didn't print out a "wrote BUILD in ./vendor" message.

This is fixed by https://github.com/mikedanese/gazel/pull/2

Automatic merge from submit-queue

Avoid double decoding all client responses

Fixes #35982 

The linked issue uncovered that we were always double decoding the response in restclient for get, list, update, create, and patch.  That's fairly expensive, most especially for list.  This PR refines the behavior of the rest client to avoid double decoding, and does so while minimizing the changes to rest client consumers.

restclient must be able to deal with multiple types of servers. Alter the behavior of restclient.Result#Raw() to not process the body on error, but instead to return the generic error (which still matches the error checking cases in api/error like IsBadRequest). If the caller uses
.Error(), .Into(), or .Get(), try decoding the body as a Status.

For older servers, continue to default apiVersion "v1" when calling restclient.Result#Error(). This was only for 1.1 servers and the extensions group, which we have since fixed.

This removes a double decode of very large objects (like LIST) - we were trying to DecodeInto status, but that ends up decoding the entire result and then throwing it away.  This makes the decode behavior specific to the type of action the user wants.

```release-note
The error handling behavior of `pkg/client/restclient.Result` has changed.  Calls to `Result.Raw()` will no longer parse the body, although they will still return errors that react to `pkg/api/errors.Is*()` as in previous releases.  Callers of `Get()` and `Into()` will continue to receive errors that are parsed from the body if the kind and apiVersion of the body match the `Status` object.

This more closely aligns rest client as a generic RESTful client, while preserving the special Kube API extended error handling for the `Get` and `Into` methods (which most Kube clients use).
```

Automatic merge from submit-queue

[Federation][join-01] Implement `kubefed join` command.

Supersedes PR #35155.

Please review only the last commit here. This is based on PR #35492 which will be reviewed independently.

I will add a release note separately for this entire feature, so please don't worry too much about the release note here in the PR.

Design Doc: PR #34484

cc @kubernetes/sig-cluster-federation @quinton-hoole @mwielgus

Automatic merge from submit-queue

Making the pod.alpha.kubernetes.io/initialized annotation optional in PetSet pods

**What this PR does / why we need it**: As of now, the absence of the annotation `pod.alpha.kubernetes.io/initialized` in PetSets causes the PetSet controller to effectively "pause". Being a debug hook, users expect that its absence has no effect on the working of a PetSet. This PR inverts the logic so that we let the PetSet controller operate as expected in the absence of the annotation.
Letting the annotation remain alpha seems ok. Renaming it to something more meaningful needs further discussion.

**Which issue this PR fixes** _(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)_: fixes https://github.com/kubernetes/kubernetes/issues/35498

**Special notes for your reviewer**: 

**Release note**:

``` release-note
The annotation "pod.alpha.kubernetes.io/initialized" on StatefulSets (formerly PetSets) is now optional and only encouraged for debug use.
```

cc @erictune @smarterclayton @bprashanth @kubernetes/sig-apps 
@kow3ns The examples will need to be cleaned up as well I think later on to remove them.

Automatic merge from submit-queue

Scheduler Reduce functions can use PriorityMetadata

Automatic merge from submit-queue

Extend etcd migration logs

Ref #20504

Automatic merge from submit-queue

Fix build break on non-Linux OS introduced in 87aaf4c0

**What this PR does / why we need it**: simple fix for build breakage on non-Linux OS.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: 
**Special notes for your reviewer**:

```console
+++ [1102 11:16:58] linux/amd64: go build started
+++ [1102 11:20:11] linux/amd64: go build finished
+++ [1102 11:16:58] darwin/amd64: go build started
# k8s.io/kubernetes/pkg/kubelet/dockershim/cm
pkg/kubelet/dockershim/cm/container_manager_unsupported.go:33: undefined: fmt in fmt.Errorf
+++ [1102 11:16:58] windows/amd64: go build started
# k8s.io/kubernetes/pkg/kubelet/dockershim/cm
pkg/kubelet/dockershim/cm/container_manager_unsupported.go:33: undefined: fmt in fmt.Errorf
Makefile:79: recipe for target 'all' failed
make[1]: *** [all] Error 1
Makefile:255: recipe for target 'cross' failed
make: *** [cross] Error 1
Makefile:239: recipe for target 'release' failed
make: *** [release] Error 1
```

**Release note**:
```release-note
NONE
```

Revert "Make master components in kubemark monitored by supervisord"

Automatic merge from submit-queue

split scheduler priorities into separate files

In the current state it's really hard to find a thing one is looking for, if he doesn't know already know where to look. cc @davidopp

Automatic merge from submit-queue

Create multiple clients in Density test and remove unused 'interval'

Fix #36039

Automatic merge from submit-queue

Hide groups with new versions from old kubectl

Fix https://github.com/kubernetes/kubernetes/issues/35791

**What caused the bug?**

In 1.5, we are going to graduate Policy and Apps to beta. Old version kubectl doesn't has the new versions built-in, its TRP dynamic discover thinks Policy/v1beta1 is a TPR, and tried to register it in kubectl's scheme. The registration failed because Policy group already exist, because kubectl had registered Policy.v1alpha1.

**How does this PR fix the bug?**

This PR let the API server hides Policy and Apps from old version kubectl, so TPR discovery won't see them.

Old version kubectl doesn't know about Policy/v1beta1 or Apps/v1beta1, and v1alpha1 will be removed, so old version kubectl won't work for Policy or Apps anyway, so this PR does not cause any function loss.

@kubernetes/sig-api-machinery @liggitt @smarterclayton @deads2k @janetkuo @mwielgus

Automatic merge from submit-queue

Make master components in kubemark monitored by supervisord

Automatic merge from submit-queue

kubeadm: move defaulting to the api group

ref #35796

@pires @kubernetes/sig-cluster-lifecycle

Automatic merge from submit-queue

Support events in restore script

Ref #20504

Automatic merge from submit-queue

Log portal IP in error info.

Log portal IP in error info. This will help when debugging.

Automatic merge from submit-queue

Align with other cli descriptions

**What this PR does / why we need it**:
Align with other cli descriptions

**Special notes for your reviewer**:
![image](https://cloud.githubusercontent.com/assets/20062886/19887803/adbeee16-a065-11e6-9933-98bfff784086.png)
Signed-off-by: yupeng <yu.peng36@zte.com.cn>

Automatic merge from submit-queue

Fix typo: attachement -> attachment

Fix panics in density test.

Automatic merge from submit-queue

dockershim: move docker to the given cgroup

Automatic merge from submit-queue

Only set sysctls in infra container in dockershim

Dockershim counterpart of https://github.com/kubernetes/kubernetes/pull/32383.

Fix typos

Automatic merge from submit-queue

Update verify-openapi-spec script to check for extra generated spec

hack/verify-openapi-spec.sh only check for existing spec changes. If for some reason (here most probably I forgot to delete a file in api/openapi-spec folder in #35388 after a rebase) there is an old spec exists in the spec folder, it won't panic but it should. This resulted in an unused out of date v1.spec file in the api/openapi-spec folder that this PR also removes.

Automatic merge from submit-queue

CRI: Handle empty container name in dockershim.

Fixes https://github.com/kubernetes/kubernetes/issues/35924.

Dead container may have no name, we should handle this properly.

@yujuhong @bprashanth

Automatic merge from submit-queue

make ./pkg/client/listers compile

currently compilation is broken

```
$ go install ./pkg/client/listers/...
# k8s.io/kubernetes/pkg/client/listers/apps/v1alpha1
pkg/client/listers/apps/v1alpha1/zz_generated.statefulset.go:89: undefined: apps in apps.Resource
# k8s.io/kubernetes/pkg/client/listers/autoscaling/v1
pkg/client/listers/autoscaling/v1/zz_generated.horizontalpodautoscaler.go:89: undefined: autoscaling in autoscaling.Resource
# k8s.io/kubernetes/pkg/client/listers/batch/v2alpha1
pkg/client/listers/batch/v2alpha1/zz_generated.job.go:89: undefined: batch in batch.Resource
pkg/client/listers/batch/v2alpha1/zz_generated.scheduledjob.go:89: undefined: batch in batch.Resource
# k8s.io/kubernetes/pkg/client/listers/authentication/v1beta1
pkg/client/listers/authentication/v1beta1/zz_generated.tokenreview.go:63: undefined: authentication in authentication.Resource
# k8s.io/kubernetes/pkg/client/listers/batch/v1
pkg/client/listers/batch/v1/zz_generated.job.go:89: undefined: batch in batch.Resource
# k8s.io/kubernetes/pkg/client/listers/authorization/v1beta1
pkg/client/listers/authorization/v1beta1/zz_generated.localsubjectaccessreview.go:89: undefined: authorization in authorization.Resource
pkg/client/listers/authorization/v1beta1/zz_generated.selfsubjectaccessreview.go:63: undefined: authorization in authorization.Resource
pkg/client/listers/authorization/v1beta1/zz_generated.subjectaccessreview.go:63: undefined: authorization in authorization.Resource
# k8s.io/kubernetes/pkg/client/listers/certificates/v1alpha1
pkg/client/listers/certificates/v1alpha1/zz_generated.certificatesigningrequest.go:63: undefined: certificates in certificates.Resource
# k8s.io/kubernetes/pkg/client/listers/policy/v1alpha1
pkg/client/listers/policy/v1alpha1/zz_generated.poddisruptionbudget.go:89: undefined: policy in policy.Resource
# k8s.io/kubernetes/pkg/client/listers/core/v1
pkg/client/listers/core/v1/zz_generated.componentstatus.go:62: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.configmap.go:89: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.endpoints.go:89: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.event.go:89: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.limitrange.go:89: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.namespace.go:62: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.node.go:62: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.persistentvolume.go:62: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.persistentvolumeclaim.go:89: undefined: api in api.Resource
pkg/client/listers/core/v1/zz_generated.pod.go:89: undefined: api
pkg/client/listers/core/v1/zz_generated.pod.go:89: too many errors
# k8s.io/kubernetes/pkg/client/listers/imagepolicy/v1alpha1
pkg/client/listers/imagepolicy/v1alpha1/zz_generated.imagereview.go:63: undefined: imagepolicy in imagepolicy.Resource
# k8s.io/kubernetes/pkg/client/listers/rbac/v1alpha1
pkg/client/listers/rbac/v1alpha1/zz_generated.clusterrole.go:63: undefined: rbac in rbac.Resource
pkg/client/listers/rbac/v1alpha1/zz_generated.clusterrolebinding.go:63: undefined: rbac in rbac.Resource
pkg/client/listers/rbac/v1alpha1/zz_generated.role.go:89: undefined: rbac in rbac.Resource
pkg/client/listers/rbac/v1alpha1/zz_generated.rolebinding.go:89: undefined: rbac in rbac.Resource
# k8s.io/kubernetes/pkg/client/listers/storage/v1beta1
pkg/client/listers/storage/v1beta1/zz_generated.storageclass.go:63: undefined: storage in storage.Resource
# k8s.io/kubernetes/pkg/client/listers/extensions/v1beta1
pkg/client/listers/extensions/v1beta1/zz_generated.daemonset.go:89: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.deployment.go:89: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.ingress.go:89: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.job.go:89: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.podsecuritypolicy.go:63: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.replicaset.go:89: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.scale.go:89: undefined: extensions in extensions.Resource
pkg/client/listers/extensions/v1beta1/zz_generated.thirdpartyresource.go:63: undefined: extensions in extensions.Resource
```
cc @ncdc @caesarxuchao

Automatic merge from submit-queue

Disable gci-mounter in cri node e2e tests

gci-mounter is still being validated and there are known issues. Do not enable it
for cri tests for now.

Automatic merge from submit-queue

CRI: Add kuberuntime container logs

Based on https://github.com/kubernetes/kubernetes/pull/34858.

The first 2 commits are from #34858. And the last 2 commits are new.

This PR added kuberuntime container logs support and add unit test for it.

I've tested all the functions manually, and I'll send another PR to write a node e2e test for container log.

**_Notice: current implementation doesn't support log rotation**_, which means that:
- It will not retrieve logs in rotated log file.
- If log rotation happens when following the log:
  - If the rotation is using create mode, we'll still follow the old file.
  - If the rotation is using copytruncate, we'll be reading at the original position and get nothing.

To solve these issues, kubelet needs to rotate the log itself, or at least kubelet should be able to control the the behavior of log rotator. These are doable but out of the scope of 1.5 and will be addressed in future release.

@yujuhong @feiskyer @yifan-gu 
/cc @kubernetes/sig-node

Automatic merge from submit-queue

Node controller to not force delete pods

Fixes https://github.com/kubernetes/kubernetes/issues/35145

- [x] e2e tests to test Petset, RC, Job.
- [x] Remove and cover other locations where we force-delete pods within the NodeController.

**Release note**:

<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->

``` release-note
Node controller no longer force-deletes pods from the api-server.

* For StatefulSet (previously PetSet), this change means creation of replacement pods is blocked until old pods are definitely not running (indicated either by the kubelet returning from partitioned state, or deletion of the Node object, or deletion of the instance in the cloud provider, or force deletion of the pod from the api-server). This has the desirable outcome of "fencing" to prevent "split brain" scenarios.
* For all other existing controllers except StatefulSet , this has no effect on the ability of the controller to replace pods because the controllers do not reuse pod names (they use generate-name).
* User-written controllers that reuse names of pod objects should evaluate this change.
```

Status is exposed as v1 in the current schema (so all groups are
returning v1.Status).  However, if you give a codec only "mygroup"
"myversion", it will fail to convert Status to v1. For now, unversioned
types should be allowed to be projected into all group versions, and
when we add the server group we'll rip out the unversioned concept
entirely.