Automatic merge from submit-queue

Add hack/update-staging-* to update-all.sh

Automatic merge from submit-queue

clean up generic apiserver options

Clean up generic apiserver options before we tag any levels.  This makes them more in-line with "normal" api servers running on the platform.

Also remove dead example code.

@sttts

Automatic merge from submit-queue

kubeadm: Make the CLI arguments for the control plane overridable

Automatic merge from submit-queue

Move push-federation-images.sh to federation and implement similar functionality in jenkins build directory for presubmits.

This is required for federation presubmit e2es.

```release-note
NONE
```

Automatic merge from submit-queue

Reserve kubernetes.io and k8s.io namespace for flex volume options

Split from https://github.com/kubernetes/kubernetes/pull/39488.

Flex volume already stuffs system information into the options map, and assumes it is free to do so:
```
	optionFSType    = "kubernetes.io/fsType"
	optionReadWrite = "kubernetes.io/readwrite"
	optionKeySecret = "kubernetes.io/secret"
```

this formalizes that by reserving the `kubernetes.io` and `k8s.io` namespaces so that user-specified options are never stomped by the system, and flex plugins can know that options with those namespaces came from the system, not user-options.

```release-note
Parameter keys in a StorageClass `parameters` map may not use the `kubernetes.io` or `k8s.io` namespaces.
```

Automatic merge from submit-queue (batch tested with PRs 41937, 41151, 42092, 40269, 42135)

[Federation] ReplicaSet e2es should let the API server to generate the names to avoid collision while running tests in parallel.

cc @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 41937, 41151, 42092, 40269, 42135)

Improve code coverage for scheduler/algorithmprovider/defaults

**What this PR does / why we need it**:

Improve code coverage for scheduler/algorithmprovider/defaults from #39559

Thanks for your review.
**Special notes for your reviewer**:

**Release note**:

```release-note
```

Automatic merge from submit-queue (batch tested with PRs 41937, 41151, 42092, 40269, 42135)

[Federation] Make --dns-provider mandatory for kubefed

Targets #40757

I thought about adding a test for this but I decided it wasn't worth it.  There's too much setup involved in being able to run ``Complete`` for such a simple change.

**Release note**:

```release-note
The --dns-provider argument of 'kubefed init' is now mandatory and does not default to `google-clouddns`. To initialize a Federation control plane with Google Cloud DNS, use the following invocation: 'kubefed init --dns-provider=google-clouddns'
```

cc: @kubernetes/sig-federation-pr-reviews @madhusudancs

Automatic merge from submit-queue (batch tested with PRs 41937, 41151, 42092, 40269, 42135)

Add a unit test for idempotent applys to the TPR entries.

The test in apply_test follows the general pattern of other tests.
We load from a file in test/fixtures and mock the API server in the
function closure in the HttpClient call.
The apply operation expects a last-modified-configuration annotation.
That is written verbatim in the test/fixture file.

References #40841



**What this PR does / why we need it**:
Adds one unit test for TPR's using applies. 

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
References: 
https://github.com/kubernetes/features/issues/95
https://github.com/kubernetes/kubernetes/issues/40841#issue-204769102


**Special notes for your reviewer**:

I am not super proud of the tpr-entry name. 
But I feel like we need to call the two objects differently. 
The one which has Kind:ThirdPartyResource 
and the one has Kind:Foo. 

Is the name "ThirdPartyResource" used interchangeably for both ? I used tpr-entry for the Kind:Foo object.

Also I !assume! this is testing an idempotent apply because the last-applied-configuration annotation is the same as the object itself. 

This is the state I see in the logs of kubectl if I do a proper idempotent apply of a third party resource entry. 

I guess I will know more once I start playing around with apply command that change TPR objects. 

**Release note**:

```release-note
```

Automatic merge from submit-queue (batch tested with PRs 41937, 41151, 42092, 40269, 42135)

GCE will properly regenerate basic_auth.csv on kube-apiserver start.

**What this PR does / why we need it**:

If basic_auth.csv does not exist we will generate it as normal.
If basic_auth.csv exists we will remove the old admin password before adding the "new" one. (Turns in to a no-op if the password exists).
This did not work properly before because we were replacing by key, where the key was the password. New password would not match and so not replace the old password.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #41935

**Special notes for your reviewer**:

**Release note**:

```release-note
```

Automatic merge from submit-queue (batch tested with PRs 41234, 42186, 41615, 42028, 41788)

apimachinery: handle duplicated and conflicting type registration

Double registrations were leading to duplications in  `KnownKinds()`. Conflicting registrations with same gvk, but different types were not detected.

Automatic merge from submit-queue (batch tested with PRs 41234, 42186, 41615, 42028, 41788)

Make DaemonSet respect critical pods annotation when scheduling

**What this PR does / why we need it**: #41612

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #41612

**Special notes for your reviewer**:

**Release note**:

```release-note
Make DaemonSet respect critical pods annotation when scheduling. 
```

cc @kubernetes/sig-apps-feature-requests @erictune @vishh @liggitt @kargakis @lukaszo @piosz @davidopp

Automatic merge from submit-queue (batch tested with PRs 41234, 42186, 41615, 42028, 41788)

Additional upgrade e2e tests

**What this PR does / why we need it**: Add basic upgrade tests for DaemonSet and Job, and add "during upgrade" testing to ConfigMap test. Add a simple harness for testing upgrade tests.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:

**Special notes for your reviewer**: continuation of #41296 @krousey please review, thanks

**Release note**: `NONE`

Automatic merge from submit-queue (batch tested with PRs 41234, 42186, 41615, 42028, 41788)

Restore wait for etcd in rbac initialization

Wait for etcd was accidentally removed in https://github.com/kubernetes/kubernetes/commit/26b42d350d8071bf84840023065bc600ac9ad273#diff-efb2aa0040291a41fcf922efc886ea13L136

Related to https://github.com/kubernetes/kubernetes/issues/37704 and https://github.com/kubernetes/kubernetes/pull/39821

Automatic merge from submit-queue

Enforce Node Allocatable via cgroups

This PR enforces node allocatable across all pods using a top level cgroup as described in https://github.com/kubernetes/community/pull/348

This PR also provides an option to enforce `kubeReserved` and `systemReserved` on user specified cgroups. 

This PR will by default make kubelet create top level cgroups even if `kubeReserved` and `systemReserved` is not specified and hence `Allocatable = Capacity`.

```release-note
New Kubelet flag `--enforce-node-allocatable` with a default value of `pods` is added which will make kubelet create a top level cgroup for all pods to enforce Node Allocatable. Optionally, `system-reserved` & `kube-reserved` values can also be specified separated by comma to enforce node allocatable on cgroups specified via `--system-reserved-cgroup` & `--kube-reserved-cgroup` respectively. Note the default value of the latter flags are "".
This feature requires a **Node Drain** prior to upgrade failing which pods will be restarted if possible or terminated if they have a `RestartNever` policy.
```

cc @kubernetes/sig-node-pr-reviews @kubernetes/sig-node-feature-requests 

TODO:

- [x] Adjust effective Node Allocatable to subtract hard eviction thresholds
- [x] Add unit tests
- [x] Complete pending e2e tests
- [x] Manual testing
- [x] Get the proposal merged

@dashpole is working on adding support for evictions for enforcing Node allocatable more gracefully. That work will show up in a subsequent PR for v1.6

Automatic merge from submit-queue

Change etcd test image to 3.0.17.

Follow-up from #41540

Automatic merge from submit-queue

Enqueue controllers after minreadyseconds when all pods are ready

@janetkuo this should address https://github.com/kubernetes/kubernetes/issues/41697#issuecomment-281851377. Impossible to unit test this but it should stabilize some of our deployment e2e tests that occasionally fail because of availableReplicas not being updated.

It should also fix https://github.com/kubernetes/kubernetes/issues/41641

Eventually I would like AddAfter to be able to cancel previous invocations of the same key so I opened https://github.com/kubernetes/client-go/issues/131

@kubernetes/sig-apps-bugs

Signed-off-by: Vishnu kannan <vishnuk@google.com>

Signed-off-by: Vishnu kannan <vishnuk@google.com>

Signed-off-by: Vishnu kannan <vishnuk@google.com>

Signed-off-by: Vishnu Kannan <vishnuk@google.com>

Signed-off-by: Vishnu Kannan <vishnuk@google.com>

Automatic merge from submit-queue (batch tested with PRs 41205, 42196, 42068, 41588, 41271)

Implements an upgrade test for Job

**What this PR does / why we need it**:
This PR implements a cluster upgrade test for Job. Some functionality for Job testing has been moved from the e2e package to the framework package to facilitate code reuse between the e2e package and the upgrade package without introducing cyclic dependencies.
We need this PR to help automate the testing of cluster upgrades between versions.
**Release note**
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41205, 42196, 42068, 41588, 41271)

[CRI] enable kubenet traffic shaping

ref: https://github.com/kubernetes/kubernetes/issues/37316

Another way to do this is to expose another interface in network host to allow network plugins to retrieve annotation. But that seems unnecessary and more complicated.

Automatic merge from submit-queue (batch tested with PRs 41205, 42196, 42068, 41588, 41271)

Openapigen should process types in a consistent order

Types are sorted by name only in openapi-gen. This makes problems like #42051 for types with the same name. This PR adds an identity namer that includes full package name for the types and use that to sort types before processing them.

fixes #42051

Move push-federation-images.sh to federation and implement similar functionality in jenkins build directory for presubmits.

Automatic merge from submit-queue

AWS: Kill bash deployment

c.f. #38772, #42194 and https://k8s-testgrid.appspot.com/google-aws#aws

cluster/kube-up.sh with KUBERNETES_PROVIDER=aws has been broken on 1.6
for a couple of months now. No one is supporting it. Nuke.

```release-note
Deployment of AWS Kubernetes clusters using the in-tree bash deployment (i.e. cluster/kube-up.sh or get-kube.sh) is obsolete. v1.5.x will be the last release to support cluster/kube-up.sh with AWS. For a list of viable alternatives, see: http://kubernetes.io/docs/getting-started-guides/aws/ 
```

Automatic merge from submit-queue

fed: Add marun as reviewer

cc: @kubernetes/sig-federation-pr-reviews

c.f. #38772, #42194 and https://k8s-testgrid.appspot.com/google-aws#aws

cluster/kube-up.sh with KUBERNETES_PROVIDER=aws has been broken on 1.6
for a couple of months now. No one is supporting it. Nuke.

Automatic merge from submit-queue (batch tested with PRs 42053, 41282, 42056, 41663, 40927)

Allow getting logs directly from deployment, job and statefulset

**Special notes for your reviewer**:
@smarterclayton you asked for it in OpenShift


```release-note
kubectl logs allows getting logs directly from deployment, job and statefulset
```