Automatic merge from submit-queue

Fixing federation secret controller unit test flakiness

Fixes https://github.com/kubernetes/kubernetes/issues/36422

Adding a wait for the secret to be updated in the store to fix flakiness.
It was failing ~once in 3 to 5 runs before this change. I now have had 30 local runs without a failure.

cc @kubernetes/sig-cluster-federation @mwielgus

Automatic merge from submit-queue

Use the new name for cassandra-statefulset.yaml in e2e tests.

Automatic merge from submit-queue

Add retry to node scheduability marking.

Fixes https://github.com/kubernetes/kubernetes/issues/32391

@justinsb @janetkuo @kubernetes/sig-cli

Merge pull request #36040 from bruceauyeung/add-master-address-desc-into-kubeadm-join-and-some-validations

Automatic merge from submit-queue

add master address into kubeadm join help message and some validations

**What this PR does / why we need it**:

1, add master address into kubeadm join help message. looks like :

>Usage:
>  kubeadm join <master address> [flags]

2, when user provides more than one master address, return an error.

3, since `kubeadm join` not only support ip addresses but also host names or domain names, so i delete the word `ip` from error message `must specify master ip address (see --help)`
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>

Automatic merge from submit-queue

Fix find pattern in generating kubernetes src tarball

#36407 wasn't supposed to merge. This fixes a small issue in it.

Automatic merge from submit-queue

specify custom ca file to verify the keystone server

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:

Sometimes the keystone server's certificate is self-signed, mainly used for internal development, testing and etc.

For this kind of ca, we need a way to verify the keystone server.

Otherwise, below error will occur.

> x509: certificate signed by unknown authority

This patch provide a way to pass in a ca file to verify the keystone server when starting `kube-apiserver`.

**Which issue this PR fixes** : fixes #22695, #24984

**Special notes for your reviewer**:

**Release note**:

<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->

``` release-note
```

Automatic merge from submit-queue

Make kubernetes-src.tar.gz contain source code again

**What this PR does / why we need it**: This PR fixes `kubernetes-src.tar.gz` by reviving the logic for `kube-source.tar.gz` from before #30787.

**Which issue this PR fixes**: fixes #36403

@saad-ali @jbeda @zmerlynn

Automatic merge from submit-queue

Node E2E: Extend the default ci node e2e test timeout to 1h.

With more and more test added into node e2e, 45m seems to be not enough now.
I saw sometimes the test takes 40+m:
* https://storage.googleapis.com/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10942/build-log.txt (44m4.917870119s)
* https://storage.googleapis.com/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10965/build-log.txt (40m2.37254827s)

And sometimes even timeout:
* https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10968
* https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10941

Although it's quite likely that the timeout happened because the limit is too tight, we are not 100% sure.
This PR extends the test timeout of regular ci node e2e test to 1h. Let's see whether the timeout will happen again.

@yujuhong

Automatic merge from submit-queue

Fail kubelet creation if swap enabled and configured with memory eviction thresholds

/cc @vishh @derekwaynecarr

**Release note**:

```
This adds an opt-in Kubelet flag (--fail-swap-on) that causes the Kubelet to fail to start if swap is enabled on the node. This is a temporary opt-in, and the Kubelet will by default fail with swap enabled starting in 1.6.0. The KubeletConfiguration equivalent to the flag is FailSwapOn.
```

Automatic merge from submit-queue

register default functions for federation/apis/core

fix issue #35667 and all related e2e test fail on federation.

Provides an opt-in flag, --experimental-fail-swap-on (and corresponding
KubeletConfiguration value, ExperimentalFailSwapOn), which is false by default.

Automatic merge from submit-queue

Fix flunetd-gcp image Dockerfile

A mistake was made during rebasing this PR: https://github.com/kubernetes/kubernetes/pull/36370

One line, earlier deleted, was restored and broke the image build.

@piosz

Automatic merge from submit-queue

 Deprecate extensions/v1beta1.Jobs related stuff

This PR supersedes https://github.com/kubernetes/kubernetes/pull/33861, it's a pre-req for removing `extensions/v1beta1.Jobs` (#32763) in the next release. 

@kubernetes/kubectl @kubernetes/api-review-team ptal
@bgrant0607 @erictune @janetkuo fyi

```release-note
Deprecate extensions/v1beta1.Jobs
```

Automatic merge from submit-queue

Fix kubelet's PodAdmitResult's message

There is no test added for it since it's a straightforward change.

Automatic merge from submit-queue

fix e2e delete namespace bug

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What:**

Fix `CreateNamespace()` bug in `test/e2e/framework/framework.go`.

**Why:**

If we successfully create a namespace but fail to create a ServiceAccount in it, we should delete the namespace after test.

The current implement of `CreateNamespace()` int e2e test will forget to delete the namespace which we fail to create serviceAccount in it(but the namespace has been successfully created!).

Automatic merge from submit-queue

Simplifies NFS and hostPath plugin code

Simplifies NFS and hostPath plugin code.

cc: @jsafrane

Automatic merge from submit-queue

AWS: Support default value for ExternalHost

Automatic merge from submit-queue

Make a consistent name ( GlusterFS instead of Gluster) in variables a…

Signed-off-by: Humble Chirammal hchiramm@redhat.com

Automatic merge from submit-queue

Switch cluster autoscaler to 0.4.0-beta1

Switch Kubernetes to new 0.4.0-beta1 Cluster Autoscaler. The release contains mainly bugfixes:
* unschedulable nodes don't stop cluster autoscaler
* better logging
* events for deltions
* bulk delete for empty nodes

cc: @fgrzadkowski @piosz @jszczepkowski

Automatic merge from submit-queue

HPA: Consider unready pods separately

**Release note**:

``` release-note
The Horizontal Pod Autoscaler now takes the readiness of pods into account when calculating desired replicas.
```

Currently, the HPA considers unready pods the same as ready pods when
looking at their CPU and custom metric usage.  However, pods frequently
use extra CPU during initialization, so we want to consider them
separately.

This commit causes the HPA to consider unready pods as having 0 CPU
usage when scaling up, and ignores them when scaling down.  If, when
scaling up, factoring the unready pods as having 0 CPU would cause a
downscale instead, we simply choose not to scale.  Otherwise, we simply
scale up at the reduced amount calculated by factoring the pods in at
zero CPU usage.

Similarly, if we are missing metrics for any pods, those pods will be
considered as having 0% CPU when scaling up, and 100% CPU when
scaling down.  As with the unready pods calculation, this cannot change
the direction of the scale.

The effect is that unready pods cause the autoscaler to be a bit more
conservative -- large increases in CPU usage can still cause scales,
even with unready pods in the mix, but will not cause the scale factors
to be as large, in anticipation of the new pods later becoming ready and
handling load.

Automatic merge from submit-queue

options, kube-apiserver: clarify scheme on etcd endpoints

**What this PR does / why we need it**:

Fix typo in `kube-apiserver` flag.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

None

Automatic merge from submit-queue

'Max' and 'MIn' don't seem to used anywhere, so I would suggest removing them

Signed-off-by: Yanqiang Miao miao.yanqiang@zte.com.cn

Automatic merge from submit-queue

improve and modify log

1, the content of a unified writing, compared to the following line of failure (314th lines)
2, “instance” should be “node”

Automatic merge from submit-queue

Print osImage and kubeletVersion for nodes before and after GCE upgrade

This will print, e.g.:
```
== Pre-Upgrade Node OS and Kubelet Versions ==
name: "e2e-test-mtaufen-master", osImage: "Google Container-VM Image", kubeletVersion: "v1.4.5-beta.0.45+90d209221ec8dc-dirty"
name: "e2e-test-mtaufen-minion-group-jo79", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.5-beta.0.45+90d209221ec8dc-dirty"
name: "e2e-test-mtaufen-minion-group-ox5l", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.5-beta.0.45+90d209221ec8dc-dirty"
name: "e2e-test-mtaufen-minion-group-qvbq", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.5-beta.0.45+90d209221ec8dc-dirty"
```

Let me know what output format you prefer and I'll see if I can make it work, I have the extent of flexibility allowed by jsonpath.

Automatic merge from submit-queue

Add timeout argument to ExecInContainer

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**: This is related to https://github.com/kubernetes/kubernetes/issues/26895. It brings a timeout to the signature of `ExecInContainer` so that we can take timeouts into account in the future. Unlike my first attempt in https://github.com/kubernetes/kubernetes/pull/27956, it doesn't immediately observe the timeout, because it is impossible to do it with the current state of the Docker Remote API (the default exec handler implementation).

**Special notes for your reviewer**: This shares commits with https://github.com/kubernetes/kubernetes/pull/27956, but without some of them that have more controversial implications (actually supporting the timeouts). The original PR shall be closed in the current state to preserve the history (instead of dropping commits in that PR).

Pinging the original people working on this change: @ncdc @sttts @vishh @dims 

**Release note**:

<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->

``` release-note
NONE
```

Automatic merge from submit-queue

Fix non-default etcd image handling in kubemark

Doesn't touch production code.

Automatic merge from submit-queue

Deleting dead code from federation service controller

Automatic merge from submit-queue

Node Conformance Test: Containerize the node e2e test

For #30122, #30174.
Based on #32427, #32454.

**Please only review the last 3 commits.**

This PR packages the node e2e test into a docker image:
- 1st commit: Add `NodeConformance` flag in the node e2e framework to avoid starting kubelet and collecting system logs. We do this because:
  - There are all kinds of ways to manage kubelet and system logs, for different situation we need to mount different things into the container, run different commands. It is hard and unnecessary to handle the complexity inside the test suite.
- 2nd commit: Remove all `sudo` in the test container. We do this because:
  - In most container, there is no `sudo` command, and there is no need to use `sudo` inside the container.
  - It introduces some complexity to use `sudo` inside the test. (https://github.com/kubernetes/kubernetes/issues/29211, https://github.com/kubernetes/kubernetes/issues/26748) In fact we just need to run the test suite with `sudo`.
- 3rd commit: Package the test into a docker container with corresponding `Makefile` and `Dockerfile`. We also added a `run_test.sh` script to start kubelet and run the test container. The script is only for demonstration purpose and we'll also use the script in our node e2e framework. In the future, we should update the script to start kubelet in production way (maybe with `systemd` or `supervisord`).

@dchen1107 @vishh 
/cc @kubernetes/sig-node @kubernetes/sig-testing



**Release note**:

<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->

``` release-note
Release alpha version node test container gcr.io/google_containers/node-test-ARCH:0.1 for users to verify their node setup.
```

Fixes #36323.

Automatic merge from submit-queue

Adding cadcading deletion support for federated secrets

Ref https://github.com/kubernetes/kubernetes/issues/33612

Adding cascading deletion support for federated secrets.
The code is same as that for namespaces.  Just ensuring that DeletionHelper functions are called at right places in secret_controller.
Also added e2e tests.

cc @kubernetes/sig-cluster-federation @caesarxuchao

```release-note
federation: Adding support for DeleteOptions.OrphanDependents for federated secrets. Setting it to false while deleting a federated secret also deletes the corresponding secrets from all registered clusters.
```

Currently, the HPA considers unready pods the same as ready pods when
looking at their CPU and custom metric usage.  However, pods frequently
use extra CPU during initialization, so we want to consider them
separately.

This commit causes the HPA to consider unready pods as having 0 CPU
usage when scaling up, and ignores them when scaling down.  If, when
scaling up, factoring the unready pods as having 0 CPU would cause a
downscale instead, we simply choose not to scale.  Otherwise, we simply
scale up at the reduced amount caculated by factoring the pods in at
zero CPU usage.

The effect is that unready pods cause the autoscaler to be a bit more
conservative -- large increases in CPU usage can still cause scales,
even with unready pods in the mix, but will not cause the scale factors
to be as large, in anticipation of the new pods later becoming ready and
handling load.

Similarly, if there are pods for which no metrics have been retrieved,
these pods are treated as having 100% of the requested metric when
scaling down, and 0% when scaling up.  As above, this cannot change the
direction of the scale.

This commit also changes the HPA to ignore superfluous metrics -- as
long as metrics for all ready pods are present, the HPA we make scaling
decisions.  Currently, this only works for CPU.  For custom metrics, we
cannot identify which metrics go to which pods if we get superfluous
metrics, so we abort the scale.

Automatic merge from submit-queue

kuberuntime: remove unused GetNs code

Automatic merge from submit-queue

Deploy kube-dns with cluster-proportional-autoscaler

This PR integrates [cluster-proportional-autoscaler](https://github.com/kubernetes-incubator/cluster-proportional-autoscaler) with kube-dns for DNS horizontal autoscaling. 

Fixes #28648 and #27781.