Automatic merge from submit-queue

Enable OpenAPI spec validation

Spec validation was failing on Jenkins. I am enabling it in this PR to figure out if we can get it pass. No review is necessary until all test passes.

Automatic merge from submit-queue (batch tested with PRs 37693, 38085)

fix typo in scale

fix typo.

Automatic merge from submit-queue (batch tested with PRs 37693, 38085)

Pipe get options to storage

Ref #37473

Automatic merge from submit-queue (batch tested with PRs 38173, 38151, 38197, 38221)

Revert "Disable "overlapping deployment" test on GKE temporarily. #32947"

Reverts kubernetes/kubernetes#38157

Automatic merge from submit-queue (batch tested with PRs 38173, 38151, 38197, 38221)

test: wait for ready replica set before adopting

Reworked version of https://github.com/kubernetes/kubernetes/pull/36439 which was reverted in https://github.com/kubernetes/kubernetes/pull/38049. This PR doesn't use any of the new status API added in replica sets so it should cause no trouble with upgrade tests.

@kubernetes/deployment @smarterclayton

Automatic merge from submit-queue (batch tested with PRs 38173, 38151, 38197, 38221)

Include resource type headers in diff report

**What this PR does / why we need it**: when checking for leaked resources, we report a failure only if there were additions to the diff - we don't want resources that were deleted during the run to count as a failure.

The logic for this removes the resource type headers from the diff report, however, so it's harder to understand what leaked.

For example, in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features-release-1.5/196/, the test failure shows
```
Error: 4 leaked resources
+NAME                              REGION       IP_ADDRESS      IP_PROTOCOL  TARGET
+a66878dbaad1f11e695b342010af0002  us-central1  104.198.182.83  TCP          us-central1/targetPools/a66878dbaad1f11e695b342010af0002
+NAME                              REGION       SESSION_AFFINITY  BACKUP  HEALTH_CHECKS
+a66878dbaad1f11e695b342010af0002  us-central1                            a66878dbaad1f11e695b342010af0002
```

but the [actual diff](https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features-release-1.5/196/artifacts/gcp-resources-diff.txt) contains more information:
```
--- /workspace/_artifacts/gcp-resources-before.txt	2016-12-01 14:39:14.260049037 -0800
+++ /workspace/_artifacts/gcp-resources-after.txt	2016-12-01 14:39:14.260049037 -0800
@@ -34,8 +34 @@ [ addresses ]
-NAME                              REGION       ADDRESS          STATUS
-a03d0e020a6bc11e6855f42010af0001  us-central1  104.197.135.88   RESERVED
-a23f20a47a6a011e6829a42010af0004  us-central1  104.197.147.212  RESERVED
-a32291b51a5e611e6b73342010af0002  us-central1  104.198.171.178  RESERVED
-a3c93cf79a65811e6b7c242010af0001  us-central1  104.198.58.57    RESERVED
-a9ac5bd1ba62811e68d1142010af0004  us-central1  104.154.134.135  RESERVED
-adae4c1a7a69511e6b2df42010af0000  us-central1  104.198.182.83   RESERVED
-adb4a80caa60311e689a942010af0001  us-central1  104.154.116.195  RESERVED
+
@@ -64 +57,2 @@ [ forwarding-rules ]
-
+NAME                              REGION       IP_ADDRESS      IP_PROTOCOL  TARGET
+a66878dbaad1f11e695b342010af0002  us-central1  104.198.182.83  TCP          us-central1/targetPools/a66878dbaad1f11e695b342010af0002
@@ -68 +62,2 @@ [ target-pools ]
-
+NAME                              REGION       SESSION_AFFINITY  BACKUP  HEALTH_CHECKS
+a66878dbaad1f11e695b342010af0002  us-central1                            a66878dbaad1f11e695b342010af0002
```

This PR will include the `[ type header ]`s in the report if resources are leaked.

cc @fejta

Automatic merge from submit-queue

fix typo in ingress

fix typo.

Automatic merge from submit-queue

Fix pods/log OpenAPI/Swagger spec to return string instead of v1.Pod

log operation is part of pod and had the assumption of returning a v1.Pod. First I added the support for an operation to be able to customize its return object then add the support to pods/log operation to return an string as an object sample.

Fixes #37881
Cc @kubernetes/sig-api-machinery

Automatic merge from submit-queue

add a configuration for kubelet to register as a node with taints

and deprecate --register-schedulable

ref #28687 #29178

cc @dchen1107 @davidopp @roberthbailey

Automatic merge from submit-queue (batch tested with PRs 37032, 38119, 38186, 38200, 38139)

New ns param for NewClusterVerification

**What this PR does / why we need it**: Allows the test to specify alternate namespaces to when waiting for pods to be in a specific state.

**Which issue this PR fixes**: fixes #38138

**Special notes for your reviewer**: Minor fix

**Release note**: None

Automatic merge from submit-queue (batch tested with PRs 37032, 38119, 38186, 38200, 38139)

etcd2: remove unnecessary PrevValue in SetOption

ref: https://github.com/kubernetes/kubernetes/issues/37994

Summary:
- PrevValue is set in HTTP header, and large value (>1MB) could exceed check limit
- We don't need PrevValue indeed since we already use PrevIndex in SetOptions and each PrevIndex corresponds to each PrevValue.

I don't really think we need extra tests for this. There is already test for GuaranteedUpdate covering its use cases.

Automatic merge from submit-queue (batch tested with PRs 37032, 38119, 38186, 38200, 38139)

Remove genericapiserver.Options.MasterServiceNamespace

Deprecated and not used anymore.

Automatic merge from submit-queue (batch tested with PRs 37032, 38119, 38186, 38200, 38139)

Detect long-running requests from parsed request info

Follow up to https://github.com/kubernetes/kubernetes/pull/36064

Uses parsed request info to more tightly match verbs and subresources

Removes regex-based long-running request path matching (which is easily fooled)

```release-note
The --long-running-request-regexp flag to kube-apiserver is deprecated and will be removed in a future release. Long-running requests are now detected based on specific verbs (watch, proxy) or subresources (proxy, portforward, log, exec, attach).
```

Automatic merge from submit-queue

Add integration tests for desire state of world populator

Add integration tests for desire state of world populator
    
This adds tests for code introduced here :
https://github.com/kubernetes/kubernetes/issues/26994
    
Via integration test we can now verify that if pod delete
event is somehow missed by AttachDetach controller - it still
get cleaned up by Desired State of World populator.

Automatic merge from submit-queue (batch tested with PRs 38194, 37594, 38123, 37831, 37084)

[kubeadm] Fail fast if there already is a node in the cluster with that name

kubeadm Fail fast if there is another node with the same name already in the cluster. 

Fixes #36255

cc @kubernetes/sig-cluster-federation

Automatic merge from submit-queue (batch tested with PRs 38194, 37594, 38123, 37831, 37084)

Improve kubeadm reset

Depends on: https://github.com/kubernetes/kubernetes/pull/36474
Broken out from: https://github.com/kubernetes/kubernetes/pull/37568
Carries: https://github.com/kubernetes/kubernetes/pull/35709, @camilocot

This makes the `kubeadm reset` command more robust and user-friendly.
I'll rebase after #36474 merges...

cc-ing reviewers: @mikedanese @errordeveloper @dgoodwin @jbeda

Automatic merge from submit-queue (batch tested with PRs 38194, 37594, 38123, 37831, 37084)

remove unnecessary fields from genericapiserver config

Cleans up some unnecessary fields in the genericapiserver config.

Automatic merge from submit-queue (batch tested with PRs 38194, 37594, 38123, 37831, 37084)

Better compat with very old iptables (e.g. CentOS 6)

Fixes reported issue with CentOS6 iptables 1.4.7 (ancient)

Older iptables expanded things like 0x4000 into 0x00004000, which defeats the
fallback "check" logic.

Fixes #37416

Automatic merge from submit-queue

e2e: mark portforward tests as flaky

As long as we don't get https://github.com/kubernetes/kubernetes/issues/27673 and https://github.com/kubernetes/kubernetes/issues/27680 under control, this PR marks them as `[Flaky]`.

There is https://github.com/kubernetes/kubernetes/pull/37072 in flight which might improve the situation, but is blocked on socat issues right now with half-open connections.

cc @kubernetes/sig-testing

Automatic merge from submit-queue

Fix GCI mounter issue

Automatic merge from submit-queue

Skip not registered nodes in labeling in CA e2e tests

This PR fixes problems with querying for not yet registered nodes. The underlying problem is related to the way the test is written. So we apply labels to the existing nodes, create pods that require N+1 nodes  with the labels and expect a new node to be added. But the new node is created without the labels. As soon as the node is spotted it is labeled. But sometimes it is too late. CA notices that the new node doesn't solve the problem and ask for another, hoping that this time it will get the node with the labels.  The node is added by MIG but it takes a minute or more for the node to start and register in kubernetes. At this moment the labeling is started. The list of nodes to be labeled is taken from MIG. The extra node is there. But it is not in kubernetes yet. So 404 error is returned on labeling attempt and test fails.

This PR filters the list of nodes to be labeled and applies the labels only on the fully registered nodes.

Fixes 404 in #33754

cc: @jszczepkowski @piosz @fgrzadkowski

Automatic merge from submit-queue

enable the kubernetes-metadata-plugin in a case of daemonSet

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
```

instruction to users that would like to enable the fluentd kubernetes-metadata plugin.
Relevant only if you are using daemon-set

Automatic merge from submit-queue (batch tested with PRs 36990, 37494, 38152, 37561, 38136)

join client CA bundles

Last commit grabs client CA bundles from disparate parts of the auth config and makes a pool that contains all of them.

I suspect a rebase broke this because of ordering.  I'll keep these separate to make it easier for me to debug.

Automatic merge from submit-queue (batch tested with PRs 36990, 37494, 38152, 37561, 38136)

api federation types

First commit adds types that can back the kubernetes-discovery server with an `kubectl` compatible way of adding federated servers.  Second commit is just generated code.

After we have types, I'd like to start splitting `kubernetes-discovery` into a "legacy" mode which will support what we have today and a "normal" mode which will provide an API federation server like this: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federated-api-servers.md that includes both discovery and proxy in a single server.  Something like this: https://github.com/openshift/kube-aggregator .

@kubernetes/sig-api-machinery @nikhiljindal

Automatic merge from submit-queue (batch tested with PRs 36990, 37494, 38152, 37561, 38136)

Node E2E: Move ssh related functions into ssh.go.

This PR moves all ssh related functions and variables into a separate file `ssh.go`.

This is a minor cleanup preparing for my test framework refactoring work. Will send out the refactor PR later.

/cc @kubernetes/sig-node

Automatic merge from submit-queue (batch tested with PRs 36990, 37494, 38152, 37561, 38136)

Pass proxy environment variables to static pods

**What this PR does / why we need it**:

To access outside world or cloud provider APIs it might be
required to use proxy. kubeadm will be passing proxy variables to static pods during init stage.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes kubernetes/kubernetes#36573, kubernetes/kubeadm#5

**Special notes for your reviewer**:

**Release note**:
```release-note
- kubeadm will pass proxy environment variables to static pods.
```

Automatic merge from submit-queue (batch tested with PRs 36990, 37494, 38152, 37561, 38136)

Never set hairpin mode on every interface

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:

Abandon setting hairpin mode if finding the peer interface fails; simply return an error.

There are many reasons why finding the peer could fail - "`ethtool` not installed" is popular.  Going ahead and changing the hairpin setting on every bridge-connected interface on the machine may have unwanted effects on other things installed on the machine (e.g. https://github.com/kubernetes/kops/issues/879)

**Which issue this PR fixes** : fixes #19766

**Special notes for your reviewer**:

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
Kubelet will no longer set hairpin mode on every interface on the machine when an error occurs in setting up hairpin for a specific interface.
```

/cc @thockin who appears to have requested this implementation at https://github.com/kubernetes/kubernetes/pull/13628#issuecomment-138128180

instruction to users that would like to enable the fluentd kubernetes-metadata plugin.
Relevant only if you are using daemon-set

Automatic merge from submit-queue

Add "make help" to list all make targets and help info

As discussed in [PR#29320comment](https://github.com/kubernetes/kubernetes/pull/29320#issuecomment-234420145), add a `make help` to make the use of `make` easy. Though it works well on my Ubuntu now (see the output as below, not sure if it still works on other systems), I believe the scripts are somewhat ugly, so, any suggestion for optimization is welcome.

BTW, I intended to list targets by groups, but it's hard to do that automatically. So I just list them in alphabetical order. I think this may be enough.

There are still some TODOs (also mentioned in the Makefile):
1. make EXCLUDE_TARGET auto-generated when there are other files in cmd/
2. should we exclude the target "cmd/libs" but include "cmd/libs/go2idl/*"?
3. should we let `help` be the first/default target? It will show the help when we just type `make`.

1 and 2 are to exclude the invalid targets generated by `$(notdir $(abspath $(wildcard cmd/*/)))`:
- OWNERS is just a file, it can't be a target
- libs itself cannot be built

/cc @thockin @jfrazelle @MHBauer @dims 

Output:

```
root@vm:/home/paas/zxp/code/k8s/fork/kubernetes# make help
--------------------------------------------------------------------------------
all
# Build code.
#
# Args:
#   WHAT: Directory names to build.  If any of these directories has a 'main'
#     package, the build will produce executable files under _output/go/bin.
#     If not specified, "everything" will be built.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags passed to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make
#   make all
#   make all WHAT=cmd/kubelet GOFLAGS=-v
#   make all GOGCFLAGS="-N -l"
#     Note: Use the -N -l options to disable compiler optimizations an inlining.
#           Using these build options allows you to subsequently use source
#           debugging tools like delve.
---------------------------------------------------------------------------------
check
# Build and run tests.
#
# Args:
#   WHAT: Directory names to test.  All *_test.go files under these
#     directories will be run.  If not specified, "everything" will be tested.
#   TESTS: Same as WHAT.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags to pass to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make check
#   make test
#   make check WHAT=pkg/kubelet GOFLAGS=-v
---------------------------------------------------------------------------------
clean
# Remove all build artifacts.
#
# Example:
#   make clean
#
# TODO(thockin): call clean_generated when we stop committing generated code.
---------------------------------------------------------------------------------
clean_generated
# Remove all auto-generated artifacts.
#
# Example:
#   make clean_generated
---------------------------------------------------------------------------------
clean_meta
# Remove make-related metadata files.
#
# Example:
#   make clean_meta
---------------------------------------------------------------------------------
cross
# Cross-compile for all platforms
#
# Example:
#   make cross
---------------------------------------------------------------------------------
federation-apiserver
federation-controller-manager
genfeddocs
# Add rules for all directories in federation/cmd/
#
# Example:
#   make federation-apiserver federation-controller-manager
---------------------------------------------------------------------------------
gendocs
genkubedocs
genman
genswaggertypedocs
genutils
genyaml
hyperkube
kube-apiserver
kube-controller-manager
kubectl
kube-dns
kubelet
kubemark
kube-proxy
kubernetes-discovery
libs
linkcheck
mungedocs
# Add rules for all directories in cmd/
#
# Example:
#   make kubectl kube-proxy
---------------------------------------------------------------------------------
generated_files
# Produce auto-generated files needed for the build.
#
# Example:
#   make generated_files
---------------------------------------------------------------------------------
ginkgo
# Build ginkgo
#
# Example:
# make ginkgo
---------------------------------------------------------------------------------
help
# Print make targets and help info
#
# Example:
# make help
---------------------------------------------------------------------------------
quick-release
# Build a release, but skip tests
#
# Example:
#   make release-skip-tests
---------------------------------------------------------------------------------
release
# Build a release
#
# Example:
#   make release
---------------------------------------------------------------------------------
release-skip-tests
# Build a release, but skip tests
#
# Example:
#   make release-skip-tests
---------------------------------------------------------------------------------
test
# Build and run tests.
#
# Args:
#   WHAT: Directory names to test.  All *_test.go files under these
#     directories will be run.  If not specified, "everything" will be tested.
#   TESTS: Same as WHAT.
#   GOFLAGS: Extra flags to pass to 'go' when building.
#   GOLDFLAGS: Extra linking flags to pass to 'go' when building.
#   GOGCFLAGS: Additional go compile flags passed to 'go' when building.
#
# Example:
#   make check
#   make test
#   make check WHAT=pkg/kubelet GOFLAGS=-v
---------------------------------------------------------------------------------
test-cmd
# Build and run cmdline tests.
#
# Example:
#   make test-cmd
---------------------------------------------------------------------------------
test-e2e
# Build and run end-to-end tests.
#
# Example:
#   make test-e2e
---------------------------------------------------------------------------------
test-e2e-node
# Build and run node end-to-end tests.
#
# Args:
#  FOCUS: Regexp that matches the tests to be run.  Defaults to "".
#  SKIP: Regexp that matches the tests that needs to be skipped.  Defaults
#    to "".
#  RUN_UNTIL_FAILURE: If true, pass --untilItFails to ginkgo so tests are run
#    repeatedly until they fail.  Defaults to false.
#  REMOTE: If true, run the tests on a remote host instance on GCE.  Defaults
#    to false.
#  IMAGES: For REMOTE=true only.  Comma delimited list of images for creating
#    remote hosts to run tests against.  Defaults to a recent image.
#  LIST_IMAGES: If true, don't run tests.  Just output the list of available
#    images for testing.  Defaults to false.
#  HOSTS: For REMOTE=true only.  Comma delimited list of running gce hosts to
#    run tests against.  Defaults to "".
#  DELETE_INSTANCES: For REMOTE=true only.  Delete any instances created as
#    part of this test run.  Defaults to false.
#  ARTIFACTS: For REMOTE=true only.  Local directory to scp test artifacts into
#    from the remote hosts.  Defaults to ""/tmp/_artifacts".
#  REPORT: For REMOTE=false only.  Local directory to write juntil xml results
#    to.  Defaults to "/tmp/".
#  CLEANUP: For REMOTE=true only.  If false, do not stop processes or delete
#    test files on remote hosts.  Defaults to true.
#  IMAGE_PROJECT: For REMOTE=true only.  Project containing images provided to
#  IMAGES.  Defaults to "kubernetes-node-e2e-images".
#  INSTANCE_PREFIX: For REMOTE=true only.  Instances created from images will
#    have the name "-".  Defaults to "test".
#  INSTANCE_METADATA: For REMOTE=true and running on GCE only.
#
# Example:
#   make test-e2e-node FOCUS=Kubelet SKIP=container
#   make test-e2e-node REMOTE=true DELETE_INSTANCES=true
#   make test-e2e-node TEST_ARGS="--cgroups-per-qos=true"
# Build and run tests.
---------------------------------------------------------------------------------
test-integration
# Build and run integration tests.
#
# Example:
#   make test-integration
---------------------------------------------------------------------------------
verify
# Runs all the presubmission verifications.
#
# Args:
#   BRANCH: Branch to be passed to verify-godeps.sh script.
#
# Example:
#   make verify
#   make verify BRANCH=branch_x
---------------------------------------------------------------------------------
vet
# Run 'go vet'.
#
# Args:
#   WHAT: Directory names to vet.  All *.go files under these
#     directories will be vetted.  If not specified, "everything" will be
#     vetted.
#
# Example:
#   make vet
#   make vet WHAT=pkg/kubelet
---------------------------------------------------------------------------------
```

this is a workaround for the unmount device issue caused by gci mounter. In GCI cluster, if gci mounter is used for mounting, the container started by mounter script will cause additional mounts created in the container. Since these mounts are irrelavant to the original mounts, they should be not considered when checking the mount references. By comparing the mount path prefix, those additional mounts can be filtered out.

Plan to work on better approach to solve this issue.

and deprecate register-schedulable