lengthen pod deletion timeout to prevent flakes

Automatic merge from submit-queue (batch tested with PRs 40873, 40948, 39580, 41065, 40815)

Make DeltaFIFO Resync atomic

Make DeltaFIFO's Resync operation atomic, so it enqueues the entire
queue before allowing adds/updates/deletes.

I'm hoping to use this to help with custom resync periods for multiple event handlers against a single shared informer (see https://github.com/kubernetes/kubernetes/pull/40759#pullrequestreview-19598213 for the motivation).

@lavalamp @smarterclayton @deads2k @liggitt @sttts @timothysc @wojtek-t @gmarek @kubernetes/sig-api-machinery-pr-reviews @kubernetes/sig-scalability-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 40873, 40948, 39580, 41065, 40815)

Default target storage in etcd

To make etcd v2->v3 upgrade work correctly, we need to correctly set the "TARGET_STORAGE" env var. Since in head we are defaulting to etcd v3, this PR is defaulting also that env var to etcd3, so that by default upgrade works fine.

@fgrzadkowski @gmarek

Automatic merge from submit-queue (batch tested with PRs 40873, 40948, 39580, 41065, 40815)

Upgrade Hazelcast example with the latest hazelcast-kubernetes-bootst…

…rapper 0.8.0, using Deployments instead of Replication Controller.

**What this PR does / why we need it**:
This PR upgrade the Hazelcast example in storage folder. It would be great to be aligned with the latest version of hazelcast-kubernetes-bootstrapper

Automatic merge from submit-queue (batch tested with PRs 40873, 40948, 39580, 41065, 40815)

[CRI] Enable Hostport Feature for Dockershim

Commits:
1. Refactor common hostport util logics and add more tests

2. Add HostportManager which can ADD/DEL hostports instead of a complete sync.

3. Add Interface for retreiving portMappings information of a pod in Network Host interface. 
Implement GetPodPortMappings interface in dockerService. 

4. Teach kubenet to use HostportManager

Automatic merge from submit-queue

Remove 'get node' call during bootstrapping

Kubelet bootstrapping should have minimal permissions until it obtains an approved client certificate.

@luxas PTAL

/cc @mikedanese @cjcullen https://github.com/kubernetes/kubernetes/pull/40760#issuecomment-276832957

Automatic merge from submit-queue

[Kubelet] Delay deletion of pod from the API server until volumes are deleted

Previous PR that was reverted: #40239.

To summarize the conclusion of the previous PR after reverting:
- The status manager has the most up-to-date status, but the volume manager uses the status from the pod manager, which only is as up-to-date as the API server.
- Because of this, the previous change required an additional round trip between the kubelet and API server.
- When few pods are being added or deleted, this is only a minor issue.  However, when under heavy load, the QPS limit to the API server causes this round trip to take ~60 seconds, which is an unacceptable increase in latency. Take a look at the graphs in #40239 to see the effect of QPS changes on timing.
- To remedy this, the volume manager looks at the status from the status manager, which eliminates the round trip.

cc: @vishh @derekwaynecarr @sjenning @jingxu97 @kubernetes/sig-storage-misc

Automatic merge from submit-queue

add deads2k to approvers for controllers

I've done significant maintenance on these for a while and introduced new patterns like shared informers and rate limited work queues.

Automatic merge from submit-queue

avoid repeated length calculation and some other code improvements

**What this PR does / why we need it**:
1. in function `ParsePairs`,  calculating `invalidBuf`'s length over and over again brings performance penalty. a `invalidBufNonEmpty` bool value can fix this.
2. pairArg is not a string template and also there is no other arguments for `fmt.Sprintf`, so i remove `fmt.Sprintf`
3. in function `DumpReaderToFile`, we must check nil error first before defer statement, otherwise there maybe a potential nil error on `f.Close()`
4. add nil checks into `GetWideFlag` function
5. some other minor code improvements for better readability.
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>

Automatic merge from submit-queue

Removed a space in portforward.go.

**What this PR does / why we need it**:

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
```

Automatic merge from submit-queue

Add more logs in the progress check path

Follow-up to https://github.com/kubernetes/kubernetes/pull/41097

@mfojtik quick look here

Revert "remove second CA used for kubelet auth in favor of webhook auth"

Automatic merge from submit-queue (batch tested with PRs 40175, 41107, 41111, 40893, 40919)

[Federation][e2e] Move Cluster Registration to federation-up.sh

**What this PR does / why we need it**:
Remove cluster register/unregister calls from test case BeforeEach/AfterEach blocks.
Register clusters once in federation-up.sh


**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #40768

**Special notes for your reviewer**:

**Release note**: `NONE`

cc: @madhusudancs @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 40175, 41107, 41111, 40893, 40919)

remove second CA used for kubelet auth in favor of webhook auth

partial fixes upgrade test.

Automatic merge from submit-queue (batch tested with PRs 40175, 41107, 41111, 40893, 40919)

Default the version to the information in federation versions file if $KUBERNETES_RELEASE isn't set.

Also, slightly unrelated fix: copy the output from the build container
to the host filesystem while building hyperkube image. The recent change
in the build scripts has caused the binaries to be not copied to the
required locations. It must be explicitly copied by calling the build copy
function.

cc @kubernetes/sig-federation-pr-reviews @perotinus @csbell @nikhiljindal

Automatic merge from submit-queue (batch tested with PRs 40175, 41107, 41111, 40893, 40919)

Node E2E: Remove angle brackets in the test name.

Ref https://github.com/kubernetes/test-infra/issues/1827#issuecomment-278187834.

Remove the angle brackets in the test name.

/cc @krzyzacy @mtaufen

Automatic merge from submit-queue (batch tested with PRs 40175, 41107, 41111, 40893, 40919)

kubeadm: skip integration tests if kubeadm-cmd-skip flag passed

Will skip integration tests for token generation if it can't find a file by the given --kubeadm-path or default value.

**What this PR does / why we need it**: Tests would fail if just running `go test` in the dir because it expects to have more values. This won't change the behavior of `make test-cmd` which gets run here:
https://github.com/kubernetes/kubernetes/blob/master/Makefile#L258

**Which issue this PR fixes**: fixes #40155 

**Special notes for your reviewer**: /cc @pires @pipejakob @liggitt 

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 38796, 40823, 40756, 41083, 41105)

Let ReadLogs return when there is a read error.

Fixes a bug in kuberuntime log.

Today, @yujuhong found that once we cancel `kubectl logs -f` with `Ctrl+C`, kuberuntime will keep complaining:
```
27939 kuberuntime_logs.go:192] Failed with err write tcp 10.240.0.4:10250->10.240.0.2:53913: write: broken pipe when writing log for log file "/var/log/pods/5bb76510-ed71-11e6-ad02-42010af00002/busybox_0.log": &{timestamp:{sec:63622095387 nsec:625309193 loc:0x484c440} stream:stdout log:[84 117 101 32 70 101 98 32 32 55 32 50 48 58 49 54 58 50 55 32 85 84 67 32 50 48 49 55 10]}
```

This is because kuberuntime keeps writing to the connection even though it is already closed. Actually, kuberuntime should return and report error whenever there is a writing error.

Ref the [docker code](https://github.com/docker/docker/blob/3a4ae1f661ac9c542d16f9086dd9b6ab21ef2666/pkg/stdcopy/stdcopy.go#L159-L167)

I'm still creating the cluster and verifying this fix. Will post the result here after that.

/cc @yujuhong @kubernetes/sig-node-bugs

Automatic merge from submit-queue (batch tested with PRs 38796, 40823, 40756, 41083, 41105)

Add more options to the RecommendedOptions struct.

Builds on https://github.com/kubernetes/kubernetes/pull/41028

Adds `AuditOptions` to the `RecommendedOptions`

@sttts @kubernetes/sig-api-machinery-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 38796, 40823, 40756, 41083, 41105)

e2e tests for vSphere cloud provider

**What this PR does / why we need it**:
This PR contains changes for existing e2e volume provisioning test cases for running on vsphere cloud provider.

**Following is the summary of changes made in  existing e2e test cases**

**Added test/e2e/persistent_volumes-vsphere.go**
- This test verifies deleting a PVC before the pod does not cause pod deletion to fail on PD detach and deleting the PV before the pod does not cause pod deletion to fail on PD detach.

**test/e2e/volume_provisioning.go**
- This test creates a StorageClass and claim with dynamic provisioning and alpha dynamic provisioning annotations and verifies that required volumes are getting created. Test also verifies that created volume is readable and retaining data.
- Added vsphere as supported cloud provider. Also set pluginName to "kubernetes.io/vsphere-volume" for vsphere cloud provider.

**test/e2e/volumes.go**
- Added test spec for vsphere
-  This test creates requested volume, mount it on the pod, write some random content at /opt/0/index.html and verifies file contents are perfect to make sure we don't see the content from previous test runs.
- This test also passes "1234" as fsGroup to mount volume and verifies fsGroup is set correctly.

**added  test/e2e/vsphere_utils.go** 
- Added function verifyVSphereDiskAttached - Verify the persistent disk attached to the node.
- Added function waitForVSphereDiskToDetach - Wait until vsphere vmdk is deteched from the given node or time out after 5 minutes
- Added getVSpherePersistentVolumeSpec - create vsphere volume spec with given VMDK volume path, Reclaim Policy and labels
- Added getVSpherePersistentVolumeClaimSpec - get vsphere persistent volume spec with given selector labels
- createVSphereVolume - function to create vmdk volume

**Following is the summary of new e2e tests added with this PR**

**test/e2e/vsphere_volume_placement.go**
- contains volume placement tests using node label selector
-  Test Back-to-back pod creation/deletion with the same volume source on the same worker node
- Test Back-to-back pod creation/deletion with the same volume source attach/detach to different worker nodes

**test/e2e/pv_reclaimpolicy.go**
- contains tests for PV/PVC - Reclaiming Policy
- Test verifies persistent volume should be deleted when reclaimPolicy on the PV is set to delete and associated claim is deleted
- Test also verified that persistent volume should be retained when reclaimPolicy on the PV is set to retain and associated claim is deleted 

**test/e2e/pvc_label_selector.go** 
- This is function test for Selector-Label Volume Binding Feature.
- Verify volume with the matching label is bounded with the PVC.

Other changes
Updated  pkg/cloudprovider/providers/vsphere/BUILD  and test/e2e/BUILD 


**Which issue this PR fixes** *
fixes # 41087

**Special notes for your reviewer**:
Updated tests were executed on kubernetes v1.4.8 release on vsphere.
Test steps are provided in comments 


@kerneltime @BaluDontu

Automatic merge from submit-queue (batch tested with PRs 38796, 40823, 40756, 41083, 41105)

Add unit tests for interactive edit command

Before updating edit to use unstructured objects and use generic JSON patching, we need better test coverage of the existing paths. This adds unit tests for the interactive edit scenarios. 

This PR adds:
* Simple framework for recording tests for interactive edit:
  * record.go is a tiny test server that records editor and API inputs as test expectations, and editor and API outputs as playback stubs
  * record_editor.sh is a shell script that sends the before/after of an interactive `vi` edit to the test server
  * record_testcase.sh (see README) starts up the test server, sets up a kubeconfig to proxy to the test server, sets EDITOR to invoke record_editor.sh, then opens a shell that lets you use `kubectl edit` normally
* Adds test cases for the following scenarios:
  - [x] no-op edit (open and close without making changes)
  - [x] try to edit a missing object
  - [x] edit single item successfully
  - [x] edit list of items successfully
  - [x] edit a single item, submit with an error, re-edit, submit fixed successfully
  - [x] edit list of items, submit some with errors and some good, re-edit errors, submit fixed
  - [x] edit trying to change immutable things like name/version/kind, ensure preconditions prevent submission
  - [x] edit in "create mode" successfully (`kubectl create -f ... --edit`)
  - [x] edit in "create mode" introducing errors (`kubectl create -f ... --edit`)
* Fixes a bug with edit printing errors to stdout (caught when testing stdout/stderr against expected output)

Follow-ups:
- [ ] clean up edit code path
- [ ] switch edit to use unstructured objects
- [ ] make edit fall back to jsonmerge for objects without registered go structs (TPR, unknown versions of pods, etc)
- [ ] add tests:
  - [ ] edit TPR
  - [ ] edit mix of TPR and known objects
  - [ ] edit known object with extra field from server
  - [ ] edit known object with new version from server

Automatic merge from submit-queue (batch tested with PRs 38796, 40823, 40756, 41083, 41105)

kubelet/network-cni-plugin: modify the log's info

**What this PR does / why we need it**:

  Checking the startup logs of kubelet, i can always find a error like this:
    "E1215 10:19:24.891724    2752 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d"
 
 It will appears, neither i use cni network-plugin or not. 
After analysis codes, i thought it should be a warn log, because it will not produce any actions like as exit or abort, and just ignored when not any valid plugins exit.

 thank you!

Automatic merge from submit-queue (batch tested with PRs 41061, 40888, 40664, 41020, 41085)

move --runtime-config to kubeapiserver

`--runtime-config` is only useful if you have a lot of API groups in one server.  If you have a single API group in your server (the vast majority of aggregated API servers), then the flag is unneeded and relatively complex.  This moves it to closer to point of use.

@sttts