Automatic merge from submit-queue

Adding cascading deletion support to more federation controllers

Ref #33612

Adding cascading deletion support for federated daemonsets and ingress.
The code is same as that for namespaces. Just ensuring that DeletionHelper functions are called at right places in these controllers.
e2e tests coming up in another PR.

cc @kubernetes/sig-cluster-federation @caesarxuchao @madhusudancs @mwielgus


```release-note
federation: Adding support for DeleteOptions.OrphanDependents for federated daemonsets and ingresses. Setting it to false while deleting a federated daemonset or ingress also deletes the corresponding resource from all registered clusters.
```

Automatic merge from submit-queue

Enable NFS and GlusterFS tests in both node and cluster e2e tests

This PR is to enable NFS and GlusterFS tests on both node and cluster
e2e tests.

It also change the code to use ExecCommandInPod instead of kubectl since
node does not have kubectl available

Automatic merge from submit-queue

Use indirect streaming path for remote CRI shim

Last step for https://github.com/kubernetes/kubernetes/issues/29579

- Wire through the remote indirect streaming methods in the docker remote shim
- Add the docker streaming server as a handler at `<node>:10250/cri/{exec,attach,portforward}`
- Disable legacy streaming for dockershim

Note: This requires PR https://github.com/kubernetes/kubernetes/pull/34987 to work.

Tested manually on an E2E cluster.

/cc @euank @feiskyer @kubernetes/sig-node

Automatic merge from submit-queue

Use new fluentd-gcp image version

In #35618 we used new version of fluentd agent, which includes new version of jeamalloc, allowing us to use it.

Additionally, we came up with a hacky way to encourage Ruby GC to be invoked more often by using RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR variable.

@piosz

Automatic merge from submit-queue

support kubectl apply --force

Support `kubectl apply --force` which is first delete the resource and then re-apply the resource, when the patch fails.

Fixes: #16569

Automatic merge from submit-queue

Add authz to psp admission

Add authz integration to PSP admission to enable granting access to use specific PSPs on a per-user and per-service account basis.  This allows an administrator to use multiple policies in a cluster that grant different levels of access for different types of users.

Builds on https://github.com/kubernetes/kubernetes/pull/32555.  Second commit adds authz check to matching policy function in psp admission.

@deads2k @sttts @timstclair

Automatic merge from submit-queue

Better messaging for missing volume binaries on host

**What this PR does / why we need it**:
When mount binaries are not present on a host, the error returned is a generic one.
This change is to check the mount binaries before the mount and return a user-friendly error message.

This change is specific to GCI and the flag is experimental now.

https://github.com/kubernetes/kubernetes/issues/36098

**Release note**:
Introduces a flag `check-node-capabilities-before-mount` which if set, enables a check (`CanMount()`) prior to mount operations to verify that the required components (binaries, etc.) to mount the volume are available on the underlying node. If the check is enabled and `CanMount()` returns an error, the mount operation fails. Implements the `CanMount()` check for NFS.















Sample output post change :


rkouj@rkouj0:~/go/src/k8s.io/kubernetes$ kubectl describe pods
Name:		sleepyrc-fzhyl
Namespace:	default
Node:		e2e-test-rkouj-minion-group-oxxa/10.240.0.3
Start Time:	Mon, 07 Nov 2016 21:28:36 -0800
Labels:		name=sleepy
Status:		Pending
IP:		
Controllers:	ReplicationController/sleepyrc
Containers:
  sleepycontainer1:
    Container ID:	
    Image:		gcr.io/google_containers/busybox
    Image ID:		
    Port:		
    Command:
      sleep
      6000
    QoS Tier:
      cpu:	Burstable
      memory:	BestEffort
    Requests:
      cpu:		100m
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Environment Variables:
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	False 
  PodScheduled 	True 
Volumes:
  data:
    Type:	NFS (an NFS mount that lasts the lifetime of a pod)
    Server:	127.0.0.1
    Path:	/export
    ReadOnly:	false
  default-token-d13tj:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-d13tj
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----						-------------	--------	------		-------
  7s		7s		1	{default-scheduler }						Normal		Scheduled	Successfully assigned sleepyrc-fzhyl to e2e-test-rkouj-minion-group-oxxa
  6s		3s		4	{kubelet e2e-test-rkouj-minion-group-oxxa}			Warning		FailedMount	Unable to mount volume kubernetes.io/nfs/32c7ef16-a574-11e6-813d-42010af00002-data (spec.Name: data) on pod sleepyrc-fzhyl (UID: 32c7ef16-a574-11e6-813d-42010af00002). Verify that your node machine has the required components before attempting to mount this volume type. Required binary /sbin/mount.nfs is missing

Automatic merge from submit-queue

Get kernel logs with timestamps

Automatic merge from submit-queue

Implement external provisioning proposal

In other words, add "provisioned-by" annotation to all PVCs that should be provisioned dynamically.

Most of the changes are actually in tests.

@kubernetes/sig-storage

Automatic merge from submit-queue

Fix startup script bug in kibana image

Big thanks to @lhopki01 for noticing this!

As mention in discussion in https://github.com/kubernetes/kubernetes/pull/36103 current image crashes if we don't want to work behind proxy because of string interpolation in bash.

@piosz

Automatic merge from submit-queue

Add rescheduler logs to the fluentd-elasticsearch configuration

Same as https://github.com/kubernetes/kubernetes/pull/36359 for elasticsearch plugin

@piosz

Automatic merge from submit-queue

Support for PodDisruptionBudget in Kubectl

Based on:

https://github.com/kubernetes/kubernetes/pull/35287

cc: @davidopp @soltysh @wojtek-t

Without the timestamps, the log is not very useful.

Automatic merge from submit-queue

implement kubectl procelain csr commands

cc @gtank

ref #30163

Automatic merge from submit-queue

Fixes token_found bug in addon manager

From #35832.

Above PR exposed addon manager's logs on Jenkins, found below error on the gce e2e test artifacts:
```
Error from server: serviceaccounts "default" not found
error executing template "{{with index .secrets 0}}{{.name}}{{end}}": template: output:1:7: executing "output" at <index .secrets 0>: error calling index: index of untyped nil
== default service account in the kube-system namespace has token Error executing template: template: output:1:7: executing "output" at <index .secrets 0>: error calling index: index of untyped nil. Printing more information for debugging the template:
	template was:
		{{with index .secrets 0}}{{.name}}{{end}}
	raw data was:
		{"kind":"ServiceAccount","apiVersion":"v1","metadata":{"name":"default","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/serviceaccounts/default","uid":"de3f2f85-9d6a-11e6-9df3-42010af00002","resourceVersion":"48","creationTimestamp":"2016-10-29T00:01:40Z"}}
	object given to template engine was:
		map[apiVersion:v1 metadata:map[selfLink:/api/v1/namespaces/kube-system/serviceaccounts/default uid:de3f2f85-9d6a-11e6-9df3-42010af00002 resourceVersion:48 creationTimestamp:2016-10-29T00:01:40Z name:default namespace:kube-system] kind:ServiceAccount] ==
```

Seems like the script failed to retrieve service token at the first time and mistakenly used the error message as the token content. Fixes by replacing `|| true` with if condition.

This PR is to enable NFS and GlusterFS tests on both node and cluster
e2e tests for gci and containervm distro.

It also change the code to use ExecCommandInPod instead of kubectl since
node does not have kubectl available

Automatic merge from submit-queue

add nodes role to RBAC bootstrap policy

Add a nodes role.  

@sttts @pweil-

Disabling flaky federation unit tests

Automatic merge from submit-queue

Fixing script to bring up federation control plane

Fixes https://github.com/kubernetes/kubernetes/issues/36287

Adding a wait to check if load balancer status is set before checking the ingress field.

cc @kubernetes/sig-cluster-federation

Automatic merge from submit-queue

Fix build break

Problem introduced in #31996

Fixes #36454

Automatic merge from submit-queue

Use generation in pod disruption budget

Fixes #35324

Previously it was possible to use allowedDirsruptions calculated for the previous spec with the current spec. With generation check API servers always make sure that allowedDisruptions were calculated for the current spec. 

At the same time I set the registry policy to only accept updates if the version based on which the update was made matches to the current version in etcd. That ensures that parallel eviction executions don't use the same allowed disruption.

cc: @davidopp @kargakis @wojtek-t

Automatic merge from submit-queue

Increase max-requests-inflight in large clusters

Fix #35402

Automatic merge from submit-queue

Increase initialization timeout for podStore

Fix #33839 - see https://github.com/kubernetes/kubernetes/issues/33839#issuecomment-259442714 for more details.

Automatic merge from submit-queue

Use the correct time field to estimate progress in deployments

Fixes https://github.com/kubernetes/kubernetes/issues/36427

@kubernetes/deployment

Automatic merge from submit-queue

Added e2e test for HA master replicas in different zones.

Automatic merge from submit-queue

Updated test_owners.csv

@kargakis you asked for it

Automatic merge from submit-queue

controller: fix panic in deployments

Fixes https://github.com/kubernetes/kubernetes/issues/36488

@kubernetes/deployment

Automatic merge from submit-queue

extensions: invalidate progress deadline less than minreadyseconds

@kubernetes/deployment ptal

Automatic merge from submit-queue

Use available informers in quota replenishment

more iteration on the goal to use informers where available in quota system.  this time adding persistent volume claims so the same informer is used here and https://github.com/kubernetes/kubernetes/pull/36316