Automatic merge from submit-queue

Ensure that init containers are preserved during pruning

Pods with multiple init containers were getting the wrong containers
pruned. Fix an error message and add a test.

Fixes #26131

Automatic merge from submit-queue

Add some extra checking in the tests to prevent flakes.

Attempts to fix https://github.com/kubernetes/kubernetes/issues/25967

The hypothesis is that somehow waitTest() catches an idle that occurs before all changes have been applied.  This will block until the expected number of changes have arrived.

e2e: fix tests that are broken because of the image prepull pod

Automatic merge from submit-queue

use monotonic now in TestDelNode

Fixes https://github.com/kubernetes/kubernetes/issues/24971.

Briefly, the rate_limited_queue uses a `container/heap` to store values, and use this data structure to ensure we can always fetch the value with the minimum `processAt`. However, in some extreme condition, the continuous call to `time.Now()` would get the same value, which causes some unpredictable order in the queue, this fix uses a monotonic `now()` to avoid that.

@smarterclayton please take a look.

Many tests expect all kube-system pods to be running and ready. The newly
added image prepull add-on pod can in the "succeeded" state. This commit fixes
the tests to allow kube-system pods to be succeeded.

Automatic merge from submit-queue

rkt: Support alternate stage1's via annotation

This provides a basic implementation for setting a stage1 on a per-pod
basis via an annotation.

This provides a basic implementation for setting a stage1 on a per-pod
basis via an annotation. See discussion here for how this approach was arrived at: https://github.com/kubernetes/kubernetes/issues/23944#issuecomment-212653776

It's possible this feature should be gated behind additional knobs, such
as a kubelet flag to filter allowed stage1s, or a check akin to what
priviliged gets in the apiserver.
Currently, it checks `AllowPrivileged`, as a means to let people disable
this feature, though overloading it as stage1 and privileged isn't
ideal.

Fixes #23944

Testing done (note, unfortunately done with some additional ./cluster changes merged in):

```
$ cat examples/stage1-fly/fly-me-to-the-moon.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: exit
  name: exit-fast
  annotations: {"rkt.alpha.kubernetes.io/stage1-name-override": "coreos.com/rkt/stage1-fly:1.3.0"}
spec:
  restartPolicy: Never
  containers:
    - name: exit
      image: busybox
      command: ["sh", "-c", "ps aux"]
$ kubectl create -f examples/stage1-fly
$ ssh core@minion systemctl status -l --no-pager k8s_2f169b2e-c32a-49e9-a5fb-29ae1f6b4783.service
...
failed
...
May 04 23:33:03 minion rkt[2525]: stage0: error writing /etc/rkt-resolv.conf: open /var/lib/rkt/pods/run/2f169b2e-c32a-49e9-a5fb-29ae1f6b4783/stage1/rootfs/etc/rkt-resolv.conf: no such file or directory
...
# Restart kubelet with allow-privileged=false
$ kubectl create -f examples/stage1-fly
$ kubectl describe exit-fast
...
  1m		19s		5	{kubelet euank-e2e-test-minion-dv3u}	spec.containers{exit}	Warning		Failed		Failed to create rkt container with error: cannot make "exit-fast_default(17050ce9-1252-11e6-a52a-42010af00002)": running a custom stage1 requires a privileged security context
....
```

Note as well that the "success" here is rkt spitting out an [error message](https://github.com/coreos/rkt/issues/2141) which indicates that the right stage1 was being used at least.

cc @yifan-gu @aaronlevy

Automatic merge from submit-queue

reduce conflict retries

Eliminates quota admission conflicts due to latent caches on the same API server.

@derekwaynecarr 

Merge pull request #24179 from aveshagarwal/master-downward-api-resources-limits-requests-implementation

Automatic merge from submit-queue

Downward API implementation for resources limits and requests

This is an implementation of Downward API for resources limits and requests, and it works with environment variables and volume plugin.

This is based on proposal https://github.com/kubernetes/kubernetes/pull/24051. This implementation follows API with magic keys approach as discussed in the proposal.

@kubernetes/rh-cluster-infra

<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24179)
<!-- Reviewable:end -->

Automatic merge from submit-queue

Speed up load test

Automatic merge from submit-queue

Node e2e junit test artifacts


[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

- Add junit test reported
- Write etcd.log, kubelet.log and kube-apiserver.log to files instead of stdout
- Scp artifacts to the jenkins WORKSPACE

Fixes #25966

Automatic merge from submit-queue

Bumped Cluster Autoscaler to v0.1.0

[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

Automatic merge from submit-queue

Prepull images in e2e 

Quick and dirty image puller because the SQ stalled multiple times just *today* on image pull flake (https://github.com/kubernetes/kubernetes/issues/25277).
@kubernetes/sig-node @kubernetes/sig-testing wdyt?

Automatic merge from submit-queue

e2e-runner: Drop Trusty support in favor of GCI

Now that GCI (Google Container-VM Image) is out, we will start running e2e tests
with it instead of Ubuntu Trusty. This change updates `e2e-runner.sh` to replace
the Trusty related logic with GCI.

_Note that this change has to go in the same time as https://github.com/kubernetes/test-infra/pull/54_

@spxtr Can you review?

cc/ @andyzheng0831 @kubernetes/goog-image 

Automatic merge from submit-queue

Wait for RC to stabilize in rolling-update e2e

https://github.com/kubernetes/kubernetes/issues/25170

Increase expected fluent CPU usage

Replace containervm with GCI as default master image for GCE clusters

Automatic merge from submit-queue

Fix node e2e log print

Fix bad log print:
```
W0524 18:20:41.679642    2069 container_list.go:56] Could not pre-pull image %s %v output:  %sgcr.io/google_containers/pause-amd64:3.0exit status 1 [69 114 114 111 114 32 114 101 115 112 111 110 115 101 32 102 114 111 109 32 100 97 101 109 111 110 58 32 71 101 116 32 104 116 116 112 115 58 47 47 103 99 114 46 105 111 47 118 50 47 103 111 111 103 108 101 95 99 111 110 116 97 105 110 101 114 115 47 112 97 117 115 101 45 97 109 100 54 52 47 109 97 110 105 102 101 115 116 115 47 51 46 48 58 32 71 101 116 32 104 116 116 112 115 58 47 47 103 99 114 46 105 111 47 118 50 47 116 111 107 101 110 63 115 99 111 112 101 61 114 101 112 111 115 105 116 111 114 121 37 51 65 103 111 111 103 108 101 95 99 111 110 116 97 105 110 101 114 115 37 50 70 112 97 117 115 101 45 97 109 100 54 52 37 51 65 112 117 108 108 38 115 101 114 118 105 99 101 61 103 99 114 46 105 111 58 32 110 101 116 47 104 116 116 112 58 32 114 101 113 117 101 115 116 32 99 97 110 99 101 108 101 100 32 119 104 105 108 101 32 119 97 105 116 105 110 103 32 102 111 114 32 99 111 110 110 101 99 116 105 111 110 32 40 67 108 105 101 110 116 46 84 105 109 101 111 117 116 32 101 120 99 101 101 100 101 100 32 119 104 105 108 101 32 97 119 97 105 116 105 110 103 32 104 101 97 100 101 114 115 41 10]
```

/cc @pwittrock 

[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

- Add junit test reported
- Write etcd.log, kubelet.log and kube-apiserver.log to files instead of stdout
- Scp artifacts to the jenkins WORKSPACE

Fixes #25966

Automatic merge from submit-queue

Renaming federated-apiserver to federation-apiserver

Its not a "federated" apiserver. Its an apiserver for federation.

The corresponding controller manager is also called federation-controller-manager.

If we are going to rename, it should be done in 1.3 before people write scripts referencing the binary name.

Automatic merge from submit-queue

in e2e test, when kubectl exec fails to find the container to run a command, it should retry

fix #26076 
Without retrying upon "container not found" error, `Pod Disks` test failed on the following error:
```console
[k8s.io] Pod Disks 
  should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession [Slow]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:271
[BeforeEach] [k8s.io] Pod Disks
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:108
STEP: Creating a kubernetes client
May 23 19:18:02.254: INFO: >>> TestContext.KubeConfig: /root/.kube/config

STEP: Building a namespace api object
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [k8s.io] Pod Disks
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:69
[It] should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession [Slow]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:271
STEP: creating PD1
May 23 19:18:06.678: INFO: Successfully created a new PD: "rootfs-e2e-11dd5f5b-211b-11e6-a3ff-b8ca3a62792c".
STEP: creating PD2
May 23 19:18:11.216: INFO: Successfully created a new PD: "rootfs-e2e-141f062d-211b-11e6-a3ff-b8ca3a62792c".
May 23 19:18:11.216: INFO: PD Read/Writer Iteration #0
STEP: submitting host0Pod to kubernetes
W0523 19:18:11.279910    4984 request.go:347] Field selector: v1 - pods - metadata.name - pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c: need to check if this is versioned correctly.
STEP: writing a file in the container
May 23 19:18:39.088: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- /bin/sh -c echo '1394466581702052925' > '/testpd1/tracker0''
May 23 19:18:40.250: INFO: Wrote value: "1394466581702052925" to PD1 ("rootfs-e2e-11dd5f5b-211b-11e6-a3ff-b8ca3a62792c") from pod "pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c" container "mycontainer"
STEP: writing a file in the container
May 23 19:18:40.251: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- /bin/sh -c echo '1740704063962701662' > '/testpd2/tracker0''
May 23 19:18:41.433: INFO: Wrote value: "1740704063962701662" to PD2 ("rootfs-e2e-141f062d-211b-11e6-a3ff-b8ca3a62792c") from pod "pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c" container "mycontainer"
STEP: reading a file in the container
May 23 19:18:41.433: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- cat /testpd1/tracker0'
May 23 19:18:42.585: INFO: Read file "/testpd1/tracker0" with content: 1394466581702052925

STEP: reading a file in the container
May 23 19:18:42.585: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- cat /testpd2/tracker0'
May 23 19:18:43.779: INFO: Read file "/testpd2/tracker0" with content: 1740704063962701662

STEP: deleting host0Pod
May 23 19:18:44.048: INFO: PD Read/Writer Iteration #1
STEP: submitting host0Pod to kubernetes
W0523 19:18:44.132475    4984 request.go:347] Field selector: v1 - pods - metadata.name - pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c: need to check if this is versioned correctly.
STEP: reading a file in the container
May 23 19:18:45.186: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- cat /testpd1/tracker0'
May 23 19:18:46.290: INFO: error running kubectl exec to read file: exit status 1
stdout=
stderr=error: error executing remote command: error executing command in container: container not found ("mycontainer")
)
May 23 19:18:46.290: INFO: Error reading file: exit status 1
May 23 19:18:46.290: INFO: Unexpected error occurred: exit status 1
```
Now I've run this fix on e2e pd test 5 times and no longer see any failure

Automatic merge from submit-queue

Add a 'kubectl clusterinfo dump' option

Ref: #3500 

@bgrant0607 @smarterclayton @jszczepkowski 

Usage:
```
  # Dump current cluster state to stdout
  kubectl clusterinfo dump
  
  # Dump current cluster state to /tmp
  kubectl clusterinfo dump --output-directory=/tmp
  
  # Dump all namespaces to stdout
  kubectl clusterinfo dump --all-namespaces
  
  # Dump a set of namespaces to /tmp
  kubectl clusterinfo dump --namespaces default,kube-system --output-directory=/tmp
```

<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/20672)
<!-- Reviewable:end -->

docker: don't set timeout for image pulling requests

Don't try executing bash variable in assignment.

Removing unreferenced code in github/com/emicklei

Pods with multiple init containers were getting the wrong containers
pruned. Fix an error message and add a test.

Log requests to GCE

Automatic merge from submit-queue

plumb Update resthandler to allow old/new comparisons in admission

Rework how updated objects are passed to rest storage Update methods (first pass at https://github.com/kubernetes/kubernetes/pull/23928#discussion_r61444342)

* allows centralizing precondition checks (uid and resourceVersion)
* allows admission to have the old and new objects on patch/update operations (sets us up for field level authorization, differential quota updates, etc)
* allows patch operations to avoid double-GETting the object to apply the patch

Overview of important changes:
* pkg/api/rest/rest.go
  * changes `rest.Update` interface to give rest storage an `UpdatedObjectInfo` interface instead of the object directly. To get the updated object, the storage must call `UpdatedObject()`, passing in the current object
* pkg/api/rest/update.go
  * provides a default `UpdatedObjectInfo` impl
  * passes a copy of the updated object through any provided transforming functions and returns it when asked
  * builds UID preconditions from the updated object if they can be extracted
* pkg/apiserver/resthandler.go
  * Reworks update and patch operations to give old objects to admission
* pkg/registry/generic/registry/store.go
  * Calls `UpdatedObject()` inside `GuaranteedUpdate` so it can provide the old object

Todo:
- [x] Update rest.Update interface:
  * Given the name of the object being updated
  * To get the updated object data, the rest storage must pass the current object (fetched using the name) to an `UpdatedObject(ctx, oldObject) (newObject, error)` func. This is typically done inside a `GuaranteedUpdate` call.
- [x] Add old object to admission attributes interface
- [x] Update resthandler Update to move admission into the UpdatedObject() call
- [x] Update resthandler Patch to move the patch application and admission into the UpdatedObject() call
- [x] Add resttest tests to make sure oldObj is correctly passed to UpdatedObject(), and errors propagate back up

Follow-up:
* populate oldObject in admission for delete operations?
* update quota plugin to use `GetOldObject()` in admission attributes
* admission plugin to gate ownerReference modification on delete permission
* Decide how to handle preconditions (does that belong in the storage layer or in the resthander layer?)

Automatic merge from submit-queue

Make node-instance-group base names unique to prevent collisions

We create multiple IGMs for >1000 Node clusters. When we have a conflict on base name IGMs will fight over ownership of the VM that happen to have the name belonging to multiple IGMs.

This change will increase reliability of starting big clusters.

cc @wojtek-t @alex-mohr @roberthbailey @mikedanese 

Signed-off-by: Huamin Chen <hchen@redhat.com>