• k8s-merge-robot's avatar
    Merge pull request #26100 from rootfs/fix-pd · db74335f
    k8s-merge-robot authored
    Automatic merge from submit-queue
    
    in e2e test, when kubectl exec fails to find the container to run a command, it should retry
    
    fix #26076 
    Without retrying upon "container not found" error, `Pod Disks` test failed on the following error:
    ```console
    [k8s.io] Pod Disks 
      should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession [Slow]
      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:271
    [BeforeEach] [k8s.io] Pod Disks
      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:108
    STEP: Creating a kubernetes client
    May 23 19:18:02.254: INFO: >>> TestContext.KubeConfig: /root/.kube/config
    
    STEP: Building a namespace api object
    STEP: Waiting for a default service account to be provisioned in namespace
    [BeforeEach] [k8s.io] Pod Disks
      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:69
    [It] should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession [Slow]
      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:271
    STEP: creating PD1
    May 23 19:18:06.678: INFO: Successfully created a new PD: "rootfs-e2e-11dd5f5b-211b-11e6-a3ff-b8ca3a62792c".
    STEP: creating PD2
    May 23 19:18:11.216: INFO: Successfully created a new PD: "rootfs-e2e-141f062d-211b-11e6-a3ff-b8ca3a62792c".
    May 23 19:18:11.216: INFO: PD Read/Writer Iteration #0
    STEP: submitting host0Pod to kubernetes
    W0523 19:18:11.279910    4984 request.go:347] Field selector: v1 - pods - metadata.name - pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c: need to check if this is versioned correctly.
    STEP: writing a file in the container
    May 23 19:18:39.088: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- /bin/sh -c echo '1394466581702052925' > '/testpd1/tracker0''
    May 23 19:18:40.250: INFO: Wrote value: "1394466581702052925" to PD1 ("rootfs-e2e-11dd5f5b-211b-11e6-a3ff-b8ca3a62792c") from pod "pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c" container "mycontainer"
    STEP: writing a file in the container
    May 23 19:18:40.251: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- /bin/sh -c echo '1740704063962701662' > '/testpd2/tracker0''
    May 23 19:18:41.433: INFO: Wrote value: "1740704063962701662" to PD2 ("rootfs-e2e-141f062d-211b-11e6-a3ff-b8ca3a62792c") from pod "pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c" container "mycontainer"
    STEP: reading a file in the container
    May 23 19:18:41.433: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- cat /testpd1/tracker0'
    May 23 19:18:42.585: INFO: Read file "/testpd1/tracker0" with content: 1394466581702052925
    
    STEP: reading a file in the container
    May 23 19:18:42.585: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- cat /testpd2/tracker0'
    May 23 19:18:43.779: INFO: Read file "/testpd2/tracker0" with content: 1740704063962701662
    
    STEP: deleting host0Pod
    May 23 19:18:44.048: INFO: PD Read/Writer Iteration #1
    STEP: submitting host0Pod to kubernetes
    W0523 19:18:44.132475    4984 request.go:347] Field selector: v1 - pods - metadata.name - pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c: need to check if this is versioned correctly.
    STEP: reading a file in the container
    May 23 19:18:45.186: INFO: Running '/srv/dev/kubernetes/_output/dockerized/bin/linux/amd64/kubectl kubectl --server=https://130.211.199.187 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-pod-disks-3t3g8 pd-test-16d3653c-211b-11e6-a3ff-b8ca3a62792c -c=mycontainer -- cat /testpd1/tracker0'
    May 23 19:18:46.290: INFO: error running kubectl exec to read file: exit status 1
    stdout=
    stderr=error: error executing remote command: error executing command in container: container not found ("mycontainer")
    )
    May 23 19:18:46.290: INFO: Error reading file: exit status 1
    May 23 19:18:46.290: INFO: Unexpected error occurred: exit status 1
    ```
    Now I've run this fix on e2e pd test 5 times and no longer see any failure
    db74335f
Name
Last commit
Last update
.github Loading commit data...
Godeps Loading commit data...
api/swagger-spec Loading commit data...
build Loading commit data...
cluster Loading commit data...
cmd Loading commit data...
contrib Loading commit data...
docs Loading commit data...
examples Loading commit data...
federation Loading commit data...
hack Loading commit data...
hooks Loading commit data...
logo Loading commit data...
pkg Loading commit data...
plugin Loading commit data...
release Loading commit data...
test Loading commit data...
third_party Loading commit data...
vendor Loading commit data...
www Loading commit data...
.generated_docs Loading commit data...
.gitignore Loading commit data...
.travis.yml Loading commit data...
CHANGELOG.md Loading commit data...
CONTRIB.md Loading commit data...
CONTRIBUTING.md Loading commit data...
DESIGN.md Loading commit data...
LICENSE Loading commit data...
Makefile Loading commit data...
README.md Loading commit data...
Vagrantfile Loading commit data...
code-of-conduct.md Loading commit data...