Skip to content

K
k3s
Commit 0fbb246c authored by Peter Hornyack's avatar Peter Hornyack
Browse files
Options

Update GCE Windows README steps for running e2e tests.

parent 583ff363
Hide whitespace changes
Inline Side-by-side
Showing with 60 additions and 89 deletions
cluster/gce/windows/README-GCE-Windows-kube-up.md
View file @ 0fbb246c
...@@ -2,9 +2,9 @@ ...@@ -2,9 +2,9 @@
## IMPORTANT PLEASE NOTE! ## IMPORTANT PLEASE NOTE!
Any time the file structure in the `windows` directory changes, `windows/BUILD` Any time the file structure in the `windows` directory changes, `windows/BUILD`
and `k8s.io/release/lib/releaselib.sh` must be manually updated with the changes. and `k8s.io/release/lib/releaselib.sh` must be manually updated with the
We HIGHLY recommend not changing the file structure, because consumers of changes. We HIGHLY recommend not changing the file structure, because consumers
Kubernetes releases depend on the release structure remaining stable. of Kubernetes releases depend on the release structure remaining stable.
## Bring up the cluster ## Bring up the cluster
...@@ -31,28 +31,32 @@ The most straightforward approach to build those binaries is to run `make ...@@ -31,28 +31,32 @@ The most straightforward approach to build those binaries is to run `make
release`. However, that builds binaries for all supported platforms, and can be release`. However, that builds binaries for all supported platforms, and can be
slow. You can speed up the process by following the instructions below to only slow. You can speed up the process by following the instructions below to only
build the necessary binaries. build the necessary binaries.
```
# Fetch the PR: https://github.com/pjh/kubernetes/pull/43
git remote add pjh https://github.com/pjh/kubernetes
git fetch pjh pull/43/head
# Get the commit hash and cherry-pick the commit to your current branch ```
BUILD_WIN_COMMIT=$(git ls-remote pjh | grep refs/pull/43/head | cut -f 1) # Apply https://github.com/pjh/kubernetes/pull/43 to your tree:
git cherry-pick $BUILD_WIN_COMMIT curl \
https://patch-diff.githubusercontent.com/raw/pjh/kubernetes/pull/43.patch | \
git apply
# Build binaries for both Linux and Windows # Build binaries for both Linux and Windows:
make quick-release make quick-release
``` ```
### 2 Create a Kubernetes cluster ### 2. Create a Kubernetes cluster
You can create a regular Kubernetes cluster or an end-to-end test cluster. You can create a regular Kubernetes cluster or an end-to-end test cluster.
End-to-end test clusters support running the Kubernetes e2e tests and enable
some debugging features such as SSH access on the Windows nodes.
Please make sure you set the environment variables properly following the Please make sure you set the environment variables properly following the
instructions in the previous section. instructions in the previous section.
First, set the following environment variables which are required for First, set the following environment variables which are required for
controlling the number of Linux and Windows nodes in the cluster and for controlling the number of Linux and Windows nodes in the cluster and for
enabling IP aliases (which are required for Windows pod routing): enabling IP aliases (which are required for Windows pod routing). At least one
Linux worker node is required and two are recommended because many default
cluster-addons (e.g., `kube-dns`) need to run on Linux nodes. The master control
plane only runs on Linux.
``` ```
export NUM_NODES=2 # number of Linux nodes export NUM_NODES=2 # number of Linux nodes
...@@ -60,19 +64,9 @@ export NUM_WINDOWS_NODES=2 ...@@ -60,19 +64,9 @@ export NUM_WINDOWS_NODES=2
export KUBE_GCE_ENABLE_IP_ALIASES=true export KUBE_GCE_ENABLE_IP_ALIASES=true
``` ```
If you wish to use `netd` as the CNI plugin for Linux nodes, set these
variables:
```
export KUBE_ENABLE_NETD=true
export KUBE_CUSTOM_NETD_YAML=$(curl -s \
https://raw.githubusercontent.com/GoogleCloudPlatform/netd/master/netd.yaml \
| sed -e 's/^/ /')
```
Now bring up a cluster using one of the following two methods: Now bring up a cluster using one of the following two methods:
#### 2.a Create a regular Kubernetes cluster #### 2a. Create a regular Kubernetes cluster
``` ```
# Invoke kube-up.sh with these environment variables: # Invoke kube-up.sh with these environment variables:
...@@ -87,13 +81,15 @@ To teardown the cluster run: ...@@ -87,13 +81,15 @@ To teardown the cluster run:
PROJECT=${CLOUDSDK_CORE_PROJECT} KUBERNETES_SKIP_CONFIRM=y ./cluster/kube-down.sh PROJECT=${CLOUDSDK_CORE_PROJECT} KUBERNETES_SKIP_CONFIRM=y ./cluster/kube-down.sh
``` ```
#### 2.b Create a Kubernetes end-to-end (E2E) test cluster #### 2b. Create a Kubernetes end-to-end (E2E) test cluster
``` ```
PROJECT=${CLOUDSDK_CORE_PROJECT} go run ./hack/e2e.go -- --up PROJECT=${CLOUDSDK_CORE_PROJECT} go run ./hack/e2e.go -- --up
``` ```
This command, by default, tears down the existing E2E cluster and create a new
one. This command, by default, tears down any existing E2E cluster and creates a new
one. To teardown the cluster run the same command with `--down` instead of
`--up`.
No matter what type of cluster you chose to create, the result should be a No matter what type of cluster you chose to create, the result should be a
Kubernetes cluster with one Linux master node, `NUM_NODES` Linux worker nodes Kubernetes cluster with one Linux master node, `NUM_NODES` Linux worker nodes
...@@ -108,87 +104,62 @@ brought up correctly: ...@@ -108,87 +104,62 @@ brought up correctly:
cluster/gce/windows/smoke-test.sh cluster/gce/windows/smoke-test.sh
``` ```
## Running tests against the cluster Sometimes the first run of the smoke test will fail because it took too long to
pull the Windows test containers. The smoke test will usually pass on the next
These steps are based on attempt.
[kubernetes-sigs/windows-testing](https://github.com/kubernetes-sigs/windows-testing).
* TODO(pjh): use patched `cluster/local/util.sh` from ## Running e2e tests against the cluster
https://github.com/pjh/kubernetes/blob/windows-up/cluster/local/util.sh.
* If necessary run `alias kubectl=client/bin/kubectl` . If you brought up an end-to-end test cluster using the steps above then you can
use the steps below to run K8s e2e tests. These steps are based on
[kubernetes-sigs/windows-testing](https://github.com/kubernetes-sigs/windows-testing).
* Set the following environment variables (these values should make sense if * Build the necessary test binaries. This must be done after every change to
you built your cluster using the kube-up steps above): test code.
``` ```
export KUBE_HOME=$(pwd) make WHAT=test/e2e/e2e.test
export KUBECONFIG=~/.kube/config
export KUBE_MASTER=local
export KUBE_MASTER_NAME=kubernetes-master
export KUBE_MASTER_IP=$(kubectl get node ${KUBE_MASTER_NAME} -o jsonpath='{.status.addresses[?(@.type=="ExternalIP")].address}')
export KUBE_MASTER_URL=https://${KUBE_MASTER_IP}
export KUBE_MASTER_PORT=443
``` ```
* Download the list of Windows e2e tests: * Set necessary environment variables and fetch the `run-e2e.sh` script:
``` ```
curl https://raw.githubusercontent.com/e2e-win/e2e-win-prow-deployment/master/repo-list.txt -o ${KUBE_HOME}/repo-list.yaml export KUBECONFIG=~/.kube/config
export KUBE_TEST_REPO_LIST=${KUBE_HOME}/repo-list.yaml export WORKSPACE=$(pwd)
``` export ARTIFACTS=${WORKSPACE}/e2e-artifacts
* Download and configure the list of tests to exclude:
curl \
https://raw.githubusercontent.com/yujuhong/gce-k8s-windows-testing/master/run-e2e.sh \
-o ${WORKSPACE}/run-e2e.sh
chmod u+x run-e2e.sh
``` ```
curl https://raw.githubusercontent.com/e2e-win/e2e-win-prow-deployment/master/exclude_conformance_test.txt -o ${KUBE_HOME}/exclude_conformance_test.txt
export EXCLUDED_TESTS=$(cat exclude_conformance_test.txt |
tr -d '\r' | # remove Windows carriage returns
tr -s '\n' '|' | # coalesce newlines into |
tr -s ' ' '.' | # coalesce spaces into .
sed -e 's/[]\[()]/\\&/g' | # escape brackets and parentheses
sed -e 's/.$//g') # remove final | added by tr
```
* Taint the Linux nodes so that test pods will not land on them:
``` NOTE: `run-e2e.sh` begins with a 5 minute sleep to wait for container images
export LINUX_NODES=$(kubectl get nodes -l beta.kubernetes.io/os=linux,kubernetes.io/hostname!=${KUBE_MASTER_NAME} -o name) to be pre-pulled. You'll probably want to edit the script and remove this.
export LINUX_NODE_COUNT=$(echo ${LINUX_NODES} | wc -w)
for node in $LINUX_NODES; do
kubectl taint node $node node-under-test=false:NoSchedule
done
```
* Build necessary test binaries: * The canonical arguments for running all Windows e2e tests against a cluster
on GCE can be seen by searching for `--test-cmd-args` in the [test
configuration](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-gcp/sig-gcp-windows.yaml#L78)
for the `ci-kubernetes-e2e-windows-gce` continuous test job. These arguments
should be passed to the `run-e2e` script; escape the ginkgo arguments by
adding quotes around them. For example:
``` ```
make WHAT=test/e2e/e2e.test ./run-e2e.sh --node-os-distro=windows \
--ginkgo.focus="\[Conformance\]|\[NodeConformance\]|\[sig-windows\]" \
--ginkgo.skip="\[LinuxOnly\]|\[Serial\]|\[Feature:.+\]" --minStartupPods=8
``` ```
* Run the tests with flags that point at the "local" (already-running) cluster * Run a single test by setting the ginkgo focus to match your test name; for
and that permit the `NoSchedule` Linux nodes: example, the "DNS should provide DNS for the cluster" test can be run using:
``` ```
export KUBETEST_ARGS="--ginkgo.noColor=true "\ ./run-e2e.sh --node-os-distro=windows \
"--report-dir=${KUBE_HOME}/e2e-reports "\ --ginkgo.focus="provide\sDNS\sfor\sthe\scluster"
"--allowed-not-ready-nodes=${LINUX_NODE_COUNT} "\
"--ginkgo.dryRun=false "\
"--ginkgo.focus=\[Conformance\] "\
"--ginkgo.skip=${EXCLUDED_TESTS}"
go run ${KUBE_HOME}/hack/e2e.go -- --verbose-commands \
--ginkgo-parallel=4 \
--check-version-skew=false --test --provider=local \
--test_args="${KUBETEST_ARGS}" &> ${KUBE_HOME}/conformance.out
``` ```
TODO: copy log files from Windows nodes using some command like: Make sure to always include `--node-os-distro=windows` for testing against
Windows nodes.
```
scp -r -o PreferredAuthentications=keyboard-interactive,password \
-o PubkeyAuthentication=no \
user@kubernetes-minion-windows-group-mk0p:C:\\etc\\kubernetes\\logs \
kubetest-logs/
```
After the test run completes, log files can be found under the `${ARTIFACTS}`
directory.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment