Use rsync to get source into build container

.gitignore

@@ -62,6 +62,9 @@ network_closure.sh
 
 .tags*
 
+# Version file for dockerized build
+.dockerized-kube-version-defs
+
 # Web UI
 /www/master/node_modules/
 /www/master/npm-debug.log

build/BUILD_IMAGE_VERSION

+3

build/README.md

@@ -4,12 +4,12 @@ Building Kubernetes is easy if you take advantage of the containerized build env
 
 ## Requirements
 
-1. Docker, using one of the two following configurations:
+1. Docker, using one of the following configurations:
   1. **Mac OS X** You can either use Docker for Mac or docker-machine. See installation instructions [here](https://docs.docker.com/installation/mac/).
-     **Note**: You will want to set the Docker VM to have at least 3GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)) and do not `make quick-release` from `/tmp/` (See: [#14773]( https://github.com/kubernetes/kubernetes/issues/14773))
-  2. **Linux with local Docker**  Install Docker according to the [instructions](https://docs.docker.com/installation/#installation) for your OS.  The scripts here assume that they are using a local Docker server and that they can "reach around" docker and grab results directly from the file system.
-2. [Python](https://www.python.org)
-3. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/)
+     **Note**: You will want to set the Docker VM to have at least 3GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)).
+  2. **Linux with local Docker**  Install Docker according to the [instructions](https://docs.docker.com/installation/#installation) for your OS.
+  3. **Remote Docker engine** Use a big machine in the cloud to build faster. This is a little trickier so look at the section later on.
+2. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/)
 
 You must install and configure Google Cloud SDK if you want to upload your release to Google Cloud Storage and may safely omit this otherwise.
 
@@ -17,8 +17,6 @@ You must install and configure Google Cloud SDK if you want to upload your relea
 
 While it is possible to build Kubernetes using a local golang installation, we have a build process that runs in a Docker container.  This simplifies initial set up and provides for a very consistent build and test environment.
 
-There is also early support for building Docker "run" containers
-
 ## Key scripts
 
 The following scripts are found in the `build/` directory. Note that all scripts must be run from the Kubernetes root directory.
@@ -29,10 +27,68 @@ The following scripts are found in the `build/` directory. Note that all scripts
   *  `build/run.sh make test`: Run all unit tests
   *  `build/run.sh make test-integration`: Run integration test
   *  `build/run.sh make test-cmd`: Run CLI tests
-* `build/copy-output.sh`: This will copy the contents of `_output/dockerized/bin` from any the Docker container to the local `_output/dockerized/bin`.
-* `build/make-clean.sh`: Clean out the contents of `_output/dockerized`, remove any container images and the data container
-* `build/shell.sh`: Drop into a `bash` shell in a build container with a snapshot of the current repo code.
-* `build/release.sh`: Build everything, test it, and (optionally) upload the results to a GCS bucket.
+* `build/copy-output.sh`: This will copy the contents of `_output/dockerized/bin` from the Docker container to the local `_output/dockerized/bin`. It will also copy out specific file patterns that are generated as part of the build process. This is run automatically as part of `build/run.sh`.
+* `build/make-clean.sh`: Clean out the contents of `_output`, remove any locally built container images and remove the data container.
+* `/build/shell.sh`: Drop into a `bash` shell in a build container with a snapshot of the current repo code.
+
+## Basic Flow
+
+The scripts directly under `build/` are used to build and test.  They will ensure that the `kube-build` Docker image is built (based on `build/build-image/Dockerfile`) and then execute the appropriate command in that container.  These scripts will both ensure that the right data is cached from run to run for incremental builds and will copy the results back out of the container.
+
+The `kube-build` container image is built by first creating a "context" directory in `_output/images/build-image`.  It is done there instead of at the root of the Kubernetes repo to minimize the amount of data we need to package up when building the image.
+
+There are 3 different containers instances that are run from this image.  The first is a "data" container to store all data that needs to persist across to support incremental builds. Next there is an "rsync" container that is used to transfer data in and out to the data container.  Lastly there is a "build" container that is used for actually doing build actions.  The data container persists across runs while the rsync and build containers are deleted after each use.
+
+`rsync` is used transparently behind the scenes to efficiently move data in and and of the container.  This will use an ephemeral port picked by Docker.  You can modify this by setting the `KUBE_RSYNC_PORT` env variable.
+
+All Docker names are suffixed with a hash derived from the file path (to allow concurrent usage on things like CI machines) and a version number.  When the version number changes all state is cleared and clean build is started.  This allows the build infrastructure to be changed and signal to CI systems that old artifacts need to be deleted.
+
+## Proxy Settings
+
+If you are behind a proxy and you are letting these scripts use `docker-machine` to set up your local VM for you on macOS, you need to export proxy settings for kubernetes build, the following environment variables should be defined.
+
+```
+export KUBERNETES_HTTP_PROXY=http://username:password@proxyaddr:proxyport
+export KUBERNETES_HTTPS_PROXY=https://username:password@proxyaddr:proxyport
+```
+
+Optionally, you can specify addresses of no proxy for kubernetes build, for example
+
+```
+export KUBERNETES_NO_PROXY=127.0.0.1
+```
+
+If you are using sudo to make kubernetes build for example make quick-release, you need run `sudo -E make quick-release` to pass the environment variables.
+
+## Really Remote Docker Engine
+
+It is possible to use a Docker Engine that is running remotely (under your desk or in the cloud).  Docker must be configured to connect to that machine and the local rsync port must be forwarded (via SSH or nc) from localhost to the remote machine.
+
+To do this easily with GCE and `docker-machine`, do something like this:
+```
+# Create the remote docker machine on GCE.  This is a pretty beefy machine with SSD disk.
+KUBE_BUILD_VM=k8s-build
+KUBE_BUILD_GCE_PROJECT=<project>
+docker-machine create \
+  --driver=google \
+  --google-project=${KUBE_BUILD_GCE_PROJECT} \
+  --google-zone=us-west1-a \
+  --google-machine-type=n1-standard-8 \
+  --google-disk-size=50 \
+  --google-disk-type=pd-ssd \
+  ${KUBE_BUILD_VM}
+
+# Set up local docker to talk to that machine
+eval $(docker-machine env ${KUBE_BUILD_VM})
+
+# Pin down the port that rsync will be exposed on on the remote machine
+export KUBE_RSYNC_PORT=8370
+
+# forward local 8730 to that machine so that rsync works
+docker-machine ssh ${KUBE_BUILD_VM} -L ${KUBE_RSYNC_PORT}:localhost:8730 -N &
+```
+
+Look at `docker-machine stop`, `docker-machine start` and `docker-machine rm` to manage this VM.
 
 ## Releasing
 
@@ -63,41 +119,6 @@ Env Variable | Default | Description
 `KUBE_GCS_NO_CACHING` | `y` | Disable HTTP caching of GCS release artifacts.  By default GCS will cache public objects for up to an hour.  When doing "devel" releases this can cause problems.
 `KUBE_GCS_DOCKER_REG_PREFIX` | `docker-reg` | *Experimental* When uploading docker images, the bucket that backs the registry.
 
-## Basic Flow
-
-The scripts directly under `build/` are used to build and test.  They will ensure that the `kube-build` Docker image is built (based on `build/build-image/Dockerfile`) and then execute the appropriate command in that container.  If necessary (for Mac OS X), the scripts will also copy results out.
-
-The `kube-build` container image is built by first creating a "context" directory in `_output/images/build-image`.  It is done there instead of at the root of the Kubernetes repo to minimize the amount of data we need to package up when building the image.
-
-Everything in `build/build-image/` is meant to be run inside of the container.  If it doesn't think it is running in the container it'll throw a warning.  While you can run some of that stuff outside of the container, it wasn't built to do so.
-
 When building final release tars, they are first staged into `_output/release-stage` before being tar'd up and put into `_output/release-tars`.
 
-## Proxy Settings
-
-
-If you are behind a proxy, you need to export proxy settings for kubernetes build, the following environment variables should be defined.
-
-```
-export KUBERNETES_HTTP_PROXY=http://username:password@proxyaddr:proxyport
-export KUBERNETES_HTTPS_PROXY=https://username:password@proxyaddr:proxyport
-```
-
-Optionally, you can specify addresses of no proxy for kubernetes build, for example
-
-```
-export KUBERNETES_NO_PROXY=127.0.0.1
-```
-
-If you are using sudo to make kubernetes build for example make quick-release, you need run `sudo -E make quick-release` to pass the environment variables.
-
-## TODOs
-
-These are in no particular order
-
-* [X] Harmonize with scripts in `hack/`.  How much do we support building outside of Docker and these scripts?
-* [X] Deprecate/replace most of the stuff in the hack/
-* [ ] Finish support for the Dockerized runtime. Issue [#19](http://issue.k8s.io/19).  A key issue here is to make this fast/light enough that we can use it for development workflows.
-
-
 [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/build/README.md?pixel)]()

build/build-image/Dockerfile

@@ -15,6 +15,8 @@
 # This file creates a standard build environment for building Kubernetes
 FROM gcr.io/google_containers/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG
 
+ADD localtime /etc/localtime
+
 # Mark this as a kube-build container
 RUN touch /kube-build-image
 
@@ -25,19 +27,13 @@ RUN chmod -R a+rwx /usr/local/go/pkg ${K8S_PATCHED_GOROOT}/pkg
 # of operations.
 ENV HOME /go/src/k8s.io/kubernetes
 WORKDIR ${HOME}
-# We have to mkdir the dirs we need, or else Docker will create them when we
-# mount volumes, and it will create them with root-only permissions.  The
-# explicit chmod of _output is required, but I can't really explain why.
-RUN mkdir -p ${HOME} ${HOME}/_output \
-    && chmod -R a+rwx ${HOME} ${HOME}/_output
-
-# Propagate the git tree version into the build image
-ADD kube-version-defs /kube-version-defs
-RUN chmod a+r /kube-version-defs
-ENV KUBE_GIT_VERSION_FILE /kube-version-defs
 
 # Make output from the dockerized build go someplace else
 ENV KUBE_OUTPUT_SUBPATH _output/dockerized
 
-# Upload Kubernetes source
-ADD kube-source.tar.gz /go/src/k8s.io/kubernetes/
+# Pick up version stuff here as we don't copy our .git over.
+ENV KUBE_GIT_VERSION_FILE ${HOME}/.dockerized-kube-version-defs
+
+ADD rsyncd.password /
+RUN chmod a+r /rsyncd.password
+ADD rsyncd.sh /

build/build-image/rsyncd.sh

+#!/bin/bash
+
+# Copyright 2016 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script will set up and run rsyncd to allow data to move into and out of
+# our dockerized build system.  This is used for syncing sources and changes of
+# sources into the docker-build-container.  It is also used to transfer built binaries
+# and generated files back out.
+#
+# When run as root (rare) it'll preserve the file ids as sent from the client.
+# Usually it'll be run as non-dockerized UID/GID and end up translating all file
+# ownership to that.
+
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+# The directory that gets sync'd
+VOLUME=${HOME}
+
+# Assume that this is running in Docker on a bridge.  Allow connections from
+# anything on the local subnet.
+ALLOW=$(ip route | awk  '/^default via/ { reg = "^[0-9./]+ dev "$5 } ; $0 ~ reg { print $1 }')
+
+CONFDIR="/tmp/rsync.k8s"
+PIDFILE="${CONFDIR}/rsyncd.pid"
+CONFFILE="${CONFDIR}/rsyncd.conf"
+SECRETS="${CONFDIR}/rsyncd.secrets"
+
+mkdir -p "${CONFDIR}"
+
+if [[ -f "${PIDFILE}" ]]; then
+  PID=$(cat "${PIDFILE}")
+  echo "Cleaning up old PID file: ${PIDFILE}"
+  kill $PID &> /dev/null || true
+  rm "${PIDFILE}"
+fi
+
+PASSWORD=$(</rsyncd.password)
+
+cat <<EOF >"${SECRETS}"
+k8s:${PASSWORD}
+EOF
+chmod go= "${SECRETS}"
+
+USER_CONFIG=
+if [[ "$(id -u)" == "0" ]]; then
+  USER_CONFIG="  uid = 0"$'\n'"  gid = 0"
+fi
+
+cat <<EOF >"${CONFFILE}"
+pid file = ${PIDFILE}
+use chroot = no
+log file = /dev/stdout
+reverse lookup = no
+munge symlinks = no
+port = 8730
+[k8s]
+  numeric ids = true
+  $USER_CONFIG
+  hosts deny = *
+  hosts allow = ${ALLOW}
+  auth users = k8s
+  secrets file = ${SECRETS}
+  read only = false
+  path = ${VOLUME}
+  filter = - /.make/ - /.git/ - /_tmp/
+EOF
+
+exec /usr/bin/rsync --no-detach --daemon --config="${CONFFILE}" "$@"

build/copy-output.sh

@@ -14,10 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Copies any built binaries off the Docker machine.
-#
-# This is a no-op on Linux when the Docker daemon is local.  This is only
-# necessary on Mac OS X with docker-machine.
+# Copies any built binaries (and other generated files) out of the Docker build contianer.
 set -o errexit
 set -o nounset
 set -o pipefail

build/make-clean.sh

@@ -23,5 +23,4 @@ KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
 source "${KUBE_ROOT}/build/common.sh"
 
 kube::build::verify_prereqs
-kube::build::clean_output
-kube::build::clean_images
+kube::build::clean

hack/update-generated-protobuf.sh

@@ -18,6 +18,8 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
+
+
 KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
 source "${KUBE_ROOT}/build/common.sh"
 
@@ -35,11 +37,11 @@ function prereqs() {
   KUBE_BUILD_IMAGE_TAG="build-${KUBE_ROOT_HASH}"
   KUBE_BUILD_IMAGE="${KUBE_BUILD_IMAGE_REPO}:${KUBE_BUILD_IMAGE_TAG}"
   KUBE_BUILD_CONTAINER_NAME="kube-build-${KUBE_ROOT_HASH}"
-  KUBE_BUILD_DATA_CONTAINER_NAME="kube-build-data-${KUBE_ROOT_HASH}"
+  KUBE_DATA_CONTAINER_NAME="kube-build-data-${KUBE_ROOT_HASH}"
   DOCKER_MOUNT_ARGS=(
     --volume "${REPO_DIR:-${KUBE_ROOT}}:/go/src/${KUBE_GO_PACKAGE}"
     --volume /etc/localtime:/etc/localtime:ro
-    --volumes-from "${KUBE_BUILD_DATA_CONTAINER_NAME}"
+    --volumes-from "${KUBE_DATA_CONTAINER_NAME}"
   )
   LOCAL_OUTPUT_BUILD_CONTEXT="${LOCAL_OUTPUT_IMAGE_STAGING}/${KUBE_BUILD_IMAGE}"
 }

hack/update-generated-runtime.sh

@@ -35,11 +35,11 @@ function prereqs() {
   KUBE_BUILD_IMAGE_TAG="build-${KUBE_ROOT_HASH}"
   KUBE_BUILD_IMAGE="${KUBE_BUILD_IMAGE_REPO}:${KUBE_BUILD_IMAGE_TAG}"
   KUBE_BUILD_CONTAINER_NAME="kube-build-${KUBE_ROOT_HASH}"
-  KUBE_BUILD_DATA_CONTAINER_NAME="kube-build-data-${KUBE_ROOT_HASH}"
+  KUBE_DATA_CONTAINER_NAME="kube-build-data-${KUBE_ROOT_HASH}"
   DOCKER_MOUNT_ARGS=(
     --volume "${REPO_DIR:-${KUBE_ROOT}}:/go/src/${KUBE_GO_PACKAGE}"
     --volume /etc/localtime:/etc/localtime:ro
-    --volumes-from "${KUBE_BUILD_DATA_CONTAINER_NAME}"
+    --volumes-from "${KUBE_DATA_CONTAINER_NAME}"
   )
   LOCAL_OUTPUT_BUILD_CONTEXT="${LOCAL_OUTPUT_IMAGE_STAGING}/${KUBE_BUILD_IMAGE}"
 }
@@ -50,4 +50,3 @@ cp "${KUBE_ROOT}/cmd/libs/go2idl/go-to-protobuf/build-image/Dockerfile" "${LOCAL
 kube::build::update_dockerfile
 kube::build::docker_build "${KUBE_BUILD_IMAGE}" "${LOCAL_OUTPUT_BUILD_CONTEXT}" 'false'
 kube::build::run_build_command hack/update-generated-runtime-dockerized.sh "$@"
-

hack/verify-generated-protobuf.sh

@@ -34,8 +34,8 @@ trap "cleanup" EXIT SIGINT
 
 cleanup
 for APIROOT in ${APIROOTS}; do
-  mkdir -p "${_tmp}/${APIROOT%/*}"
-  cp -a "${KUBE_ROOT}/${APIROOT}" "${_tmp}/${APIROOT}"
+  mkdir -p "${_tmp}/${APIROOT}"
+  cp -a -T "${KUBE_ROOT}/${APIROOT}" "${_tmp}/${APIROOT}"
 done
 
 "${KUBE_ROOT}/hack/update-generated-protobuf.sh"
@@ -44,7 +44,7 @@ for APIROOT in ${APIROOTS}; do
   echo "diffing ${APIROOT} against freshly generated protobuf"
   ret=0
   diff -Naupr -I 'Auto generated by' -x 'zz_generated.*' "${KUBE_ROOT}/${APIROOT}" "${TMP_APIROOT}" || ret=$?
-  cp -a "${TMP_APIROOT}" "${KUBE_ROOT}/${APIROOT%/*}"
+  cp -a -T "${TMP_APIROOT}" "${KUBE_ROOT}/${APIROOT}"
   if [[ $ret -eq 0 ]]; then
     echo "${APIROOT} up to date."
   else

hack/verify-generated-swagger-docs.sh

@@ -42,15 +42,21 @@ DIFFROOT="${KUBE_ROOT}/pkg"
 TMP_DIFFROOT="${KUBE_ROOT}/_tmp/pkg"
 _tmp="${KUBE_ROOT}/_tmp"
 
+cleanup() {
+  rm -rf "${_tmp}"
+}
+trap "cleanup" EXIT SIGINT
+
+cleanup
+
 mkdir -p "${_tmp}"
-trap "rm -rf ${_tmp}" EXIT SIGINT
-cp -a "${DIFFROOT}" "${TMP_DIFFROOT}"
+cp -a -T "${DIFFROOT}" "${TMP_DIFFROOT}"
 
 "${KUBE_ROOT}/hack/update-generated-swagger-docs.sh"
 echo "diffing ${DIFFROOT} against freshly generated swagger type documentation"
 ret=0
 diff -Naupr -I 'Auto generated by' "${DIFFROOT}" "${TMP_DIFFROOT}" || ret=$?
-cp -a "${TMP_DIFFROOT}" "${KUBE_ROOT}/"
+cp -a -T "${TMP_DIFFROOT}" "${DIFFROOT}"
 if [[ $ret -eq 0 ]]
 then
   echo "${DIFFROOT} up to date."