Skip to content

K
k3s
Commit 4e203399 authored by Kubernetes Submit Queue's avatar Kubernetes Submit Queue Committed by GitHub
Browse files
Options

Merge pull request #34513 from MrHohn/addon-manager-upgrade

Automatic merge from submit-queue Upgrade addon-manager with kubectl apply The first step of #33698. Use `kubectl apply` to replace addon-manager's previous logic. The most important issue this PR is targeting is the upgrade from 1.4 to 1.5. Procedure as below: 1. Precondition: After the master is upgraded, new addon-manager starts and all the old resources on nodes are running normally. 2. Annotate the old ReplicationController resources with kubectl.kubernetes.io/last-applied-configuration="" 3. Call `kubectl apply --prune=false` on addons folder to create new addons, including the new Deployments. 4. Wait for one minute for new addons to be spinned up. 5. Enter the periodical loop of `kubectl apply --prune=true`. The old RCs will be pruned at the first call. Procedure of a normal startup: 1. Addon-manager starts and no addon resources are running. 2. Annotate nothing. 3. Call `kubectl apply --prune=false` to create all new addons. 4. No need to explain the remain. Remained Issues: - Need to add `--type` flag to `kubectl apply --prune`, mentioned [here](https://github.com/kubernetes/kubernetes/pull/33075#discussion_r80814070). - This addon manager is not working properly with the current Deployment heapster, which runs [addon-resizer](https://github.com/kubernetes/contrib/tree/master/addon-resizer) in the same pod and changes resource limit configuration through the apiserver. `kubectl apply` fights with the addon-resizers. May be we should remove the initial resource limit field in the configuration file for this specific Deployment as we removed the replica count. @mikedanese @thockin @bprashanth --- Below are some logical things that may need to be clarified, feel free to **OMIT** them as they are too verbose: - For upgrade, the old RCs will not fight with the new Deployments during the overlap period even if they use the same label in template: - Deployment will not recognize the old pods because it need to match an additional "pod-template-hash" label. - ReplicationController will not manage the new pods (created by deployment) because the [`controllerRef`](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/controller-ref.md) feature. - As we are moving all addons to Deployment, all old RCs would be removed. Attach empty annotation to RCs is only for letting `kubectl apply --prune` to recognize them, the content does not matter. - We might need to also annotate other resource types if we plan to upgrade them in 1.5 release: - They don't need to be attached this fake annotation if they remain in the same name. `kubectl apply` can recognize them by name/type/namespace. - In the other case, attaching empty annotations to them will still work. As the plan is to use label selector for annotate, some innocence old resources may also be attached empty annotations, they work as below two cases: - Resources that need to be bumped up to a newer version (mainly due to some significant update --- change disallowed fields --- that could not be managed by the update feature of `kubectl apply`) are good to go with this fake annotation, as old resources will be deleted and new sources will be created. The content in annotation does not matter. - Resources that need to stay inside the management of `kubectl apply` is also good to go. As `kubectl apply` will [generate a 3-way merge patch](https://github.com/kubernetes/kubernetes/blob/master/pkg/util/strategicpatch/patch.go#L1202-L1226). This empty annotation is harmless enough.
parents 936bb64a 0e4e54a6
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 131 additions and 10 deletions
cluster/addons/addon-manager/kube-addons.sh
View file @ 4e203399
......@@ -14,6 +14,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# LIMITATIONS
# 1. Exit code is probably not always correct.
# 2. There are no unittests.
# 3. Will not work if the total length of paths to addons is greater than
# bash can handle. Probably it is not a problem: ARG_MAX=2097152 on GCE.
# cosmetic improvements to be done
# 1. Improve the log function; add timestamp, file name, etc.
# 2. Logging doesn't work from files that print things out.
# 3. Kubectl prints the output to stderr (the output should be captured and then
# logged)
# The business logic for whether a given object should be created
# was already enforced by salt, and /etc/kubernetes/addons is the
# managed result is of that. Start everything below that directory.
......@@ -21,10 +33,66 @@ KUBECTL=${KUBECTL_BIN:-/usr/local/bin/kubectl}
KUBECTL_OPTS=${KUBECTL_OPTS:-}
ADDON_CHECK_INTERVAL_SEC=${TEST_ADDON_CHECK_INTERVAL_SEC:-60}
ADDON_PATH=${ADDON_PATH:-/etc/kubernetes/addons}
SYSTEM_NAMESPACE=kube-system
trusty_master=${TRUSTY_MASTER:-false}
# Remember that you can't log from functions that print some output (because
# logs are also printed on stdout).
# $1 level
# $2 message
function log() {
# manage log levels manually here
# add the timestamp if you find it useful
case $1 in
DB3 )
# echo "$1: $2"
;;
DB2 )
# echo "$1: $2"
;;
DBG )
# echo "$1: $2"
;;
INFO )
echo "$1: $2"
;;
WRN )
echo "$1: $2"
;;
ERR )
echo "$1: $2"
;;
* )
echo "INVALID_LOG_LEVEL $1: $2"
;;
esac
}
# $1 command to execute.
# $2 count of tries to execute the command.
# $3 delay in seconds between two consecutive tries
function run_until_success() {
local -r command=$1
local tries=$2
local -r delay=$3
local -r command_name=$1
while [ ${tries} -gt 0 ]; do
log DBG "executing: '$command'"
# let's give the command as an argument to bash -c, so that we can use
# && and || inside the command itself
/bin/bash -c "${command}" && \
log DB3 "== Successfully executed ${command_name} at $(date -Is) ==" && \
return 0
let tries=tries-1
log WRN "== Failed to execute ${command_name} at $(date -Is). ${tries} tries remaining. =="
sleep ${delay}
done
return 1
}
# $1 filename of addon to start.
# $2 count of tries to start the addon.
# $3 delay in seconds between two consecutive tries
......@@ -35,7 +103,7 @@ function start_addon() {
local -r delay=$3;
local -r namespace=$4
create-resource-from-string "$(cat ${addon_filename})" "${tries}" "${delay}" "${addon_filename}" "${namespace}"
create_resource_from_string "$(cat ${addon_filename})" "${tries}" "${delay}" "${addon_filename}" "${namespace}"
}
# $1 string with json or yaml.
......@@ -43,7 +111,7 @@ function start_addon() {
# $3 delay in seconds between two consecutive tries
# $4 name of this object to use when logging about it.
# $5 namespace for this object
function create-resource-from-string() {
function create_resource_from_string() {
local -r config_string=$1;
local tries=$2;
local -r delay=$3;
......@@ -51,19 +119,51 @@ function create-resource-from-string() {
local -r namespace=$5;
while [ ${tries} -gt 0 ]; do
echo "${config_string}" | ${KUBECTL} ${KUBECTL_OPTS} --namespace="${namespace}" apply -f - && \
echo "== Successfully started ${config_name} in namespace ${namespace} at $(date -Is)" && \
return 0;
log INFO "== Successfully started ${config_name} in namespace ${namespace} at $(date -Is)" && \
return 0;
let tries=tries-1;
echo "== Failed to start ${config_name} in namespace ${namespace} at $(date -Is). ${tries} tries remaining. =="
log WRN "== Failed to start ${config_name} in namespace ${namespace} at $(date -Is). ${tries} tries remaining. =="
sleep ${delay};
done
return 1;
}
# $1 resource type.
function annotate_addons() {
local -r obj_type=$1;
# Annotate to objects already have this annotation should fail.
# Only try once for now.
${KUBECTL} ${KUBECTL_OPTS} annotate ${obj_type} --namespace=${SYSTEM_NAMESPACE} -l kubernetes.io/cluster-service=true \
kubectl.kubernetes.io/last-applied-configuration='' --overwrite=false
if [[ $? -eq 0 ]]; then
log INFO "== Annotate resources completed successfully at $(date -Is) =="
else
log WRN "== Annotate resources completed with errors at $(date -Is) =="
fi
}
# $1 enable --prune or not.
# $2 additional option for command.
function update_addons() {
local -r enable_prune=$1;
local -r additional_opt=$2;
run_until_success "${KUBECTL} ${KUBECTL_OPTS} apply --namespace=${SYSTEM_NAMESPACE} -f ${ADDON_PATH} \
--prune=${enable_prune} -l kubernetes.io/cluster-service=true --recursive ${additional_opt}" 3 5
if [[ $? -eq 0 ]]; then
log INFO "== Kubernetes addon update completed successfully at $(date -Is) =="
else
log WRN "== Kubernetes addon update completed with errors at $(date -Is) =="
fi
}
# The business logic for whether a given object should be created
# was already enforced by salt, and /etc/kubernetes/addons is the
# managed result is of that. Start everything below that directory.
echo "== Kubernetes addon manager started at $(date -Is) with ADDON_CHECK_INTERVAL_SEC=${ADDON_CHECK_INTERVAL_SEC} =="
log INFO "== Kubernetes addon manager started at $(date -Is) with ADDON_CHECK_INTERVAL_SEC=${ADDON_CHECK_INTERVAL_SEC} =="
# Create the namespace that will be used to host the cluster-level add-ons.
start_addon /opt/namespace.yaml 100 10 "" &
......@@ -75,22 +175,43 @@ while [ -z "${token_found}" ]; do
token_found=$(${KUBECTL} ${KUBECTL_OPTS} get --namespace="${SYSTEM_NAMESPACE}" serviceaccount default -o go-template="{{with index .secrets 0}}{{.name}}{{end}}" || true)
done
echo "== default service account in the ${SYSTEM_NAMESPACE} namespace has token ${token_found} =="
log INFO "== Default service account in the ${SYSTEM_NAMESPACE} namespace has token ${token_found} =="
# Create admission_control objects if defined before any other addon services. If the limits
# are defined in a namespace other than default, we should still create the limits for the
# default namespace.
for obj in $(find /etc/kubernetes/admission-controls \( -name \*.yaml -o -name \*.json \)); do
start_addon "${obj}" 100 10 default &
echo "++ obj ${obj} is created ++"
log INFO "++ obj ${obj} is created ++"
done
# Fake the "kubectl.kubernetes.io/last-applied-configuration" annotation on old resources
# in order to clean them up by `kubectl apply --prune`.
# RCs have to be annotated for 1.4->1.5 upgrade, because we are migrating from RCs to deployments for all default addons.
# Other types resources will also need this fake annotation if their names are changed,
# otherwise they would be leaked during upgrade.
log INFO "== Annotating the old addon resources at $(date -Is) =="
annotate_addons ReplicationController
# Create new addon resources by apply (with --prune=false).
# The old RCs will not fight for pods created by new Deployments with the same label because the `controllerRef` feature.
# The new Deployments will not fight for pods created by old RCs with the same label because the additional `pod-template-hash` label.
# Apply will fail if some fields are modified but not are allowed, in that case should bump up addon version and name (e.g. handle externally).
log INFO "== Executing apply to spin up new addon resources at $(date -Is) =="
update_addons false
# Wait for new addons to be spinned up before delete old resources
log INFO "== Wait for addons to be spinned up at $(date -Is) =="
sleep ${ADDON_CHECK_INTERVAL_SEC}
# Start the apply loop.
# Check if the configuration has changed recently - in case the user
# created/updated/deleted the files on the master.
log INFO "== Entering periodical apply loop at $(date -Is) =="
while true; do
start_sec=$(date +"%s")
#kube-addon-update.sh must be deployed in the same directory as this file
`dirname $0`/kube-addon-update.sh /etc/kubernetes/addons ${ADDON_CHECK_INTERVAL_SEC}
# Only print stderr for the readability of logging
update_addons true ">/dev/null"
end_sec=$(date +"%s")
len_sec=$((${end_sec}-${start_sec}))
# subtract the time passed from the sleep time
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment