Now rootless mode can be used with cgroup v2 resource limitations.
A pod is executed in a cgroup like "/user.slice/user-1001.slice/user@1001.service/k3s-rootless.service/kubepods/podd0eb6921-c81a-4214-b36c-d3b9bb212fac/63b5a253a1fd4627da16bfce9bec58d72144cf30fe833e0ca9a6d60ebf837475".

This is accomplished by running `kubelet` in a cgroup namespace, and enabling `cgroupfs` driver for the cgroup hierarchy delegated by systemd.

To enable cgroup v2 resource limitation, `k3s server --rootless` needs to be launched as `systemctl --user` service.
Please see the comment lines in `k3s-rootless.service` for the usage.

Running `k3s server --rootless` via a terminal is not supported.
When it really needs to be launched via a terminal, `systemd-run --user -p Delegate --tty` needs to be prepended to create a systemd scope.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

put etcd bootstrap save call in goroutine and update comment

Addresses k3s-io/k3s#3066 and CVE-2021-21334
Signed-off-by: Jacob Blain Christen <jacob@rancher.com>

Signed-off-by: Brian Downs <brian.downs@gmail.com>

Signed-off-by: Martin Norrsken <martin.norrsken@gmail.com>

* remove etcd data dir when etcd is disabled
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

* fix comment
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

* more fixes
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

* use debug instead of info logs
Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

Support repository regex rewrite rules when fetching image content.

Example configuration:
```yaml
# /etc/rancher/k3s/registries.yaml
mirrors:
  "docker.io":
    endpoint:
    - "https://registry-1.docker.io/v2"
    rewrite:
      "^library/alpine$": "my-org/alpine"
```

This will instruct k3s containerd to fetch content for `alpine` images
from `docker.io/my-org/alpine` instead of the default
`docker.io/library/alpine` locations.
Signed-off-by: Jacob Blain Christen <jacob@rancher.com>

* have state stored in etcd at completed start and remove unneeded code

Signed-off-by: Chris Kim <oats87g@gmail.com>

get() is called in a loop until client configuration is successfully
retrieved. Each iteration will try to configure the apiserver proxy,
which will in turn create a new load balancer. Skip creating a new
load balancer if we already have one.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

If the port wanted by the client load balancer is in TIME_WAIT, startup
will fail. Set SO_REUSEPORT so that it can be listened on again
immediately.

The configurable Listen call wants a context, so plumb that through as
well.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

* Always use static ports for the load-balancers

This fixes an issue where RKE2 kube-proxy daemonset pods were failing to
communicate with the apiserver when RKE2 was restarted because the
load-balancer used a different port every time it started up.

This also changes the apiserver load-balancer port to be 1 below the
supervisor port instead of 1 above it. This makes the apiserver port
consistent at 6443 across servers and agents on RKE2.

Additional fixes below were required to successfully test and use this change
on etcd-only nodes.

* Actually add lb-server-port flag to CLI
* Fix nil pointer when starting server with --disable-etcd but no --server
* Don't try to use full URI as initial load-balancer endpoint
* Fix etcd load-balancer pool updates
* Update dynamiclistener to fix cert updates on etcd-only nodes
* Handle recursive initial server URL in load balancer
* Don't run the deploy controller on etcd-only nodes

Docs housekeeping

Signed-off-by: David Nuzik <david.nuzik@rancher.com>

* BUILDING.md
* CODE_OF_CONDUCT.md
* CONTRIBUTING.md
* MAINTAINERS
* README.md
Signed-off-by: David Nuzik <david.nuzik@rancher.com>

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

* Add functionality for etcd snapshot/restore to and from S3 compatible backends.
* Update etcd restore functionality to extract and write certificates and configs from snapshot.

Fix etcd only nodes

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Servers should always be upgraded before agents, but generally this
isn't required because things are compatible between versions. In this
case we're OK with failing closed if the user upgrades out of order, but
we should give a clearer message about what steps are required to fix
the issue.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

We have had a couple issues with newer agents not working with old
servers or vice versa. Add a CI test to test variations on
uplevel/downlevel server/agent against latest, stable, and the previous
branch.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

Traefik v2 integration

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

Logging cleanup
Co-authored-by: Brad Davidson <brad@oatmail.org>

Allow writing static files regardless of the version.
Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

K3s upgrade via watch over file change of static file and manifest
and triggers helm-controller for change. It seems reasonable to
only allow upgrade traefik v1->v2 when there is no existing custom
traefik HelmChartConfig in the cluster to avoid any
incompatibility.

Here also separate the CRDs and put them into a different chart
to support CRD upgrade.
Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

It is possible that the apiserver may serve read requests but not allow
writes yet, in which case flannel will crash on startup when trying to
configure the subnet manager.

Fix this by waiting for the apiserver to become fully ready before
starting flannel and the network policy controller.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

mark v1.20.4-k3s1 as stable

Signed-off-by: galal-hussein <hussein.galal.ahmed.11@gmail.com>

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Adds support for retagging images to appear to have been sourced from
one or more additional registries as they are imported from the tarball.
This is intended to support RKE2 use cases with system-default-registry
where the images need to appear to have been pulled from a registry
other than docker.io.
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>