• Kubernetes Submit Queue's avatar
    Merge pull request #59898 from Random-Liu/add-log-rotation · b38f1b90
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 60214, 58762, 59898, 59897, 60204). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
    
    Add CRI container log rotation support
    
    Fixes https://github.com/kubernetes/kubernetes/issues/58823.
    
    This PR:
    1) Added `pkg/kubelet/logs/container_log_manager.go` which manages and rotates container logs.
    2) Added a feature gate `CRIContainerLogRotation` to enable the alpha feature. And 2 kubelet flags `--container-log-max-size` and `--container-log-max-files` to configure the rotation behavior.
    3) Added unit test and node e2e test for container log rotation.
    
    Note that:
    1) Container log manager only starts when the container runtime is `remote` (not docker), because we can't implement `ReopenContainerLog` for docker.
    2) Rotated logs are compressed with `gzip`.
    2) The latest rotated log is not compressed. Because fluentd may still be reading the file right after rotation.
    3) `kubectl logs` still doesn't support log rotation. This is not a regression anyway, it doesn't support log rotation for docker log today. We'll probably fix this in the future. (Issue: https://github.com/kubernetes/kubernetes/issues/59902)
    
    An example of container log directory with `--container-log-max-files=3`:
    ```console
    $ ls -al /var/log/pods/57146449-11ec-11e8-90e1-42010af00002
    total 592
    drwxr-xr-x 2 root root   4096 Feb 15 01:07 .
    drwxr-xr-x 3 root root  12288 Feb 15 01:06 ..
    -rw-r----- 1 root root 176870 Feb 15 01:07 log-container_0.log
    -rw-r--r-- 1 root root  40239 Feb 15 01:07 log-container_0.log.20180215-010737.gz
    -rw-r----- 1 root root 365996 Feb 15 01:07 log-container_0.log.20180215-010747
    ```
    
    /assign @mtaufen for the config change.
    /assign @dashpole @crassirostris for the log change.
    /assign @feiskyer for CRI related change.
    /cc @yujuhong @feiskyer @abhi @mikebrow @mrunalp @runcom 
    /cc @kubernetes/sig-node-pr-reviews @kubernetes/sig-instrumentation-pr-reviews 
    
    **Release note**:
    
    ```release-note
    [Alpha] Kubelet now supports container log rotation for container runtime which implements CRI(container runtime interface).
    The feature can be enabled with feature gate `CRIContainerLogRotation`.
    The flags `--container-log-max-size` and `--container-log-max-files` can be used to configure the rotation behavior.
    ```
    b38f1b90
Name
Last commit
Last update
..
clicheck Loading commit data...
cloud-controller-manager Loading commit data...
controller-manager/app Loading commit data...
gendocs Loading commit data...
genkubedocs Loading commit data...
genman Loading commit data...
genswaggertypedocs Loading commit data...
genutils Loading commit data...
genyaml Loading commit data...
hyperkube Loading commit data...
importverifier Loading commit data...
kube-apiserver Loading commit data...
kube-controller-manager Loading commit data...
kube-proxy Loading commit data...
kube-scheduler Loading commit data...
kubeadm Loading commit data...
kubectl Loading commit data...
kubelet Loading commit data...
kubemark Loading commit data...
linkcheck Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...