• Kubernetes Submit Queue's avatar
    Merge pull request #31064 from soundcloud/grobie/filter-internal-labels · feb4d200
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue
    
    Filter internal Kubernetes labels from Prometheus metrics
    
    **What this PR does / why we need it**:
    
    Kubernetes uses Docker labels as storage for some internal labels. The
    majority of these labels are not meaningful metric labels and a few of
    them are even harmful as they're not static and cause wrong aggregation
    results.
    
    This change provides a custom labels func to only attach meaningful
    labels to cAdvisor exported metrics.
    
    **Which issue this PR fixes**
    
    google/cadvisor#1312
    
    **Special notes for your reviewer**:
    
    Depends on google/cadvisor#1429. Once that is merged, I'll update the vendor update commit.
    
    **Release note**:
    
    ```release-note
    Remove environment variables and internal Kubernetes Docker labels from cAdvisor Prometheus metric labels.
    
    Old behavior:
    
    - environment variables explicitly whitelisted via --docker-env-metadata-whitelist were exported as `container_env_*=*`. Default is zero so by default non were exported
    - all docker labels were exported as `container_label_*=*`
    
    New behavior:
    
    - Only `container_name`, `pod_name`, `namespace`, `id`, `image`, and `name` labels are exposed
    - no environment variables will be exposed ever via /metrics, even if whitelisted
    ```
    
    ---
    
    Given that we have full control over the exported label set, I shortened the pod_name, pod_namespace and container_name label names. Below an example of the change (reformatted for readability).
    
    ```
    # BEFORE
    container_cpu_cfs_periods_total{
      container_label_io_kubernetes_container_hash="5af8c3b4",
      container_label_io_kubernetes_container_name="sync",
      container_label_io_kubernetes_container_restartCount="1",
      container_label_io_kubernetes_container_terminationMessagePath="/dev/termination-log",
      container_label_io_kubernetes_pod_name="popularsearches-web-3165456836-2bfey",
      container_label_io_kubernetes_pod_namespace="popularsearches",
      container_label_io_kubernetes_pod_terminationGracePeriod="30",
      container_label_io_kubernetes_pod_uid="6a291e48-47c4-11e6-84a4-c81f66bdf8bd",
      id="/docker/68e1f15353921f4d6d4d998fa7293306c4ac828d04d1284e410ddaa75cf8cf25",
      image="redacted.com/popularsearches:42-16-ba6bd88",
      name="k8s_sync.5af8c3b4_popularsearches-web-3165456836-2bfey_popularsearches_6a291e48-47c4-11e6-84a4-c81f66bdf8bd_c02d3775"
    } 72819
    
    # AFTER
    container_cpu_cfs_periods_total{
      container_name="sync",
      pod_name="popularsearches-web-3165456836-2bfey",
      namespace="popularsearches",
      id="/docker/68e1f15353921f4d6d4d998fa7293306c4ac828d04d1284e410ddaa75cf8cf25",
      image="redacted.com/popularsearches:42-16-ba6bd88",
      name="k8s_sync.5af8c3b4_popularsearches-web-3165456836-2bfey_popularsearches_6a291e48-47c4-11e6-84a4-c81f66bdf8bd_c02d3775"
    } 72819
    ```
    
    Feedback requested on:
    * Label names. Other suggestions? Should we keep these very long ones?
    * Do we need to export io.kubernetes.pod.uid? It makes working with the metrics a bit more complicated and the pod name is already unique at any time (but not over time). The UID is aslo part of `name`.
    
    As discussed with @timstclair, this should be added to v1.4 as the current labels are harmful.
    
    PTAL @jimmidyson @fabxc @vishh
    feb4d200
Name
Last commit
Last update
..
api Loading commit data...
cadvisor Loading commit data...
client Loading commit data...
cm Loading commit data...
config Loading commit data...
container Loading commit data...
custommetrics Loading commit data...
dockershim Loading commit data...
dockertools Loading commit data...
envvars Loading commit data...
events Loading commit data...
eviction Loading commit data...
images Loading commit data...
kuberuntime Loading commit data...
leaky Loading commit data...
lifecycle Loading commit data...
metrics Loading commit data...
network Loading commit data...
pleg Loading commit data...
pod Loading commit data...
prober Loading commit data...
qos Loading commit data...
remote Loading commit data...
rkt Loading commit data...
rktshim Loading commit data...
server Loading commit data...
status Loading commit data...
sysctl Loading commit data...
types Loading commit data...
util Loading commit data...
volumemanager Loading commit data...
OWNERS Loading commit data...
active_deadline.go Loading commit data...
active_deadline_test.go Loading commit data...
container_bridge.go Loading commit data...
container_bridge_test.go Loading commit data...
disk_manager.go Loading commit data...
disk_manager_test.go Loading commit data...
doc.go Loading commit data...
flannel_helper.go Loading commit data...
kubelet.go Loading commit data...
kubelet_cadvisor.go Loading commit data...
kubelet_cadvisor_test.go Loading commit data...
kubelet_getters.go Loading commit data...
kubelet_getters_test.go Loading commit data...
kubelet_network.go Loading commit data...
kubelet_network_test.go Loading commit data...
kubelet_node_status.go Loading commit data...
kubelet_node_status_test.go Loading commit data...
kubelet_resources.go Loading commit data...
kubelet_resources_test.go Loading commit data...
kubelet_test.go Loading commit data...
kubelet_volumes.go Loading commit data...
kubelet_volumes_test.go Loading commit data...
networks.go Loading commit data...
oom_watcher.go Loading commit data...
oom_watcher_test.go Loading commit data...
pod_container_deletor.go Loading commit data...
pod_container_deletor_test.go Loading commit data...
pod_workers.go Loading commit data...
pod_workers_test.go Loading commit data...
reason_cache.go Loading commit data...
reason_cache_test.go Loading commit data...
root_context_linux.go Loading commit data...
root_context_unsupported.go Loading commit data...
runonce.go Loading commit data...
runonce_test.go Loading commit data...
runtime.go Loading commit data...
util.go Loading commit data...
volume_host.go Loading commit data...