1. 03 May, 2018 6 commits
    • Kubernetes Submit Queue's avatar
      Merge pull request #62903 from cofyc/fixfsgroupcheckinlocal · 186dd7be
      Kubernetes Submit Queue authored
      Automatic merge from submit-queue (batch tested with PRs 62657, 63278, 62903, 63375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
      
      Add more volume types in e2e and fix part of them.
      
      **What this PR does / why we need it**:
      
      - Add dir-link/dir-bindmounted/dir-link-bindmounted/bockfs volume types for e2e tests.
      - Fix fsGroup related e2e tests partially.
      - Return error if we cannot resolve volume path.
        - Because we should not fallback to volume path, if it's a symbolic link, we may get wrong results.
      
      To safely set fsGroup on local volume, we need to implement these two methods correctly for all volume types both on the host and in container:
      
      - get volume path kubelet can access
        - paths on the host and in container are different
      - get mount references
        - for directories, we cannot use its mount source (device field) to identify mount references, because directories on same filesystem have same mount source (e.g. tmpfs), we need to check filesystem's major:minor and directory root path on it
      
      Here is current status:
      
      | | (A) volume-path (host) | (B) volume-path (container) | (C) mount-refs (host) | (D) mount-refs (container) |
      | --- | --- | --- | --- | --- |
      | (1) dir | OK | FAIL | FAIL | FAIL |
      | (2) dir-link | OK | FAIL | FAIL | FAIL |
      | (3) dir-bindmounted | OK | FAIL | FAIL | FAIL |
      | (4) dir-link-bindmounted | OK | FAIL | FAIL | FAIL |
      | (5) tmpfs| OK | FAIL | FAIL | FAIL |
      | (6) blockfs| OK | FAIL | OK | FAIL |
      | (7) block| NOTNEEDED | NOTNEEDED | NOTNEEDED | NOTNEEDED |
      | (8) gce-localssd-scsi-fs| NOTTESTED | NOTTESTED | NOTTESTED | NOTTESTED |
      
      - This PR uses `nsenter ... readlink` to resolve path in container as @msau42  @jsafrane [suggested](https://github.com/kubernetes/kubernetes/pull/61489#pullrequestreview-110032850). This fixes B1:B6 and D6, , the rest will be addressed in https://github.com/kubernetes/kubernetes/pull/62102.
      - C5:D5 marked `FAIL` because `tmpfs` filesystems can share same mount source, we cannot rely on it to check mount references. e2e tests passes due to we use unique mount source string in tests.
      - A7:D7 marked `NOTNEEDED` because we don't set fsGroup on block devices in local plugin. (TODO: Should we set fsGroup on block device?)
      - A8:D8 marked `NOTTESTED` because I didn't test it, I leave it to `pull-kubernetes-e2e-gce`. I think it should be same as `blockfs`.
      
      **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
      Fixes #
      
      **Special notes for your reviewer**:
      
      **Release note**:
      
      ```release-note
      NONE
      ```
      186dd7be
    • Kubernetes Submit Queue's avatar
      Merge pull request #63278 from feiskyer/azure-lb-new-interface · 40182118
      Kubernetes Submit Queue authored
      Automatic merge from submit-queue (batch tested with PRs 62657, 63278, 62903, 63375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
      
      Use new Azure SDK APIs for load balancer and public IP operations
      
      **What this PR does / why we need it**:
      
      #63063 updated Azure SDK to a stable version. After that, we should also update existing clients to use new SDK APIs.
      
      Without this, public IP listing will be blocked forever in some case.
      
      **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
      Fixes #
      
      **Special notes for your reviewer**:
      
      A following PR will be sent for other interfaces, e.g. routes and NSGs.
      
      **Release note**:
      
      ```release-note
      NONE
      ```
      40182118
    • Kubernetes Submit Queue's avatar
      Merge pull request #62657 from matthyx/master · b5f61ac1
      Kubernetes Submit Queue authored
      Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
      
      Update all script shebangs to use /usr/bin/env interpreter instead of /bin/interpreter
      
      This is required to support systems where bash doesn't reside in /bin (such as NixOS, or the *BSD family) and allow users to specify a different interpreter version through $PATH manipulation.
      https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
      ```release-note
      Use /usr/bin/env in all script shebangs to increase portability.
      ```
      b5f61ac1
    • Kubernetes Submit Queue's avatar
      Merge pull request #62892 from liggitt/node-authorizer-index · 4558e419
      Kubernetes Submit Queue authored
      Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
      
      add index to node-authorizer for high cardinality vertices
      
      follow-up to https://github.com/kubernetes/kubernetes/pull/62856#issuecomment-382788780
      
      explores adding an index to high-cardinality vertices in the node authorizer to reduce CPU usage for high density namespaces
      
      * first commit is a refactor only - cc @mtaufen 
      * second commit adds an optional per-vertex index we can maintain when there are sufficient outgoing edges.
      
      benchmark results:
      * shared_secret_via_pod cases are ~1000x faster
      * throughput on processing of graph modifications is 50% higher
      * there is more variance on graph modifications requiring index updates (though the 100 index-impacting graph modifications per second might be a higher-than-realistic write rate)
      
      data profile (5000 pods per namespace, assigned to 5000 nodes, shared service account and secret):
      ```
              opts := sampleDataOpts{
                      // To simulate high replication in a small number of namespaces:
                     nodes:       5000,
                     namespaces:  10,
                     podsPerNode: 10,
      ...
      ```
      
      command:
      ```
      $ go test ./plugin/pkg/auth/authorizer/node/  -bench Authorization  -benchmem -v 
      ```
      
      before
      ```
      BenchmarkAuthorization/allowed_node_configmap-8                                  557 ns/op   530 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/allowed_configmap-8                                       539 ns/op   530 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/allowed_secret_via_pod-8                                  605 ns/op   529 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/allowed_shared_secret_via_pod-8                        215974 ns/op   792 B/op   19 allocs/op      5000
      BenchmarkAuthorization/disallowed_node_configmap-8                               823 ns/op   694 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_configmap-8                                    888 ns/op   691 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_secret_via_pod-8                               868 ns/op   694 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_shared_secret_via_pvc-8                       1216 ns/op   948 B/op   22 allocs/op   1000000
      BenchmarkAuthorization/disallowed_pvc-8                                          918 ns/op   691 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_pv-8                                          1095 ns/op   839 B/op   19 allocs/op   2000000
      BenchmarkAuthorization/disallowed_attachment_-_no_relationship-8                 867 ns/op   677 B/op   16 allocs/op   2000000
      BenchmarkAuthorization/disallowed_attachment_-_feature_disabled-8                220 ns/op   208 B/op    2 allocs/op  10000000
      BenchmarkAuthorization/allowed_attachment_-_feature_enabled-8                    687 ns/op   594 B/op   12 allocs/op   2000000
      BenchmarkAuthorization/contentious_allowed_node_configmap-8                      656 ns/op   530 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/contentious_allowed_configmap-8                           659 ns/op   529 B/op   11 allocs/op   2000000
      BenchmarkAuthorization/contentious_allowed_secret_via_pod-8                      654 ns/op   529 B/op   11 allocs/op   2000000
      BenchmarkAuthorization/contentious_allowed_shared_secret_via_pod-8            234308 ns/op  1022 B/op   22 allocs/op      5000
      BenchmarkAuthorization/contentious_disallowed_node_configmap-8                  1118 ns/op   692 B/op   17 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_configmap-8                       1054 ns/op   692 B/op   17 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_secret_via_pod-8                  1059 ns/op   691 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/contentious_disallowed_shared_secret_via_pvc-8           1403 ns/op   949 B/op   22 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_pvc-8                             1058 ns/op   692 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/contentious_disallowed_pv-8                              1237 ns/op   838 B/op   19 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_attachment_-_no_relationship-8    1022 ns/op   676 B/op   16 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_attachment_-_feature_disabled-8    260 ns/op   209 B/op    2 allocs/op   5000000
      BenchmarkAuthorization/contentious_allowed_attachment_-_feature_enabled-8        793 ns/op   594 B/op   12 allocs/op   2000000
      --- BENCH: BenchmarkAuthorization
         node_authorizer_test.go:596: graph modifications during non-contention test: 0
         node_authorizer_test.go:593: graph modifications during contention test: 961
         node_authorizer_test.go:594: <1ms=774, <10ms=32, <25ms=14, <50ms=29, <100ms=62, <250ms=46, <500ms=2, <1000ms=1, >1000ms=1
      ```
      
      after
      ```
      BenchmarkAuthorization/allowed_node_configmap-8                                  629 ns/op   530 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/allowed_configmap-8                                       641 ns/op   530 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/allowed_secret_via_pod-8                                  591 ns/op   530 B/op   11 allocs/op   3000000
      BenchmarkAuthorization/allowed_shared_secret_via_pod-8                           217 ns/op   160 B/op    1 allocs/op  10000000
      BenchmarkAuthorization/disallowed_node_configmap-8                               912 ns/op   693 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_configmap-8                                    913 ns/op   694 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_secret_via_pod-8                               881 ns/op   691 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_shared_secret_via_pvc-8                       1271 ns/op   952 B/op   22 allocs/op   1000000
      BenchmarkAuthorization/disallowed_pvc-8                                          903 ns/op   694 B/op   17 allocs/op   2000000
      BenchmarkAuthorization/disallowed_pv-8                                          1024 ns/op   836 B/op   19 allocs/op   1000000
      BenchmarkAuthorization/disallowed_attachment_-_no_relationship-8                1187 ns/op   678 B/op   16 allocs/op   2000000
      BenchmarkAuthorization/disallowed_attachment_-_feature_disabled-8                250 ns/op   209 B/op    2 allocs/op  10000000
      BenchmarkAuthorization/allowed_attachment_-_feature_enabled-8                    694 ns/op   594 B/op   12 allocs/op   2000000
      BenchmarkAuthorization/contentious_allowed_node_configmap-8                      732 ns/op   530 B/op   11 allocs/op   2000000
      BenchmarkAuthorization/contentious_allowed_configmap-8                           820 ns/op   530 B/op   11 allocs/op   2000000
      BenchmarkAuthorization/contentious_allowed_secret_via_pod-8                     1082 ns/op   531 B/op   11 allocs/op   1000000
      BenchmarkAuthorization/contentious_allowed_shared_secret_via_pod-8               274 ns/op   160 B/op    1 allocs/op   5000000
      BenchmarkAuthorization/contentious_disallowed_node_configmap-8                  1332 ns/op   693 B/op   17 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_configmap-8                       1534 ns/op   693 B/op   17 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_secret_via_pod-8                  1077 ns/op   692 B/op   17 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_shared_secret_via_pvc-8           1976 ns/op   949 B/op   22 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_pvc-8                             1297 ns/op   694 B/op   17 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_pv-8                              1632 ns/op   837 B/op   19 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_attachment_-_no_relationship-8    1394 ns/op   677 B/op   16 allocs/op   1000000
      BenchmarkAuthorization/contentious_disallowed_attachment_-_feature_disabled-8    320 ns/op   209 B/op    2 allocs/op   5000000
      BenchmarkAuthorization/contentious_allowed_attachment_-_feature_enabled-8       1055 ns/op   595 B/op   12 allocs/op   2000000
      --- BENCH: BenchmarkAuthorization
          node_authorizer_test.go:629: graph modifications during non-contention test: 0
          node_authorizer_test.go:626: graph modifications during contention test: 1424
          node_authorizer_test.go:627: <1ms=0, <10ms=569, <25ms=340, <50ms=145, <100ms=101, <250ms=160, <500ms=61, <1000ms=42, >1000ms=6
      ```
      
      ```release-note
      NONE
      ```
      4558e419
    • Kubernetes Submit Queue's avatar
      Merge pull request #62516 from nicksardo/expand-id · c968d99e
      Kubernetes Submit Queue authored
      Automatic merge from submit-queue (batch tested with PRs 62060, 62516). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
      
      [GCE] Add new formats to resource parser and self link creator
      
      **What this PR does / why we need it**:
      - Expands the set of parse-able strings for resource IDs, while also simplifying the code. Note that these are acceptable values for some fields in GCP API.
        - global/networks/my-network
        - regions/us-central1/subnetworks/my-sub
        - zones/us-central1-a/instances/my-hacker-instance
      - Fixes the SelfLink function to return links for regions and zones:
        - https://www.googleapis.com/compute/v1/projects/proj4/regions/us-central1
      - Generates helper functions to create a ResourceID for each resource
      - Generates a unit test that ensures all links can be generated and all generated links can be parsed.
      - Fixes an ILB test which creates a malformed URL.
      
      **Special notes for your reviewer**:
      /assign rramkumar1
      
      **Release note**:
      ```release-note
      NONE
      ```
      c968d99e
    • Kubernetes Submit Queue's avatar
      Merge pull request #62060 from WanLinghao/namespace_miss_fix · 03eb9f68
      Kubernetes Submit Queue authored
      Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
      
      fix namespace miss bug
      
      **What this PR does / why we need it**:
      This  patch fixes  the namespace miss problems.
      I am not sure if this is the correct way it should be fixed.
      Just offer a solution.
      **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
      Fixes #62059
      
      **Special notes for your reviewer**:
      
      **Release note**:
      
      ```release-note
      NONE
      ```
      03eb9f68
  2. 02 May, 2018 18 commits
  3. 01 May, 2018 16 commits