• Kubernetes Submit Queue's avatar
    Merge pull request #64049 from aveshagarwal/master-scheduler-flake · 89b73711
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 61963, 64279, 64130, 64125, 64049). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
    
    Fix TestSchedulerWithVolumeBinding to avoid setting predicate ordering.
    
    It is causing data race condition as predicate ordering is changing global
    variable `predicatesOrdering`. Infact this test does not require any special
    predicate order and should work on default predicate ordering as far as
    VolumeScheduling feature is enabled.
    
    See these logs:
    
    ```
    ==================
    ==================
    WARNING: DATA RACE
    Read at 0x00c420894180 by goroutine 156:
      k8s.io/kubernetes/pkg/scheduler/core.podFitsOnNode()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:503 +0xbb
      k8s.io/kubernetes/pkg/scheduler/core.(*genericScheduler).findNodesThatFit.func1()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:353 +0x2f0
      k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.Parallelize.func1()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:47 +0xa3
    
    Previous write at 0x00c420894180 by goroutine 186:
      k8s.io/kubernetes/pkg/scheduler.TestSchedulerWithVolumeBinding()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler_test.go:663 +0x71
      testing.tRunner()
          /usr/lib/golang/src/testing/testing.go:777 +0x16d
    
    Goroutine 156 (running) created at:
      k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.Parallelize()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:43 +0x139
      k8s.io/kubernetes/pkg/scheduler/core.(*genericScheduler).findNodesThatFit()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:378 +0xe8a
      k8s.io/kubernetes/pkg/scheduler/core.(*genericScheduler).Schedule()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:131 +0x385
      k8s.io/kubernetes/pkg/scheduler.(*Scheduler).schedule()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:192 +0xcd
      k8s.io/kubernetes/pkg/scheduler.(*Scheduler).scheduleOne()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:447 +0x598
      k8s.io/kubernetes/pkg/scheduler.(*Scheduler).(k8s.io/kubernetes/pkg/scheduler.scheduleOne)-fm()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:182 +0x41
      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x61
      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xcd
      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
          /home/avagarwa/upstream-code/gocode/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a
    
    Goroutine 186 (running) created at:
      testing.(*T).Run()
          /usr/lib/golang/src/testing/testing.go:824 +0x564
      testing.runTests.func1()
          /usr/lib/golang/src/testing/testing.go:1063 +0xa4
      testing.tRunner()
          /usr/lib/golang/src/testing/testing.go:777 +0x16d
      testing.runTests()
          /usr/lib/golang/src/testing/testing.go:1061 +0x4e1
      testing.(*M).Run()
          /usr/lib/golang/src/testing/testing.go:978 +0x2cd
      main.main()
          _testmain.go:52 +0x22a
    ==================
    --- FAIL: TestSchedulerWithVolumeBinding (18.04s)
    	testing.go:730: race detected during execution of test
    FAIL
    ```
    
    It is pretty easy to reproduce this race by following these steps:
    
    ```
    cd pkg/scheduler
    go test -c -race
    stress -p 100 ./scheduler.test
    ```
    
    Predicate ordering to this unit test was added here: https://github.com/kubernetes/kubernetes/pull/57168
    Since the whole scheduler instance uses just one ordering at time, not sure what is the advantage. 
    
    @kubernetes/sig-scheduling-bugs @bsalamat @k82cn @frobware @smarterclayton @sjenning 
    
    ```release-note
    None
    ```
    89b73711
Name
Last commit
Last update
..
api Loading commit data...
apis Loading commit data...
auth Loading commit data...
capabilities Loading commit data...
client Loading commit data...
cloudprovider Loading commit data...
controller Loading commit data...
credentialprovider Loading commit data...
features Loading commit data...
fieldpath Loading commit data...
generated Loading commit data...
kubeapiserver Loading commit data...
kubectl Loading commit data...
kubelet Loading commit data...
kubemark Loading commit data...
master Loading commit data...
printers Loading commit data...
probe Loading commit data...
proxy Loading commit data...
quota Loading commit data...
registry Loading commit data...
routes Loading commit data...
scheduler Loading commit data...
security Loading commit data...
securitycontext Loading commit data...
serviceaccount Loading commit data...
ssh Loading commit data...
util Loading commit data...
version Loading commit data...
volume Loading commit data...
watch/json Loading commit data...
windows/service Loading commit data...
.import-restrictions Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...