Automatic merge from submit-queue

Close websocket stream when client closes

Same problem as #30736, but for websocket streaming

Automatic merge from submit-queue

[GarbageCollector] Make Rate Limiter registration more efficient in GC

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:
Decrease the CPU consumption of the garbage collector

**Which issue this PR fixes** 
#30759

**Special notes for your reviewer**:
I observed dramatic improvement (dropped from 0.8cpu to 0.3cpu) in load test.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
```



@wojtek-t @lavalamp @gmarek

Automatic merge from submit-queue

	Don't validate selector that is already validated

Ref #30875

@timothysc

Automatic merge from submit-queue

Tweak resource consumption in density test

Fix: #30275

Automatic merge from submit-queue

Do not hold the lock for a long time

Followup to #30839.

I'm not convinced this is a super great idea but I'll throw it out and let others decide.

Ref https://github.com/kubernetes/minikube/issues/368
Ref #30759

Revert "Use netlink.SetPromiscOn instead of iproute2 command"

Automatic merge from submit-queue

AWS: Support HTTP->HTTP mode for ELB

**What this PR does / why we need it**:

Right now it is not possible to create an AWS ELB that listens for HTTP and where the backend pod also listens for HTTP.
I asked @justinsb in slack and he said that this seems to be an oversight, so I'd like to use this PR as a step towards solving this.

**Special notes for your reviewer**:

I've only added a simple unit test. Are any integration tests needed? I'm not familiar with the code base.

cc @therc

Automatic merge from submit-queue

Parameterize vpc name

Automatic merge from submit-queue

Fix caching of go dirs during build

Sometimes when you press `^C` during `make` the subsequent attempt to run `make` hangs due to zero-sized cache file for `ALL_GO_DIRS` var in Makefile.generated_files:
```
vagrant@devbox:~/work/kubernetes/src/k8s.io/kubernetes (master *%) $ KUBE_JUNIT_REPORT_DIR=/tmp/art KUBE_COVER=y make test
# hangs...
^CMakefile:279: recipe for target 'generated_files' failed
make: *** [generated_files] Interrupt

vagrant@devbox:~/work/kubernetes/src/k8s.io/kubernetes (master *%) $ ls -l .make/all_go_dirs.mk
-rw-rw-r-- 1 vagrant vagrant 0 Aug 18 15:03 .make/all_go_dirs.mk
```

 Corresponding process subtree looks like the following:
```
└─make test
    └─make -f Makefile.generated_files generated_files
        └─bash -c grep --color=never -l '+k8s:deepcopy-gen='  | xargs -n1 dirname | sort -u
            ├─grep --color=never -l +k8s:deepcopy-gen=
            ├─sort -u
            └─xargs -n1 dirname
```

Let's remove the cache file if `cache_go_dirs.sh` gets interrupted.

Automatic merge from submit-queue

Rename IsEmpty to Empty

addresses #21076

Automatic merge from submit-queue

e2e-runner: Get GCI builtin k8s version from GCS

**What this PR does / why we need it**:
The GCI release qualification tests use builtin k8s version to run tests. They used to get the builtin version by parsing descriptions of the test images, but that's unreliable because the description format could change. This PR makes it to read a file checked in GCS that has the exact k8s version number.

@adityakali @spxtr Can you review?

cc/ @kubernetes/goog-image

Automatic merge from submit-queue

Use netlink.SetPromiscOn instead of iproute2 command

Depends on https://github.com/vishvananda/netlink/pull/157
Related https://github.com/kubernetes/kubernetes/issues/26093

Automatic merge from submit-queue

Validate AppArmor annotations in the API server

This is important because it applies the same validation to the annotations that we would eventually want for the AppArmor fields, which will smooth the upgrade path.

/cc @pmorie @pweil- @jfrazelle @vishh

Automatic merge from submit-queue

[e2e test] Fix e2e test pause image hard code

Use `framework.GetPauseImageName(f.Client)` instead of hard code(such as `"gcr.io/google_containers/pause-amd64:3.0"`) to represent pause image name.

Related issus is #30967

Automatic merge from submit-queue

Add encryption to EBS dynamic provisioner

Resolves https://github.com/kubernetes/kubernetes/issues/30792

Adds encryption to the EBS cloud provider and provisioner.

Follow up to #29006 (all commits but the one in this PR will drop out).

@kubernetes/sig-storage 


```release-note
```

Automatic merge from submit-queue

kube-up: Allow IP restrictions for SSH and HTTPS API access on AWS.

<!--
Checklist for submitting a Pull Request

Please remove this comment block before submitting.

1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md).
2. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md).
3. If you want this PR to automatically close an issue when it is merged,
   add `fixes #<issue number>` or `fixes #<issue number>, fixes #<issue number>`
   to close multiple issues (see: https://github.com/blog/1506-closing-issues-via-pull-requests).
4. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below.
-->

```release-note

```


[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

Closes #26661

Automatic merge from submit-queue

Add readyReplicas to replica sets

@bgrant0607 for the api changes

@bprashanth for the controllers changes

@deads2k fyi

Automatic merge from submit-queue

Node E2E: Remove fatal error in e2e_node_suite_test.go

Addresses https://github.com/kubernetes/kubernetes/issues/30779#issuecomment-240532190.

Currently we run node e2e test in parallel, and ginkgo makes sure that we only initialize test framework in the first test node.
However, because we throw out some fatal error during the initialization. Once there is an fatal error, the first test node will die immediately without reporting any error, and the other nodes will exit because the first node is gone with meaningless error.

If kubelet start fails, we'll get something like:
```
------------------------------
Failure [132.485 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
------------------------------
......
------------------------------
Failure [132.465 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
```

This PR replaces these fatal errors with gomega assertion, with this PR, we'll get:
```
Failure [132.482 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  should be able to start node services.
  Expected success, but got an error:
      <*errors.errorString | 0xc8203351b0>: {
          s: "failed to run server start command \"/tmp/ginkgo869068712/e2e_node.test --run-services-mode --server-start-timeout 2m0s --report-dir  --node-name lantaol0.mtv.corp.google.com --disable-kubenet=true --cgroups-per-qos=false --manifest-path /tmp/node-e2e-pod221291440 --eviction-hard memory.available<250Mi\": exit status 255",
      }
      failed to run server start command "/tmp/ginkgo869068712/e2e_node.test --run-services-mode --server-start-timeout 2m0s --report-dir  --node-name lantaol0.mtv.corp.google.com --disable-kubenet=true --cgroups-per-qos=false --manifest-path /tmp/node-e2e-pod221291440 --eviction-hard memory.available<250Mi": exit status 255

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:117
------------------------------
Failure [132.485 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
------------------------------
......
------------------------------
Failure [132.465 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
```

This is much more informative.

/cc @kubernetes/sig-node

Automatic merge from submit-queue

Add Get() to cache listers

@kargakis @deads2k  PTAL

Automatic merge from submit-queue

Remove implicit Prometheus metrics from client

**What this PR does / why we need it**: This PR starts to cut away at dependencies that the client has.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
The implicit registration of Prometheus metrics for request count and latency have been removed, and a plug-able interface was added. If you were using our client libraries in your own binaries and want these metrics, add the following to your imports in the main package: "k8s.io/pkg/client/metrics/prometheus". 
```

cc: @kubernetes/sig-api-machinery @kubernetes/sig-instrumentation @fgrzadkowski  @wojtek-t

Automatic merge from submit-queue

Don't bind pre-bound pvc & pv if size request not satisfied

as discussed briefly here https://github.com/kubernetes/kubernetes/pull/30522 , volume size ought to be verified before binding a pv & pvc regardless of what's in the pv's claimRef. @thockin

Automatic merge from submit-queue

Support for creation/removal of master replicas.

HA master: initial support for creation/removal of masters replicas by
kube-up/kube-down scripts for GCE on gci (other distributions, including debian, are not supported yet).

Automatic merge from submit-queue

Fix data race in PVC Run/Stop methods

Fixes #30947
Fixes #30775

Automatic merge from submit-queue

kubectl run add pull-policy flag to control image pull policy

```release-note
Add support for --image-pull-policy to 'kubectl run'
```

Fix #30493 
@pwittrock @thockin ptal

Automatic merge from submit-queue

Object meta fixes in federation controllers

This PR adds helper functions for handling object meta structs and applies them to the existing controllers.

cc: @quinton-hoole @jianhuiz @kubernetes/sig-cluster-federation

Automatic merge from submit-queue

x509 authenticator: get groups from subject's organization field

Note that the current X509 tests provide a bunch of certs but no private keys or commands to reproduce the testdata, so the new test case isn't added to the certificate chain.

Closes  #30260

cc @treed @gtank @mikedanese @deads2k  @kubernetes/sig-auth

Automatic merge from submit-queue

Node E2E: Wait for node ready before the node e2e test started.

Fixes https://github.com/kubernetes/kubernetes/issues/30252.

This PR makes node e2e test wait for exactly one node ready before running other test.

@ronnielai @mtaufen

Automatic merge from submit-queue

Kubelet: add --container-runtime-endpoint and --image-service-endpoint

Flag `--container-runtime-endpoint` (overrides `--container-runtime`) is introduced to identify the unix socket file of the remote runtime service. And flag `--image-service-endpoint` is introduced to identify the unix socket file of the image service.

This PR is part of #28789 Milestone 0. 

CC @yujuhong @Random-Liu

Automatic merge from submit-queue

Add initial support for TokenFile to to the client config file.

@smarterclayton @deads2k @cjcullen @krousey 

Part of the fix for #28260

Automatic merge from submit-queue

fix path handling in hack/lib/init.sh

Jenkinsfile pipeline jobs get cloned into "\<project\> (\<branch\>)". As a result, I can't use certain things in `hack/lib/init.sh`.

This is a small fix for that problem.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
NONE
```

Automatic merge from submit-queue

pkg/genericapiserver/options: don't import pkg/apiserver

Refactor the authorization options for the API server so
pkg/apiserver isn't directly imported by the options package.

Closes #28544

cc @smarterclayton

@madhusudancs, @nikhiljindal I've updated `federation/cmd/federation-apiserver/app/server.go` to include the RBAC options with this change. I don't know if this was intentionally left out in the first place but would like your feedback.