Automatic merge from submit-queue

Do not hold the lock for a long time

Followup to #30839.

I'm not convinced this is a super great idea but I'll throw it out and let others decide.

Ref https://github.com/kubernetes/minikube/issues/368
Ref #30759

Revert "Use netlink.SetPromiscOn instead of iproute2 command"

Automatic merge from submit-queue

AWS: Support HTTP->HTTP mode for ELB

**What this PR does / why we need it**:

Right now it is not possible to create an AWS ELB that listens for HTTP and where the backend pod also listens for HTTP.
I asked @justinsb in slack and he said that this seems to be an oversight, so I'd like to use this PR as a step towards solving this.

**Special notes for your reviewer**:

I've only added a simple unit test. Are any integration tests needed? I'm not familiar with the code base.

cc @therc

Automatic merge from submit-queue

Parameterize vpc name

Automatic merge from submit-queue

Fix caching of go dirs during build

Sometimes when you press `^C` during `make` the subsequent attempt to run `make` hangs due to zero-sized cache file for `ALL_GO_DIRS` var in Makefile.generated_files:
```
vagrant@devbox:~/work/kubernetes/src/k8s.io/kubernetes (master *%) $ KUBE_JUNIT_REPORT_DIR=/tmp/art KUBE_COVER=y make test
# hangs...
^CMakefile:279: recipe for target 'generated_files' failed
make: *** [generated_files] Interrupt

vagrant@devbox:~/work/kubernetes/src/k8s.io/kubernetes (master *%) $ ls -l .make/all_go_dirs.mk
-rw-rw-r-- 1 vagrant vagrant 0 Aug 18 15:03 .make/all_go_dirs.mk
```

 Corresponding process subtree looks like the following:
```
└─make test
    └─make -f Makefile.generated_files generated_files
        └─bash -c grep --color=never -l '+k8s:deepcopy-gen='  | xargs -n1 dirname | sort -u
            ├─grep --color=never -l +k8s:deepcopy-gen=
            ├─sort -u
            └─xargs -n1 dirname
```

Let's remove the cache file if `cache_go_dirs.sh` gets interrupted.

Automatic merge from submit-queue

Rename IsEmpty to Empty

addresses #21076

Automatic merge from submit-queue

e2e-runner: Get GCI builtin k8s version from GCS

**What this PR does / why we need it**:
The GCI release qualification tests use builtin k8s version to run tests. They used to get the builtin version by parsing descriptions of the test images, but that's unreliable because the description format could change. This PR makes it to read a file checked in GCS that has the exact k8s version number.

@adityakali @spxtr Can you review?

cc/ @kubernetes/goog-image

Automatic merge from submit-queue

Use netlink.SetPromiscOn instead of iproute2 command

Depends on https://github.com/vishvananda/netlink/pull/157
Related https://github.com/kubernetes/kubernetes/issues/26093

Automatic merge from submit-queue

Validate AppArmor annotations in the API server

This is important because it applies the same validation to the annotations that we would eventually want for the AppArmor fields, which will smooth the upgrade path.

/cc @pmorie @pweil- @jfrazelle @vishh

Automatic merge from submit-queue

[e2e test] Fix e2e test pause image hard code

Use `framework.GetPauseImageName(f.Client)` instead of hard code(such as `"gcr.io/google_containers/pause-amd64:3.0"`) to represent pause image name.

Related issus is #30967

Automatic merge from submit-queue

Add encryption to EBS dynamic provisioner

Resolves https://github.com/kubernetes/kubernetes/issues/30792

Adds encryption to the EBS cloud provider and provisioner.

Follow up to #29006 (all commits but the one in this PR will drop out).

@kubernetes/sig-storage 


```release-note
```

Automatic merge from submit-queue

kube-up: Allow IP restrictions for SSH and HTTPS API access on AWS.

<!--
Checklist for submitting a Pull Request

Please remove this comment block before submitting.

1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md).
2. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md).
3. If you want this PR to automatically close an issue when it is merged,
   add `fixes #<issue number>` or `fixes #<issue number>, fixes #<issue number>`
   to close multiple issues (see: https://github.com/blog/1506-closing-issues-via-pull-requests).
4. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below.
-->

```release-note

```


[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

Closes #26661

Automatic merge from submit-queue

Add readyReplicas to replica sets

@bgrant0607 for the api changes

@bprashanth for the controllers changes

@deads2k fyi

Automatic merge from submit-queue

Node E2E: Remove fatal error in e2e_node_suite_test.go

Addresses https://github.com/kubernetes/kubernetes/issues/30779#issuecomment-240532190.

Currently we run node e2e test in parallel, and ginkgo makes sure that we only initialize test framework in the first test node.
However, because we throw out some fatal error during the initialization. Once there is an fatal error, the first test node will die immediately without reporting any error, and the other nodes will exit because the first node is gone with meaningless error.

If kubelet start fails, we'll get something like:
```
------------------------------
Failure [132.485 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
------------------------------
......
------------------------------
Failure [132.465 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
```

This PR replaces these fatal errors with gomega assertion, with this PR, we'll get:
```
Failure [132.482 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  should be able to start node services.
  Expected success, but got an error:
      <*errors.errorString | 0xc8203351b0>: {
          s: "failed to run server start command \"/tmp/ginkgo869068712/e2e_node.test --run-services-mode --server-start-timeout 2m0s --report-dir  --node-name lantaol0.mtv.corp.google.com --disable-kubenet=true --cgroups-per-qos=false --manifest-path /tmp/node-e2e-pod221291440 --eviction-hard memory.available<250Mi\": exit status 255",
      }
      failed to run server start command "/tmp/ginkgo869068712/e2e_node.test --run-services-mode --server-start-timeout 2m0s --report-dir  --node-name lantaol0.mtv.corp.google.com --disable-kubenet=true --cgroups-per-qos=false --manifest-path /tmp/node-e2e-pod221291440 --eviction-hard memory.available<250Mi": exit status 255

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:117
------------------------------
Failure [132.485 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
------------------------------
......
------------------------------
Failure [132.465 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
```

This is much more informative.

/cc @kubernetes/sig-node

Automatic merge from submit-queue

Add Get() to cache listers

@kargakis @deads2k  PTAL

Automatic merge from submit-queue

Remove implicit Prometheus metrics from client

**What this PR does / why we need it**: This PR starts to cut away at dependencies that the client has.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
The implicit registration of Prometheus metrics for request count and latency have been removed, and a plug-able interface was added. If you were using our client libraries in your own binaries and want these metrics, add the following to your imports in the main package: "k8s.io/pkg/client/metrics/prometheus". 
```

cc: @kubernetes/sig-api-machinery @kubernetes/sig-instrumentation @fgrzadkowski  @wojtek-t

Automatic merge from submit-queue

Don't bind pre-bound pvc & pv if size request not satisfied

as discussed briefly here https://github.com/kubernetes/kubernetes/pull/30522 , volume size ought to be verified before binding a pv & pvc regardless of what's in the pv's claimRef. @thockin

Automatic merge from submit-queue

Support for creation/removal of master replicas.

HA master: initial support for creation/removal of masters replicas by
kube-up/kube-down scripts for GCE on gci (other distributions, including debian, are not supported yet).

Automatic merge from submit-queue

Fix data race in PVC Run/Stop methods

Fixes #30947
Fixes #30775

Automatic merge from submit-queue

kubectl run add pull-policy flag to control image pull policy

```release-note
Add support for --image-pull-policy to 'kubectl run'
```

Fix #30493 
@pwittrock @thockin ptal

Automatic merge from submit-queue

Object meta fixes in federation controllers

This PR adds helper functions for handling object meta structs and applies them to the existing controllers.

cc: @quinton-hoole @jianhuiz @kubernetes/sig-cluster-federation

Automatic merge from submit-queue

x509 authenticator: get groups from subject's organization field

Note that the current X509 tests provide a bunch of certs but no private keys or commands to reproduce the testdata, so the new test case isn't added to the certificate chain.

Closes  #30260

cc @treed @gtank @mikedanese @deads2k  @kubernetes/sig-auth

Automatic merge from submit-queue

Node E2E: Wait for node ready before the node e2e test started.

Fixes https://github.com/kubernetes/kubernetes/issues/30252.

This PR makes node e2e test wait for exactly one node ready before running other test.

@ronnielai @mtaufen

Automatic merge from submit-queue

Kubelet: add --container-runtime-endpoint and --image-service-endpoint

Flag `--container-runtime-endpoint` (overrides `--container-runtime`) is introduced to identify the unix socket file of the remote runtime service. And flag `--image-service-endpoint` is introduced to identify the unix socket file of the image service.

This PR is part of #28789 Milestone 0. 

CC @yujuhong @Random-Liu

Automatic merge from submit-queue

Add initial support for TokenFile to to the client config file.

@smarterclayton @deads2k @cjcullen @krousey 

Part of the fix for #28260

Automatic merge from submit-queue

fix path handling in hack/lib/init.sh

Jenkinsfile pipeline jobs get cloned into "\<project\> (\<branch\>)". As a result, I can't use certain things in `hack/lib/init.sh`.

This is a small fix for that problem.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
NONE
```

Automatic merge from submit-queue

pkg/genericapiserver/options: don't import pkg/apiserver

Refactor the authorization options for the API server so
pkg/apiserver isn't directly imported by the options package.

Closes #28544

cc @smarterclayton

@madhusudancs, @nikhiljindal I've updated `federation/cmd/federation-apiserver/app/server.go` to include the RBAC options with this change. I don't know if this was intentionally left out in the first place but would like your feedback.

Automatic merge from submit-queue

update kubectl help output for better organization

## Pull Request Guidelines

1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md).
1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md).
1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below.

```release-note
* Use the release-note-* labels to set the release note state 
* Clear this block to use the PR title as the release note 
-OR-
* Enter your extended release note here
```


[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

Automatic merge from submit-queue

Add benchmark to jenkins

This PR contains the following changes:

1. Add more tests in density benchmark test;
2. Add the peak value (100%) in latency and CPU usage statistic data;
3. Move the Ginkgo focus flag from e2e_remote.go to run_e2e.go;
4. Support running benchmark in run_e2e.go. The benchmark configuration file is an extension of image configuration. Each item requires additional GCE machine type (e.g. n1-standard-1, default value will be used if empty) and test names (Ginkgo focus regex strings). A test item is regarded as benchmark if the tests field is non-empty.

Automatic merge from submit-queue

Node Conformance Test: Statically link apiserver

For #30122, #30174.
Based on #30116, #30198, #30200.

**Please only review the 4th PR.**

This PR is part of our roadmap to package node conformance test.
The 1st commit is from #30116, which started e2e services in a separate process.
The 2nd commit is from #30198, it statically linked etcd into the node e2e framework.
The 3rd commit is from #30200, it moved namespace controller into e2e services.

The 4th commit is new, it statically linked apiserver into the node e2e framework.

@dchen1107 @vishh 
/cc @kubernetes/sig-node @kubernetes/sig-testing

Automatic merge from submit-queue

Kubelet rkt CRI stubs & fakes

Adds stubs for CRI interfaces and creates a fake for `ContainerRuntime` interface

Automatic merge from submit-queue

daemonset controller should respect taints

cc @dchen1107 @davidopp

Automatic merge from submit-queue

Move init-container feature from alpha to beta.

```release-note
Moved init-container feature from alpha to beta.

In 1.3, an init container is specified with this annotation key
on the pod or pod template: `pods.alpha.kubernetes.io/init-containers`.

In 1.4, either that key or this key: pods.beta.kubernetes.io/init-containers`,
can be used.

When you GET an object, you will see both annotation keys with the same values.

You can safely roll back from 1.4 to 1.3, and things with init-containers
will still work (pods, deployments, etc).

If you are running 1.3, only use the alpha annotation, or it may be lost when
rolling forward.

The status has moved from annotation key
`pods.beta.kubernetes.io/init-container-statuses` to
`pods.beta.kubernetes.io/init-container-statuses`.
Any code that inspects this annotation should be changed to use the new key.

State of Initialization will continue to be reported in both pods.alpha.kubernetes.io/initialized
and in `podStatus.conditions.{status: "True", type: Initialized}`
```

Mini-design for this change:

Goals:
1. A user can create an object with the beta annotation
  on 1.4, and it works.  The fact that the annotation has beta
  in it communicates to the user that the feature is beta,
  and so the user should have confidence in using it.  Preferably,
  when the user gets the annotation back, he see the beta
  annotation.
1) If someone had an existing alpha object in their apiserver,
  such as a RS with a pod template with an init-containers
  annotation on it, it should continue to work (init containers
  run) when stack upgraded to 1.4.
2) If someone is using a chart or blog post that has alpha
  annotation on it and they create it on a 1.4 cluster, it should
  work.
3) If someone had something with an init container in 1.4
  and they roll back stack to 1.3, it should not silently stop
  working (init containers don't run anymore).

To meet all these, we mirror an absent beta label from the alpha
key and vice versa.  If they are out of sync, we use the alpha
one. We do this in conversion since there was already logic there.

In 1.3 code, all annotations are preserved across a round trip
(v1 -> api -> v1), and the alpha annotation turns into the internal
field that kubelet uses.

In 1.4 code, the alpha annotation is always preserved across
a round trip, and a beta annotation is always set equal to
the alpha one, after a round trip.

Currently, the kubelet always sees the object after a round trip
when it GETs it.  But, we don't want to rely on that behavior,
since it will break when fastpath is implemented.

So, we rely on this:
all objects either are created with an alpha annotation (1.3 or 1.4
code) or are created with a beta annotation under 1.4.  In the later
case, they are round tripped at creation time, and so get both
annotations.  So all subsequent GETs see both labels.