Automatic merge from submit-queue (batch tested with PRs 41306, 42187, 41666, 42275, 42266)

Implement bulk polling of volumes

This implements Bulk volume polling using ideas presented by
justin in https://github.com/kubernetes/kubernetes/pull/39564

But it changes the implementation to use an interface
and doesn't affect other implementations.

cc @justinsb

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Make Downward API test table-driven

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

kubeadm: Add --cert-dir, --cert-altnames instead of --api-external-dns-names

**What this PR does / why we need it**:

 - For the beta kubeadm init UX, we need this change
 - Also adds the `kubeadm phase certs selfsign` command that makes the phase invokable independently

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

This PR depends on https://github.com/kubernetes/kubernetes/pull/41897

**Release note**:

```release-note
```
@dmmcquay @pires @jbeda @errordeveloper @mikedanese @deads2k @liggitt

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Updated scheduler name for multi-scheduler.

fixes #41859

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

controller: statefulsets respect observed generation

StatefulSets do not update ObservedGeneration even though the API field is in place. This means that clients can never be sure whether the StatefulSet controller has observed the latest spec of a StatefulSet.

@kubernetes/sig-apps-bugs

Automatic merge from submit-queue (batch tested with PRs 42365, 42429, 41770, 42018, 35055)

Remove kube-dns ConfigMap modification code from federation-{up,down}.sh scripts

PR #39338 has merged now. This shouldn't be necessary. This unbreaks federation tests.

```release-note
NONE
```

Handle potential conflicts on service endpoint test deletion

Cleanup should ignore conflicts and notfound errors

e2e/framework: Deflake cleanup of RCs in service_util.

The previous Get/Update pattern with no retry on resource version mismatch
would flake with the following error:

"the object has been modified; please apply your changes to the latest
version and try again"

Automatic merge from submit-queue

Remove defaults from string flags

- The default is printed automatically
 - The string text did not match the actual default

**What this PR does / why we need it**:
Adjust the documentation for flags on `client-gen`.

**Special notes for your reviewer**:
Doc change. String text only.

**Release note**:
```release-note
NONE
```

Before:
```
client-gen  --help
Usage of ./client-gen:
      --build-tag string                       A Go build tag to use to identify files generated by this command. Should be unique. (default "ignore_autogenerated")
      --clientset-api-path string              the value of default API path.
  -n, --clientset-name string                  the name of the generated clientset package. (default "internalclientset")
      --clientset-only                         when set, client-gen only generates the clientset shell, without generating the individual typed clients
      --clientset-path string                  the generated clientset will be output to <clientset-path>/<clientset-name>. Default to "k8s.io/kubernetes/pkg/client/clientset_generated/" (default "k8s.io/kubernetes/pkg/client/clientset_generated/")
      --fake-clientset                         when set, client-gen will generate the fake clientset that can be used in tests (default true)
  -h, --go-header-file string                  File containing boilerplate header text. The string YEAR will be replaced with the current 4-digit year. (default "/Users/mhb/go/src/k8s.io/gengo/boilerplate/boilerplate.go.txt")
      --included-types-overrides stringSlice   list of group/version/type for which client should be generated. By default, client is generated for all types which have genclient=true in types.go. This overrides that. For each groupVersion in this list, only the types mentioned here will be included. The default check of genclient=true will be used for other group versions.
      --input stringSlice                      group/versions that client-gen will generate clients for. At most one version per group is allowed. Specified in the format "group1/version1,group2/version2...". Default to "api/,extensions/,autoscaling/,batch/,rbac/" (default [api/,authentication/,authorization/,autoscaling/,batch/,certificates/,extensions/,rbac/,storage/,apps/,policy/])
      --input-base string                      base path to look for the api group. Default to "k8s.io/kubernetes/pkg/apis" (default "k8s.io/kubernetes/pkg/apis")
  -i, --input-dirs stringSlice                 Comma-separated list of import paths to get input types from.
  -o, --output-base string                     Output base; defaults to $GOPATH/src/ or ./ if $GOPATH is not set. (default "/Users/mhb/go/src")
  -O, --output-file-base string                Base name (without .go suffix) for output files.
  -p, --output-package string                  Base package path.
  -t, --test                                   set this flag to generate the client code for the testdata
      --verify-only                            If true, only verify existing output, do not write anything.
```
After:
```
client-gen  --help
Usage of ./client-gen:
      --build-tag string                       A Go build tag to use to identify files generated by this command. Should be unique. (default "ignore_autogenerated")
      --clientset-api-path string              the value of default API path.
  -n, --clientset-name string                  the name of the generated clientset package. (default "internalclientset")
      --clientset-only                         when set, client-gen only generates the clientset shell, without generating the individual typed clients
      --clientset-path string                  the generated clientset will be output to <clientset-path>/<clientset-name>. (default "k8s.io/kubernetes/pkg/client/clientset_generated/")
      --fake-clientset                         when set, client-gen will generate the fake clientset that can be used in tests (default true)
  -h, --go-header-file string                  File containing boilerplate header text. The string YEAR will be replaced with the current 4-digit year. (default "/Users/mhb/go/src/k8s.io/gengo/boilerplate/boilerplate.go.txt")
      --included-types-overrides stringSlice   list of group/version/type for which client should be generated. By default, client is generated for all types which have genclient=true in types.go. This overrides that. For each groupVersion in this list, only the types mentioned here will be included. The default check of genclient=true will be used for other group versions.
      --input stringSlice                      group/versions that client-gen will generate clients for. At most one version per group is allowed. Specified in the format "group1/version1,group2/version2...". (default [api/,authentication/,authorization/,autoscaling/,batch/,certificates/,extensions/,rbac/,storage/,apps/,policy/])
      --input-base string                      base path to look for the api group. (default "k8s.io/kubernetes/pkg/apis")
  -i, --input-dirs stringSlice                 Comma-separated list of import paths to get input types from.
  -o, --output-base string                     Output base; defaults to $GOPATH/src/ or ./ if $GOPATH is not set. (default "/Users/mhb/go/src")
  -O, --output-file-base string                Base name (without .go suffix) for output files.
  -p, --output-package string                  Base package path.
  -t, --test                                   set this flag to generate the client code for the testdata
      --verify-only                            If true, only verify existing output, do not write anything.
```

This implements Bulk volume polling using ideas presented by
justin in https://github.com/kubernetes/kubernetes/pull/39564

But it changes the implementation to use an interface
and doesn't affect other implementations.

Automatic merge from submit-queue

Describers with pod templates should have consistent output

Added a test to verify it.

Fixes #38698

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

Move node problem detector test into node e2e.

Move current NPD e2e test into node e2e.

In fact, current NPD e2e test is only a functionality test for NPD. It creates test NPD pod, sets test configuration, generates test logs and verifies test result.
It doesn't actually test the NPD really deployed in the cluster.

So it doesn't actually need to run in cluster e2e. Running it in node e2e will:
1) Make it easier to run the test.
2) Make it more light weight to introduce this as a pre/post submit test in NPD repo in the future.

Except this, I'm working on a cluster e2e to run some basic functionality test and benchmark test against the real NPD deployed in the cluster. Will send the PR later.

/cc @dchen1107 @kubernetes/node-problem-detector-reviewers

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

make edit respect --save-config

New behavior:

edit:
1) the flag is set to true: it will create or update the last-applied-config annotation
2) the flag is set to false or is unspecified: NOP

replace:
1) If saveConfig is true, create or update the annotation.
2) If saveConfig is false and the local config file doesn't have the annotation, we save the annotation from the live object if there is one (Try to not surprise the users).
3) If saveConfig is false and the local config file has the annotation, we use the annotation in the config file.
4) Same behavior for force replacing

fixes #40626

```release-note
stop kubectl edit from updating the last-applied-configuration annotation when --save-config is unspecified or false.
```

@pwittrock @liggitt 

This is a bug fix that prevented `edit` from being respected by `apply`

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

Add options to kubefed telling it to generate HTTP Basic and/or token credentials for the Federated API server

fixes #41265.

**Release notes**:
```release-note
Adds two options to kubefed, `-apiserver-enable-basic-auth` and `-apiserver-enable-token-auth`, which generate an HTTP Basic username/password and a token respectively for the Federated API server.
```

Automatic merge from submit-queue (batch tested with PRs 41984, 41682, 41924, 41928)

RC/RS: Fully Respect ControllerRef

**What this PR does / why we need it**:

This is part of the completion of the [ControllerRef](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/controller-ref.md) proposal. It brings ReplicaSet and ReplicationController into full compliance with ControllerRef. See the individual commit messages for details.

**Which issue this PR fixes**:

Although RC/RS had partially implemented ControllerRef, they didn't use it to determine which controller to sync, or to update expectations. This could lead to instability or controllers getting stuck.

Ref: https://github.com/kubernetes/kubernetes/issues/24433

**Special notes for your reviewer**:

**Release note**:
```release-note
```
cc @erictune @kubernetes/sig-apps-pr-reviews

kubeadm: Add --cert-dir, --apiserver-cert-extra-sans, remove --api-external-dns-names and add the phase command for certs. Also use the CertificatesDir var everywhere instead of the HostPKIPath variable and fix some bugs in certs.go

 - The default is printed automatically
 - The string text did not match the actual default

Automatic merge from submit-queue

StorageClass should not print the namespace column

Fixes #40091

get fresh ports on startup failure for integration test

Automatic merge from submit-queue

Only set a groupresource on errors if a resource was specified

@deads2k this gets rid of the weird .meta.k8s.io error on unauthorized errors

Images field didn't include init containers. Use pod template output
consistently and ensure labels and annotations are in the right spot

Added a test to verify it.

Automatic merge from submit-queue (batch tested with PRs 42128, 42064, 42253, 42309, 42322)

Default E2E_ZONES to empty string.

Also print an error message and exit if host cluster context cannot be derived.

I don't think we should do any of this magic. This is a stop gap solution to get all our tests to stable state. I will actively get rid of all this defaulting magic once our main tests are stable.

cc @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 42128, 42064, 42253, 42309, 42322)

Fix godep race in local builds

**What this PR does / why we need it**:
This fixes a small race condition when building a godep version other than the latest. As you're building godep once, and then building it again after checkout, this can cause inconsistency in the version selected for use. This fix tells the first `go get` to only download and not build. Then we build manually after checking out the target version.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 42128, 42064, 42253, 42309, 42322)

Fix panic on nil invalid field error

bug fix for validation panic

if a field.Invalid is constructed with a nil badvalue, the Error() method panics, since reflect.TypeOf() returns nil

Automatic merge from submit-queue (batch tested with PRs 42128, 42064, 42253, 42309, 42322)

kubeadm: Rename some flags for beta UI and fixup some logic

**What this PR does / why we need it**:

In this PR:
 - `--api-advertise-addresses` becomes `--apiserver-advertise-address`
   - The API Server's logic here is that if the address is `0.0.0.0`, it chooses the host's default interface's address. kubeadm here uses exactly the same logic. This arg is then passed to `--advertise-address`, and the API Server will advertise that one for the service VIP.
 - `--api-port` becomes `--apiserver-bind-port` for clarity

ref the meeting notes: https://docs.google.com/document/d/1deJYPIF4LmhGjDVaqrswErIrV7mtwJgovtLnPCDxP7U/edit#

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
```
@jbeda @dmmcquay @pires @lukemarsden @dgoodwin @mikedanese

Automatic merge from submit-queue (batch tested with PRs 42128, 42064, 42253, 42309, 42322)

Add storage.k8s.io/v1 API

This is combined version of reverted #40088 (first 4 commits) and #41646. The difference is that all controllers and tests use old `storage.k8s.io/v1beta1` API so in theory all tests can pass on GKE.

Release note:
```release-note
StorageClassName attribute has been added to PersistentVolume and PersistentVolumeClaim objects and should be used instead of annotation `volume.beta.kubernetes.io/storage-class`. The beta annotation is still working in this release, however it will be removed in a future release.
```

Automatic merge from submit-queue (batch tested with PRs 41672, 42084, 42233, 42165, 42273)

ExecProbes should be able to do simple env var substitution

For containers that don't have bash, we should support env substitution
like we do on command and args. However, without major refactoring
valueFrom is not supportable from inside the prober. For now, implement
substitution based on hardcoded env and leave TODOs for future work.

Improves the state of #40846, will spawn a follow up issue for future refactoring after CRI settles down

Automatic merge from submit-queue (batch tested with PRs 41672, 42084, 42233, 42165, 42273)

Don't sync IPtables before underlying store/reflector is fully synced

Ref #42108

Build on top of #42108 - only the second commit is unique.

Automatic merge from submit-queue (batch tested with PRs 41672, 42084, 42233, 42165, 42273)

sort token usages in kubeadm

```
kubeadm token list
TOKEN                     TTL         EXPIRES   USAGES                   DESCRIPTION
0f97a4.2230bb81a54f470f   <forever>   <never>   authentication,signing   <none>
1ee905.3ffae0f3e189ebf3   <forever>   <never>   signing,authentication   <none>
2fc984.66a220428aed0794   <forever>   <never>   authentication,signing   <none>
6b8f54.11d2bc3cec1c2b40   <forever>   <never>   signing,authentication   <none>
81a8f9.c3fa30dd1ed05d96   <forever>   <never>   authentication,signing   The default bootstrap token generated by 'kubeadm init'.
b04eec.2b2337a87558d3f7   <forever>   <never>   signing,authentication   <none>
b186e5.7a19ae1a82652643   <forever>   <never>   authentication,signing   <none>
```

Automatic merge from submit-queue (batch tested with PRs 41672, 42084, 42233, 42165, 42273)

Keep entries of apimachinery in client-go's Godeps.json

1. copy.sh now copies staging/ to a temp dir, creates snapshots for them, and then adds the temp dir to GOPATH before running `godep save`. This way, we don't need the `go list` check added in #41987 so that's removed @sttts.
2. copy.sh fills the entries for k8s.io/apimachinery with dummy SHA1 in client-go's Godeps.json. The publish robot will later fill the actually published one.
3. copy.sh removes the entire `_vendor` from client-go. We cannot keep `_vendor` because it makes hard to use the `git filter-branch` magic for client-go, because the filtered commits might contain changes to `_vendor` while client-go only has `vendor/`.

Some notes on point 3: the robot will restore the vendor/ later, but exclude k8s.io/* and glog, so that users retrieving client-go with `go get` won't end up with multiple copies of them. I think this should be a ***general rule*** for published k8s.io/* repos: every published repos should have complete Godeps.json with latest commit hashes of other k8s.io/* repos, but vendor/k8s.io/* and vendor/glog will be removed from all repos.

@sttts @lavalamp @deads2k

Automatic merge from submit-queue (batch tested with PRs 41672, 42084, 42233, 42165, 42273)

remove azure getting kube-ups.

Haven't been touched in > 7 months. 

@colemickens , i"m going to send out an email about this.

```release-note
Remove Azure kube-up as the Azure community has focused efforts elsewhere.
```

Automatic merge from submit-queue (batch tested with PRs 42126, 42130, 42232, 42245, 41932)

apiserver/pkg/server: include scheme in insecure self client config

Noticed this during bootkube development: https://github.com/kubernetes-incubator/bootkube/issues/325

In Go 1.8's `url.Parse` became more strict, and `url.Parse("127.0.0.1:8080")` now fails.

https://beta.golang.org/doc/go1.8#net_url
https://play.golang.org/p/dw_cPeotG4

Accidentally compiled bootkube with 1.8 and tracked a panic down to the loopback client config. Though we're still using the old "genericapiserver", this seems to be translated from the old one.

The actual panic we observed was

```
E0222 19:40:11.364949       5 server.go:254] Failed to create clientset: parse 127.0.0.1:8080: first path segment in URL cannot contain colon
panic: parse 127.0.0.1:8080: first path segment in URL cannot contain colon
goroutine 35 [running]:
github.com/kubernetes-incubator/bootkube/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/internalversion.NewForConfigOrDie(0xc420728ea0, 0xc420738e30)
        /home/eric/src/github.com/kubernetes-incubator/bootkube/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/internalversion/core_client.go:132 +0x62
github.com/kubernetes-incubator/bootkube/vendor/k8s.io/kubernetes/pkg/master.(*Config).Complete(0xc420739db8, 0x0)
        /home/eric/src/github.com/kubernetes-incubator/bootkube/vendor/k8s.io/kubernetes/pkg/master/master.go:180 +0x40a
github.com/kubernetes-incubator/bootkube/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc42007a3c0, 0x0, 0x0)
        /home/eric/src/github.com/kubernetes-incubator/bootkube/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:347 +0x1e8d
github.com/kubernetes-incubator/bootkube/pkg/bootkube.(*bootkube).Run.func1(0xc4206b01e0, 0xc420164300)
        /home/eric/src/github.com/kubernetes-incubator/bootkube/pkg/bootkube/bootkube.go:124 +0x2f
created by github.com/kubernetes-incubator/bootkube/pkg/bootkube.(*bootkube).Run
        /home/eric/src/github.com/kubernetes-incubator/bootkube/pkg/bootkube/bootkube.go:124 +0xb0
```

I don't actually know if this is the correct fix or if there should be changes to `NewForConfigOrDie`. Am looking for comments more than anything.

edit: @abourget pointed out over on bootkube that the actual panic was fixed in the internalclient by https://github.com/kubernetes/kubernetes/pull/38519.

cc @deads2k @sttts @kubernetes/sig-api-machinery-pr-reviews