Automatic merge from submit-queue

[RFC] Prepare for deprecating NodeLegacyHostIP

Ref https://github.com/kubernetes/kubernetes/issues/9267#issuecomment-257994766

*What this PR does*
- Add comments saying "LegacyHostIP" will be deprecated in 1.7;
- Add v1.NodeLegacyHostIP to be consistent with the internal API (useful for client-go migration #35159)
- Let cloudproviders who used to only set LegacyHostIP set the IP as both InternalIP and ExternalIP
- Master used to ssh tunnel to node's ExternalIP or LegacyHostIP to do [healthz check](https://github.com/kubernetes/kubernetes/blame/master/pkg/master/master.go#L328-L332). OTOH, if on-prem, kubelet only [sets](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go#L430-L431) LegacyHostIP or InternalIP. In order to deprecate LegacyHostIP in 1.7, I let healthz check to use InternalIP if ExternalIP is not available. (The healthz check is the only consumer of LegacyHostIP in k8s.)

@liggitt @justinsb @bgrant0607 

```release-note
LegacyHostIP will be deprecated in 1.7.
```

Automatic merge from submit-queue

[Federation][init-10c] Fix hyperkube binary path in the image and disable auth mechanisms we don't support for now.

Please review only the last commit here. This is based on PR #36047 which will be reviewed independently.

Design Doc: PR #34484

cc @kubernetes/sig-cluster-federation @nikhiljindal

Automatic merge from submit-queue

Add caching for discovery info with invalidation on cache-miss

TODO:
- [x] write tests for `CachedDiscoveryClient`
- [x] write tests for `DeferredDiscoveryRESTMapper` on cache-miss
- [x] find better way/structure to get rid of `invalidateCh` in https://github.com/kubernetes/kubernetes/pull/35418/commits/c06ba3175b0571ee1085e1dab2146b67d8325acf

Automatic merge from submit-queue

Fix LBaaS version detection in openstack cloudprovider

`lbversion` is the local variable used for version detection when `os.lbOpts.LBVersion` is not specified.

xref https://bugzilla.redhat.com/show_bug.cgi?id=1391837

@ncdc @derekwaynecarr @anguslees

Automatic merge from submit-queue

Fix possible race in operationNotSupportedCache

Because we can run multiple workers to delete namespaces simultaneously, the
operationNotSupportedCache needs to be guarded with a mutex to avoid concurrent
map read/write errors.

Automatic merge from submit-queue

Reduce spam in Events from scheduler by counter aggregation of failure

Fixes #35842
Part of overall #35555

Automatic merge from submit-queue

Node Conformance Test: Add system verification

For #30122 and #29081.

This PR introduces system verification test in node e2e and conformance test. It will run before the real test. Once the system verification fails, the test will just fail. The output of the system verification is like this:

```
I0909 23:33:20.622122    2717 validators.go:45] Validating os...
OS: Linux
I0909 23:33:20.623274    2717 validators.go:45] Validating kernel...
I0909 23:33:20.624037    2717 kernel_validator.go:79] Validating kernel version
KERNEL_VERSION: 3.16.0-4-amd64
I0909 23:33:20.624146    2717 kernel_validator.go:93] Validating kernel config
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
I0909 23:33:20.679328    2717 validators.go:45] Validating cgroups...
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
I0909 23:33:20.679454    2717 validators.go:45] Validating docker...
DOCKER_GRAPH_DRIVER: aufs
```

It verifies the system following a predefined `SysSpec`:

``` go
// DefaultSysSpec is the default SysSpec.
 var DefaultSysSpec = SysSpec{
    OS:            "Linux",
    KernelVersion: []string{`3\.[1-9][0-9].*`, `4\..*`}, // Requires 3.10+ or 4+
    // TODO(random-liu): Add more config
    KernelConfig: KernelConfig{
        Required: []string{
            "NAMESPACES", "NET_NS", "PID_NS", "IPC_NS", "UTS_NS",
            "CGROUPS", "CGROUP_CPUACCT", "CGROUP_DEVICE", "CGROUP_FREEZER",
            "CGROUP_SCHED", "CPUSETS", "MEMCG",
        },
        Forbidden: []string{},
    },
    Cgroups: []string{"cpu", "cpuacct", "cpuset", "devices", "freezer", "memory"},
    RuntimeSpec: RuntimeSpec{
        DockerSpec: &DockerSpec{
            Version: []string{`1\.(9|\d{2,})\..*`}, // Requires 1.9+
            GraphDriver: []string{"aufs", "overlay", "devicemapper"},
        },
    },
 }
```

Currently, it only supports:
- Kernel validation: version validation and kernel configuration validation
- Cgroup validation: validating whether required cgroups subsystems are enabled.
- Runtime Validation: currently, only validates docker graph driver.

The validating framework is ready. The specific validation items could be added over time.

@dchen1107 
/cc @kubernetes/sig-node

Automatic merge from submit-queue

Fix the crossbuild that #35132 broke

@dashpole @dchen1107 @vishh 

A quick LGTM would be nice in order to not block any releases.

Automatic merge from submit-queue

Verify generated files

Fixes #33751 

Once I un-check-in generated files, this becomes moot again.

Automatic merge from submit-queue

Rename PetSet to StatefulSet in docs and examples.

**What this PR does / why we need it**: Addresses some of the pre-code-freeze changes for implementing the PetSet --> StatefulSet rename. (#35534)

**Special notes for your reviewer**: This PR only changes docs and examples, as #35731 hasn't been merged yet and I don't want to create merge conflicts. I'll open another PR for any remaining code changes needed after that PR is merged. /cc @erictune @janetkuo @chrislovecnm

Automatic merge from submit-queue

E2e test for federated deployment controller

Based on e2e tests for federated deployment.

cc: @quinton-hoole

Automatic merge from submit-queue

kubelet bootstrap: start hostNetwork pods before we have PodCIDR

Network readiness was checked in the pod admission phase, but pods that
fail admission are not retried.  Move the check to the pod start phase.

Issue #35409 
Issue #35521

Automatic merge from submit-queue

Add DisruptedPod list to PodDisruptionBudgetStatus

To ensure that PodDisruptionBudget is race free a list of pods that were planned to be disrupted needs to be added to the status. ApiServer when evicting a pod will add it to this list. Disruption controller will skip pods from that list when calculating the number of healthy pods. The pods from the list are removed either when they are gone or when they were not actually disrupted.

Automatic merge from submit-queue

Add missing expansion files to versioned clientset

I copied the expansion functions that only existed in the internalclientset to release_1_5.

Most changes are mechanical. This is needed for migrating k8s to use versioned clientset, so I add the 1.5 milestone.

Automatic merge from submit-queue

Update Secret.type examples used for provisioning.

Kubernetes now checks that secrets used for provisioning of volumes
have correct type (PR #35675), all examples should reflect it.

Automatic merge from submit-queue

CRI: rearrange kubelet rutnime initialization

Consolidate the code used by docker+cri and remote+cri for consistency, and to
prevent changing one without the other.  Enforce that
`--experimental-runtime-integration-type` has to be set in order for kubelet
use the CRI interface, *even for out-of-process shims`. This simplifies the
temporary `if` logic in kubelet while CRI still co-exists with older logic.

Automatic merge from submit-queue

Increase quota-bytes for etcd in v3 mode

Ref #20504

Automatic merge from submit-queue

E2e tests for federated daemonset

Based on federated secrets e2e tests.

cc: @quinton-hoole

Automatic merge from submit-queue

Cleanup auth logging, allow starting secured kubelet in local-up-cluster.sh

Cleanup for https://github.com/kubernetes/features/issues/89

Automatic merge from submit-queue

Made fluentd-gcl config consitent for GCI and ContainerVM

Automatic merge from submit-queue

Enable HPA controller based on autoscaling/v1 api group

ref #29778

``` release-note
Enable HPA controller based on autoscaling/v1 api group.
```

Automatic merge from submit-queue

SetSelfLink is inefficient

Generating self links, especially for lists, is inefficient.  Replace
use of net.URL.String() call with direct encoding that reduces number of
allocations. Switch from calling meta.ExtractList|SetList to a function
that iterates over each object in the list.

In steady state for nodes performing frequently small get/list
operations, and for larger LISTs significantly reduces CPU and
allocations.

@wojtek-t this is the next big chunk of CPU use during the large N nodes simulation test (11% of master CPU). Takes a few allocations out of the critical path

Automatic merge from submit-queue

lister-gen updates

- Remove "zz_generated." prefix from generated lister file names
- Add support for expansion interfaces
- Switch to new generated JobLister

@deads2k @liggitt @sttts @mikedanese @caesarxuchao for the lister-gen changes
@soltysh @deads2k for the informer / job controller changes

Automatic merge from submit-queue

cluster-addons: enable Jemalloc for Fluentd based images

**What this PR does / why we need it**:

This Pull Request includes two patches that enable the recommended use of Jemalloc memory allocator for container images that are based in Fluentd. The patches applies to the following cluster-addons:
- fluentd-es-image
- fluentd-gcp-image

**Which issue this PR fixes** 

This PR is part of the solution for issues:
-  kubernetes/kubernetes/issues/32762
-  GoogleCloudPlatform/fluent-plugin-google-cloud/issues/87

When Fluentd runs in high load environments, it's likely the default operating system memory allocator will generate a high fragmentation ending up in a high memory usage. In order to reduce fragmentation and decrease memory usage an alternative memory allocator as Jemalloc is used. 

![](https://cloud.githubusercontent.com/assets/369718/19498577/eaa9f324-954e-11e6-9a6b-6b30310a66a3.png)

For the record: fluentd-es-image uses [td-agent](https://docs.treasuredata.com/articles/td-agent) Fluentd package maintained by Treasure Data, which contains Jemalloc 4.2.1 (latest stable version). The google-fluentd package used in fluentd-gcp-image comes with Jemalloc 2.2.5, which have many known issues, I strongly suggest google-fluentd package gets updated.

**Special notes for your reviewer**:

In the research of this topic have been involved @piosz and @Crassirostris.

Automatic merge from submit-queue

Remove PV annotations for quobyte provisioner

This is the last provisioner that uses annotations to pass secrets from provisioner to deleter.

Fixes #34822

@johscheuer, I don't have access to quobyte, please take look and retest the plugin. An e2e test for quobyte would be nice!

@kubernetes/sig-storage

Automatic merge from submit-queue

Remove unused WaitForDetach from Detacher interface and plugins

See issue #33128 and PR #33270

We can't rely on the device name provided by OpenStack Cinder, and thus
must perform detection based on the drive serial number (aka It's cinder ID)
on the kubelet itself.

This needs to be removed now, as part of #33128, as the code can't be
updated to attempt device detection and fallback through to the Cinder
provided deviceName, as detection "fails" when the device is gone, and
if cinder has reported a deviceName that another volume has used in
relaity, then this will block forever (or until the other, unreleated,
volume has been detached)

Automatic merge from submit-queue

CRI: Add Status into CRI.

For https://github.com/kubernetes/kubernetes/issues/35701.
Fixes https://github.com/kubernetes/kubernetes/issues/35701.

This PR added a `Status` call in CRI, and the `RuntimeStatus` is defined as following:

``` protobuf
message RuntimeCondition {
    // Type of runtime condition.
    optional string type = 1;
    // Status of the condition, one of true/false.
    optional bool status = 2;
    // Brief reason for the condition's last transition.
    optional string reason = 3;
    // Human readable message indicating details about last transition.
    optional string message = 4;
}

message RuntimeStatus {
    // Conditions is an array of current observed runtime conditions.
    repeated RuntimeCondition conditions = 1;
}
```

Currently, only `conditions` is included in `RuntimeStatus`, and the definition is almost the same with `NodeCondition` and `PodCondition` in K8s api.

@yujuhong @feiskyer @bprashanth If this makes sense, I'll send a follow up PR to let dockershim return `RuntimeStatus` and let kubelet make use of it.
@yifan-gu @euank Does this make sense to rkt?
/cc @kubernetes/sig-node

Automatic merge from submit-queue

Add rescheduler.log to the logs exported from master

Related to https://github.com/kubernetes/kubernetes/issues/36227

@piosz

Automatic merge from submit-queue

Kubedns logging

fixes 
https://github.com/kubernetes/kubernetes/issues/29053

may resolve https://github.com/kubernetes/kubernetes/issues/29054, but depends on what the specific ask is

Automatic merge from submit-queue

We only report diskpressure to users, and no longer report inodepressure

See #36180 for more information on why #33218 was reverted.

Automatic merge from submit-queue

CRI: stop sandbox before removing it

Stopping a sandbox includes reclaiming the network resources.  By always
stopping the sandbox before removing it, we reduce the possibility of leaking
resources in some corner cases.

Automatic merge from submit-queue

Add cmd support to gcp auth provider plugin

**What this PR does / why we need it**:

Adds ability for gcp auth provider plugin to get access token by shelling out to an external command. We need this because for GKE, kubectl should be using gcloud credentials. It currently uses google application default credentials, which causes confusion if user has configured both with different permissions (previously the two were almost always identical).

**Which issue this PR fixes**:
Addresses #35530 with gcp-only solution, as generic cmd plugin was deemed not useful for other providers.

**Special notes for your reviewer**:

Configuration options are to support whatever future command gcloud provides for printing access token of active user. Also works with existing command (`gcloud auth print-access-token`)

```release-note
```

Automatic merge from submit-queue

e2e: bump limits for kubelet's cpu usage

The numbers are chosen based on the past tests results.

Automatic merge from submit-queue

Remove GetRootContext method from VolumeHost interface

Remove the `GetRootContext` call from the `VolumeHost` interface, since Kubernetes no longer needs to know the SELinux context of the Kubelet directory.

Per #33951 and #35127.

Depends on #33663; only the last commit is relevant to this PR.

Automatic merge from submit-queue

Initial work on running windows containers on Kubernetes

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

This is the first stab at getting the Kubelet running on Windows (fixes #30279), and getting it to deploy network-accessible pods that consist of Windows containers. Thanks @csrwng, @jbhurat for helping out.

The main challenge with Windows containers at this point is that container networking is not supported. In other words, each container in the pod will get it's own IP address. For this reason, we had to make a couple of changes to the kubelet when it comes to setting the pod's IP in the Pod Status. Instead of using the infra-container's IP, we use the IP address of the first container.

Other approaches we investigated involved "disabling" the infra container, either conditionally on `runtime.GOOS` or having a separate windows-docker container runtime that re-implemented some of the methods (would require some refactoring to avoid maintainability nightmare). 

Other changes:
- The default docker endpoint was removed. This results in the docker client using the default for the specific underlying OS.

More detailed documentation on how to setup the Windows kubelet can be found at https://docs.google.com/document/d/1IjwqpwuRdwcuWXuPSxP-uIz0eoJNfAJ9MWwfY20uH3Q. 

cc: @ikester @brendandburns @jstarks