Generate yaml docs for kubectl

Remove Set() from storage.Interface.

Return more useful error information when a persistent volume fails to mount

Trusty: Avoid unnecessary in-memory temp files

Automatic merge from submit-queue

Refactor E2E framework & utils to test/e2e/framework package

Refactor E2E framework & utils to test/e2e/framework package to pave the way for further refactoring and code sharing with the node e2e suite.

I've split this PR into 2 commits to try to make it more manageable: The first moves the code to the framework package (which builds on it's own), and the second fixes the test/e2e package. Unfortunately the code is so tangled up that moving it all at once this way is much easier than separating out individual pieces. Although the separation would probably be a worthwhile exercise on it's own, the goal here is to unblock the e2e_node hackathon.

One way I could reduce the number of changes is by importing the framework package as `.`. I didn't do this since I think it's not great practice - but I'm open to making this change if you (reviewers) think otherwise.

LMK if there's any way I can help to make this more reviewable.

@kubernetes/sig-node @kubernetes/sig-testing @vishh @pwittrock @jayunit100 

Automatic merge from submit-queue

Rewrite docker-checker.sh to make it less kill-happy.

e2e: fix error checking in kubelet stats

Automatic merge from submit-queue

Fixed mounting with containerized kubelet

`NsenterMounter.IsLikelyNotMountPoint()` should return `ErrNotExist` when the
checked directory does not exists - the regular mounted does this and
some volume plugins depend on this behavior.

See for example: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/aws_util.go#L72

Automatic merge from submit-queue

Adding nodeports services to quota

To fix #21677
@derekwaynecarr 

Automatic merge from submit-queue

Implement a streaming serializer for watch

Changeover watch to use streaming serialization. Properly version the
watch objects. Implement simple framing for JSON and Protobuf (but not
YAML).

@wojtek-t @lavalamp

Automatic merge from submit-queue

Increase 5 node kubemark to a 60m timeout, schedule every 5m

Closes https://github.com/kubernetes/kubernetes/issues/24183 and should reduce mergebot flakes

http://kubekins.dls.corp.google.com/view/Submit%20Queue/job/kubernetes-kubemark-5-gce/1354/consoleFull is an example where everything seemed to be running find but ran out of time after 30m, which is only slightly slower than its average passing runtime of 20m. I'm increasing to 3x average runtime.

Automatic merge from submit-queue

Add godoc to kubelet/volumes.go

Noticed that `mountExternalVolumes`, of all things, was missing Godoc while working w/ @screeley44.  Decided to add some tonight since I have been making noise about grokkability of the kubelet lately.

@kubernetes/sig-storage  

Automatic merge from submit-queue

IPerf container to support network perfomance testing

Simple iperf container.

Issue:
We want to run iperf from the e2e tests for a network baseline, but there are no gcr images for this.

Solution:
Curate our own iperf container from source in kubernetes and copy it as a top level microservice.  So long as these are injected into GCR, we can then run this container from the e2e tests.

 cc @sig-testing this can be used along side #22869 

Automatic merge from submit-queue

Add suite pages to test-history

This changes http://storage.googleapis.com/kubernetes-test-history/static/index.html to look like:

![image](https://cloud.githubusercontent.com/assets/211868/14445317/f263e1da-0000-11e6-9bd9-01150d9ae4ba.png)

And adds sub-pages to show test results for specific suites:

![image](https://cloud.githubusercontent.com/assets/211868/14445339/1bb8555c-0001-11e6-9e5c-41d671af39a1.png)

Revert "Run most GCE, GKE, and Kubemark e2e jobs inside Docker"

This makes it easier to determine which tests cause particular suites to
fail.

All static HTML pages are now generated by one invocation of gen_html.py.

- make index include good/flake/fail numbers for each link
- consistently use % for string interpolation

e2e-runner: don't clean old binaries before upgrade

Run most GCE, GKE, and Kubemark e2e jobs inside Docker

Automatic merge from submit-queue

Start running [Serial] tests in Trusty CI jobs and the release job of Trusty-dev

Adding these tests as requested in https://github.com/kubernetes/kubernetes/issues/23956#issuecomment-207096136

@spxtr Can you review this?
cc/ @dchen1107 @andyzheng0831 

Automatic merge from submit-queue

disable nfsv4 and use nfsv3

fix #22529 

Automatic merge from submit-queue

e2e: adapt kubelet_perf.go to use the new summary metrics API

This commit switch most functions in kubelet_stats.go to use the new API.
However, the functions that perform one-time resource usage retrieval remain
unchanged to be compatible with reource_usage_gatherer.go. They should be
handled separately.

Also, the new summary API does not provide the RSS memory yet, so all memory
checking tests will *always* pass. We plan to add this metrics in the API and
restore the functionality of the test.

Automatic merge from submit-queue

Remove SecurityContextDeny from vagrant setup

This should not be needed in the vagrant setup.

Automatic merge from submit-queue

Make watch cache treat resourceVersion consistent with uncached watch

Fixes #24004

This makes the watch cache handle resourceVersion consistent with an uncached watch API call, and the documented behavior. Watching from resourceVersion=X delivers watch events *after* version X (X is not included):

> // When specified with a watch call, shows changes that occur after that particular version of a resource.
> // Defaults to changes from the beginning of history.
> ResourceVersion string

Automatic merge from submit-queue

Convert the node conformance tests to use dedicated images.

This PR will de-flake the node e2e.

Add an e2e test suite for multi-zone GKE clusters.

This commit switch most functions in kubelet_stats.go to use the new API.
However, the functions that perform one-time resource usage retrieval remain
unchanged to be compatible with reource_usage_gatherer.go. They should be
handled separately.

Also, the new summary API does not provide the RSS memory yet, so all memory
checking tests will *always* pass. We plan to add this metrics in the API and
restore the functionality of the test.

Automatic merge from submit-queue

Move predicates into library

This PR tries to implement #12744 

Any suggestions/ideas are welcome. @davidopp 

current state: integration test fails if including podCount check in Kubelet.
    
DONE:
    1. refactor all predicates: predicates return fitOrNot(bool) and error(Error) in which the latter is of type PredicateFailureError or InsufficientResourceError
    2. GeneralPredicates() is a predicate function, which includes serveral other predicate functions (PodFitsResource, PodFitsHost, PodFitsHostPort). It is registered as one of the predicates in DefaultAlgorithmProvider, and is also called in canAdmitPod() in Kubelet and should be called by other components (like rescheduler, etc if necessary. See discussion in issue #12744
TODO:
    1. determine which predicates should be included in GeneralPredicates()
    2. separate GeneralPredicates() into: a.) GeneralPredicatesEvictPod() and b.) GeneralPredicatesNotEvictPod()
    3. DaemonSet should use GeneralPredicates()

Automatic merge from submit-queue

Preserve int data when unmarshaling

There are several places we use `json.Unmarshal` into an unstructured map (StrategicMergePatch, UnstructuredJSONScheme, many others).

In this scenario, the json package converts all numbers to float64. This exposes many of the int64 fields in our API types to corruption when the unstructured map is marshalled back to json.

A simple example is a pod with an `"activeDeadlineSeconds": 1000000`. Trying to use `kubectl label`, `annotate`, `patch`, etc results in that int64 being converted to a float64, submitted to the server, and the server rejecting it with an error about "cannot unmarshal number 1e+6 into Go value of type int64"

The json package provides a way to defer conversion of numbers (`json.Decoder#UseNumber`), but does not actually do conversions to int or float. This PR makes use of that feature, and post-processes the unmarshalled map to convert json.Number objects to either int64 or float64 values

Automatic merge from submit-queue

Additional go vet fixes

Mostly:
- pass lock by value
- bad syntax for struct tag value
- example functions not formatted properly

Automatic merge from submit-queue

kubele: add RSS memroy to the summary API

Automatic merge from submit-queue

Fix valid CI version regex in parse_and_validate_release_version().

Ref: #23759
cc @ihmccreery