Automatic merge from submit-queue

Small Makefile cleanup on build flags

Automatic merge from submit-queue

E2E & Node E2E: Move configmap, docker_containers, downward_api, expansion and secrets test into common directory.

This is the 3rd part of #29494.

For #29081.
Based on #29092, #29806.

The first commit is squash of all dependent commits. Please only review the second commit.

The second PR added 17 lines.

@vishh @timstclair 

Automatic merge from submit-queue

Log more info of container for debugging.

Automatic merge from submit-queue

For e2e_node tests tell etcd to listen on ports 2379 and 4001

This is the default for etcd2, but etcd3 only listens on 2379.
Specifying the ports keeps things consistent no matter which version the user has installed.

Fixes #29117

Automatic merge from submit-queue

Scheduledjob controller

Supersedes #25952; first 7 commits come from #25816

<!--
Checklist for submitting a Pull Request

Please remove this comment block before submitting.

1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md).
2. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md).
3. If you want this PR to automatically close an issue when it is merged,
   add `fixes #<issue number>` or `fixes #<issue number>, fixes #<issue number>`
   to close multiple issues (see: https://github.com/blog/1506-closing-issues-via-pull-requests).
4. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below.
-->

```release-note
```

Automatic merge from submit-queue

Bug fix: Use p.Name instead of pod.Name

For example, if you used `pod.GenerateName`, `pod.Name` might be the empty
string while `p.Name` contains the actual name of your pod. Thus passing
`pod.Name` can result in a `resource name may not be empty` error.

Automatic merge from submit-queue

Mark "downward API volume should update annotations / labels on modification" as flaky test.

Because of a known race https://github.com/kubernetes/kubernetes/issues/29633, the test is flaky.
https://github.com/kubernetes/kubernetes/issues/28462
https://github.com/kubernetes/kubernetes/issues/28416

After we move the test to be shared by e2e and node e2e https://github.com/kubernetes/kubernetes/pull/29807, it flakes more in node e2e.
https://github.com/kubernetes/kubernetes/issues/30132
https://github.com/kubernetes/kubernetes/issues/30124

Mark the test as flaky and we should reenable it after #29633 is fixed.

@saad-ali 

Automatic merge from submit-queue

Run goimport for the whole repo

While removing GOMAXPROC and running goimports, I noticed quite a lot of other files also needed a goimport format. Didn't commit `*.generated.go`, `*.deepcopy.go` or files in `vendor`

This is more for testing if it builds.
The only strange thing here is the gopkg.in/gcfg.v1 => github.com/scalingdata/gcfg replace.
cc @jfrazelle @thockin

Automatic merge from submit-queue

optimize conditions of ServiceReplenishmentUpdateFunc to replenish service

Originally, the  replenishQuota method didn't focus on the third parameter object even if others transfered to it, i think the function is not efficient and perfect. then i use the third param to get MatchResources, it will be more exact. for example, if the old pod was quota tracked and the new was not, the replenishQuota only focus on usage resource of the old pod, still if  the third parameter object is nil, the process will be same as before

Added image-policy proposal

For example, if you used pod.GenerateName, pod.Name might be the empty
string while p.Name contains the actual name of your pod. Thus passing
pod.Name can result in a `resource name may not be empty` error.

Automatic merge from submit-queue

add subjectaccessreviews resource

Adds a subjectaccessreviews endpoint that uses the API server's authorizer to determine if a subject is allowed to perform an action.

Part of kubernetes/features#37 

modification" as flaky.

Automatic merge from submit-queue

enhancements to run local-up-cluster.sh

- fix containerized kubelet: bind mount /dev and cloud credential directories
- add CLOUD_CONFIG env to get openstack credentials

@kubernetes/sig-storage @jayunit100 

Automatic merge from submit-queue

Detail unit testing workflow

Automatic merge from submit-queue

docs: Detail possible transitions in CRI

Right now the document doesn't make it clear that transitions are unidirectional and a exited container won't be restarted, but replaced by a fresh copy.

cc @yujuhong @feiskyer @kubernetes/sig-node 

Automatic merge from submit-queue

Allow starting test etcd with http

There are cases where we'll want to start a test etcd without https (external tools monitoring tests, tests with lots of parallelism, etc). This makes it an option.

Helps with https://github.com/kubernetes/kubernetes/issues/23127

<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/22919)
<!-- Reviewable:end -->

Automatic merge from submit-queue

Fix Incorrect response model for pods/{name}/log

The swagger spec for pods/{name}/log does not include
"text/plain" as a possible content-type for the the response.
So we implement ProducesMIMETypes to make sure "text/plain"
gets added to the default list ot content-types.

the v1.json was generated by running:
hack/update-generated-swagger-docs.sh;./hack/update-swagger-spec.sh;

Automatic merge from submit-queue

Wrong format is used with single quotes in log

Wrong format is used with single quotes in log, should use ""

Automatic merge from submit-queue

Allow empty value for env in kubectl run

Also add a TODO for myself.

Automatic merge from submit-queue

Update the Pod Resource Management Proposal with Implementation status

This is tied to the upstream issue #27204 for adding pod level cgroups into Kubernetes.

@vishh @derekwaynecarr @Random-Liu  PTAL.

Automatic merge from submit-queue

Federation - common libs - cluster lifecycle handlers in FederatedInformer

Adds cluster lifecycle handlers to FederatedInformer to execute some actions when a cluster becomes available or unavailable.

ref: #29347

cc: @nikhiljindal @wojtek-t @kubernetes/sig-cluster-federation

The swagger spec for pods/{name}/log does not include
"text/plain" as a possible content-type for the the response.
So we implement ProducesMIMETypes to make sure "text/plain"
gets added to the default list ot content-types.

the v1.json was generated by running:
hack/update-generated-swagger-docs.sh;./hack/update-swagger-spec.sh;

Fixes #14071

Fixes #28734

Automatic merge from submit-queue

Fix PodAffinity test

Automatic merge from submit-queue

Cleanup k8s script noise with a verbosity concept

Fixes https://github.com/kubernetes/kubernetes/issues/30109

The KUBE_VERBOSE environment variable sets the verbosity level to
use. Log messages can specify a verbosity by setting the V
variable. e.g.

    V=2 kube::log::info foo bar

Would only print "foo bar" if $KUBE_VERBOSE >= 2.

Examples:

Default verbosity (1 for make commands):
```
$ make kubelet
+++ [0804 17:23:32] Generating bindata:
    /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go
+++ [0804 17:23:37] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
+++ [0804 17:23:37] Building go targets for linux/amd64:
    cmd/kubelet
# k8s.io/kubernetes/pkg/kubelet
pkg/kubelet/kubelet.go:247: undefined: a
make: *** [kubelet] Error 1
```

Extra verbose (5, comparable to previous levels):
<details>
```
$ make kubelet KUBE_VERBOSE=5
I0804 17:31:05.083395    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/trace.go:151:30: cannot use (traceBufHeader literal) (value of type traceBufHeader) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.083503    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/trace.go:151:7: array length 64 << 10 - unsafe.Sizeof((traceBufHeader literal)) (value of type uintptr) must be constant
I0804 17:31:05.083600    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/mgcwork.go:269:37: cannot use (workbufhdr literal) (value of type workbufhdr) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.083654    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/mgcwork.go:269:7: array length (_WorkbufSize - unsafe.Sizeof((workbufhdr literal))) / sys.PtrSize (value of type uintptr) must be constant
I0804 17:31:05.084006    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/chan.go:21:28: cannot use (hchan literal) (value of type hchan) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.084040    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/chan.go:21:66: cannot use (hchan literal) (value of type hchan) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.084076    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/chan.go:21:14: unsafe.Sizeof((hchan literal)) + uintptr(-int(unsafe.Sizeof((hchan literal))) & (maxAlign - 1)) (value of type uintptr) is not constant
I0804 17:31:05.085536    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/hashmap.go:80:31: cannot use (struct{b bmap; v int64} literal).v (value of type int64) as unsafe.ArbitraryType value in argument to unsafe.Offsetof
I0804 17:31:05.085567    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/hashmap.go:80:15: unsafe.Offsetof((struct{b bmap; v int64} literal).v) (value of type uintptr) is not constant
I0804 17:31:05.085788    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/hashmap.go:1053:45: cannot convert &zeroinitial (value of type *[1024]byte) to unsafe.Pointer
I0804 17:31:05.086995    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/mfinal.go:20:65: cannot use (finalizer literal) (value of type finalizer) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.087031    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/mfinal.go:20:11: array length (_FinBlockSize - 2 * sys.PtrSize - 2 * 4) / unsafe.Sizeof((finalizer literal)) (value of type uintptr) must be constant
I0804 17:31:05.087957    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/mstats.go:170:39: cannot use memstats.by_size (variable of type [67]struct{size uint32; nmalloc uint64; nfree uint64}) as unsafe.ArbitraryType value in argument to unsafe.Offsetof
I0804 17:31:05.087999    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/mstats.go:170:76: cannot use memstats.by_size[0] (variable of type struct{size uint32; nmalloc uint64; nfree uint64}) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.088483    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/panic.go:118:34: cannot use (_defer literal) (value of type _defer) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.088510    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/panic.go:118:20: unsafe.Sizeof((_defer literal)) (value of type uintptr) is not constant
I0804 17:31:05.089812    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/sema.go:42:42: cannot use (semaRoot literal) (value of type semaRoot) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.089845    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/runtime/sema.go:42:8: array length sys.CacheLineSize - unsafe.Sizeof((semaRoot literal)) (value of type uintptr) must be constant
I0804 17:31:05.094634    2601 parse.go:307] type checking encountered some errors in "runtime", but ignoring.
I0804 17:31:05.875185    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/davecgh/go-spew/spew/bypass.go:33:26: cannot use (*byte)(nil) (value of type *byte) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.875234    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/davecgh/go-spew/spew/bypass.go:33:12: unsafe.Sizeof((*byte)(nil)) (value of type uintptr) is not constant
I0804 17:31:05.875838    2601 parse.go:307] type checking encountered some errors in "github.com/davecgh/go-spew/spew", but ignoring.
I0804 17:31:05.897216    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/crypto/cipher/xor.go:12:36: cannot use uintptr(0) (constant 0 of type uintptr) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I0804 17:31:05.897261    2601 parse.go:353] type checker error: /usr/local/google/home/stclair/.gvm/gos/go1.6.2/src/crypto/cipher/xor.go:12:18: int(unsafe.Sizeof(uintptr(0))) (value of type int) is not constant
I0804 17:31:05.897360    2601 parse.go:307] type checking encountered some errors in "crypto/cipher", but ignoring.
I0804 17:31:06.400904    2601 conversion.go:227] considering pkg "k8s.io/kubernetes/federation/apis/core/v1"
I0804 17:31:06.401138    2601 conversion.go:243]   tags: ["k8s.io/kubernetes/federation/apis/core"]
I0804 17:31:06.427408    2601 conversion.go:283]   no viable conversions, not generating for this package
I0804 17:31:06.427508    2601 main.go:73] Completed successfully.
Go version: go version go1.6.2 linux/amd64
+++ [0804 17:31:06] Generating bindata:
    /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go
Generated bindata file : 11536 ../../..//test/e2e/generated/bindata.go lines of lovely automated artifacts
+++ [0804 17:31:12] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
+++ [0804 17:31:12] Building go targets for linux/amd64:
    cmd/kubelet
# k8s.io/kubernetes/pkg/kubelet
pkg/kubelet/kubelet.go:247: undefined: a
!!! Error in /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/hack/lib/golang.sh:506
  'go install "${goflags[@]:+${goflags[@]}}" -gcflags "${gogcflags}" -ldflags "${goldflags}" "${nonstatics[@]:+${nonstatics[@]}}"' exited with status 2
Call stack:
  1: /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries_for_platform(...)
  2: /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/hack/lib/golang.sh:692 kube::golang::build_binaries(...)
  3: hack/make-rules/build.sh:27 main(...)
Exiting with status 1
!!! Error in /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/hack/lib/golang.sh:596
  '( kube::golang::setup_env; echo "Go version: $(go version)"; local host_platform; host_platform=$(kube::golang::host_platform); local goflags goldflags gogcflags; eval "goflags=(${KUBE_GOFLAGS:-})"; goldflags="${KUBE_GOLDFLAGS:-} $(kube::version::ldflags)"; gogcflags="${KUBE_GOGCFLAGS:-}"; local use_go_build; local -a targets=(); local arg; readonly BINDATAS=("${KUBE_ROOT}/test/e2e/framework/gobindata_util.go"); kube::log::status "Generating bindata:" "${BINDATAS[@]}"; for bindata in ${BINDATAS[@]};
do
    if [[ -f $bindata ]]; then
        go generate "${bindata}";
    fi;
done; for arg in "$@";
do
    if [[ "${arg}" == "--use_go_build" ]]; then
        use_go_build=true;
    else
        if [[ "${arg}" == -* ]]; then
            goflags+=("${arg}");
        else
            targets+=("${arg}");
        fi;
    fi;
done; if [[ ${#targets[@]} -eq 0 ]]; then
    targets=("${KUBE_ALL_TARGETS[@]}");
fi; local -a platforms=(${KUBE_BUILD_PLATFORMS:-}); if [[ ${#platforms[@]} -eq 0 ]]; then
    platforms=("${host_platform}");
fi; local binaries; binaries=($(kube::golang::binaries_from_targets "${targets[@]}")); local parallel=false; if [[ ${#platforms[@]} -gt 1 ]]; then
    local gigs; gigs=$(kube::golang::get_physmem); if [[ ${gigs} -ge ${KUBE_PARALLEL_BUILD_MEMORY} ]]; then
        kube::log::status "Multiple platforms requested and available ${gigs}G >= threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in parallel"; parallel=true;
    else
        kube::log::status "Multiple platforms requested, but available ${gigs}G < threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in serial"; parallel=false;
    fi;
fi; kube::golang::build_kube_toolchain; if [[ "${parallel}" == "true" ]]; then
    kube::log::status "Building go targets for ${platforms[@]} in parallel (output will appear in a burst when complete):" "${targets[@]}"; local platform; for platform in "${platforms[@]}";
    do
        ( kube::golang::set_platform_envs "${platform}"; kube::log::status "${platform}: go build started"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status "${platform}: go build finished" ) &> "/tmp//${platform//\//_}.build" &
    done; local fails=0; for job in $(jobs -p);
    do
        wait ${job} || let "fails+=1";
    done; for platform in "${platforms[@]}";
    do
        cat "/tmp//${platform//\//_}.build";
    done; exit ${fails};
else
    for platform in "${platforms[@]}";
    do
        kube::log::status "Building go targets for ${platform}:" "${targets[@]}"; kube::golang::set_platform_envs "${platform}"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};
    done;
fi )' exited with status 1
Call stack:
  1: /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/hack/lib/golang.sh:596 kube::golang::build_binaries(...)
  2: hack/make-rules/build.sh:27 main(...)
Exiting with status 1
make: *** [kubelet] Error 1
```
</details>

Remaining work: Add a verbosity label to more log messages.

/cc @kubernetes/sig-api-machinery @kubernetes/contributor-experience 

Automatic merge from submit-queue

Moving image gc to pkg/kubelet/images

Automatic merge from submit-queue

replace improper create quota alias

`q` is not a shortname for `resourcequota`, so it shouldn't be an alias for `kubectl create resourcequota`.

@sttts ptal

Automatic merge from submit-queue

AWS/GCE: Rework use of master name

* Add a pillar for `hostname` (because even if there's a good Salt function for it, I don't trust it to return the short hostname)
* Move `INITIAL_ETCD_CLUSTER` to just the GCE turn-up
* Remove `master_name`, which isn't needed

Automatic merge from submit-queue

Node E2E: Move the node name initialization to first function of SynchronizedBeforeEach

Currently, we start e2e services in the first function of `SynchronizedBeforeEach` to make sure that we only start them once even we are running test in parallel test nodes.

However, e2e services require `NodeName`, but we initialize `NodeName` in the second function.

This PR moved the initialization logic into the first function, and shared the node name with all test nodes via the `SharedContext`.