Remove usage of the aggregated clientset in the e2e testing framework
itself. We have one test that consumes the clientset in the suite
and it's in test/e2e/apimachinery/aggregator.go, which was recently
promoted to conformance in 8101b86a.

This test now obtains a local copy of the aggregated clientset.
The suite still has to compile the internal client in.
One possible solution here is to move this test in a separate suite,
yet it's unclear of how to tackle the problem now that the test
has to run as part of the conformance suite.