• Kubernetes Submit Queue's avatar
    Merge pull request #53400 from micahhausler/aws-nlb · 63d4b85b
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 54316, 53400, 55933, 55786, 55794). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
    
    Add Amazon NLB support
    
    **What this PR does / why we need it**:
    
    This adds support for AWS's NLB for `LoadBalancer` services.
    
    **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
    
    Fixes #52173
    
    **Special notes for your reviewer**:
    
    This is NOT yet ready for merge, but I'd love any feedback before it is.
    
    This requires at least `v1.10.40` of the [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go), which is not yet included in Kubernetes. Per @justinsb, I'm waiting on possibly #48314 to update to `v1.10.40`  or some other PR. 
    
    I tried to make the change as easy to review as possible, so some LoadBalancer logic is duplicated in the `if isNLB(annotations)` blocks. I can refactor that and sprinkle more `isNLB()` switches around, but it might be harder to view the diff.
    
    **Other Notes:**
    
    * NLB's subnets cannot be modified after creation (maybe look for public subnets in all AZ's?).  Currently, I'm just using `c.findELBSubnets()`
    * Health check uses TCP with all the NLB default values. I was thinking HTTP health checks via annotation could be added later. Should that go into this PR?
    * ~~`externalTrafficPolicy`/`healthCheckNodePort` are ignored. Should those be implemented for this PR?~~
    * `externalTrafficPolicy` and subsequent `healthCheckNodePort` are handled properly. This may come with uneven load balancing, as NLB doesn't support weighted backends.
    * With classic ELB, you have a security group the ELB is inside of to associate Instance (k8s node) SG rules with a LoadBalancer (k8s Service), but NLB's don't have a security group. Instead, I use the `Description` field on [`ec2.IpRange`](https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/#IpRange) with the following annotations. Is this ok? I couldn't think of another way to associate SG rule to the NLB
        * Node SG gets an rule added for VPC cidr on NodePort for Health Check with annotation in description `kubernetes.io/rule/nlb/health=<loadBalancerName>`
        * Node SG gets an rule added for `loadBalancerSourceRanges` to  NodePort for client traffic with annotation in description `kubernetes.io/rule/nlb/client=<loadBalancerName>`
        * **Note: if `loadBalancerSourceRanges` is unspecified, this opens instance security groups to traffic from `0.0.0.0/0` on the service's nodePorts**
    * Respects internal annotation
    * Creates a TargetGroup per frontend port: simplifies updates when you have same backend port for multiple front end ports.
    * Does not (yet) verify that we're under the NLB limits in terms of # of listeners
    * `UpdateLoadBalancer()`  basically just calls `EnsureLoadBalancer` for NLB's. Is this ok?
    
    **Areas for future improvement or optimization**:
    
    * A new annotation indicating a new security group should be created for NLB traffic and instances would be placed in this new SG. (Could bump up against the default limit of 5 SG's per instance)
    * Only create a client health check security group rule when the VPC cidr is not a subset of `spec.loadBalancerSourceRanges`
    * Consolidate TargetGroups if a service has 2+ frontend ports and the same nodePort.
    * A new annotation for specifying TargetGroup Health Check options.
    
    **Release note**:
    
    ```release-notes
    Add Amazon NLB support - Fixes #52173
    ```
    
    ping @justinsb @bchav
    63d4b85b
Name
Last commit
Last update
..
api Loading commit data...
apimachinery/tests Loading commit data...
apis Loading commit data...
auth Loading commit data...
bootstrap/api Loading commit data...
capabilities Loading commit data...
client Loading commit data...
cloudprovider Loading commit data...
controller Loading commit data...
credentialprovider Loading commit data...
features Loading commit data...
fieldpath Loading commit data...
generated Loading commit data...
hyperkube Loading commit data...
kubeapiserver Loading commit data...
kubectl Loading commit data...
kubelet Loading commit data...
kubemark Loading commit data...
master Loading commit data...
printers Loading commit data...
probe Loading commit data...
proxy Loading commit data...
quota Loading commit data...
registry Loading commit data...
routes Loading commit data...
security Loading commit data...
securitycontext Loading commit data...
serviceaccount Loading commit data...
ssh Loading commit data...
util Loading commit data...
version Loading commit data...
volume Loading commit data...
watch Loading commit data...
.import-restrictions Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...