Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

clean up legacy ipv4/32 in ipvs proxy

**What this PR does / why we need it**:

clean up legacy ipv4/32 in ipvs proxy

**Which issue(s) this PR fixes**: closes #51866

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
/sig network

/area ipv6

/assign @thockin 

cc @danehans