• Kubernetes Submit Queue's avatar
    Merge pull request #38865 from vwfs/ext4_no_lazy_init · 6dfe5c49
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue
    
    Enable lazy initialization of ext3/ext4 filesystems
    
    **What this PR does / why we need it**: It enables lazy inode table and journal initialization in ext3 and ext4.
    
    **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #30752, fixes #30240
    
    **Release note**:
    ```release-note
    Enable lazy inode table and journal initialization for ext3 and ext4
    ```
    
    **Special notes for your reviewer**:
    This PR removes the extended options to mkfs.ext3/mkfs.ext4, so that the defaults (enabled) for lazy initialization are used.
    
    These extended options come from a script that was historically located at */usr/share/google/safe_format_and_mount* and later ported to GO so this dependency to the script could be removed. After some search, I found the original script here: https://github.com/GoogleCloudPlatform/compute-image-packages/blob/legacy/google-startup-scripts/usr/share/google/safe_format_and_mount
    
    Checking the history of this script, I found the commit [Disable lazy init of inode table and journal.](https://github.com/GoogleCloudPlatform/compute-image-packages/commit/4d7346f7f59cb88ec934c436bd853b74f4ebbaf5). This one introduces the extended flags with this description:
    ```
    Now that discard with guaranteed zeroing is supported by PD,
    initializing them is really fast and prevents perf from being affected
    when the filesystem is first mounted.
    ```
    
    The problem is, that this is not true for all cloud providers and all disk types, e.g. Azure and AWS. I only tested with magnetic disks on Azure and AWS, so maybe it's different for SSDs on these cloud providers. The result is that this performance optimization dramatically increases the time needed to format a disk in such cases.
    
    When mkfs.ext4 is told to not lazily initialize the inode tables and the check for guaranteed zeroing on discard fails, it falls back to a very naive implementation that simply loops and writes zeroed buffers to the disk. Performance on this highly depends on free memory and also uses up all this free memory for write caching, reducing performance of everything else in the system. 
    
    As of https://github.com/kubernetes/kubernetes/issues/30752, there is also something inside kubelet that somehow degrades performance of all this. It's however not exactly known what it is but I'd assume it has something to do with cgroups throttling IO or memory. 
    
    I checked the kernel code for lazy inode table initialization. The nice thing is, that the kernel also does the guaranteed zeroing on discard check. If it is guaranteed, the kernel uses discard for the lazy initialization, which should finish in a just few seconds. If it is not guaranteed, it falls back to using *bio*s, which does not require the use of the write cache. The result is, that free memory is not required and not touched, thus performance is maxed and the system does not suffer.
    
    As the original reason for disabling lazy init was a performance optimization and the kernel already does this optimization by default (and in a much better way), I'd suggest to completely remove these flags and rely on the kernel to do it in the best way.
    6dfe5c49
Name
Last commit
Last update
..
async Loading commit data...
bandwidth Loading commit data...
cert Loading commit data...
chmod Loading commit data...
chown Loading commit data...
clock Loading commit data...
config Loading commit data...
configz Loading commit data...
crlf Loading commit data...
dbus Loading commit data...
ebtables Loading commit data...
env Loading commit data...
errors Loading commit data...
exec Loading commit data...
flag Loading commit data...
flock Loading commit data...
flowcontrol Loading commit data...
framer Loading commit data...
goroutinemap Loading commit data...
hash Loading commit data...
homedir Loading commit data...
httpstream Loading commit data...
i18n Loading commit data...
initsystem Loading commit data...
integer Loading commit data...
interrupt Loading commit data...
intstr Loading commit data...
io Loading commit data...
iptables Loading commit data...
json Loading commit data...
jsonpath Loading commit data...
keymutex Loading commit data...
labels Loading commit data...
limitwriter Loading commit data...
logs Loading commit data...
maps Loading commit data...
metrics Loading commit data...
mount Loading commit data...
net Loading commit data...
netsh Loading commit data...
node Loading commit data...
oom Loading commit data...
parsers Loading commit data...
procfs Loading commit data...
proxy Loading commit data...
rand Loading commit data...
resourcecontainer Loading commit data...
rlimit Loading commit data...
runtime Loading commit data...
selinux Loading commit data...
sets Loading commit data...
slice Loading commit data...
strategicpatch Loading commit data...
strings Loading commit data...
sysctl Loading commit data...
system Loading commit data...
taints Loading commit data...
term Loading commit data...
testing Loading commit data...
threading Loading commit data...
uuid Loading commit data...
validation Loading commit data...
version Loading commit data...
wait Loading commit data...
workqueue Loading commit data...
yaml Loading commit data...
BUILD Loading commit data...
doc.go Loading commit data...
template.go Loading commit data...
template_test.go Loading commit data...
trace.go Loading commit data...
trie.go Loading commit data...
umask.go Loading commit data...
umask_windows.go Loading commit data...
util.go Loading commit data...
util_test.go Loading commit data...