Besides z being much faster (looking through a sorted list of directories visited instead of recursively walking from the current directory) your snippet looks from the cwd
Even in the worst cases, at least with my i7 NUC and using fd instead of find, it's less than 500ms to load the entire list into fzf.