Try disabling VC over Tramp connections[1]:

  (setq vc-ignore-dir-regexp
        (format "\\(%s\\)\\|\\(%s\\)"
                vc-ignore-dir-regexp
                tramp-file-name-regexp))

VC is quite chatty and assumes that filesystem operations have a negligible cost. Before I disabled it, VC was adding >1 second to every find-file operation over Tramp.

I also recommend using the direct-async-process connection property[2], which significantly decreases the latency of async process creation.

[1] https://www.gnu.org/software/emacs/manual/html_node/tramp/Fr...

[2] https://www.gnu.org/software/emacs/manual/html_node/tramp/Re...