Ok, I edited a file in a meaningful way and tried to decipher the process (actually it become more of a normal workflow more than editing a file).
The thing is I switch all the time between vim and my terminal (it's complicated by let's say I mainly use one terminal tab for everything I'm currently writing) so using the home row I would have to switch all the time between home row and arrows when using the terminal (and other programs) (I guess I could add vim bindings to every single thing I use: web browser, terminal emulator, pry, irb, tmux, etc etc. but doesn't seem appealing to me).
The escape key isn't an issue either because I don't use the home row, so no need to remap that too.
I think this misconception comes from the manual pushing people to use the home row.
I don't claim I'm the fastest writer ever, but there are a lot of compromises I'm not willing to make (I could use vim tabs/windows, I could use Ctrl-D to switch files, I could use GVim, have the main files always open, etc.). I'm pretty sure everyone works differently and this workflow works very well for me.
What I don't like is this constant shaming of arrow vim users and people that keep saying it's only for beginners (and the manual pushing people to use the home row).