you can use terminal-bufffer settings too:

:tnoremap <Esc> <C-\><C-n>

:tnoremap <A-h> <C-\><C-n><C-w>h

:tnoremap <A-j> <C-\><C-n><C-w>j

:tnoremap <A-k> <C-\><C-n><C-w>k

:tnoremap <A-l> <C-\><C-n><C-w>l

Use <C-\><C-n> to get back into normal mode so you can use <C-w>w and and friends. Or you can use your mouse since Neovim by default does 'set mouse=a' iirc.

<C-\><C-n> leaves you in normal mode again, and then you can navigate as usual.