You can, though? If you're on Gnome, simply install Dolphin or Thunar and your file manager should change. You can revert it in Settings > Applications > Files by resetting the application handler for the application you just installed.

People seem to be confusing the file manager (nautilus) with the file picker (whatever an application chooses, often the GTK file picker) and the desktop (which can be, but isn't always, part of the file manager).

They can, but they usually control the desktop, so if you want a consistent experience it is somewhat ill-advised.