"Some binary" = the same version of VS Code (maybe without Electron and some other parts).

Makes sense as you need to - edit files, run tasks and terminals, and run extensions which access the remote filesystem (eg Jump To Definition taking you to header files installed in system folders) - basically everything you need VS Code to do when you run it locally.

The alternative would be running VS Code as a webserver remotely and the local VS Code Electron just acting like a browser window pointing at your server. It probably wouldn't feel as responsive.