Why not use FUSE?
RE: building, compiling without login etc - I have a bunch of aliases that let me do that remotely with ssh. You're still "logging in" but don't need to deal with anything major.
Big file changes are never going to be fast (even doing git operations is pretty slow) but remote interaction can be done via ssh in another tab.