I hear you - Virtual Box is getting worse and worse in terms of being able to interact with the host in anything resembling a timely manner. I wish the competition in the free VM space was better.
To get around this shortcoming, the sharing is mostly one way: from the host to the VM, via the default shared folder. I've not had problems with VBFS when managing files in this manner.
There have also been times where I have implemented watch scripts against files on the host, which triggers a rsync to the guest's normal file systems. From there, the file system behavior is more like what we can expect from a typical virtual machine.
To support workflows similar to what I do against non-local VMs, I've worked up some wrapper scripts which allow me to interact with the Vagrant VM via the usual ssh, scp, and rsync. When running "vagrant up", I also write out the VM's SSH configs and a ssh alias concatenates these with my base .ssh/config file.