The cross references underneath Google codesearch (both internal and external) are created by, roughly speaking, partially compiling everything and groveling around in the AST.
One of our teammates Luke Zarko has a decent overview of this: https://www.youtube.com/watch?v=VYI3ji8aSM0
> Bizarre, especially when you consider that the whole underlying infrastructure of google codesearch has been open-sourced.
Some of the code is also open sourced at kythe.io, though it lacks a considerable amount of the internal logic necessary to scale it out to all of google3. So it's not exactly plug & play. But, it should be enough for a company the size of Microsoft/Github to get started using it.