I didn't realize that the GC copied the surviving objects to a separate location. Come to think of it, i don't actually know how the GC keeps track of what object lies in what generation.
C# allows you to "pin" a variable to combat this. Any pinned memory will never be moved by GC, so if you pass a pointer to an object out of your managed program, the pointer address will always stay valid.