What is the "magic handshake"?

Also are the gcc extensions related to nested functions that will generate an executable stack documented? What would I search for in the documentation?