Once they have a shell they're free to do what they want. Local root exploits, etc are all fair game -- and there are plenty of 0-days out there. In fact, removing the compiler doesn't help; a lot of attackers and bots run a script to detect the system environment and auto-download what they need to move on. Remember: jails can still be broken out of, and there is some really neat work in the research space of attacking hypervisors...
There are numerous things you could remove from an OS install to make their lives a bit harder (dtrace, systemtap anyone? you can sniff passwords easily...), but once they have a shell and a directory with write access it's game over. You're better off leaving the tools there but having your auditing throw a serious fit when a compiler, etc is executed unexpectedly on a production box.
So yes, I think it's bunk to claim that removing a compiler will provide meaningful security benefit. In fact, I believe this has been suggested a few times to FreeBSD by users (don't install compilers, etc by default!) and was dismissed as ineffective for the same reasons.
Work harder to prevent the bad guys from getting that far. Keep your systems well patched. Have a thorough auditing and monitoring system in place. Use containerization and segregation everywhere possible. Limit scope of access. Some points you made apply here as well. Maybe once you've mastered all of those arts you can play with the obscurity angle.
edit: and L7 firewalls... I'm on the fence there. There has been some movement in that area by security researchers equating them to antivirus... they're only as good as their definitions, and a targeted attack will bypass it. They also seem to give a false sense of security and let people be lazy. Security in layers is important, though.
edit2: don't get me wrong, you should be hardening your production boxes (and dev... so the environments match...) but removing a compiler is not high on my list