It does come with a huge penalty. Want to get rid of your GC pauses? Good luck with that. Best case scenario like D you get to split your libraries in GC and non-GC, or C# where you can kinda skirt around it with structs and stackalloc.

Worst case scenario Java without value types - Rewrite it in another language.