L1 Cache is an order of magnitude faster than L2 cache and that's still all on the same chip. That's not "a pretty small part".

Energy consumption, heat dissipation, all of this becomes a larger problem for smaller structures. It's just easier and less error prone to build larger structures, yields are higher, so costs are lower for larger structures.

I'm certain the tradeoffs will change, but it's all limited by some physical law and while we can sometimes trade one law against the other, we can't currently break them, so tradeoffs will always exist.