There's a default limit of 2 billion cells (rows * cols) but larger partitions also mean larger indexes and more serialization, repair, compaction and memory overhead which doesn't work well with Cassandra's Java/JVM GC requirements.

It's getting better but likely will never reach the performance potential of Scylla which is Cassandra reimplemented in C++ with a much better sharded architecture.

It’s not a real requirement. Just a performance thing. Each partition has an index and as it grows the deserialization of it to find the queried start column takes longer. It is lazy loads but there a perf hit. Easier to have a rule of thumb like “100mb partitions” and have more fixed tps per core etc expectations.

No, it's just that the storage layer is low priority to improve for this use case, but it'll happen eventually.