> What happens is that each Spark partition outputs its own CSV, which is never what anyone would want!

1 file per partition is exactly what you want when the output is large, so that multiple executors can share the work of writing out the output.

> The expected (undocumented) solution is to compress into one partition, but then non-primitive data structures get output incorrectly.

By "compress" do you mean coalesce? Yes, coalescing the RDD/DataFrame into one partition is the commonly accepted solution for when you want to force 1 output file. The cost of doing so is that you lose parallelism, since only 1 task will be able to write out that file.

And what do you mean by "non-primitive data structure" in this case?