The way I see it, CSV's purpose is information storage and transfer, not presentation.
Presentation is where you decide the font face, the font size, cell background color (if you are viewing the file through a spreadsheet editor) etc. Number formatting belongs here.
In information transfer and storage it's much more important that the parsing/generation functions are as simple as possible. So let's go with one numeric format only. I think we should use the dot as a decimal separator since it's the most common separator in all programming languages. Maybe extend it to include exponential notation as well, because that is what other languages like json support. But that's it.
I hold the same opinion about dates tbh.
(The same goes for dates, btw - yyyy-mm-dd or death)