A computer may need to read the data millions of times, but humans only need to read it when things goes wrong. If you take the route of using a binary format, then you need to provide a robust set of tools to view, edit, debug that format. It can absolutely work, journald is binary, but has the tooling to help you get the plain text, when you need it. Protobuf could work well, but if you design your own binary data format you have to build some tooling as well.
For many (most) it doesn't matter much if you use XML or JSON, the computer will process it fast enough. If the concern is parsing speed, then yes, absolutely binary formats might be the best option. For things like HFT or anything else that processes vast volumes of data, its probably the only choice.