Binary data simply means that each datum encodes exactly one of two states. We represent them as 0s and 1s, but that's just an abstraction of convenience - physically, that maps to something like "high voltage" and "low voltage", or "dark" and "light", or "on" and "off", depending on the physical medium.

We could have three states - "high voltage", "medium voltage" and "low voltage" - but the advantage of having exactly two is that it makes it harder to mistake one state for another (e.g. if voltage fluctuates within a specific range). If the measurement range for each datum is (0, 100), you can decide that anything below 50 is "low voltage" and anything below 100 is "high voltage". You can also do the same thing with (0,33), <33,66), <66,100), but that requires assuming more precise tools throughout the entire pipeline. When we talk about bits getting flipped (the entire reason that checksums exist in wire protocols), that's the reason: the medium conducting the signal is imprecise, and sometimes the reading is off.

Traditionally, this represents a tradeoff between density and fidelity. If your system has high enough fidelity, you can take advantage of the additional precision and distinguish between more states, representing additional information.

If your system has 8 states (0,8.3), <8.3, 16.6)... etc., you can look at this as an octal system, or you can think of it as a binary system in which a read/write error affects an entire byte, rather than a single bit.

At the end of the day, this is a question of signal processing - binary representation is a convenient abstraction that allows us to understand the way that we're interpreting the signals we're reading, but it's fundamentally an arbitrary choice.