It's completely possible, but it's just not worth it. The information density of ternary is higher than binary, sure, but you now need 3 states to represent it. This applies to ternary vs base 4, and so on. As it turns out, it's been proven that binary has the optimal ratio of info density to symbols, and any higher of a base gives you no net benefit. Unfortunately I don't have a link to the proof on hand, but if someone else does I'd appreciate it.

We would also lose decades of knowledge on binary operations and how best to do it. Two's complement is out the window, for instance.

Additionally, the entire CS mindset is built around boolean logic. Exclusive-or, for instance, makes no sense when you remove binary operation. There's not much to gain from it either, since any sort of ternary operation can be implemented as a binary operation without that much overhead.