It doesn't matter what Turing-complete model you have; they are all equivalent, even if exponentially slower than another. And no, binary logic is not the most abstracted version possible. Clearly lambda calculus is....ok, just kidding. You can build a universal computer out of just two stacks, out of subtract-and-jump-if-zero, and a lot of other things. (https://gwern.net/turing-complete)
I don't think computer scientists really think that binary logic is the only computation paradigm that makes sense, it just happens to be one that we can make extremely fast and extremely reliable realizations of using electronic circuits. It's been phenomenally successful.