Programming (and computers in general) have nothing to do with 1s and 0s and saying stuff like "at the end of the day, its all 1s and 0s" is like saying that at the end of the day banking just comes down to scrap pieces of pressed tree pulp or a bakers job is just about a bunch of grain or a teachers job reduces to sitting in a room for a few hours or whatever silly analogy you want to dream up.
All of these things are implementation details that, really, are irrelevant to pretty much everyone everywhere. If you are dealing with low level digital systems you will encounter 1s and 0s (and analog voltage and resistance and capacitance and a whole lot more besides), but usually (especially when building something complex like a computer) this gets abstracted away pretty quickly.
I did not at all think you were joking when you asserted this:
Given this fact, I don't consider the idea of coding only with the letter 's' that much of a stretch.
Which I completely agree with.