He didn't say programming is about numbers.
His claim was that programming is a simple kind of thinking. He gives two representative examples:
* The consideration of sorting algorithms (which in all fairness _is_ a staple in teaching algorithms), and
* The use of simple logic - such is representative of programming language specs.
I'd say that this is a fair summary - real life tends to be a lot more complicated that the problems faced by most programmers.
Even more brutally: he is aiming quite high, a lot of programming is the minor stitching together of APIs without making too big of a mess.
Few programmers ever seriously actually need to analyse thousands of dimensions of information - and I'd say that it is a safe bet that the ones who do are probably using numbers.
To "We are literally expert at becoming experts.", I'd reply with https://xkcd.com/1112/