While it's a great story and is very entertainingly written, as is typical of Feynman, I don't think he was being very fair to the merits of the two approaches to mental math -- his own vs that of the soroban.
Feynman clearly hadn't done any research on the merits of the soroban method, he just had this single run in with a random soroban salesman. From this one experience, and without very much thought on the subject, he just kind of dismisses the method almost out of hand. Such offhand dismissals are unfortunately also pretty typical of Feynman regarding subjects he is mostly ignorant of and has little experience with (such as his dismissals of philosophy in other talks).
In the story, the random soroban salesman, who clearly hasn't even very much proficiency with the soroban (as evidenced by his need for a physical soroban, vs soroban-trained practitioners who no longer need the physical device but can apply the soroban methods in their head) is matched against one of the greatest physicists in history, a Physics Nobel Prize winner who has advanced math training. That's really not a fair match.
It would have been great to see how Feynman would have fared against someone like this guy: [2] or against one of the national soroban competition winners, or against a top mathematician or physicist who was soroban-trained. That would be a fairer match and the results more indicative of the merits of the two approaches.