There is a big difference between 'doing math' by repeating/elaborating on previously seen patterns, and by having an intuitive grasp of what is going on 'under the hood'. Of course our desktop calculators work (very well) on the latter principle.

As you say, both the broken and correct solutions were likely in the training corpus (and indeed the error message), so really we are doing a smoke and mirrors performance to make it look like the correct solution was 'thought out' in some sense.