But didn't LLMs read all the math books and can't really do arithmetics (they need special modes / hacks / python to do it I think)?

So why would they be able to "read" the docs and use that knowledge except up to pattern matching level. That's why I also assume, that tons of examples with results would do better than lang docs, but I haven't tested it yet.