Yes, indeed - it is very weird! Because mathics still mentions ply in change.rst, but not used in code anymore:
The main issue was that generating the tables was extremely slow (~15mins). We solved this by packaging the tables, but when the version of ply changed a table regeneration was forced on our users.
The recursive descent parser was also simpler and significantly more performant.
There's a short write-up on the PR: https://github.com/mathics/Mathics/pull/416