The mystery about how Rentec makes a ton of money is very real - nobody seem to have a clue. It's most probably indeed due to some excellent math / engineering work, but could very well be a fraud too - wouldn't be a first time that something is 'very secretive' for a reason.
Btw, another interesting trading company is XTX - founded by a Russian-born mathematician Alexander Gerko - they are huge (in terms of trading volumes and profits, not in terms of headcount), and apparently using a lot of modern machine learning.