Over the last week I tried to use a combination of Claude and OpenAI o3-mini to do a direct conversion of about 500 lines of uncommented academic modeling code from Matlab to Python. I can’t stress enough how badly these models performed. Nearly every consequential line had some variety of off by one or logic error, often very subtle. I didn’t try cursor or the more agentic systems, but I would be astounded if they properly rigged up a test harness, inspected the output and were able to respond to the runtime errors. I’d be happy to share the code if anyone wants to surprise me.

This is exactly the kind of semi-mechanical, low added value work that would greatly benefit from automation, and they really fell on their faces. I really benefit from these models on greenfield tasks where I can delegate minor drudge work, but in this case I honestly think they actually increased the difficulty.