Folks at Middleware have tried to analyse Llama 3.1 vs GPT-4o. They ingest a lot of time-series data from live sources like Github.

They tried to analyse on:

- Mathematical Accuracy - Data Analysis - Actionability - Summarisation

They have a DORA metrics product which show: Lead Time for Changes, Deployment Frequency, Mean Time to Recover (MTTR), Change Failure Rate (CFR) for engineering teams. (https://github.com/middleware/middleware)

I think still it is not the best or efficient way to compare. What do you think? Any suggestions on comparisions for specific use cases like these.