I'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems.
https://www.techrxiv.org/users/685780/articles/1375955-topol...
Yes the paper compares the new architecture (that is also a fork of my implementation of nanoGPT) with Karpathy's nanoGPT. There are also links to the code and bench used.
Transformers are universal function approximators. When well-tuned, they often start to approximate other innovations. Not always, thank god, but often enough that you have to be careful.
https://www.databricks.com/blog/mosaicbert
I'll add they should do a number of small, training runs with different architectures and data mixes. That proves generalization.