May also suggest the suite of open source DeepView tools and which are part of PyPi. Profile and predict your specific model training performance on a variety of GPUs. I wrote a linked in post with usage GIF here: https://www.linkedin.com/posts/activity-7057419660312371200-...

And links to PyPi:

=> https://pypi.org/project/deepview-profile/ => https://pypi.org/project/deepview-predict/

And you can actually do it in browser for several foundational models (more to come): => https://centml.ai/calculator/

Note: I have personal interests in this startup.