Do you guys have any reference material on how to get this started?
But there are a lot of different routes to go. If you want to do what I've done though -
1. Sign up for Oracle Cloud and try to get their free 4 core ARM VPS (these are at-capacity very often, but free if you can get them)
2. Install the Ampere Pytorch runtime: https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
(can also use the ONNX or Transformers acceleration image if you need it)
3. Install your Pytorch/ONNX/Transformers software, link in system libraries and enjoy!
Whole thing works super smooth in my experience. 7B and 13B models are very usable for chatbot type applications.
The hardest part is downloading the 30GB of weights.