GCP system to run llama2 Chat model 7B 13B and 70B

You can have load balanced inference and also fine tune models on Modal (disclaimer: I wrote this guide)

You can get a load balanced endpoint serving hundreds of tok/s per GPU using vLLM or Text Generation Inference in a few lines of code:

Look into vllm and text-generation-inference

Why not use an API though?

Some to check out are:

Replicate, Together AI, Fireworks AI, Deepinfra, Anyscale Endpoints

I can personally recommend deepinfra. Been using it the past few days and it's served me very well.