Cheapest Llama2 chatbot solution costs only $4/mon

It's $4/month to use the image, what about the monthly cost of the t3.large server it is suggesting?

The total cost comes to approx $70/month.

They should link to this from the amazon page: https://ai.meta.com/llama/use-policy/

It seems to suggest t3.large by default, which has 2 vCPUs and 8 GB of RAM.

What kind of performance can one expect to get from such a system?

They're saying it's a LLaMA-2 7B model, but I assume it's heavily quantized (Q2 or Q3 are mostly likely) because if it's got to fit in 8GB of RAM, it's going to need to be. Q2 and Q3 quantized models will require somewhere between 5 and 6GB of RAM). They might be able to go as high as a Q5 if they have extremely low overhead for the rest of the tooling (up to Q5 may fit in as little as 7GB).

Performance is going to be...poor. I would expect this to be less than 1 token a second.

Your math doesn't quite add up. An 8-bit quantized 7B parameter model would use only ~7GB of memory. That aside, I'm able to infer a quantized 13B model on a 2-core CPU at ~2 tokens/sec, so performance would likely be fine compared to actual ChatGPT.

what about 70B? What’s the lowest cost we can expect?

Eh, I've seen cheaper. I run Llama on a free 4-core Oracle system.