Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.

If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.

I estimate that the server consumes probably around 500W during inference.

In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.

But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.

The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.

EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.