Nice - btw curious how much it costs to run Llama on this type of side project? I can imagine the cost goes to some reasonable amount when things like this get shared on HN. Does Fly.io give a flat price to use their inference? Curious how it works
I'd need to do the math on this, thankfully most of it is on a GPU kind that I don't normally use. I probably over-scaled so that I didn't need to think about the scaling issues, but I can tally it all up.
that would be great... i'm curious on the cost per million token as it compares to gpt-4o-mini