actually the math is more like $1500 a month on a MI300x on a highly optimized rocm/sglang image and everyone's getting 100+ tok/sec and we have 100 active people and plenty of room for others.

but were losing money right now obviously.

we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.

People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.

its been fun though. you should check it out