Its basically Cost/Speed/Size Pick 2 or maybe even 1.
Some people have been able to run large LLM's on older slower CUDA gpus. A lot of them are truly ancient and have found their way back to ebay simply due to market conditions. They aren't good, but they work in a lot of use cases.
There are a couple of janky implementations to run on AMD instead. But reviews have been so mixed that I decided not to even test it. Ditto multi GPU setups. I thought that having access to 16 8g AMD cards from old mining rigs would have set me in good stead but apparently it benches roughly the same as just using a server with heaps of RAM because of the way the jobs are split up.
The cloud services seem to be the best option at the moment. But if you are going to spend 100 bucks vs going to spend 1000 bucks it might be worth just to fork out for the card.
Also honestly, hoping that someone else has a better idea in this thread because it will be useful to me too.