I've seen reports of people being able to run LLMs at decent speeds on old Nvidia P40s. These are 24GB Pascal GPUs and can be bought for as low as $100 (although more commonly $200) on eBay.
Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34