A few points:
* The Supermicro was on sale for around $500 a few weeks ago. That's an actual price new on Newegg. I had it in my cart (not with intent to purchase now, but spec'ing out for a future purchase).
* I've found proper refurbished to be at least as reliable as new, if bought through appropriate channels. What you're looking for is off-lease. Companies lease computers for e.g. 3 years, and then return to Dell. Those computers ran for long enough to be past infant mortality. That very much differs from consumer returns / repairs, which were returned because they were unreliable. I'd never buy a refubished Inspiron, but I very much would buy a refurbished Latitude.
* Part of it depends on what you're doing. I'm very much not performance-limited for most of the things I do. If a model takes 2x as long to run, I'll wait twice as long. No big deal. If a model can't run at all, that's where there's a huge difference.
A card like the A770, which is around $20/GB, is very exciting to me, if the drivers / documentation / etc. lands.
I'll also mention: The scaling here isn't quite as it seems:
1) Two 16GB cards with half the bandwidth per card will have the same theoretical bandwidth as one 32GB card in a faster lane. Ditto for bandwidth to VRAM. Once you start doing the math there, the cheap cards look very good, indeed!
2) If big chunks of data need to flow between cards, you're using much more of that bandwidth. The cheap cards look a lot worse.
A lot of that depends on what you're doing, and if I haven't made it clear yet, I don't quite know what I'm doing :)