You will always be able to do more and better with the premium hardware. Maybe for video games the marginal differences aren't enough to sway most consumers towards the high end cards, but I expect there will be a lot of people willing to pay top dollar to be able to run the best version of X AI tool.
For workloads though, does it really matter if it completes in 20 seconds instead of 15?
Unless there's some feature difference in the higher end cards (like AI-specific subchips), not just more of the same & faster, then a lower card of the same generation shouldn't impact your ability to run something or not.
Maybe RAM size...
Or maybe it's an automated thing that runs constantly? The difference between 20 and 15 seconds is 480 additional iterations per 8-hours. How much is that worth to you?
There's also the unmeasurable benefits. What if you're losing potential good employees applying because word on the net is your company uses last-gen hardware for everything, and your competitors use the newest.
I mean, it's no different than CPUs and RAM and such. A middle-of-the-road laptop from the last year or two is good enough for most users... no company I've ever worked for gets the top-of-the-line model every single refresh for every single dev. That'd be an absurd waste of money, IMO.
Personally, I wouldn't want to work for a company that upgrades hardware all the time for single-digit performance increases. That money is better used elsewhere, whether employee salaries/benefits or donating it to the community or just saving customers some money or paying dividends to shareholders.
Even if you're running massive continuous workloads, it is probably more cost-efficient to just parallelize a bunch of midrange systems (optimize for perf/dollars and/or perf/watt) than to just chase the highest-end chips for no reason.