Although since TensorFlow models should be trained on a GPU unlike sklearn, that's less useful, and there are better tools for CPU inference. (e.g. SavedModels or ONNX)
Historically doing inference on Intel gear was mostly about whether or not to target AVX2 or AVX512 when building Eigen or whatever. A few years ago the net win was AVX2 because the de-clock and re-clock just killed you.
What’s the game these days? Long term I doubt inference will be done on x86, but I think a lot of people still do it.
How do you know it's bad? Do you have benchmarks?
Just checked, used 3090s are going for $700-900 on eBay.
You can get the latest Linux pacakges for eg Ubuntu here: https://wiki.ubuntu.com/Kernel/MainlineBuilds - probably other distributions have similar easy ways since it's needed so often in bug reporting for users to tell if a hw support bug is in a distro specific kernel change or not.