You can find the Konftel Ego for $80 at Provantage.com (around $120 everywhere else). Konftel makes amazing speakerphones that rival polycomm, for a lot less. Indeed, I actually prefer their sound to polycomm.
The Ego will be a dramatic upgrade from whatever is built into your other gear. If you still aren't happy with it, then you can start going down the USB/XLR microphone rathole if you still prefer. But it's probably unnecessary.
If you are going for the prosumer option, I recommend Scarlett Solo + Shure SM58 + XLR cable + mic stand. This may be a bit overkill for meetings, but it's a good starting point if you want to record music, stream, record a podcast, etc.
If I were modifying my setup today, I would've gone with an XLR mic out the gate: at the end of the day I ended up running analog audio to the Blue microphone anyway because the value of having monitor audio to your headphones is incredible (and you do not want any latency on that at all).