I think HSM's are just expensive because of price gouging rather than cost of the device? Like the Yubikey HSM is the same form factor as the Nano FIPS key but over 10x the price.
There's probably also a bit of psychological biases at play, like: "if your HSM is 10x cheaper than everyone else's, it must be crappy and insecure".
For example, a normal YubiKey is unrated, a YubiKey FIPS is level 2 rated, and a Thales HSM is level 3 rated with all sorts of zeroization hardware.
I think they mostly require an outside evaluator to do a sort of documentation process that costs somewhere around $500k depending on complexity on a new product, and maybe $50k just for up-versioning.
It's generally hard to get that money back on a product since the market of organizations that need the certification is tiny and then the larger overall market for a security product is also usually small and not so happy to defray those costs.
Software was the same, hardware looked the same. The crypto module is validated only with the $$ hardware.
Sometimes the non FIPS devices will have other algorithms not on the FIPS list.