In my opinion open source hardware IP has at least two major limitations compared to software, which make it significantly less useful or important to end users. I guess your preference for open source rather than copyleft already hinted that the benefits are mostly for the ecosystem rather than the users. The first one is that even though you can modify the design, you have no practical way of using your modified design (at least in this context, because FPGAs are slow and expensive and high performance SoCs don't fit anyway).
The second limitation is that it is impractical to audit the hardware to determine whether you have actually received the unmodified open source design. For software this can be achieved using reproducible builds or just by compiling it for yourself.
Open source hardware boards (as opposed to IP) have some of these limitations as well, however both can be overcome by a hobbyist with a modest budget.