This probably is because many buyers go with the inspector recommendation of their realtor and realtors aren’t going to recommend someone that is too through to the point of risking the deal.
In most states, home inspectors are licensed and insured professionals so they can be held liable for obvious omissions from their reports. But there are a lot of holes in the system, and there's no clear guide for inexperienced home buyers to vet inspectors.
The benefit of a good inspector is that they aren't the buyer and thus have no temptation to overlook things the buyer might miss. A good inspector is thorough and will take several hours to perform an inspection of even a small home.
If you can find a very reputable inspector, it's generally a good idea to use them both when selling or purchasing homes. Inspecting your home before sale can identify issues you can quickly address and it gives comfort to buyers to see an inspection report on the counter when they do a walk through.
I got burned with a home purchase early on in my life because I used my realtor's inspector who wasn't licensed in my state. When we later discovered a foundation issue we had no recourse because the inspection report had a disclosure on it.
A general problem in the US is that it's mostly ok to be misleading as long as you have a disclaimer, and that disclaimer doesn't necessarily have to be upfront or obvious. As a result of have a lot of "Administrations", "Bureaus", and "Authorities" that superficially appear to be governmental or reputable but turn out to be scams.
A decent inspector can and will find things, but true it may be difficult to find one.
I think the solution is going to be an unpopular take--learn about the world and be able to do a lot of this stuff yourself.
I bought my house without a realtor and without an inspection. My house needed work for sure, but I got a great deal and learning experience in both the buying and repair/maintenance. I spend my evenings working on the house instead of watching tv and eating garbage. Three years later I have a few 100k in equity.
I do wish the world was full of trustworthy people and life was easy, but no number of bureaucratic requirements are going to make these problems go away. If you live in a world you don't understand, you will be taken advantage of.
We had asked for various fixes to be made but the seller only did a sewer repair and I think that is/was required for the house to be in sellable condition.
With everybody looking for houses now, a big thing is to put in a bid without requiring an inspection. I'm sure it works out almost all of the time but I'd hate to do it.
People love security theater where some professional swoops in and assures them everything is fine. In some cases you have to rely on an expert but in many cases, I've found that for anything significant, I can't outsource risk assessment because it's not worth it most of the time. No one doing the assessments have enough liability or incentive to do a deep assessment, it often simply costs too much relative to what you're assessing.
Now if you're completely naive or ignorant about something or the total risk is known and is relatively low, then it doesn't hurt to pay a small fee for the quick risk assessment passover made by an expert who will be more likely to spot things than you. Also, if someone really has significant liability for their assessment, they tend to be very direct to the point of pointing out low risks they normally would brush over.
I recently purchased a home and it was hooked up to a private well. The seller said it was fine, the inspector said everything looked fine based on old test data, and a private well inspector said it looked fine. Not a single one of them thought it was worth testing the water quality beyond the bare minimum, only concerned about quantity. It turns out, I spent a part of 4-6 years developing water quality monitoring infrastructure and performing water quality analyses and had it not been for the knowledge I had and always skeptical attitude, I wouldn't have batted an eye. I told them to all stuff it and perform a water quality test or I was rejecting on other terms. Good thing I was difficult because there were all sorts of issues (bacteria, organic and inorganic issues) and I may have even gotten sick or a guest could have gotten sick.
Years ago I had a health scare and saw three independent doctors that thought it was nothing, even after describing my symptoms but there were too many red flags for me to ignore accept their opinions. I suggested a specific condition to each of them and they brushed it off like I was crazy. They didn't have the same level of risk or incentive for their time in their assessments to push forward and didn't want to acknowledge that even a low level of uncertainty could be true. Sure enough, I found a fourth doctor to check, they listened, and did appropriate diagnostics eventually confirming via lab and in person diagnostics I was correct. This isn't something that frequently occurs, not a hypochondriac.
Professional risk assessment can be useful but you should always be skeptical of it given the lack of liability and incentives many have. They want to get paid and move to their next transaction. Such is our highly transactional society where people really don't care about other people or can't afford to care about other people in the name of efficiency.