I've been working with a robotics startup for the past few months, and I'm often struck by how much extra stuff has to come together to deliver the product (in our case, we have to integrate with phone and elevator systems, have to deal with various regulatory compliance in every country we ship to, etc.). The sales pipeline is also long and expensive, involving shipping out multi-thousand dollar devices for evaluations, sending people out to demo the devices, etc. It's a production involving a half dozen people and thousands of dollars worth of resources to make every single sale.
So, yes, I think they are harder than some kinds of businesses. At least an order of magnitude harder than software. Though, because of it, the competitive landscape is somewhat smaller...robots are cool, so a lot of people want to work on them, so there's more than the market will support, but it's less pronounced than, say, the mobile app market in terms of competition for customers.
In general, I think the hardware business is hard for all the reasons you mention. Or better stated, the hardware business has characteristics that make it difficult to get right the first time (more important in hardware than software, I’d argue) and scale. It’s not inconceivable for a single (or a small team) to build a useful software product that makes money (several successful mobile apps, companies like 37signals, etc.). It’s difficult for me to think of similar hardware analogs (I’d love examples!). This isn’t to say that scaling software—building companies like Google–isn’t incredibly hard. It is. But for a hardware company to even have a chance, it needs a minimum scale threshold from the very get-go (besides all the other things that any business needs – a useful product, luck, etc.).
I used to work for a large aerospace company in the early/mid 2000s, and the products we built were useful, and amazing (hunks of metal weighing tens or even hundreds of thousands of pounds flying in the air transporting people). I was absolutely awestruck by the fact that upcoming companies like Google and Facebook (which were “just websites” in my then-naive worldview) had fewer people, and made way more money than us.
Think about Vacuum robots on the other hand. It's still a small niche, but there's a lot of potential. I'd like to see some improvements in that direction, like if they were smarter, with better robotics & movement capabilities, they could be self-sustainable (charge themselves and empty their garbage bag etc.). Could do more tasks, remove dust, or check on things, turn off lights when you are gone... you can come up with some many things here.