No, the better question is; Should someone else be able to build a knock-off of your home?

IP is not the same as physical property. Disney isn't forced to give away they artwork they've made internally. Mickey Mouse as IP, i.e. as a concept, is currently not allowed to be reproduced.

If we're going to debate property rights, it's important to have the context and the property properly defined. Many people can rightfully ask if land should be allowed to be held by any single entity for infinite time, but again, that's not the question here.