Real estate agent here... in my market there are a number of mid-century homes that have the very square look of a manufactured home. Even looking vaguely like a manufactured home is enough to keep people away from them. My own observations are that newer pre-fab / manufactured can be very well built. We have a lot of people coming here looking for rural retreats - truth is, if you are way off the grid, a manufactured home may be a very good option for you as it may be difficult to convince a builder to work on your custom build project if it's going to be a long commute for them.
So what I'm saying is - there's a mental block for people in accepting these houses that has noting to do with reality and everything to do with perception.