Samsung's hardware today seems better, but for a long time the galaxy phones were cheap plastic builds with unwanted software customization on top. The few Motorola phones that came out during the Google ownership were nice.
If Google had built their own hardware back then (or had even gotten Nokia) I would probably have stayed on Android. For a long time starting around iPhone 5 the Apple hardware was a lot better (prior to that Apple lacked turn by turn navigation and LTE). The Nexus One was nice, but the S wasn't great and the Galaxy Nexus was terrible enough to switch.
Now Apple seems to be winning in security and new hardware features and everything else is fairly comparable.