Many Taiwanese see the ROC as invaders, they would rather have neither the PRC nor the ROC governing over them, and this is what the PRC is most against (at least if Taiwan is controlled by the ROC, it can still be seen as a civil dispute over who controls China, vs a question of an independent Taiwan).

Japan actually treated Taiwan more as a province rather than a colony, which led to a lot of distrust between the islanders and mainlanders when the ROC regained control of the island.