But in general direct or indirect retaliation is a risk and is why it is so important for you to do something ASAP.
I don't know all the details of your situation but based on what you said in OP it might be advisable to switch. The sooner you do the less time you might 'waste' in an unfruitful project and the less time and money the supervisor has invested in you, so they might be less likely to retaliate.
Further, it also might be considered 'professionally embarrassing' for the supervisor if you ask to switch during the PhD. At the stage you are now you could probably change in a way that saves you hassle and the supervisors face by diplomatically posing the change as a mere change in 'academic interests', i.e. slightly different project with a favourable supervisor.
Of course you should suss out the possible alternative supervisors first and try and meet them and their students to make sure they are actually favourable to your current prof.
These are just the first points that come to mind, happy to answer other questions if you would like (see my gmail in profile).