IANAL, but my understanding of CA law is that it's more complicated than that:
1. Just like the police, they can only go in without a warrant if they have a reasonable belief that there is imminent danger to a child. They must have a specific thing in mind, not just "vibes." Bringing your child to the door to show them they are fine would often be sufficient to quash that (but you can't do that if your child isn't home).
2. Refusing to be interviewed by CPS and/or denying them access to your children can be used against you in family court (unlike when you are being interrogated by the police).
3. From the ACLU of SoCal(https://www.aclusocal.org/en/know-your-rights/california-chi...):
> If there is no warrant and no emergency and you do not want to allow the investigator in your home, you can say something like, “Sorry, this isn’t a good time for me, but I’d be happy to schedule a visit on a later date.” By offering an alternative date like the next day or later in the week, you can plan to be more prepared for a visit, but also show cooperation. If you refuse any visit, that can be used as evidence against you later in the investigation. Depending on the situation, follow-up visits can take place either at home or the CPS office.