I think you are dramatically overestimating it.

It's one thing to vandalize some cameras or a driverless car in your neighborhood.

Traveling to a large, potentially secured facility to commit terrorism is on another level.

It being inside a building it is also much more difficult to damage and the consequences would be far greater.

I'd think one would need strong motivations for such actions.

A protest seems much more realistic, but I did not even hear of protests at data centers yet, only a small one at OpenAI headquarters.