It depends on where in the silence zone you are (I live in the NRSZ). There are increasingly strict zones the closer you get to the Observatory. Unless you were flying your drone close to the observatory itself, it's never been a problem. Folks in town have WiFi as well. Although the authorities can shut it down if they needed to, they're generally hands off unless it's a problem.
A lot of military and civilian signals intelligence outposts are out there, so they probably wouldn’t be so happy
From Edward Snowden's book I learned a lot of people living there work for the government as well.