If it helps you in talking yourself down from domestic terrorism, IRS is just law enforcement. Killing IRS agents won't lower your taxes, in the same way killing police officers won't legalize anything.
But I do give it serious thought, with the quandary normally being: do I have the capacity (and the willingness) to inflict such an impression that I can terrify people from choosing to work for the IRS?