They are not all soldiers, the corps has considerable civilian staff as well. But the reason is mostly historical. The corps existed first to build and maintain coastal fortifications and then when the western frontier opened up it made sense to assign them to secure the rivers and improve the waterways west (the western expansion was as much a military operation against the native peoples as it was a civilian migration).

The criminal division of the IRS is most certainly armed, as well as equipped with AR-15s, and are trained by Army Special Forces contractors.