When Finder is active click on the View menu, press the option key and select "Customize Control Strip", then you can just mouse down to the touchbar and remove the Siri button.
In Preferences under the keyboard settings you can customize the placement and presence of every single button on the TouchBar. Preferences also lets you disable Siri
Also you can make shortcut for "Customize Control Bar...", it helps to quickly adjust Touch Bar in new program or for experimenting