Are those the steps for Android 5/Lollipop? I don't see any of those menus in settings.
edit: Ah it's in the actual hangouts app that you have to go to settings. Also if those settings are greyed out then you might not have Hangouts handling your SMS messages, so run the Messaging app instead and go to its settings to disable auto open of MMS.