You can also graduate to removing almost all notifications from your phone. I say almost all because the beauty of Slack is that it never differentiated between being mobile or at a desktop (like Skype did) so you might need notifications from your work Slack to appear if you are not really present. Also, get rid of non-work Slacks on your phone, if you spend too much time on Slack on your phone, it could help eliminate more potential distractions.
The next level is to remove all native applications for anything that you think might distract you. It is much, much harder to fall into the rabbit hole of infinite scroll type binging with a mobile web experience. There is just enough friction with mobile web to snap you out of your trance, usually.