Also go to
Settings App -> General -> Restrictions -> Apple Music Connect (toggle to off)
Settings App -> General -> Restrictions -> Apple Music Connect (toggle to off)
So the trick is to first enable Restrictions, then toggle off Apple Music Connect, then do what bydo says in his comment above (Settings App -> Music -> Show Apple Music (off)).
Pretty intuitive really. Great work everybody.