EDIT: Judging by the upvotes, my issue below might be a common problem. Read below and be warned :)
---
Well I guess they did say Beta...
I turned this on immediately and now get HTTP 500 errors on every page on GitHub while logged in. Clear cache and login again does not solve.
Oh dear.
Thankfully their support portal does still work. For anyone as unlucky as me: