The downside: A lot of websocket APIs (and most especially the one in the browser) don't support this and will only send/receive complete messages. Which means if you want streaming support you better implement it as a layer on top of messages, since it works everywhere.
It's a little bit sad that the websocket spec is complicated through the continuation frame feature while in reality noone has a reason to use it.
And back to the question: The masking "feature" of websockets is also there to prevent browsers from speaking raw TCP. Without it Javascript could craft exact TCP payloads, which might in certain situations be used to directly talk and manipulate internal services. The masking guarantees that the remote on the TCP connection will get some random data after the websocket header.