Gitter uses a proprietary websocket protocol built on top of the Bayeux protocol: https://docs.cometd.org/current/reference/
More details can be found on the developer site: https://developer.gitter.im/docs/faye-endpoint
We also have an open-source IRC bridge: https://github.com/gitterHQ/irc-bridge