True, currently we always need some sort of signalling server to be able to connect to browser<>browser peers but once you done that, you can connect directly to other browsers (truly P2P in the browser).
Now, if we can have js-ipfs and go-ipfs nodes on desktop automatically act as signalling servers (if you start with a `--server` flag or something), we'll be able to decentralize the need for signalling servers as well, and remove another point of centralization. Getting us closer to the goal as well.
Maybe in the future we'll be able to have TCP/UDP sockets in the browser, but we'll see when/if that happens.
Disclaimer: I work fulltime on IPFS