All encryption runs clientside. The whole point of end-to-end is that you don't have to care about who the message passes through.

Your actual worries at this point are the key exchange (impersonation) and extension distribution.

The first is very hard to get right without alienating a large number of potential users, the second seems perfectly possible to have verified but seems like it's going to require some kind of changes in the Chrome extension system.