Riot can do audio and video calls, I've had good luck with it (currently gets used between my friendgroup instead of phonecalls). Check out http://matrix.org/ for the server implementation details and https://riot.im for the biggest client. Currently, it only supports 1-on-1 audiocalls/videocalls. No group calls yet.

Forewarning that while it supports end-to-end encryption for the chatting bits, the actual audio data is not encrypted. The call request/advertisement can be encrypted though!