I built a server-side WebRTC implementation for Twilio (as part of a team of two) in Java (with C bindings to libsrtp and openssl) in mid-2012 (and gradually evolved it over the next year and a half before moving on to a different team/project). I'm certainly not going to say it was easy, but it wasn't as hard as you're making it sound. It turns out you can cheat a lot here and there. You don't need TURN at all, and the ICE/STUN stuff can be dirt-simple because you don't need to deal with UDP hole punching or other esoteric things. You can cheat and hard-code most of the things that would usually need to be variable.

Honestly, most of the difficulty back in 2012 was because all of it was poorly documented and the standards were changing with every Chrome release, so it was partially a reverse-engineering effort. If I had to do it again from scratch (even though it's been at least 2 years since I've touched it), I could probably get something working in a week or two, and production-ready in a month or so.

I think a large part of the difficulty you describe is because you're trying to do this in Javascript. Doing it in a language that already has some library support for the various components cuts out a lot of the pain.