You need to implement DTLS and SCTP for client->server, just not ICE. But client->server ICE is fairly trivial with ICE lite. It's DTLS+SCTP that are complex, and they don't go away client->server.

Replacing DTLS+SCTP with QUIC would be a good reduction in complexity, and we (the WebRTC team) are experimenting with doing so (https://cs.chromium.org/chromium/src/third_party/webrtc/p2p/...). It has a long way to go, but hopefully someday just having QUIC on your server will be enough.

(I work on WebRTC)