The most elegant, protocol-agnostic solution I've seen for doing this is 1st to throttle individual clients to their allocated BW (i.e. so that one client doesn't gobble up an entire shared link), and 2nd to simply kill any long-term (12hrs+) TCP/IP connections that are not over a recognized/allowed L7 protocol. The 2nd is very effective at starving unwanted bit torrent usage.

This method is in use right now by the guys who hang 5.8GHz antennas on grain silos, i.e. rural WISPs.