> I have a 50/50 fiber connection now and even with multiple people watching HD video streams, torrents and other files I don't believe I come close to saturating it.
Unless you've throttled your torrents, or you're connected to a really slow swarm, you have almost certainly saturated your download. I have -er- 100/10 or 100/15 service from Comcast. If I'm downloading anything >= 100MB, and a member of the household is going to start a new Netflix stream, I need to pause the download, or else the stream will be degraded for a long time.
Also, unless you're using a decent router, it's very possible that your router can't route and/or NAT at 50mbit or faster. I know that Apple Airports can and Ubiquiti EdgeRouters[0] can. I would also expect that an ISP-provided router would route fast enough.
[0] If you're in the market for a new router and are somewhat network savvy, or are comfortable with running OpenWRT on your router, I strongly recommend the Ubiquiti EdgeRouter Lite. It routes and firewalls at ~1Gbit/sec (when offloading is enabled) and costs right around $100. Check out the Newegg reviews that aren't "It's slower than I expected" or "It's totally broken" for an idea of what you'll be getting in to. I've had mine for nearly a year and -other than the eventually dodgy external power supply[1] it shipped with- it has been rock solid.
[1] The unit I got had been sitting in a box in a warehouse in Florida for ~two years before it arrived on my doorstep. The failing power supply presented as infrequent spontaneous reboots with nothing unusual in syslog, which became more and more frequent until the power supply just stopped providing power entirely. Replaced it with a wall wart that I had on hand and have had no trouble since.