What can you do with a second Ethernet port?
linux.com
linux.com
nTop - http://www.ntop.org/news.php
And realtime traffic stats:
If you run SIP on your network, you can capture call setup information to debug phone issues:
And even extract RTP audio streams to investigate call quality issues:
for app-level protocol stats and/or high-end situations i'd attach an extrahop product: http://www.extrahop.com/products/