Netflow! My first "big data" project was getting netflow data from 100s of routers in a internet backbone and start to learn where traffic came and if it was worthy to do a peereng agreement with them.
It stopped working the day Google activated their proxies on mobile networks. I saw Google traffic increase 1-2% week-over-week during several months. I quitted the job when it was >30% of the backbone traffic and don't know what happened at the end...