One thing that hasn't been mentioned yet is queuing delay. All networks contain queues to smooth over bursts of packets being sent at once, and the time a packet spends occupying a queue can be significant.
Yes, this is huge, and likely one of the least appreciated aspects. Scheduling and queueing delay (not in the switches and routers) can contribute significantly to that 100ms.
No one forgot about those factors, he was only adding up the numbers to see where the lower physical limit is.
That shouldn't be significant unless:
1. There is a significant serialization/deserialization delay, which I don't think will be the based on my naive understanding of the tech
2. Buffering in time periods even relevant to discussion should really only occur under specific circumstances (like during overutilization)