A transport protocol's view of Starlink
potaroo.net
potaroo.net
> Interestingly, we observe that the Starlink OWD (one-way delay) often noticeably shifts at interval points that occur at 15 s increments. Further investigation reveals the cause to be the Starlink reconfiguration interval, which, as reported in FCC filings [71], is the time-step at which the satellite paths are reallocated to the users.
AFAIK it is not the dish itself that does the tracking but a central orchestration.
[1]: https://arxiv.org/abs/2310.09242 [2]: https://arxiv.org/abs/2306.07469
Teardown videos available
Most routers do not put ping-processing in the "fast path". That is, instead of having the ping be processed by an ASIC, the ping gets processed by the router's CPU. And ping-processing is typically a lower-priority task. Because of that, you can't assume that the high variation in latency is because of Starlink.
I wonder how true that is anymore, with ICMPv6 processing being a mandatory part of IPv6. I could totally see ICMP processing being a low-priority task, but am far less certain that it would not be done by dedicated hardware these days.
On top of that, I've never, ever, ever noticed the behavior he's observing with either my cable ISP connection, or the terrestrial microwave link provided by my local WISP. I don't have enough data to say that I'd never see that behavior if I happened to ping some router powered by a potato... but I've pinged a whole bunch of systems over the years, and have never seen variation like what he describes.
Could have been traffic shaping or prioritization (their network was in complete shambles after all), but ICMP was definitely taking some lower priority queue or path.
ICMPv6 may be mandatory according to specs, but you can still drop most of it with no ill effects. You probably shouldn't drop needs fragmentation packets, but everyone has adapted to those being dropped sometimes, so...
If you ignore neighbor discovery, you'll have trouble reaching your neighbors, though.
But neighbor discovery is low volume mostly, and can be handled by the cpu, not the asic. Needs frag likely won't be directed at the router ip, the asic can forward them just fine though.
I've certainly seen much more variable ping times for routers than for the hosts behind them. If the router's cpu is less busy, ping times are usually a bit more than a host that's right there, and as the router's cpu gets more busy, the rtt increases or pings get dropped. It's not usually a factor of how busy the links are either; routers tend to have a pretty limited amount of cpu and if they're getting a lot of traffic, it's easy to overload them.
I haven't seen it happen quite in the bands like in the article, but sometimes it varies quite a bit. The results tend to be much more consistent if you can get a rtt from full host and not a router.
I believe a better technique would be to ping _through_ the router to another endpoint under ones own control, on the same network as the origin client, and where processing latency can be controlled. Basically ping yourself but via starlink and back.
That said, in practical terms he's quite right about the jitter problem. I maintain 2 networks myself at my remote home. One my microwave WISP for tight latency and jitter, and then Starlink for much better bandwidth/throughput.
> ping across a Starlink connection from the customer side terminal to the first IP end point behind the Starlink earth station
Pretty sure that is the reason why he used
> ping across a Starlink connection from the customer side terminal to the first IP end point behind the Starlink earth station
instead of pinging the terminal itself.
(ICMP packets passing _through_ a router are handled just like any other traffic and are therefore OK for an analysis like this)
These changes are more typical of a physical path change, as suggested by the author.
CPU/soft processing latency would look more like additive noise.
I give some examples of RTT patterns in this talk: https://ripe77.ripe.net/archives/video/2250/
YMMV if you attempt the same traceroute from Bangalore or Sydney to London/Paris/NYC.
From my router in Dhaka pining a router on airtel in India, 36-38ms
Routers in the real would give very stable ping responses, no matter where they are based.
The last mile is usually the issue. Pretty much only Cogent and HE allow you to have packet loss
That’s meaningless. Control planes are often policed sure, but on overload will simply drop. In my experience they drop icmp to expired before echo response but most will generate ttl expired just fine.
Any router capable of processing a full bgp table, and to be honest any router made in the last 20 years, is perfectly capable of responding to icmp echos.
There was then a second argument that “3rd world” routers aren’t as good as ones in western country. In the majority of cases they’re exactly the same. That western arrogance is somewhere between insulting and amusing.
The final argument is about path loss/jitter/etc, specially loss on the first hop (your “my crappy 3G provider” argument)
That’s exactly what this test of starlink is showing.
Starlink is a great tool in specific cases, but the fanboyish ness often drowns out the actual benefits.
You be capping
That was a temporary situation though, and I think the worst I’ve seen in east Africa from the last 10 years.
The U.T. contains a phased array antenna that can electronically 'steer' the bore-sight (aim) of the transmitted (and received) signal at the current satellite that is in view. In ideal circumstances the U.T. antenna has approximately 110 degrees field of view (~ 35 degrees above each horizon).
The satellites pass from west to east and take approximately 15 seconds to pass through the field of view of the U.T. The satellite forms a beam aimed at a fixed location on the ground - this is called a 'cell'. All U.T. within that area share the radio link that has a fixed bandwidth, so contention is managed by the satellite.
The path length to a satellite directly overhead would be around 550km (in most cases the satellite is slightly north, or slightly south, of the U.T. but for round numbers sake assume 550km).
The path length to a satellite appearing 35 degrees above the horizon (the slant range) is ~ 2568km.
Satellites relay the packets from the U.T. to the (nearest) Earth ground station, so the path length and therefore travel-time will vary enormously over just 15 seconds.
The round-trip for the minimum case is 4 x 550km = 2200km but for the maximal case is 4 x 2568km = 10272km. These equate to a travel time of between 1.8 and 3.6ms per leg, so that gives a hard physical minimum of 4 x 1.8ms = 7.2ms to 4 x 3.6ms = 14.4ms
As more satellites are added to the constellation so the gap between satellites decreases and the angle above horizon at which a satellite is acquired can increase thus shortening the maximum path and lowering the latency.
Starlink has a publicly stated goal of less than 20ms round trip latency and published a report in March 2024 about the engineering efforts to achieve this [0]. Much of the effort that customers see focuses on two issues:
1. reducing latency between ground station and Internet connection point
2. scheduling the radio links between satellite and all U.T.s in its beam area
[0] https://api.starlink.com/public-files/StarlinkLatency.pdf
Huh? Is this right? That's really fast, like 24,800 miles per second, which is a significant fraction of c (the speed of light, 186,000 miles/sec). >10% of c, is this a typo?
Is that a typo? Orbital velocity on the surface should be 7.9km/s.
Uhm, no:
-f
Flood ping. For every ECHO_REQUEST sent a period “.” is
printed, while for every ECHO_REPLY received a backspace is
printed. This provides a rapid display of how many packets
are being dropped. If interval is not given, it sets interval
to zero and outputs packets as fast as they come back or one
hundred times per second, whichever is more. Only the
super-user may use this option with zero interval.If the latency is under 10 ms, then what op said is correct .