One thing I've not understood is why will some hops have consistently lower ping times than hops farther down the chain in the same trace?
Is it indicating that the router is faster at forwarding packets than responding to ping requests?
https://archive.nanog.org/sites/default/files/traceroute-201...
Exactly this. In most “real” routers, forwarding (usually) happens in the “data plane”. It’s handled by an ASIC that has a routing table accessible to it in RAM. A packet comes in on an interface, a routing decision is made, and it goes out another interface - all of this happens with dedicated hardware. Pings (ICMP Echo requests), however, get forwarded by this ASIC to a local CPU, where they are handled by software (in the “control plane”).
You’re really seeing different response times from the two control planes - one may be more loaded or less powerful than another, regardless of the capacity of their data planes.
Maybe the only thing I've explained more in my career than this is why it's ok that your Linux box has no "free" memory.
Beyond that technicality, your guess is often right... Routers will frequently prioritize forwarding packets over sending the TTL exceeded packets tools like MTR use to measure response times.
Obviously you know, but for anyone else reading, a modern traceroute tool (like mtr) can send icmp, udp or tcp, on generic or specific ports. Indeed the default for mtr on my laptop is to use icmp.
However, it could also be the case that the routing back to you is significantly different, so you can have a much longer path to you from router N than router N+1.
This is more likely to happen on routes that cross oceans. Say you're tracing from the US to Brazil. If router N and N+1 are both in Brazil, but N sends return packets through Europe and N+1 sends through Florida, N+1 returns will arrive significantly sooner.
I believe most of the time this is the reason indeed. Answering an ICMP error to a TTL expiration or to an echo request is very low priority.
This latency in error message generation may even be a better signal of the router load than the latency of the actualy trip through it.