Exactly this. In most “real” routers, forwarding (usually) happens in the “data plane”. It’s handled by an ASIC that has a routing table accessible to it in RAM. A packet comes in on an interface, a routing decision is made, and it goes out another interface - all of this happens with dedicated hardware. Pings (ICMP Echo requests), however, get forwarded by this ASIC to a local CPU, where they are handled by software (in the “control plane”).
You’re really seeing different response times from the two control planes - one may be more loaded or less powerful than another, regardless of the capacity of their data planes.