> Which source? The charger or the device? Because either could be at fault.

Any device acting as a power source - but you need PD for this so you can create power contracts that are compatible with the root power source, and you're not describing a PD situation, so this is not possible.

Probably your router does limit downstream power, but due to many years of devices ignoring the 500mA limit, these limits are usually very liberal so as to avoid users complaining that the USB port doesn't work with device X. PD helps with this by establishing a way for sources and sinks to negotiate a clear power delivery contract that the source can be strict about limiting.

4W is more than non-PD USB can supply, so the fact that it doesn't work is not that surprising.