This seems unlikely for the reasons you listed. Since USB is a polled bus and has shared bandwidth, implementing the flow control is likely very beneficial in case the bus can't temporarily handle the full Ethernet speed.
So, a USB-to-Ethernet gizmo just monitors its internal Ethernet-to-USB queue and if it starts to overflow, it attempts to flow control the switch. But, if no USB host is connected, no USB data requests ever arrive, the queue overflows and never gets cleared...
There may also be some WoL/USB remote wakeup stuff at play, which may be why these ICs do not drop the link on USB host disconnect by default. In fact, I wonder if messing with those settings on the host side could remediate the issue...