> `apt remove` will often leave other things lying around.

If you mean configuration files, then this is by design.

`apt purge` removes those as well.