etckeeper records the exact package version changes too. it's extremely handy for small shops (so unless you do 100% immutable infrastructure, it's worth it)
you start culling packages with apt purge, that should remove config files too, and see what breaks, if nothing, well, try to push your luck further.
also with dpkg -S /etc/some/file you can see which file belongs to which package and target that package
usually on a server you know what you don't need. it's a bit more trial and error on a desktop. (because those can break in a lot more fun, unexpected ways, harder to know what is needed)