One thought that keeps me awake at night is that my past automation work has cost others their jobs. I understand that that's "how the world works", but it still deeply affects me that people have lost their livelihoods because of my actions.
Progress enables productivity which is a net win for everyone, your work has contributed to productivity a LOT more than the work of the people who needed to move on. Without technology we’d almost all be in poverty.
In the end you're probably right for a population of people. I'm not talking about that, though - I'm talking about very specific people with families to feed.