The best way I know is to delay introducing the new abstraction until you have enough material to justify it. This means tolerating more repetition than you want the program to have in the long run, but only temporarily, while you build up a repertoire of cases that can eventually be abstracted away. You wait for the code to tell you how it 'wants' to become shorter, rather than trying to decide the primitives in advance. It's still just a bet that an abstraction will prove useful—i.e. will allow you to write future cases without thinking—and it still might not work out, but this puts the odds in your favor.

It works the other way around too: when an abstraction such as a Lisp macro turns out not to be a good fit, simply inline all its usages and delete the macro. This makes the program more repetitive, but only temporarily, while you wait to spot a different abstraction that fits the material better. This again is listening to how the code 'wants' to be: if you get resistance (increased complexity) while moving in a certain direction, backtrack.

I try to give every concrete case veto power over a new abstraction. That is, if I've got six cases of a pattern in my code, and five can nicely be shortened with a macro but one is stubborn and refuses, I tend not to write that macro. I used to, but have observed that such abstractions tend to need undoing later. Instead, I'll hold off, tolerate temporary bloat, and wait for more cases. Perhaps when there are nine or ten, I'll get a new idea for a language construct that can express all of them nicely, and then I'll add that construct in the way that pg describes. It's like the Tao Te Ching says: "If you want to shrink something, you must first allow it to expand" [1]. When you hit on a good way to compress code, it feels like a satisfying click [2].

With such an approach, the program seems to breathe: expanding a little and contracting a little, but never straying too far from simple overall. Lisps are the best medium I know of for this. You get finer and more immediate feedback from the material of the code: less is consumed by the noise of the language itself, so it can be adapted to fit the problem better. Your brain fills up more with the dance between problem and solution instead of the externalities of the tools. And you have a wider range of tools for actually working the adaptation. The result feels more like molding clay with your hands than carving bricks with a chisel.

1. http://taotechingme.com/chapter-36-if-you-want-to-shrink-som...

2. https://www.npr.org/2010/12/30/132488837/The-Behind-The-Scen...