Mostly experience.
And unfortunately it's mostly an ad-hoc experience. That is, my particular experience won't generally be applicable to your particular situation. You have a different team, different preparation, skill level... You're using different tools, workflows, methods. Your projects and features can be different to mine. You get the point.
But fortunately it's an experience that can be fairly easily acquired and organized if you make the effort. You only have to measure and follow through on your estimations and their accuracy. You start making mostly educated guesses. Then you go ahead and do the work and compare the estimates with the time you actually took. You correct for the next estimate.
Loop through this several times and you generally end up with better estimates through each iteration. If you do it in an organized and consistent way, obviously.
Some of the keys for making this work are in no particular order:
- Break down larger tasks into smaller ones and estimate the small ones. It's easier to correctly estimate a smaller task. Also, the errors should generally be smaller.
- Following the previous point: Make sure you account for "hidden tasks". That is, you need to consider those parts of the work you just assume tacitly and make them explicit. Setup, infrastructure, bureaucracy, etc. You need to write down all tasks when you break down large tasks into smaller ones. You can, to some extent, account for this through an added safety margin, like multiplying by 1.2 or whatever. If you do that, I'd say start by multiplying by a larger factor (1.5 even 2) and in time, with the added experience and better ability to spot hidden tasks, reduce that factor accordingly.
- Involve the people doing the actual work in the estimation. Don't try one person doing the estimate and a different one doing the work. Also, don't let deadlines become estimates. I mean, you may not avoid a deadline but do a proper estimation anyway. If not for anything else, at least for the consistency of the estimation process.
- Give and use actual estimations. An estimation is not a single number, but a pair of numbers -sometimes even 3 or 4-. You may organize your estimations as a "duration + confidence level" (e.g. 3 days with a confidence of 80%), as a "lower and upper bound" (e.g. at least 3 days at most 5), or as "durations at various levels of performance" (e.g. if all goes very well 3 days, normally 4 days, if X thing happens then 6 days). Different styles work better for different people and circumstances. I like the last one because it helps bring up potential hidden tasks. But whatever you do, try to stick to the idea that a single number estimation is just useless.