We constantly need to make estimates in software engineering. But, the ironic thing is not that our estimates are off, but rather that they're almost always short. There's a one-way bias.
I think this is because:
- It always looks simple in our heads. Devs are overly optimistic and plan for best case scenario, and hence get caught off-guard. I mean, who would ever guess that it takes 5 hours to fix a single line of code?
- Scope creep - The business just keeps piling on more requests (and it's hard to say "no" to the business).
- Strong bias from managers - Most managers want the lower estimate because, regardless of whether the estimate is actually accurate, it's easier to sell to their own boss or clients. For example, in consulting it's easier to give a lower estimate to "get your foot in the door", and then gradually try to pile on more services.
- Devs try to impress - I can't help but wonder if many devs try to impress their managers by providing a lower estimate "Sure Mr. Burns, I can get that 5 day task done tomorrow (because I'm a rock star developer... please appreciate me)"
- It's almost expected - At least in my experience, it almost seems like (unfortunately) the industry has resigned itself to low-ball developer estimates. So the managers always just double (if the estimate is from a senior dev) or triple (if the estimate is from a junior dev).
The best solution I can think of: read and study Steve McConnell's phenomenal book on Software Estimation: Demystifying the Black Art.