The “Moving Truck” school of project estimation
Estimation has long been a point of embarrassment for the software development community. A big part of that is unrealistic expectations set by association with construction and manufacturing industries. Software is unlike building houses and widgets because every software application is unique and unprecedented, whereas construction and manufacturing most often involve assembling well-understood components in repeatable ways. Truly innovative construction projects, like the Big Dig, and the Sydney Opera House, regularly overrun their estimates (by 3 times and 15 times respectively in these cases). They shouldn’t be compared with building out a subdivision either.
A more appropriate analogy is a moving truck. If you have ever hired movers to help you move between homes, you know that the process starts by someone walking through your house and looking over your stuff. This person may pull out a tape measure a few times but she is not measuring everything and she is certainly not doing things like seeing if your ottoman can be neatly wedged underneath your dinner table. The real purpose of the exercise is to know how big a truck (and team) to bring on moving day. It is up to the movers to make the most of the space and there is a real art to that. Sometimes the moving team can see right away that the job is going to be a real squeeze and they think through every piece like a game of Tetris. But sometimes the truck is just too small, you need to call in another truck. Either that or pulverize your furniture into very compact dust.
The main point of this analogy is that precise software estimation is impossible at the detail level. It takes people who have done lots of projects to give a decent “truck-size” estimate, and even then they can be wrong. In fact, each task estimate will likely be wrong. Accuracy is achieved only at the aggregate level when the underestimates cancel out the overestimates. Just as important, an experienced team will be quick to identify the need to adjust: either by increasing the scope (size of the truck), by leaving low value items on the curb (decreasing scope), or by changing how things are packed together. The earlier you are aware of your constraints the more options you will have. Only inexperienced teams wait until the truck is full to realize that some adjustments need to be made.