Many software projects go overtime. Software developers get blamed for that, laughed at, scorned at.

But guess what: it's not only software developers.

Many building projects also go overtime, and most noticeably of all, perhaps, opera houses and concert halls.

Here are some notorious examples.

From the saga of the Sydney Opera House (10 years late, 15 times over budget) to Los Angeles’s Walt Disney Hall (16 years in the making) and Copenhagen’s Koncerthuset (in 2007 the most expensive concert hall ever – almost destroying its client, Danish National Radio), the contemporary concert hall has become an architectural nightmare. -Â The problem with building concert halls

Cost overrun for the Channel Tunnel, the longest underwater rail tunnel in Europe, connecting the UK and France, was 80 percent in real terms. For Boston’s Big Dig, 220 percent. The Sydney Opera House, 1,400 percent. -Â Megaprojects: Over Budget, Over Time, Over and Over

Koncerthuset in Copenhagen was two years late, three times over budget, "as is tradition".

Elbphilharmonie concert hall in Hamburg was seven years late and ten times over-budget.

Philharmonie de Paris was two years late and three times over budget.

Wembley Stadium - the biggest stadium in the UK - was eight years late, three times over budget.

After nearly 15 years in the planning process, construction of Berlin’s new airport began in 2006, and in 2016Â there is still no end in sight. "No one, unless he is addicted to drugs, will give you any fixed guarantees for this airport," the spokesman told the press. That's quite an estimation.

Why do we need time estimation?

This is a reasonable question, and it certainly deserves an answer.

A customer needs to know how much time the project will take, because they need to plain their work.

Many software products are being created for business purposes, and the business most likely plans to use this software quite soon after it's been finished. For example, if they plan to hire people who are going to work with this software, these two operations should be aligned. People can't work without the software.

If it is a manager of a software development team who requires an estimate from their team, they need to be on time for the business needs; but also to plan the team's future projects. How can they pick up a new project when the old one is not done yet? The timeline of new projects is going to be skewed as well.

If it is a startup-like company, where they work for themselves, and they need to implement a project, then, perhaps, they act like their own customers. They need to plan what to do with the product: sell it, market it, evolve it. This needs to be on schedule as well.

Finally, there could be a specific calendar-date deadline, such as birthday of a person who gets a small website as a greeting card; or all emojis have to be replaced with pumpkins for Halloween; or the release date of the new project has to be announced on a public press-conference. Imagine the development goes overtime, and the greeting card is late for a week, or the Halloween chat stickers are rolled out November 1st? What is the use of them then?

In some cases the deadline could be very strict; in other cases it can be a bit more flexible, but in professional software development there is always a deadline. There is no such thing as infinite budget or infinite time.

In some cases the customer will ask developers to estimate the project. In this case they will rightfully expect the developers to adhere to the deadline.

In other cases the customer will set the deadline, and developers will be expected to implement to adhere to it as well.

In this article I am going to list some very common reasons why software projects go overtime, explain why exactly it is happening, and how to, if not stop completely, at least mitigate the risk of it. I hope that both developers and their customers will find this article helpful.

Reason 1: Uniqueness, or Complex Requirements

Almost any software is complex. Each project is unique, it has to be custom built and tailored for client's needs and wishes.

Software development is often compared to building houses. But let me dispel the myth that they should be the same.

The art of building has existed for at least tens of thousands of years. Software development - barely for 70 years. If we stretch a bit, it's 180 years, tops.

Many houses are built following one project, with no changes between each other, except for wallpaper color. It's very easy to build a 10th exactly-the-same-house, and there's not much room for error.

Construction of unique buildings, on the other hands, often goes overtime. Operas, bridges, monuments, arenas, bold and complex projects are very prone to it.

Most software projects are different from each other. Each and every one of them have their own, unique set of requirements.

If you're building 10th exactly-the-same-website, it's very easy to estimate and to implement the project.

But if requirements are somewhat complicated, entangled, or not quite clear even for the client, the developers are not going to make a good time estimation. When working on the project, they inevitably are going to understand: they need more time than planned.

For an example of particularly complex set of requirements, take a look at my article about GIS-system development.

Many projects have such a big and complex set of requirements that they could easily take up to several book volumes. No single person is able to keep them all in mind.

If you're following an instruction as complicated as that, some inaccuracy in time estimation is inevitable.

If you're a developer

If you feel the project is too complex, and you're not sure what's going to pop up next, you can negotiate some deadline flexibility.

Have a business analyst write down all requirements, to get as many details as possible. You need to be very well informed to make a good time estimation.

Make sure you understand the requirements in the same way as your customer does! Nothing can be more disastrous as building the wrong thing.

Help the customer to prioritize the requirements in order of importance. Start with the critical ones - they are the most important and the least likely to change.

Make an agreement with the customer that the requirements from the "non-critical" group will be implemented only if you have time in the end. Perhaps, it will be fine to take care of them even after the original deadline - since the critical part is going to be done on time.

If you're a client

Help your developers understand the requirements better. The clearer you explain, the better they can estimate and work. If they ask questions, answer. Provide details. Provide examples. Draw pictures.

You need to figure out the requirements for yourself first. If they are unclear for you at an early stage, that's fine: a skilled development team will help you to understand what you want. But by the time the work on your project begins, you have to understand it.

Developers have to understand the requirements in the same way you do. You need to make sure they build exactly the thing you need.

If you have a strict deadline, be prepared to drop some of your requirements. Prioritize them according to your needs, and divide them into "absolutely necessary" and "nice to have, but can live without" categories. The second one will contain likely candidates for removing from the backlog. They can always be implemented later, perhaps, as separate smaller projects, when your big dream thing is ready.

If you absolutely can't part with any of your requirements, then the only thing is left to consider is, unfortunately, to move the deadline.

Reason 2: Unrealistic Requirements

The original design was so boldly conceived that it proved structurally impossible to build. After four years of research Utzon altered his design. - About the Sydney Opera house, 10 years late

If a project is too "boldly conceived", chances are it's close to impossible to build.

A deadline is also a requirement, so if you want something to be done tomorrow, it might be unrealistic as well.

If you're a developer

Make sure your client understands it's impossible to do what they want. Maybe they don't know the technology is not quite there yet. Maybe they want a thing that's not possible to build even theoretically. Maybe they just don't know what they want.

Try to enlighten your client as much as possible. Perhaps, it's a genuine mistake, they will be happy you took the time to explain it to them, and you can modify the requirements together so that it's more realistic.

If they insist, you might consider not committing to working on such a project.

If you're a client

Check with your developers if the thing you want is possible to build at all.

Do you want a scalable distributed system where data updates instantly appear on all instances? That's not possible even theoretically.

Do you want your servers to have 100% uptime? Not very likely.

Do you want a 3-D dynamic shooter with 200 FPS on outdated laptops? Not possible.

Do you want a social network "like facebook", but hosted on one machine? Not gonna happen.

An encrypted messenger, where some messages still can be sneakily read by a third party? Nope.

An A.I. to conquer the world? No.

Still, your developers might want the job so much that they accept it. They can spend ages trying to meet your requirements, do a lot of research, try a lot of things, but never be quite there. In this case, you should be willing to understand why this is happening and make some compromises.

If your developers tell you it's impossible, but you have very good reasons to think it is, seek for a second opinion. Of course, they can be mistaken sometimes.

Reason 3: Changing Requirements

At least as much a problem as starting the construction prior to completing the revolutionary design, was the fact that the government itself changed the requirements for the building after construction had started. The original design called for two theatres. The government changed its mind and required the building to be altered and that four theatres now be incorporated into the design. -Â About the Sydney Opera house, 10 years late

If you first decide to build "two theatres", but change your mind and decide on four, guess if it takes more time?

There might be an urgent need to to add just one small feature to the project, and then another one, and then to change the first one, and the third one is definitely conflicting with the second one, so please delete the second. No, leave it be. On a second thought, delete the first one. And one more thing to add...

If you're a developer

Prioritize requirements to understand which ones you should take care of first. Start with the most critical ones.

If your client wants to have a strict deadline, the requirements should be fixed in the beginning of the project. Make sure it's documented thoroughly. You should have everything listed on a piece of parchment, it should be signed with blood and solemnly sealed with two scarlet wax seals with two coats of arms - yours and your client's. Make sure critical requirements are fixed in the first place.

Every feature the customer wants to add or change, however small it is, should lead to an official change in the requirements and to some deadline changes.

Otherwise, it's going to lead to a disaster. The client will assume all changes are included, they're going to put pressure on you, you're going to work weekends, and still won't be on time. No one will be happy.

Non-critical requirements can be taken care of later, perhaps even as an additional smaller project after the main deadline - see Reason 1 as well.

If you're a client

Prioritize your requirements, make sure you made up your mind about most critical ones. See tips in Reason 1 about requirement prioritization.

If you're making a change in the requirements, you should understand it's more work for the developers. More work means more time.

Unless you want to remove a feature. Removing features can be fine sometimes, if no one has started to work on it yet - otherwise they would be very upset.

A deadline is also a requirement, and most likely a critical one. So moving it closer means changing the requirements.

Of course, cancelling the deadline would mitigate the problem at all, but without deadlines your business probably would suffer.

Reason 4: Changes In The Team

Following the failure to open it in 2012, many of those involved, from engineers to architects, were summarily dismissed, leaving the project in yet more disarray, because much of the knowledge that had been accumulated disappeared. - About the Berlin airport, almost twice the time and still not done

Developers come and go. Next month half of the team is new people. The team leader has left. The tech leader has just started, coming from a different company with a different background. Both junior developers, who started two months ago, are doing their best to help onboarding the new QA. No one has read the documentation yet.

If the team is changing too often, they don't have the time to get to know each other's strengths and weaknesses, and don't know how to work together efficiently.

Moreover, they are just starting in a new project. It takes some time to learn things specific to the project, and to get acquainted with the clients and their requirements. The domain area knowledge is extremely valuable, and more so if the project if complex. The more complex the project is, the more time the process of acquiring knowledge takes.

On top of that, if the project is already half-way through, or if you are extending an existing project, it will take some time to get acquainted with the codebase.

In some cases, it can take as much as half a year for a developer to become familiar enough and start bringing value.

The more time the team works together and knows each other, the better their estimates are.

If you're a developer

Persuade your manager that adding a lot of new people to the project won't help to finish it faster.

If you have new people in the team, help them as much as you can. By investing more time in them now, you'll ease the work for them and for you in the nearest future.

Planning poker can help with estimates. If you're very familiar with the area, you might think the task takes an hour; whereas for a newbie it can take a week. The new person can be embarrassed to say so directly; but the planning poker forces them to speak up.

Discussing possible implementation with the team also helps new team members to get familiar with all aspects, which they probably didn't and wouldn't think of.

Pair programming and code reviews help in both maintaining the code quality and spreading the knowledge.

If you're the new developer, talk to the "old" ones and learn from them. Learn about the team and the project. Read the documentation. Talk with the customers. Soak up the information. You probably need more time to complete a task than your "older" colleagues. It's perfectly fine, there's no shame in it. But sometimes your colleagues might forget about it, and you need to remind them - they might have already forgotten you haven't spent your entire life here.

If you're a client

Try to figure out if the team has been working together for a long time, or they just got together yesterday.

If it's the second case, make a mental note: it's going to take longer. Let them become familiar with the project in their own pace, don't expect a new developer to speed up instantly, and remember that 9 women can't make a baby in a month.

Reason 5: Too Junior

Everyone in the team is on their first or second year. The senior has just graduated out of college. The manager has been working for three years and just got promoted.

They all are trying very-very hard, but they just don't have enough experience yet to start (and finish) a big and complex project.

If you're a developer

Don't overestimate yourself. (Don't underestimate too! But it's a separate topic.)

If you're just starting, and everyone else around are too, raise the question before committing to a project. Are you actually capable of doing it? Will you be able to finish it? Do you have enough skills? Can you actually implement what's being asked? Do you know what needs to be done prior to development, how to collect requirements, what to pay attention to?

It's not so bad to decline a project, as it is to start it and not being able to finish.

If you're a client

Pay attention to your team's experience. They may look overly confident, while still being too junior.

Ask for recommendations. Ask for completed projects. Check with their previous clients.

Look at them working. Do they test? Do they document? Do they over-engineer? Do they follow a process?

It depends on your case, of course. Simpler projects can very well be trusted to a more junior team. Projects that are more critical and important you would probably want to give to a more experienced team.

Reason 6:Â It's not only code

Forgot the QA department? Well, they didn't forget about you. Testing and bugfixing takes time.

Did you count the time to write unit tests? Integration tests? Regression tests? And don't forget you need to set up a build server first.

...Time to set up git, slack, and wiki. Time to get the servers ready by admins. Time to deploy. By the way, did you include time to write a proper deployment script?

...Brainstorms, plannings, stand-ups and retrospectives.

...Presentations. Preparing to presentations.

Also vacations, sick leaves, going to conferences, staying at home to wait for a delivery, remote colleagues not having any internet, power surges, and all sorts of small domestic disasters that happen once in a while.

If you're a developer

Keep in mind it's not only coding. You're not "done" the moment you type "git push"Â in your console. There're a lot of things that need to happen before that piece of code is live.

Learn not to say "it's done" when it's actually not. It is misleading and creates false expectations.

Some unforeseen things also happen. Have a buffer for them.

If you're a client

Gently remind them they need to test and deploy. Especially if you're talking to developers directly. If a developer tells you "I'm going to finish today", it doesn't mean you'll see the feature today. Or tomorrow.

It is a good idea to ask them instead when the feature is going to be shipped, rolled out, live, or visible to the users. The answer might be dramatically different.

Reason 7: Optimism

Developers tend to think optimistically about time estimates. That's so easy! I'll do it quickly. It's going to be done tomorrow. No, in an hour. Just hold my coffee and watch me pushing already. Ah no, I need to install an IDE and configure my git first. But then - just one hour!

If you're a developer

Think about your previous estimates. How did the tasks actually end up?

To do that, keep a journal. Before starting each task, write down what do you think it will take; when you finish - write down how much time it actually took. Then you can calibrate. Do you see a pattern?

A team may have a shared journal (jira, wiki, trello, post-it notes). Team calibration can happen, for example, during a sprint retrospective.

Tip Of The Day: if you don't have a journal yet, multiply your initial estimate by Pi.

If you're a client

Multiply all time estimates by Pi.

Seriously. It might sound like some sort of a cargo cult, but this empirical number is magically correct in many cases.

Think Pi is too much? Well, if they're done faster, you'll be pleasantly surprised. If not - you totally expected that. Be a pessimist! Negate the overly-optimistic developers!

Ask for a range of estimations. Pick the longer one for your usage. Again - be a pessimist.

Reason 8: Constant Pressure

Come on, it's day 2, and we still don't have anything done. Why are you so slow? No time to think the architecture design through, start coding! No time for tests, go, go, go!

People react to pressure differently. Some of them pull together and become more productive. Some - start to hesitate and question themselves.

Many people make mistakes under stress. It is more expensive to fix a mistake afterwards.

Developers often are pressured to make trade-offs. They have to pick between long-term quality and short-term speed, and can be pressured decide it is better to implement a feature dirty and quick, than slower and better.

After some time, when there are many features implemented like this, the code becomes polluted. It becomes harder to implement new features, and much easier to make a mistake and create a bug. This state of code is called a "technical debt". Debts, as is widely known, are meant to be paid, the sooner - the better. The longer you postpone the payment (cleaning up the code), the more expensive it gets (harder to add new features and fix bugs in older ones).

If you're a developer

The main questions is: what's the reason of the pressure?

You client may want to "feel the power". In this case try to explain them it's not helping and actually hurts the project.

If the deadline is approaching, or you've been slow lately, or something else is happening - when the client has the "moral right" to apply pressure - you need to find a way to be more productive without hurting the quality.

Be reasonable, don't let yourself panic. If you can, delegate dealing with such a client to your manager and let them negotiate.

If you're a client

Trust the developers. Don't pressure them if it's not absolutely needed. People make mistakes under pressure. They don't make the best decisions. And you need your developers to think clearly, because it's your project.

Reason 9: Indecisiveness

These two libraries are very nice, which one to pick? The first one has some good features, and the second one - some other interesting features. Neither seems perfect. Maybe we should write our own?

What Javascript framework to choose? Do we use a NoSQL or an SQL database? MySQL or PostgreSQL? RabbitMQ or Kafka? Ruby or Python?

Sometimes developers can't decide how exactly to implement a feature, and are stuck in an endless loop of discussions, prototyping, and comparison.

If you're a developer

Take a deep breath and think about the customer. Is it a really important piece of project? Will something bad happen if you didn't pick the absolute best technology? Maybe it's not so important which build-server you choose.

Limit the time you spend on comparison. For example, decide you shouldn't spend more than 2 days on investigating and learning about different variants, and then by the end of the time you have to make a decision.

Don't let yourself procrastinate because you can't decide. It is wasting your time and is highly non-productive.

Often the best approach is simply picking the technology someone in the team already knows. It will save you a lot of time to learn a new one, and you already have someone who can teach the rest of the team.

A decision to write your own solution from scratch should be considered as a last resort.

If you're a client

Don't let your developers spend too much time on non-critical parts. Help them understand it is a non-critical part.

If they decide to implement their own solution, library or framework, when there already are existing ones, they should have very good reasons for doing that.

Reason 10: Incompetence

That's the saddest reason of all. People implementing features wrong, breaking old working features with new code, people who don't unit-test, people with no regression tests. People who don't test at all.

Some people don't think about architecture before starting to code. Many think code is more important than product, and they end up creating a monster unusable for human beings. Many write code that is too fragile, can't be extended, is unreadable, and contains too many bugs.

Sometimes developers don't feel motivated and start to slack off. They walk around drinking coffee, have water cooler conversations, long lunches, prepare conference talks, watch youtube, play videogames, and discuss in details whether spaces or tabs will save the world. They do everything but work. In one word, they procrastinate.

Sometimes it happens that the team procrastinates until the deadline is very close, and then rushes into work. Of course, there's not enough time now.

Last, but not the least, the developers might cheat the customer. They might take the customer's money and do nothing, or do a sub par job quite consciously, without making an effort to produce quality results. They can inflate costs and ask for more time without any other reason but greed. The team might subcontract the project to someone else whose work will be cheaper, without caring about the customer, the project, and even their own reputation.

All these are symptoms of unprofessional behavior.

Again, this doesn't happen only with code.

Christmas festivities at the Palau de las Artes opera house in Valencia, Spain, were cancelled due to chunks of the mosaic façade falling off during high winds. [The architect is being] sued over the cost of repairs to the buildings, which are rapidly deteriorating only eight years after inauguration, citing "construction defects." - About the Palau de las Artes, going 4 times over budget and still crumbling just 5 years after opening

Heads rolled, and over 4,000 construction defects had to be corrected, with the responsible parties not communicating with each other for stretches of time. - About the Elbphilharmonie in Hamburg, 7 years late, 10 times over budget

Cormorants are damaging St. Petersburg Arena’s retractable roof, which is the second most expensive stadium in the world - after Wembley Stadium in London. It turns out the cover - which is meant to withstand pressures of up to 400 kg per square meter - is failing to stand up to the cormorant’s beak. In 2013, investigators found that a subcontractor inflated construction costs by $16.5m (£10.5m). One businessman was sentenced to four years for embezzling 146m roubles of state funds. - About Zenit Arena stadium in Saint-Petersburg, 7 times over budget, 6 years late

If you're a developer

Sorry. But you need to learn a lot before you can actually commit to working on important projects.

Consider trainings and learning courses. Consider getting more experience on easier projects. Get a mentor, read articles, buy books. Be more responsible for your work. Remember that your reputation depends on it.

If you notice that your fellow teammates' work is not of the best quality, or if they are slacking off lately - give them feedback, talk with your manager, take measures.

If you're a manager of such a team of developers, you should be able to notice such problems in the very beginning and address them accordingly. Maybe you need to be more strict, or you need changes in the process, or let developers know their efforts are appreciated, and reward good work.

If it's really bad, or your efforts don't help the situation, ask yourself how come you hired people not competent enough. How come your hiring process let them through?

But also - what can you do to help them improve? Or they're done with, and the only way is to hire someone else?

If you're a client

Sorry. You probably need to change the team.

Or you should be willing to watch very closely the current one. Basically, you're going to become their manager. It's very time consuming and irritating, you probably don't have all the necessary skills, and they probably are not going to be happy about it. You too. But if you're halfway through the project, you don't always have an option to cancel the contract.

I'm afraid I've been in this situation myself - see my article about hiring a team which proved to be incompetent. I can't say IÂ was enjoying this experience.

Conclusion

I have listed ten reasons why software development goes overtime (and over budget). Quite likely, a single project will involve several of there. For example, I have seen "Changing Requirements", "Changes In The Team" and "Optimism" go together more often than not.

I hope that customers, upon reading this, understand that software development is a sophisticated process, still very much in development itself. Many software projects are very complex, and understanding and designing them is a very hard task. Estimating time is more so. I hope that customers can help their development teams with that.

I hope that developers, having read this, will understand a bit better what their customers expect from them. Customers are people who we are working for, and it is very important to have good relationship with them, which includes honest discussions of problems. I hope developers will learn from this article how to see problems early and how to fix at least most popular ones.