Oversimplification and Overcomplication

Simplicity is an important architectural feature. Simple systems are easier to understand. Simple component has fewer moving parts and therefore fewer reasons to break. Simple systems are easier to change and maintain. Yet our world is a complex thing and we, naive human beings, are often making it even more worse. We are creating legislation that no common citizen can understand unless he spends at least 5 years of his life attending a law school. We are creating all kinds of rules and regulations starting ranging from non-formal social norms all the way to multi-national treaties.

Software systems are here to help us deal with the complexity. But that necessarily makes them complex. It is not the programming language or operating system that makes software complex. Novice programmers can handle technological problems quite quickly (ever seen "Teach yourself Java in 21 days"?). The difficult problem is not "how to build it". The real problem is "what to build". The environment, user expectations, past and future - these are the primary sources of complexity.

What happens if you try to solve complex problem with a simple solution? It may work - if you are genius and you figure out something that generations before you missed. But honestly most of us are not geniuses. More likely outcome is that a simple solution for a complex problem will not work. It will break in spectacular and unexpected ways. Why is that?

You start to analyse the problem, find the important pieces, the "essence". Once you have that you design it, implement it, test it, reimplement it, test it, deploy it ... and you discover that the "essence" was not enough to solve the problem. I mean the real problem of human users, not the usual problem of "how to pass acceptance testing and get the money". Therefore you rethink the problem, discover that the essence is much more complex than you expected. Most project now just start to complicate the think that they already implemented. The system get more and more difficult to develop and no amount of refactoring seems to be enough. The system more and more looks like some devious creation of Dr.Frankenstein. An abomination. Such project will run high over the budget and miss all the deadlines. And project goals change as well. The original goal of solving the real problem of system users is quickly transformed to "just finish it, get the money and get out of here".

Initial definition of the problem and scope of the project is essential. The specification needs to have a correct breadth. Specification of a problem that is too complex will make the project extremely expensive just to find out that users are using only 20% of system functionality. Specification of a problem that is too simple will lead to a system that fails to satisfy the users and is unusable in practice. The specification also needs to have a correct depth. Waterfallist-like 1000 pages of analytical documentation is a pure nonsense. Nobody will read it and it creates analysis-paralysis situation. Agilist-like one-liner "the system must work" will not do either. It will send the project into an endless loop of refactoring, competing project goals and divergence.

Small problems, inaccuracies and omissions in initial project specification are easy to fix during the project. But if the breadth and depth of the specification is wrong, the project is doomed to failure from the very beginning. If the goals are wrong is it a success if such goals are reached?