It makes far more business sense to develop these solutions within an agile framework where the design and architecture themselves evolve and the code is refactored constantly. This also provides external stakeholders visibility into what is being developed and provides them a greater sense of control.
I feel that iterative development, also provides the software architects and engineers validation for the design choices made. This is important to ensure that you do not end up looking like the self-absorbed architect who was afraid to ask for directions.
Another important aspect is that as new information becomes obvious to you and the end client, different stakeholders start zooming in on aspects of the system they had not thought before. As requirements start getting re prioritized, architectural requirements also evolve.
It should be clear by now, which side of the debate I increasingly see myself on. However, I was looking for ways that would provide me the comfort of upfront design while allowing a solution to develop in an incremental fashion. An approach to analyze the sub-systems in the design while not getting locked in to sub-system choices made initially. In short, I was looking for ways to reconcile Upfront design with Incremental design.
This lead me to a rather interesting article by Steve Jurvetson in the July 2006 issue of MIT technology review. Titled Technology Design or Evolution. To quote Steve...
"Designed systems offer predictability, efficiency, and control. Their subsystems are easily understood, which allows their reuse in different contexts. But designed systems also tend to break easily, and they have conquered only simple problems so far. ...
By contrast, evolved systems demonstrate that simple, iterative algorithms, distributed over time and space, can accumulate design and create complexity that is robust, resilient, and well adapted to its environment. In fact, biological evolution provides the only "existence proof" that an algorithm can produce complexity transcending that of its antecedents. Biological evolution is so inspiring that engineers have mimicked its operations in areas such as genetic programming, artificial life, and the iterative training of neural networks.
But evolved systems have their disadvantages. For one, they suffer from "subsystem inscrutability." That is, when we direct the evolution of a system, we may know how the evolutionary process works, but we will not necessarily understand how the resulting system works internally."
Steve goes on to quote Wolfram to imply that it is not easy to discover rules for evolution without going through all the steps of evolution. This is interesting. It means that there are no easy solutions.
One idea that I am toying with, is to focus constantly on maintainability and extensibility of the solution each iteration. The goal, to ensure that resulting designs can be actually partitioned into sub-systems rather than ending up as spaghetti code that cannot be drawn on the white board.
