Introduction

If you are writing software, it is almost inevitable that you are familiar with Agile software development. In fact, you are probably familiar with it even if you have nothing to do whatsoever with software engineering. “Agile” is hard to avoid these days.

Back in 2001 a “Manifesto for Agile Software Development” was written and has popularised the term but “agile practices” that constitute it have existed prior to that. So it is not a new kid on the block.

Decades later “Agile” is everywhere and has many features of a religion: a set of beliefs one is not allowed to question, rituals (yes, much of Agile is about “practices”), a certainty that it is the only true way, sacred meetings (like a stand-up meeting in Scrum), special jobs which have no meaning in the outside world (like Scrum master).

Just like in so many major religions today, Agile has various denominations (although they are called “frameworks”): Extreme Programming (almost gone by now), Scrum, Kanban, Scrumban, Feature-driven development, and so on.

Here is my heretical and rather critical view on Agile. I will aim to be objective in my assessments but naturally, there will be a bit of bias.

Why do I write this?

Firstly, being a somewhat altruistic person, I believe a more balanced view on various matters helps people. Agile is a matter which has an impact on many things: the work environment, the efficiency of teams, and the success of the business.

Secondly, being a somewhat egoistic person, I like speaking my mind: it is a pleasant thing to do.

So here we go…

What is it?

According to Agile Manifesto, the following is true regarding values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Also, twelve principles are defined:

1. Customer satisfaction by early and continuous delivery of valuable software.
2. Welcome changing requirements, even in late development.
3. Deliver working software frequently (weeks rather than months).
4. Close, daily cooperation between business people and developers.
5. Projects are built around motivated individuals, who should be trusted.
6. Face-to-face conversation is the best form of communication (co-location).
7. Working software is the primary measure of progress.
8. Sustainable development, able to maintain a constant pace.
9. Continuous attention to technical excellence and good design.
10. Simplicity — the art of maximizing the amount of work not done — is essential.
11. Best architectures, requirements, and designs emerge from self-organizing teams.
12. Regularly, the team reflects on how to become more effective and adjusts accordingly.

To me, this looks reasonable, at least on a superficial level. Of course, the devil lies in details, and we shall come to that.

Now let’s consider the history of Agile as knowing history helps to understand where Agile and its creeds come from.

History

History is the reason why the Agile evangelists talk so much about the merits of Agile compared to Waterfall even though nobody (?) has been using Waterfall for a couple of decades. Actually, if you do use Waterfall, please let me know. I would be delighted to know that I am mistaken.

In the olden days, when hard disk drives were the size of an average washing machine, people used the dreaded Waterfall: the activities were split into phases like collecting business requirements, analysis, design, actual coding, testing, and deployment. The phases went one by one, each phase requiring the previous one to be done, or in modern English “done-done”.

Yes, it may sound impossible, but back in those days people did not have smartphones, were not hooked on flashy video clips, and consequently did not suffer so much from attention deficit. The result of it was that they were actually able to focus on their work and get it done before moving to the next phase. It was not a common situation that a team reached the stage of testing only to realise that they have forgotten to collect business requirements. Those days are over, and we cannot step into the same river again.

I should admit, people did feel a bit uneasy about Waterfall. Its strict and inevitable sequence of steps from one phase to another must have felt heavy. Eventually, “lightweight” methods have been adopted where Waterfall was called “heavyweight” (and rightfully so). These new methods were RAD, XP, UP, and so on.

Eventually, a meeting of several software engineers in 2001 and their discussion on lightweight software development methods resulted in the “Manifesto for Agile Software Development”.

With nice branding — you can’t argue against a point that the very word “agile” sounds attractive — and many bright minds focused on refining the methods a revolution of sorts has happened and Agile is everywhere nowadays, even in areas that have nothing to do with software engineering.

Nowadays Agile is much more than a method of getting the job done. It is a whole ecosystem of various consultants, publishers, experts, certifications, and so on. It mandates its rules, introduces its own specific job roles, and moulds its customers to suit its needs.

Why do people like it, what does it promise, and are those promises false?

Promises and Reality

The very name “Agile Software Development” promises agility.

People are often frustrated with the process of software development, they feel it is slow and rigid and something needs to be done. This is particularly true for those who are not taking part in this process. Unlike a traditional activity like making a statue, it is hard to quantify how much is done so far.

Of course, one can use various metrics, business process diagrams, charts and so on. In fact, part of the job of a project manager is to be able to demonstrate to the higher management that the project is progressing. However, such tools and efforts are often not enough to reassure the stakeholders.

Does it work?

Best case scenario

In some contexts, it works perfectly, almost like magic.

If your software is pretty generic (yet another online shop, for example), your stakeholders do not know (or cannot know) upfront all the requirements, the business is in the early stages and is looking for its niche, software engineers are perhaps inexperienced in the problem domain, then yes, Agile is a rational choice. Many Silicon Valley startups match this description. There is no way to know from day one which features would their software need. They need to start from a minimum viable product and hope that the rest can be identified and added incrementally. In a way, it is a kind of recon by fire, because in the early days of a start-up, there are too many unknowns and the only way to get knowledge is through experimentation.

Worst case scenario

In other contexts, it does not work at all.

If your software is unusual and hard (something like software for thermal analysis, microchip design, etc.), your engineers really need to understand in detail what they are going to create, the management is also aware of what the goal is, the budget is not limitless, and mistakes are costly then Agile won’t cut it. Seriously, it won’t.

Here is an example. Let’s say you are creating a software for microchip design.

1. Your minimum viable product is 90% of the fully-functional product.
2. Your problem is not deciding whether a button has to be on the left or right, it is figuring out which algorithms would work, clarifying the problem domain in detail from day one, carefully designing relevant test examples, and hard, exhausting, eyes-bleed-due-to-strain effort to verify in the most complete and formal way possible that the software is working exactly to the specs. Because it is not an online shop. Your mistake does not translate in 4 starts out of five in a review. It translates into a mistake during manufacture which costs half a year of work and ten million dollars in expenses for the end-user. The end-user will go bankrupt, and you will go down as well too as your reputation will be ruined forever.
3. You cannot start by writing code, and then change the requirements at will. A change of requirements may mean that the code has to be thrown out in its entirety.

Everything in between

There is also a middle ground.

In that middle ground, people tend to use common sense: a healthy approach to analysis, design and planning; frequent and open communication with the stakeholders and customers; steering the development sensibly; reacting to reasonable changes in requirements and being clear that some changes are unreasonable; spending enough time on important research without anyone shouting in the ear and changing the plans completely again; and so on.

Believe it or not, people do not need to call it Agile or adhere to any specific “practices” for the process to be agile. All they need is common sense.

But common sense belongs to endangered species in the world where emotional intelligence is the king. So the realistic scenario is this one:

1. Someone in the top management decides that everything is slow and inflexible.
2. The suspicions are confirmed as people do feel that there is a lack of flexibility (the root causes are ignored of course).
3. A consultant comes along and says that many companies have already embraced the Agile way and benefited immensely (as always, without any real proof).
4. To the management, switching to “Agile Software Development” seems synonymous with making thing to be actually agile, so they pay the consultants to help with the transition.
5. Months of valuable time are wasted on “educating” the professionals on how to do things the hard way. Some of them leave for places where they won’t be distracted by this nonsense.
6. Agile is not adopted, it is enforced (as it comes top-to-bottom).
7. More rules and regulations are introduced, leading to more waste of time.
8. Some people have their roles changed, so they are no longer programmers. Now they are Scrum masters, product owners, etc. and their time is spent on supporting the ritual rather than the creation of actual value.
9. Productivity and morale drop. The best people leave for greener pastures.
10. Failures are never attributed to the introduction of Agile, the conclusion is that “people need to be educated more”. As a result, more consultancy work is requested, more money is spent on it, and more time of engineers is wasted.

What’s the deal with Waterfall?

Now let me tell you something about Waterfall that others do not.

There is a very good reason why Waterfall was inevitable in the days of punched cards and teletype.

In those days programming was hard. You could not just hire a random guy from the street and expect that he would write you the software you need or indeed write anything. The programmer had to understand well how computers work, be well-versed in Assembly, understand mathematics on a very decent level, and so on. Otherwise, you have no chance of that person writing usable software to run on a computer with literally a few kilobytes of RAM.

You could not change code easily, and there were strong technological limitations. Therefore, it was crucial to understand and document what should that program do before you start. It was also terribly important to carefully analyse the requirements and capabilities, and then code and test based on that work. Yes, it resembles a house of cards and a mistake on any phase means the following phases automatically fail. But there really is no choice, given the constraints one is dealing with.

Was Waterfall an epic success? Yes, absolutely. If you do not believe me, read a bit about Margaret Hamilton and her team at MIT who worked on the Apollo project. That pile of documentation and presumably code you can see on the photo is necessary. Without it, one cannot have a chance of sending people to the Moon, which among other things means writing rather non-trivial software for a computer like this.

Now if one tries to write a text editor for PDP-11 and do it according to “Agile practices”, I think the result would be a complete mess. All the talk about welcoming changing requirements, valuing individuals over tools, and delivering working software frequently will break at the soulless machine PDP-11 is.

The world has changed, however. Code is being written in IDEs. My favourite one — with plugins — is around 4GB large (wait, what?!). Assembly is forgotten and languages like JavaScript and Python rule. Everyone can code but few understand how to do it and what is that they need to create in the end.

So yes, Waterfall is dead. It has been dead for a very long time. However, it will never be forgotten, because an average Agile consultant will be sure to tell you how much better Agile is when compared to Waterfall. He would also carefully forget to mention that both Waterfall and Agile are nothing but tools for their specific niches, and the niche Waterfall was created for is gone decades ago. Indeed, a hammer is so much better than an apple for hitting nails, but a few hours ago you were hungry.

The merits of Agile and failures to achieve them

Values

Individuals and interactions over processes and tools

Meaning: “A group of effectively co-operating competent individuals is more valuable than following specific processes or using specific tools.”

It is hard to argue with it. Although to be fair, competent individuals value quality tools because they know the tools improve their efficiency. Also, ditching a good process that is optimal for the people and the task at hand is not a productive thing to do.

In real life, the competent part is ignored, and the line is interpreted as “ignore tools and processes (unless they are the enforced kind of Agile), assemble a team of people, and demand that they have various regular meetings”.

As a result, people waste their time just because the Agile process demands it. Some of the best engineers cannot stand these endless meetings, so they leave. Remember, we have started from “individuals and interactions over processes and tools”. Isn’t it ironic?

Working software over comprehensive documentation

Meaning: “Ultimately, the software needs to be written and communication is a secondary concern.”

This also makes sense in many situations. In fact, some people are guilty of producing an excessive amount of largely unnecessary documentation.

But of course, in real life, this often leads to no documentation whatsoever. It is not always bad, by the way. Sometimes, documentation is completely unnecessary. However, where software is a little bit non-trivial, high-quality documentation is important and must not be ignored. I shall leave the matter of what should be documented and how it should be done for another post.

Customer collaboration over contract negotiation

Meaning: “A contract has to be in place, but it is important to communicate with customers regularly to understand what they need.”

This works really well where an outsourcing company has signed a contract that allows them to send bills to the customer according to some metrics (like man-hours) and there is an efficient and well-established process where the work is split into small bits, and then each bit is estimated, agreed with the customer, and delivered. I worked in an environment like that, and it was good for everyone. Of course, nobody bothered to call it Agile.

The approach does not work where the job to be delivered to the client cannot be split into small iterations or where seemingly small changes to requirements may mean massive changes to the software. This is the case with R&D-intensive projects.

Which case is yours?

Responding to change over following a plan

Meaning: “The plan is important, but it is more important to be flexible to changes in technology, requirements, priorities, etc.”

Once again, it is hard to argue with that.

However, what happens in reality, is that people tend to throw out the plan. They forget the old wisdom that one should resist the scope creep and then drown in never-ending changes. Changes in priorities are all right but someone would have to pay the cost of the change in direction. And finally, people forget that changes in technologies must not dictate what you are creating and how you do it, certainly not on the scale of weeks and months.

Principles

1. Customer satisfaction by early and continuous delivery of valuable software.

Absolutely. Although it is not always possible to have a minimum viable product and iterate from there.

2. Welcome changing requirements, even in late development.

If that change is a change in font size — of course, by all means. If it is adding an ability to edit videos to an email client — no, thanks.

3. Deliver working software frequently (weeks rather than months).

Again, this is desirable, but not always possible.

4. Close, daily cooperation between business people and developers.

… often means that rather than focusing on a problem and getting it done, engineers are being distracted for hours every day.

Also, “close cooperation” is not a substitute for a deep understanding of the business and its problem domain. Does Agile say anything about it? I have not seen it.

5. Projects are built around motivated individuals, who should be trusted.

This misses an important word: “competent”. I would not trust an individual simply because he or she is motivated. A motivated and incompetent person suffering from Dunning–Kruger effect is much worse than a competent and not particularly motivated one. For a start, the damage would be smaller. And then you can always try to motivate someone who is not motivated. That takes much less time than waiting for an individual to gain years of experience and become hopefully become competent.

6. Face-to-face conversation is the best form of communication (co-location).

Yes, but group meetings are not the best form, not even a bit. They are a waste of time.

The best kind of communication by far is the face-to-face meetings between two-three individuals which happen naturally and informally.

There is also an extremely useful kind of communication which is not of a face-to-face kind but that is another story.

7. Working software is the primary measure of progress.

Perhaps yes, but the statement is too philosophical and undisputedly binary, which makes it be rather useless.

8. Sustainable development, able to maintain a constant pace.

There is no scientific definition of pace and everything about it is smoke and mirrors. Software engineering is not the same as laying brinks. You cannot measure it in the same way.

Yes, sometimes you can decompose a problem into smaller parts and estimate it in hours. I did. Sometimes it works. But it works only if the work is sufficiently small and easy: an hour to a couple of days of solving a clearly understood problem. For larger or less understood problems this does not work but that is a matter for another post.

9. Continuous attention to technical excellence and good design.

Absolutely. Except it is hard to note that when the team is required to pay close attention to meetings, various metrics, and rituals, there is no capacity to pay real attention to technical excellence and design.

10. Simplicity — the art of maximizing the amount of work not done — is essential.

I suppose it means “avoid doing the work which is actually unnecessary“. Yes, half of success is in not doing work that is commercially irrelevant.

11. Best architectures, requirements, and designs emerge from self-organizing teams.

They do indeed. Although there are “Agile” teams where requirements, architectures, and designs are enforced as a policy. You are lucky if you do not have to take part in that.

12. Regularly, the team reflects on how to become more effective and adjusts accordingly.

It is easy to be more effective. Usually, it is a matter of removing distractions and obstacles rather than changing or adding something. I do not think that “regularly” is the right word here. Isn’t “occasionally” more appropriate?

Conclusion

Agile Software Development is a way of getting things done, an ecosystem, a road to success for some, and a road to hell for others. It is here to stay, and it is not because it is “better” in some objective sense. It is here because it is well suited for many problems worked on today.

It is not, however, a universal tool. In fact, Agile can harm your teams and destroy their productivity offering you nothing in return. Whether Agile is a good choice depends on the nature of the business and the projects you are working on. There is no one-size-fits-all approach.

If you do adopt Agile, it is important to avoid enforcing it from the top. It should emerge from the bottom as a shared understanding among the members of the team(s) that they should be doing things this way. Self-organisation ensures that the adoption is done in as much appropriate way as possible.

Lastly, remember that common sense is more important than all twelve principles of Agile. They are a guidance, not a law. Your specific situation — not a consultant — is what should dictate how you work.