The Waterfall Days
If you have been in the software development industry for a few years, then you have probably worked on a large project. Large is a relative term, but lets just assume that it means the scope of the project contains more information than could fit in one person’s head. Some project’s scope couldn’t fit in a book or even an encyclopedia.
I know we have all heard the stories of waterfall and where every single part of a project was documented on paper then handed to programmers to create. We all think this is a bad idea. I’m not sure that younger people really know why. The reason is, because software evolves as its being created. The solutions designed on paper won’t always work in the real world. There are many things we learn throughout the lifetime of a software project. Its impossible to anticipate all the ways in which users will use your software, and with waterfall the requirements are not allowed to change.
Agile to the Rescue
Today’s software projects don’t have these issues, they are typically built by agile teams who are flexible and can adapt to change. There would be a lot less software developers, and a lot less buggy software if that were really true. The truth is the way agile is typically implemented there is a concept of a backlog.
A backlog is a list of items that can be prioritized for the next iteration of software development. That doesn’t sound too bad right? What are the characteristics of these items? Backlog items contain information pertaining to a specific behavior desired by the project stake holder. They contain a clear description of one behavior of a software system. So far so good right? right. There are a few issues with that simple definition of a back log.
A backlog typically has few or no constraints placed on the items it contains. For example the limit of back log items is usually non-existent or infinite. An infinitely long list of backlog items can have some of the same side effects as the waterfall projects. They can be defined once and never changed. Having an infinite number of backlog items makes it difficult to know what items are in the backlog. It also makes it hard to have a priority order. By limiting the number of items in your backlog, you can make prioritization decisions much easier. You can also more easily be aware of all the items in the backlog, and if the specified behaviors are still correct. Having a manageable size of a backlog is only one step to getting control of your backlog.
The relative size of items in a backlog typically varies. In my experience there is typically not a limit to the size of backlog items. In a software project it can be difficult to measure relative size of system behaviors. This fact leads to a lot of waste. The problem is the term relative. Relative to what? Software is intangible you can’t often compare it to physical objects and be successful. Although we often try to do that. When I was a child I would play 20 questions with my father and his first question was, "Is it bigger than a bread box?". Software developers often talk about abstract behavior size in t-shirt sizes and things of that nature. The reason these types of things go wrong is for a simple reason that I mentioned before. We learn as we create software. We don’t know what we don’t know.
Let’s do an experiment. I’m going to give you an example backlog item, and ask you to think about the solution to it and how relatively big you think it is. In this instance imagine the solution imagine the lines of code, the servers required, how many files you would have, and how long you might think it would take in hours, days, months, years etc. Ok, here is the requirement. "As a user I need to be able to log in to the application. " Ready set think for at least a minute how you would solve this problem. Think of each line of code, how many functions and files this would require, in your favorite language and using whatever technologies and infrastructure you want.
Ok… now that you have thought about it do you have a solution in your head? What language did you use? What database? How many tables did you create? How many lines of code, how many files, and servers do you guess you might need? How much time do you think that might take to write all that code, set up the servers and whatever else you need to do? 2 weeks, 2 months, 6 months? Do you need to get with the dbas, and operations to make this happen?
Now me show you the implementation.
You might be asking the question. How does that meet the requirements? I’d argue that the requirements were lacking any context or notion of what logging in is. It didn’t specify I needed a username, or password, or what the expected behavior was once I am logged in. I did what all developers do. If I don’t know an answer to something, I just make it up and wait for a bug to be filed. That is why you thought of lines of code, and infrastructure, and user names, and passwords, user profiles, frameworks, and protocols.
I’m guessing most of you thought this was a relatively small requirement even with all overhead of infrastructure and code required to complete the task, but you still didn’t realize the simplest solution that might meet the specified requirements. Sadly I would have considered this type of backlog item one of the better ones I have seen even without any real detail.
Often backlog items are medium or large and without detail. One example might be something like, "As a customer I need to purchase and item and have it delivered to my house." That has the same types of issues that the login requirement has, as in it is not very specific, but it also has some much worse requirements. At a glance if you were asked the relative size of this, you would likely say it is medium or large. I think we could agree without further information the relative size of this is different than that of the login item.
The issue with this is its very difficult to size at all because there is way too much missing information. First of all you have no idea what the items are that need to be purchased, how to facilitate taking payments, nor how the products will get delivered. Not to mention all the stuff that happens in between like fulfillment and inventory management. I’d go as far to say any estimation of time will be off exponentially. There is just too much assumed in this backlog item.
Now that we know the bad parts of backlogs and backlog items, lets try to think of how we can solve these issues.
The first one is simple. Limit the size of your backlog enough so that you have at good understanding of the work that is too be completed in it and so that you have enough information to be able to update them and prioritize them as needed. I’d suggest that you should have no more items in your backlog than one or two iterations worth of work. That will be determined by the relative size of the backlog items. If you want to start with a random limit choose a limit between 3 and 8 items. This might get you started on the right path. You can adjust as you fix the next problem.
It’s impossible to correctly limit the size of your backlog without limiting the size of your backlog items, so lets discuss that now. The biggest problem with backlog items is that they can vary in relative size. Imagine if all your backlog items were the same relative size, and you know from experience the each of those items takes 1-3 days of work to complete. Occasionally you will have some outliers that take 5 minutes or two weeks, but for the most part each item can be completed to the specification in a couple of days. If that were the case you could know exactly how many items could be completed in an iteration or how frequently you can get feedback on the changes that you are making. The time from the start of development of a backlog item to its completion would be come predictable. Everyone likes predictability. Now imagine if the most important bug fixes or feature requests could be put to the top of the backlog. This should allow you to accurately predict when you could deliver the latest bug fix or feature to whomever is requesting it.
I have kind of skipped over the most important part of backlog item sizing. How do you make the items the same size? How do you make a medium or large feature small. The answer is you don’t. You break them up into smaller backlog items. Small enough backlog items should share a common set of characteristics.
- Each small backlog item should unambiguously specify the details of a specific behavior of a system.
- The behavior should be with in a specific context.
- The specified behavior should detail the expected outcome of the behavior.
- Unless the backlog item is a technology specific item, any person in the business should be able to understand how the system will act, or react in the given situation given specific types of input.
For most software developers this probably sounds a lot like how you think about and write a unit test.
There is no reason that our system should not be specified by developers and stakeholders together at that level of detail. This will leave no room for ambiguity, make software development much cheaper, and allow the stake holders to get exactly what they want and nothing else.
This is likely an impossible dream, because most stakeholders don’t want to be involved in the creation of the software they poor all of their money into that their business relies on to function.
Developer’s tend to build ivory thrones around their overly polished waste, and stakeholders seem content not to peek inside of the white walls, oblivious to what actually hides within them.