I dedicate this post to all the QA departments I had to deal with early in my career. This post is all about how typical project management styles, especially the so called waterfall or even worse “half-assed waterfall” makes things hell for QA.
So first, we’ll talk about the problem, then we’ll talk about how Agile can make it better.
Prayer
The most popular project management style in the world is what I call the  Set the Date and Pray method, or Prayer. I have a few other silly names for this style: Death March, Are We Done Yet?, and The CEO Says “Ship It Already”. Here is an illustration of what happens with this sort of project:
It works as follows.
- Engineers write some code.
- They throw the code at QA.
- QA tests the code, which doesn’t work.
- Engineers fix the code and hand it back to QA.
- Repeat steps 1-4 until someone intervenes.
- At some point, the product ships.
Some people like to call this the waterfall model. It’s not. In a true waterfall, you do design, analysis and planning up front so you have some idea what you’re going to be actually building. A complete waterfall model has 10 phases and this only has 3. I call this the shit-runs-downhill model when I’m feeling snarky. If I’m feeling not quite so snarky, I call it the “throw stuff over the cubicle wall at the next guy” model.
I call it this because at every company where I’ve worked that used this model it was QA’s job to prevent engineering from shipping shitty products, while it was Engineering’s job to try to mark as many features as “done” as possible. End result, QA was the company’s Sphincter muscle, they prevented the shit from coming out. I remember pointing this out at my first professional software engineering job (I blame the burrito I had for lunch), and the QA department thought it was hilarious that they were the sphincter. “We’re the sphincter.” they went around saying.
I think this gatekeeper function came about because Software QA was originally modeled on QA departments in manufacturing. But Software Engineering is about inventing new widgets, not making 10,000 of the same widget and trying to catch the 1% that come out wrong. So the gatekeeper model that Software QA copied from Manufacturing isn’t really appropriate. Software QA has to be much more collaborative with Software Engineering.
You can actually get away with this management style on small projects, and with a good team as long as there isn’t too much pressure. If the team is just you, you’re probably using this model. You design the product in your head, you write the code, and you’re done. But getting away with a style is not the same thing as having good project management style or performing well. This style can’t scale.
What happens in a Prayer project that’s beyond scale is that as the magic date approaches, Engineering starts to get stressed out. Engineering and QA start to get antagonistic. Engineering wants to ship, but QA keeps finding bugs. Engineering and QA have different motivations:
- Engineering: They’re under pressure. If the product ships, they get praised, the pressure comes off, and they can move onto the next thing.
- QA: If the product ships, and they missed something, they get yelled at because they get blamed. They never get praise.
So Engineering is seeking positive feedback and trying to avoid the negative pressure from the schedule. Meanwhile QA is trying to avoid the negative feedback that almost inevitably happens once they ship, and they never get any positive feedback. As long as QA says the product isn’t ready the Engineers get stressed, but the QA guys seem like they’re doing their job, so aren’t under as much pressure. So QA is never really motivated to say “Yep, its done”, because shipping means that they get to sit around waiting for the other shoe to drop.
This can drag on for quite awhile. A VP of Engineering I used to work for would watch for what he called the “constant-time-to-completion” phase of QA testing. That phase would come when QA would insist on a certain amount of “quiet” where no new bugs were introduced, and then they would find a new bug just often enough that the product could never ship. At that point he would intervene and push the product into the next phase. Essentially, this took the ship/no ship decision out of QA’s hands.
Now all the stuff I’ve described so far is rather dysfunctional and extreme. It’s unfortunately true, but what I’m talking about generally takes years of dysfunction to get to that point. Most companies who are successful aren’t quite that bad, they’ve done various things to make things better.
Some organizations have integrated additional phases of the SDLC. Hopefully, the first one they integrate is planning, because my model above is missing what I consider step Zero of project management: Make a list of everything you have to do! Just doing that can immediately help with dysfunctions, because for one thing, you can tell if you will actually be able to make the magic date or not.
When planning is applied to QA, that means “Creating a Test Plan”. In theory, what happens is after the Requirements phase of the project, QA can go off and write a master list of everything they have to test. For every dialog box, window, whatever, QA will dutifully document what exactly they’re going to do with that dialog box: What bad inputs they’re going to try, etc. They can then show that plan to Engineering, Engineering can make suggestions, and when Engineering is ready for them to test, they can start executing the Test Plan.
That’s the theory. In practice, I have yet to see a well written Test Plan, and if I did see one, I seriously doubt it would work. Because Software Engineering is a matter of inventing-on-a-schedule, and its very hard to write a test plan for something that hasn’t been invented yet. Having a Test Plan for the entire project is something I call the illusion of control. Everyone feels very virtuous because they have a plan. They feel like they’re in control. But as the project evolves, the test plan gets less and less relevant. Eventually, it’s just a document on a shelf somewhere. You never see the testers going step-by-step through the test plan while they stare at a dialog box.
Ok, so this model doesn’t work. So why does the Agile model work?
The Agile model works for a number of reasons. No matter who you are, you need a 3rd party to double check your work. But that first check needs to be done by you. I have probably found 10 times more bugs in my code than any QA department I’ve ever worked with has been able to find. It’s not because I’m a particularly gifted tester. I’m not. In fact, I’m a lousy tester, for the same reason I’m a lousy proofreader: Things work how I expect them to work, and it never occurs to me to put “bad values” in a dialog box to see what happens! But I have a huge advantage over any tester: I know exactly how the system works, so I know how to exercise it. So having Engineering committed to doing at least some testing leads to a huge increase in quality. At the very least, you can rely on engineers to make sure that the stuff that’s supposed to work actually works.
Of course, engineers generally don’t like testing, so they want to automate it with unit tests and so on. Which leads to Test Driven Development. Test Driven Development works, because the alternative is Jenga Driven Development.
The final reason Agile works is because you’ve already gotten everyone into a room to plan the next iteration, which means that essentially, QA and Engineering are sitting down together and building a test plan incrementally. I’m not against Test Plans. What I’m against is the all-encompassing-document-developed-in-a-vacuum test plan. Testers are testers, not engineers. They have a different set of skills, and expecting them to approach testing like an engineer would is just not going to happen. In my experience, testers tend to be even more anal than the engineers. Meanwhile, the engineers are a bit looser, but they have the critical analysis skills the testers need to produce a good test plan.
So in summary, throwing stuff over the cubicle wall to the QA department is never going to work, no matter how well QA organizes. If you want to ship really great software on time, Agile isn’t necessary. But making Engineering responsible for quality, integrating unit tests, and having Engineering and QA work together to develop Test plans as-needed on an incremental basis is necessary, and Agile helps make that happen.
Note: This is the first in a series of posts by me on Agile Project Management and Project Management in general I am doing while job hunting. To find out about the next post in the series, follow @twinforces on twitter.