Book Review: A Practical Guide to Testing in DevOps by Katrina Clokie

I just finished reading Katrina Clokie’s new book, A Practical Guide to Testing in DevOps and I wanted to do a quick write-up for anyone considering picking up the e-book.

Do it.

The book is a great primer on DevOps practices and concepts but unlike most books on the subject, it’s written from the tester’s perspective. Hallelujah! I’ve been waiting for a book just like this. The material is easy to digest while still being chock full of quotes, research, and real industry examples that are easily referenced thanks to the clickable links in the e-book format. After reading this book, I feel like I have a good handle on what to expect as a tester as my responsibilities change in the DevOps shift.

I especially liked that she gave several examples of exercises to try running with your team to help nail down the expectations of tester responsibilities such as test retrospectives and risk workshops.  I plan to run both of these with my team in the near future.

Whether your company has made the DevOps shift, plans to in the future, or you just want to take a look at what’s happening in the industry right now, I cannot recommend this book enough.

 

 

On teaching developers to test

WHAT

Recently, I’ve begun to teach my developers to help test our stories within the sprint. Crazy, right?

WHY

So why would I ever want to teach developers how to test? Developers develop, and tester test. That’s the way of the world. But because other duties started taking up a significant portion of my time, I was no longer able to be available for my team as much as I’d like. To prevent testing from becoming a bottleneck, and in an attempt to cross-train and become more Agile, the team decided that the developers would begin picking up some of the testing work in each sprint.

This wasn’t a decision we made lightly. There is a significant stigma against developers testing their own code and we knew we’d need to be conscious of that. We also had to overcome the idea that testing belongs to the tester – the developers were already writing unit and integration automation but exploratory testing was a whole new beast.

We discussed the potential ramifications – how would the quality of our work be affected? Did we think these senior developers, but junior testers, could provide value in the scope of testing?

HOW

The first thing we decided was that no developer would test his own code without pairing with another developer. Bringing in the pair testing technique helped each developer challenge their assumptions and biases and do more thorough testing. When I’m available, I also pair/mob with the developers during test sessions.

For helping my developer’s get started, I leaned a lot on Katrina Clokie’s “Testing for Non-Testers”. I also gave them a presentation on using heuristics in testing. After each story was completed, we talked about the testing that was done and how it could be improved in the next iteration.

For each story, I spend time with the developers at the whiteboard. We mindmap the story and highlight areas of risk that needed the most testing. We come up with questions about the story that would need to be answered during the test sessions and prioritize our plan. This is the same test planning I would normally do then review with the developers, but now we are doing it together as a group from the beginning.

The developers follow the test plan and make notes in the story about what they tested and what they found – whether it be the code functioning as expected, an issue they discovered, or just a behavior they didn’t expect. If you’re a tester, this all probably sounds very familiar – this is the same general outline I use for my own testing notes.

Finally, I review their testing notes and do any further testing I felt was necessary and report our findings to the Product Owner so they can sign off on acceptance.

THE CHALLENGES

One challenge has been getting the developers to “test outside the box”. In the beginning, they often just tested that the code did what they expected it to do – basically just acceptance testing. As a result, we had an issue where our service was outputting a checkbox from the UI as a string instead of the boolean it should have been. As a team, we discussed “building the right thing” vs “building the thing right” and how to question if what we built genuinely serves our customer’s needs.

We also struggled with “over-testing” some areas. In some cases, they were spending significant time testing areas that were well covered by automation, or running several variations of a test that had a low chance of returning unique results. When working within a sprint, all time is precious and must be spent wisely. The tests they were performing had a diminishing return of value. In this case, we discussed considering the value provided by a test before running it – what information am I expecting to learn from this test? What is the risk that this test will perform differently than I expect?

This experiment also required a lot of trust and vulnerability on all sides. The team as a whole needed to be able to trust that the developers could do a sufficient job at test, a role they were unused to performing. And because of their inexperience, the developers had to be open to trying a new role where they don’t have their usual level of expertise. I also had to be vulnerable enough to let go of feeling like I owned testing on my team.

RESULTS

This has been an ongoing, growing, and changing process. Just like how we iterate on our code, we’ve been iterating on our testing. Yes, some things have been missed but many were caught during the review part of the process. Do I think that the developers are skilled testers? No, they are still developers who are doing testing. But making them into testers was never the goal. The goal was to get enough testing on our stories to be able to get them accepted and into production. In that light, I’d call this a success.

One thing to note is that as much as I’ve been teaching the developers about test techniques and heuristics and other testing concepts, I have also been learning. The developers approach testing in a way very different than how I would do it and it’s opened my eyes to new ways of testing. This, too, has been a success.