Consultative QA has the ability to transform your engineering team, putting the responsibility on the engineers for the quality of their work, whilst still having a level of assurance that a seasoned tester can bring.
Many companies run one of two strategies when it comes to the quality assurance (QA) work their engineering teams produce: traditional QA and no QA. Both strategies can be seen as opposite ends of the QA spectrum, with the no QA movement (or rather, developers doing QA) being a reaction to the heavy-handed QA processes experienced in the past. Having developers being responsible for QA might work in some organizations but it depends on the level of risk you’re willing to accept.
At my company, a high degree of accuracy is paramount in everything that we do. The QA process is therefore integral to our workflows. To that end, we’ve created a strategy that gives developers more QA responsibility, while still keeping a QA team looped in; a strategy we’ve dubbed as consultative QA.
Traditional QA
In traditional QA, features are usually carved up into a series of sub-tasks which are presented to a team of developers for timeline estimation. The developers review each sub-task, usually with a set of acceptance criteria, and come to a consensus on how long each will take to implement.
Each sub-task is worked on by a developer, then handed over to a tester to check that it meets what was specified. If any issues are found, the tester will raise a bug ticket for the defect (usually a separate ticket for each issue found), assigning them and the original sub-task back to the developer. Once all bugs have been fixed and the tester is happy to sign off testing, the code is bundled into a release that will eventually go into production.
In this workflow, the carving up of a feature into a set of sub-tasks can prevent the developer and tester from seeing the bigger picture. Their focus has been artificially constrained, preventing them from being able to exercise their creativity and reinforcing a reliance on the person who created the sub-tasks (which could be a tech lead, product, or business analyst). Preventing a developer from being creative in finding solutions removes the passion from their work, results in lower quality work, work that contains defects, and work that may well not actually hold together when combined with the rest of the associated sub-tasks.
The lack of quality results in a low trust environment between developer and tester because the tester always finds obvious flaws, leaving the tester to feel like they need to check everything the developer produces, or risk a bug being shipped to production.
Testers are normally correct in this belief. This, combined with the fact that testers are typically in high demand and that companies usually settle on a ratio of around one tester for every three developers, means that hiring can quickly get expensive.
The no QA movement
An approach some companies use to combat this problem is to remove QA as a function altogether. By removing the safety net of a QA department, the developer is forced to take more responsibility for the quality of their work, the threat of a production failure being a powerful motivator. This may be an acceptable approach for some companies but it ultimately comes down to your appetite for risk.
Great QAs add value by approaching problems differently, by being curious, and by not doing what they’re meant to. Due to reduced cognitive load, they’re able to keep a larger part of the overall system in their heads and are therefore in a better position to spot when a change in one area could lead to unintended consequences in another. In short, if the price of reversing a bug mistakenly shipped to production is high, you should have a tester assuring the quality of what you release.
Consultative QA
At its heart, consultative QA aims to push the responsibility of quality control onto the developer whilst keeping the overarching support and assurance of a seasoned tester.
To make this work, QA needs to be engaged right from the beginning. At my company, this engagement point is when the developer has completed an architectural decision record (ADR) – a document that outlines the technical solution to a project brief, completed by the developer before they begin building.
It’s important to note that the ADR is the lowest level the feature exists at, and should result in a product that can be shipped to your customers. The developer may choose to build and release the feature in parts but, if this is the case, it’s described in the ADR rather than through sub-tasks in a ticketing system.
The benefits of this approach are threefold:
- It ensures that the team is always looking at the bigger picture, driving ownership and overall responsibility.
- Thinking isn’t constrained to a singular sub-task and developers are free to come up with innovative solutions that may not have been otherwise thought of.
- It prevents an explosion of tickets that need to be managed, tested, triaged, and reported on.
The QA review process
There are two distinct phases during this process.
First, the developer brings together all interested parties (QA, product, or business analysts) and walks them through the proposed design in an ADR review. This allows attendees to ensure that the proposed design meets the requirements and that it fits into the overall architecture. Once feedback has been incorporated and the ADR iterated if necessary, we move on to the second phase: the QA review.
The QA review was conceived by Antony Konstantinou, a talented QA lead at my company, who sits at the heart of consultative QA. The QA review is run by QA and takes place after the ADR review, usually with the same set of testers involved with the ADR review.
There should be a gap between the ADR and QA review allowing the tester time to think holistically about the proposed solution, what processes could be affected by the changes, and what tests should be run to deliver the work within the risk appetite.
Once the tester has come to a conclusion on the above, the developer(s) should be walked through each test case. The discussion of each test case serves a few purposes:
- It helps clarify the requirements with the tester playing back their understanding and checking the group is in agreement.
- It ensures that testability is baked in from the start and that the automated test cases created by the developer are meaningful.
- It fosters discussion about risk and who the best person to run the test is.
Point c) above is very important. Consultative QA does not necessarily mean the tester completes the testing themselves; it could mean someone else in the business who’s closer to the feature, for example, an experienced end-user of the system. Conversely, the developer may be able to simply automate the test case themselves.
Conversations, not tickets
If a bug is discovered, the tester discusses the issue with the developer, either in person or in an open channel. Bug tickets aren’t raised and assigned – all communication happens outside a ticketing tool via conversation between developer and tester.
During these discussions, you may decide that serious defects are raised to ensure that all the relevant information is captured. As a rule of thumb, it is a “serious” issue if it will take longer than 24 hours to fix.
The benefits of this are twofold: a) the burden of creating and managing tickets through a development flow is completely removed, and b) it encourages the developer and tester to work in harmony rather than the stop-start style of communication that takes place through a ticketing system.
However, the one benefit that the old ticketed approach does offer is that it proves that testing has taken place. This is especially important if, for instance, you work in a regulated industry. In this scenario, rather than reverting to old habits, ticket the test cases and attach the evidence to each ticket. These tickets can then be grouped together and attached to the ADR, keeping the documentation and test cases in one place.
Implementing consultative QA within your organization
Using this approach, we’ve managed to get a developer-to-QA ratio as high as 1:6 at my company, while increasing the quality of our releases. This is because quality is baked in from the start rather than being dusted on at the end.
Implementing consultative QA in your organization can be emotionally taxing for those involved as you’re getting people to change the way they work. The trick is to introduce change gradually but consistently over a period of time to allow the team to get comfortable with the new ways of working.
Below is a suggested order of implementation:
- Drive the message that everyone is responsible for quality, not just QA. Encourage the developers to test the code themselves before they then release it to QA and production, encourage discussion of possible bugs with the team before they’re raised in the bug tracking system, and encourage a culture of responsibility rather than blame. Importantly, keep repeating these messages.
- Reduce the number of tickets that comprise a feature, ideally to just one ticket per major flow of the feature. This will help get the team out of the weeds, allowing them to see the bigger picture and have more meaningful conversations.
- Get the developers to start creating ADRs before cutting code. Make sure to review them as a team.
- Finally, start implementing the QA review process.
Remember, this is a journey to becoming a more mature engineering team, not an overnight transformation. If you take a team who are reliant on traditional QA and suddenly remove their safety net, it will cause upset and likely result in bugs being shipped to production.
Final thoughts
The adoption of consultative QA encourages developers to embrace accountability for the quality of their work while benefiting from the expertise of seasoned testers. Traditional QA and no QA strategies represent two sides of the QA spectrum, but there is a middle ground to be reached. Consultative QA is the approach that strikes a balance by involving QA professionals from the inception of projects and facilitating an environment where developers drive ownership and innovative problem-solving.
