When teams are looking to transform, optimize, or cut costs in testing, where do they first look? More often than not, they follow the advice given by outside consultancies: “You haven’t got enough test automation”, “your peers have achieved xx% tests executed through automation”.
There are a number of problems with this advice:
- It focuses solely on test execution.
- It usually comes with zero context as to what that figure even means, or how testing approaches at one organization are comparable to another.
- It assumes that the way you do testing today is effective, and simply needs automating. Sadly this is rarely the case and you likely have a ton of waste, duplication and inefficiency in there.
Don’t automate broken approaches
So, what are some of the telltale signs that your testing efforts today aren’t as effective and efficient as they could be? Let’s explore some of the warning signs that the way you test today could do with some changes and a fresh outlook on your test approach:
- You have a large team of testers doing mainly e2e tests (even if they call it something else).
- You have a test pack largely made up of UI based e2e tests which always grows after each release – you suspect there’s lots of duplicated effort in there.
- Testing is delayed because you are waiting on e2e environments to be built, or waiting for your slot in the few e2e environments that persist in your organization.
- Test coverage relies on traceability back to requirements that you know are incomplete or out of date.
- Test design is largely an activity that involves testers creating test cases without input from the wider development team. Alternatively, you don’t design tests at all; you just write test cases/automation.
- Despite trying many different tools, you are struggling to increase the rate of tests executed through automation as the tests are just too flaky!
- A large % of your defects are environment/config related, causing large delays in test phases.
- You are finding fundamental design issues in late, e2e test phases.
These antipatterns lead to a poor return on your testing efforts. Testing becomes too costly, too time-consuming, and leads to questionable outcomes in terms of quality and your ability to implement change.
You probably recognize some of these challenges. Yet, if you are suffering from one of them, you likely have them all, but haven’t looked from the right angle. This is because all of these challenges stem from one common cause: Not setting testing up properly. They are all signs that you don’t have solid foundations in your test approach and strategy.
First, fix your test approach and strategy
If you don’t consciously have a test approach today, then the chances are that you have unwittingly implemented the “upside-down test pyramid” in some form:
With this, all of your testing is likely more expensive, slower, and less reliable than it needs to be. Yet, this is unfortunately common in test teams across organizations today, regardless of their delivery approach or technology being tested.
Lean approaches for moving beyond the upside-down test pyramid
The upside-down test pyramid will slow you down and cost you more the longer you persist with it.
So, what can you do? Take the first step in consciously implementing a test approach that matches the IT vision, technology and change methodology of your organization. And be lean about it.
A mismatch between “agile” and test?
Many organizations have been adopting smaller, iterative ways of working. Common goals include boosting agility and speed to market, while lowering cost and improving feedback (quality).
While these organizations have introduced plenty of new technologies and design patterns, testing has often remained large, cumbersome, and poorly suited to automation. As dev teams work so hard to keep design and build small, they are frustrated as test doesn’t (or wont) follow those same principles. This is fundamentally and evidently wrong.
Uncoupling monolithic architecture
If your team are trying to adopt agile, but your wider team struggle to carve out small iterations of delivery, you have more than a test problem. You have a “big ball of mud”. This will either be a tightly couple architecture, or a distributed monolith if you are in the cloud. You need to start decoupling.
Decoupling means establishing boundaries within your architecture. A good place to start is by understanding your interfaces, building models by analyzing how interfaces work today. BDD-style collaboration and monitoring production are both example methods of understanding how an interface works. Then, you should validate that these interfaces work in the way they were intended.
Once you know that interfaces are working as intended, you have an opportunity to rearchitect your big ball of mud. More relevant to this article, you also have an opportunity to change your inefficient test approach. You can start targeting the newly understood, observable and testable interfaces through API testing. And then, you are starting to turn that testing pyramid the right way up.
Isolating blast radiuses for testing
Yet, a good test approach involves more than doing different types of tests. Just as we want small, interactive delivery, we also want fast feedback. For that, we need to have a full stack test approach:
Full stack testing should not test everything every time. The architecture should act like a Lego kit for constructing test approaches, assembling “blast radiuses” that we need to test. Testing then becomes appropriately lean and targeted at the things we are changing.
In other words, we want to follow the architectural best practices that are used around lose coupling and isolation, so that we don’t end up unnecessarily testing interfaces that aren’t changing and so pose zero risk. Targeted tests should live and run alongside the component they test. They then provide immediate and specifically directed feedback on what’s changed, and which might have gone wrong. If those best practices don’t exist, we can establish them through our test approach.
The diagram and table below indicates the broadened scope that testing should consider during a more full stack approach. You could easily go further than this to make testing more agile. Have a think for yourself about where you might add further separation:
Want to learn more about isolating blast radiuses for testing your systems? Join my next webinar.
From testing phases to Continuous Testing
Being able to isolate creases in your system architecture and use different types of testing in turn replaces the traditional phases of testing from waterfalled projects of old, and those that carried into newly formed agile teams.
An effective test approach in this context no longer talks about System Test or System Integration Test as phases. Instead, you have types of tests that you execute continually as part of your pipeline, and these tests are harmonized and targeted to the area of the system you are changing, including testing platforms.
Reaching the whole system
Testing has been traditionally focused on functionality, but being able to isolate change involves getting closer to the system stack where the change is happening. This is not just API testing; it involves testing things like the platform and moving towards compliance as code.
Testing continuously at the platform level will be especially important as organizations transition to cloud, risking challenges in test environment builds and production outages due to manual and ambiguous config changes. We’ve all been there, yet many test approaches still don’t include such test types.
The problem is, a fully automated test pack of your user interface is going to fail as a non-starter if the environment you run it in is incorrectly configured. Check out the bugs you raised in your last release: Are environment problems ranking as a top cause of defects? Are you looking for these problems in your testing? Or you are finding these things far to late in your testing efforts, leading to substantial delays as your teams must spend time triaging the problems?
Plan a lean test approach – then automate
If this sounds familiar, your testing isn’t targeted sufficiently. Taking a lean approach to your test approach will do more than just improve the effectiveness and efficiency of your testing; it will make your whole development effort more effective. It will also create a change in mindset to help your team collaborate and strive for continual improvement.
Fast feedback is the key thing your test approach must provide within newer development practices. Automating test execution alone misses the mark of providing this critical capability. Before you automate, take a step back to revise your test approach. Go slower to go faster…..
To learn more about continuously targeting your testing across your whole system architecture, join my upcoming webinar: “Isolating Blast Radiuses For Testing: How to not blow up your DevOps world!”
