Problems with Test Automation and Modern QA

What are some common problems with test automation in agile and DevOps? Modern Software Development and QA focus too much on test automation and not enough on exploratory testing.

Are we releasing better quality software with more automated tests?

I think not!

I recently came across a post on a social media network that said

What I see in most testing and QA events today is mostly DevOps, Continuous Integration and Test Automation.

Although that is all very nice, I see a lot of crappy test cases being automated.

I see few bugs reported during integration tests and functional tests although they are all automated.

In UAT the users are finding more and more bugs because the test teams fail to identify them in previous phases.

If we do not teach people how to write good test cases we will end up with fully automated …

And my interpretation of … is “crap”. 🙂

Anyways, let’s see what is really happening in the world of Modern QA and Test Automation.

Problems with Modern QA

Most of the “Test Automation” within the agile development is in a dire state. The software industry is pouring in huge sums of money to hire “Test Automation Experts” mostly to gain a sense of confidence that the software they’re building is of high quality. Yet, noticeable bugs and/or other issues are found during UAT or slip through into production environments. So, what’s going on?

N.B. By Test Automation, I am mostly referring to UI Test Automation.

Automated testing is now at the heart of any modern software development process. Its purpose is to help deliver high-quality software on a repeatable basis, but is it really?

Do Testers Still Test?

The truth of the matter is that in most agile teams, testers are not testing anymore.

Manual testing has lost its virtue, thanks to development practices and cultures such as agile and DevOps, which have created a divide in the QA space – those who can code and those who can’t.

You’d often hear things like, “I’m a 100% automation engineer”, or “80% automation 20% manual”, or even worse, “I hate manual testing”. Shocking!

In DevOps, we are led to believe that everything should be automated. There is no place for manual intervention, e.g. manual testing.

Nowadays, most testers in an agile team struggle to keep up with the “Test Automation” demand. There is pressure to automate every story in the sprint, and there is not enough time for thorough exploratory testing.

The problem, especially in Agile development, is that QAs take a user story and automate its acceptance criteria. While doing so, their main and only focus is to wrestle with their limited coding skills just to get the test passing.

Naturally, this creates a narrow focus when you’re only interested in automating the test and see it passing in the build pipeline. This only proves what was in the acceptance criteria – right or wrong – is working and you tend to forget about the big picture.

The Decline in Manual Testing

More and more “traditional testers” are transitioning to “agile testing” by taking some coding lessons and becoming more technical.

Don’t get me wrong; this is all good. I believe as testers, we should always strive to learn new and emerging technologies to stay resourceful. We should understand the tech stack if we want to test a system to a high degree of quality.

However, the real reason why most manual testers take these initiatives is that there is a common belief that “automated testing” is superior to manual testing and hey, coding is fun, right?

N.B. By manual testing, I am NOT referring to the old school way of following a script and executing the steps. I am referring to the so-called “exploratory testers” – those who do the real testing and are curious to examine the system’s behavior by exercising interesting and thoughtful scenarios.

Unfortunately, there seems to be a big decline in the market for exploratory testers. This is quite evident. Just run a couple of search queries for “manual tester” and “automation tester” in any IT job site, and see the result for yourselves.

Problems with Test Automation

Now, let’s see why most of the test automation effort is not delivering any value.

Common mistakes I see happening repeatedly:

• Having a wrong expectation of automated tests
• Automating tests at the wrong layer, at the wrong time and using wrong tools
• Automating useless tests
• Neglecting important areas
• Missing key scenarios

Wrong Expectations

A while back I wrote a blog post on why would you want to automate a test? If you haven’t read it, it’s worth a read.

The summary of that article is that you automate tests which you want to run on a regular basis. By definition, these are your regression tests which confirm the system is still functioning.

However, if automated checks do find a lot of regression issues, I would question the skills of the developers and the development process. UI Automated Tests should not be [held at the expense of] or [compensated] for lousy coding.

Wrong Layer, Wrong Tools and Wrong Time

Majority of “Test Automation Engineers” in agile teams, look at a user story and automate it’s acceptance criteria. Most of the time this is done by a combination of Selenium and Cucumber.

Modern web applications are now clearly divided between backend and frontend. The backend is mostly composed of a number of REST web services or APIs with easily accessible end-points.

The application’s logic can be tested at the API layer. However, most test automation engineers resort to validating functionality at the UI layer which is at best cumbersome.

There are testing tools out there, such as Karate and Rest-Assured, that simplify API testing. We should be making use of these tools during development. Sadly, some test automation engineers don’t even know the basics of HTTP, let alone be able to write API test scenarios.

As for UI tests automation, Cypress is a great tool. It is more like a TDD tool for the front-end developers. The developers get a very quick feedback on the behavior of the new UI components.

Both Karate and Cypress serve as “development test tools”, i.e. tools that guide and support development. Both are lightweight, easy to integrate and can provide confidence in development.

We can then use Selenium or repurpose Cypress to design only a handful of scenarios that exercise the system end-to-end. These scenarios form our light-weight regression pack and provide confidence in business continuity.

Quite often I hear things like, “we wait until the feature is fully developed and stable, before automating tests”. Any conscious tester knows that the new-feature bugs outnumber regression bugs. There is a higher chance of finding issues with the current developing feature, than a stable feature.

If you are going to spend time automating tests, do them in parallel with development when they can provide more value.

Automating Useless Tests

Don’t automate every “test” just for the sake of it. Put some thinking process into the game. Study the high-level and low-level architectural diagrams. Ask what can possibly go wrong. Examine the integration points and look for potential failure points.

Take a risk-based approach in automation as you would (hopefully) do with your overall testing approach. What is the likelihood of something failing, and what is the impact of the failure? If the answer is high, then those scenarios should be automated and executed on every build.

In each sprint, we often end up writing automated tests around user stories for that sprint and forget about integration with other features. There are either weak or no integration tests.

Remember automating “tests” takes time. Do also bear in mind that by automating a test, you are not really testing, you’re only merely checking that the feature in question is satisfying some acceptance criteria.

You cannot automate testing, but you can automate the checking of known facts.

Therefore, every time you spend automating a “test”, think about the time you are wasting by not testing!

Neglecting Important Areas

I see more and more of this negligence since the birth of the DevOps culture.

In DevOps, the delivery pipeline, along with the deployment scripts are the spine of the software development and delivery, yet they hardly ever get tested.

In the past few years, I could easily say, that I have seen a lot more “environmental issues” than functional bugs. Environment issues such as problems with the CI server, the deployment scripts, the test environments, and so on.

Environment issues have a serious impact on the development and testing effort. They consume a lot of developer and DevOps time and massively slow down the deployment process, yet there is no consideration for testing and thus preventing these issues.

We just accept them as part of the modern software delivery.

We spend a lot of effort automating functional behavior and completely disregard the “things” that matter the most. Even worse is having to rely on Selenium tests to indicate if deployments are working or not!

Missing Key Scenarios

Scenarios are king! After all, it is the scenarios that reveal bugs.

Quite often, a serious issue leaks into production because no one thought about that particular scenario. The number of executed automated tests doesn’t matter. If a scenario was not thought of or tested, sod’s law tells us there is a bug in there.

Unfortunately, in most agile development environments, not enough dedication is given to this all important “Scenario Workshop” activity.

Problems with the Process

Let’s see how the above problems manifest themselves in a typical development scenario:

• Product owner writes user stories with either no or minimum acceptance criteria.
• Not enough time dedicated to story refinement sessions to discuss various scenarios for a user story.
• Acceptance criteria are interpreted as acceptance tests – Yes, there is a difference between two!
• Testers only automate the acceptance criteria in the user stories mostly using Selenium and/or Cucumber.
• Automated testing is almost always the responsibility of “automation testers”.
• Developers have no idea what is covered in the test packs or don’t even know how to execute the automated tests.
• The automated tests are added to an ever-expanding “regression pack” therefore taking longer and longer to run each time.
• The UI automated functional tests are integrated into the build pipeline, which is good but…

A developer pushes a simple change and has to wait 30 minutes for the automated tests to go green before the new feature or bug fix can be deployed to production. The 30 minutes wait is only if the tests pass the first time. If they fail due to some test or environment issues, it can take longer.

As the automated tests are running and the QA is investigating random failures, the developer and/or the product owner have verified the new implementation and are happy to release, but they can’t because the build is not green.

After a while, either the build goes green or the management becomes frustrated with the failing tests and makes a decision to release anyway. Then, BOOM, after a few minutes in production, there is a spike in 500 server errors.

Infrastructure Failures

The failures seem to show a similar pattern

• Failure in integration points.
• Failure in communicating with 3rd party apps.
• Web services not being “up” and requests to the API endpoints failing.
• A wrong configuration on one of the VMs or nodes, thus resulting in intermittent issues.

And yet, there is no process in place for checking these issues as part of the development or delivery process.

The focus of the test automation engineers is to automate functional tests. There is no focus on performance, security or resiliency. And there is certainly no testing of the infrastructure!

Summary:

Time has come to shift our focus from automating functional tests that have little chance of catching functional issues to the more serious and common environmental issues that plague development.

Test Automation, if done wrong or with no thought process, is a waste of time and provides no value to anyone. Crappy automated tests can incur huge maintenance costs and impede development. In the end, the only solution is to bin the tests.

In the modern software development, most of the effort of the “Test Automation Engineers” is spent battling with automation code and getting the “tests” to work rather than focusing on proper testing and exploring the system.

There is literally not enough time to write automation code and perform exploratory testing. We automate story after story and forget about integration tests, forget about the big picture.

Often we end up executing tons of automated tests, yet exploratory testing finds the majority of bugs. Then retrospectively, we write an automated test for the bugs that were found by exploratory testing, to catch regression bugs.

We should be selective on what to automate and judge our decision based on risk. What can go wrong, what is the likelihood of it going wrong and what will be the impact on the user or the business if it did go wrong?

If you are in the business of “Test Automation” then please don’t use Selenium to test the functionality of APIs or UI components. Instead, use Selenium to automate only a handful of useful and business-critical scenarios to provide confidence in business continuity before each release.

And lastly, every time you spend automating a “test”, think about the time you are wasting by not testing!

Common Myths of Test Automation

It is not difficult to imagine the benefits of having automated testing alongside product development – faster releases, increased test coverage, frequent test execution, faster feedback to the development team, just to name a few, yet many organizations have not made the move or are resistant in investing in test automation.

In this article, we shall examine some of the most common myths of test automation and how these prevent organizations from succeeding in test automation.

Setting Realistic Expectations

Possibly the most difficult and challenging aspect of any test automation endeavor is to understand the limitations of automated testing and setting realistic goals and expectations to avoid disappointments. With that in mind, let’s see some of the most common misunderstandings and myths about test automation:

Automated Testing is Better than Manual Testing

Referring to Michael Bolton’s blog post “Testing vs. Checking”, automated testing is not really testing. It is checking of facts. When we have an understanding of the system, we can enforce that understanding in forms of checks and then by running the automated checks, we confirm our understanding. Testing, on the other hand, is an investigation exercise where we aim to obtain new information about the system under test through exploration.

Testing requires a human to make a sound judgment on the usability of the system. We can spot anomalies when we were not anticipating. We should not be lenient towards one or the other, as both methods are required to get insight into the quality of the application.

Achieving 100% Automated Testing

Just as there is no practical way of achieving 100% test coverage (due to endless possible permutations), the same applies to test automation. We can increase test coverage by running automated tests with more data, more configurations, covering a whole variety of operating systems, browsers, but achieving 100% is still an unrealistic goal. When it comes to automated testing, more tests do not necessarily mean better quality or better confidence. It all depends on how good a test is designed. Rather than aiming for full coverage instead, focus on the most important area of functionality which is crucial to the business.

Quick ROI

When implementing a test automation solution, there are other interrelated development activities than just scripting test cases. Normally a framework needs to be developed that can support bespoke operations which are useful and meaningful for the business, such as test case selection, reporting, data-driven, etc.

The development of the framework is a project on its own and requires skilled developers and takes time to build. Even when a fully functional framework is in place, scripting automated checks initially takes longer than executing the same test manually. Therefore when we require quick feedback on the new feature that’s just developed, checking it manually is usually quicker than automating the test. However, the ROI is returned in the long run when we need to execute the same tests at regular intervals.

Higher Rate of Defect Detection through Automated Checks

Although many of the vendor-supplied and home-brewed test automation solutions are very sophisticated and highly capable in performing complex operations, they will never be able to compete with the intelligence of a human tester who can spot unexpected anomalies in the application while exploring or executing a set of scripted tests against the system under test. Ironically, people expect automated testing to find lots of bugs because of allegedly increased test coverage, but in reality, this is not the case.

True, automated tests are good at catching regression issues – after a new feature has been added to existing code base, we need to ensure that we haven’t broken current functionality and we need that information fast – but, the number of regression issues, in most cases, tends to be far less than new functionality that’s being developed.

Another point to bear in mind is that the automated checks only check what they have been programmed to check by the person who wrote the script. The scripts are as good as the person who wrote them. All automated checks could happily pass but major flaws can go unnoticed which can give a false impression of the quality of the product. In essence, checking can prove the existence of defects, but it cannot prove their absence.

We Only Require Unit Test Automation

So, if the likelihood of finding defects is greater in testing new features, why aren’t we running our automated tests against the new functionality as it is being developed? Well, this is somewhat the case for teams that practice TDD.

The developers write a unit test first, watch it fail and then write enough code to get the unit test pass and the cycle is repeated until the intended functionality is delivered. In essence, these automated unit tests are checking new functionality and over time they form the unit regression pack which is executed repeatedly as new functionality is delivered.

But, there is a caveat to this. While TDD is highly encouraged and is a strong development practice in building quality from the grounds up, unit tests are only good at finding programmer errors, not failures. There is a much larger aspect of testing that happens when all the components are tied together and form a system.

In fact, many organizations have the majority of their automated checks at the system UI layer. However, scripting automated checks for the UI or system, while the features are being developed is at best a daunting task, as the new functionality tends to be volatile (subject to many changes) during development. Also, the expected functionality might not be known until later, so spending time automating a changing functionality is not encouraged.

We only Require System UI Automation

There are values in running automated checks at the UI and system level. We get to see what the user experiences when interacting with the application; we can test end-to-end flows and 3^rd party integration when we could not test otherwise; we can also demo the tests to clients and end-users so they can get a feel of test coverage. However, relying solely on the automated checks at the UI layer has its own problems.

UI is constantly changing to enhance visual design and usability and having automated checks failing due to the UI changes and not changes in the functionality can give a false impression of the state of the application.

UI automated checks are also much slower in the speed of execution than at unit or API layer and because of this, the feedback loop to the team is slow. It might take a few hours before a defect is spotted and reported back to the developers. And when something does go wrong, the root cause analysis takes longer because it is not easily apparent where the bug is.

Understanding the context of each test and at which layer the test should be automated is important. Test automation should be part of the development activity, so the whole team is responsible for test automation, with developers writing executing unit tests, Software Developers in Test writing executing and maintaining acceptance tests at API and/or UI.

Losing Faith and Trust in Test Automation

This last one is not a myth about test automation, but a side effect when test automation goes wrong. You spend many hours developing a perfect test automation solution, using the best tools and best practices, but if the automated checks do not help the team it is worthless.

If the team has no visibility or knowledge on what is automated and executing, they either release with fear of unknown or duplicate their regression testing efforts. If the automated checks are flaky, slow, give intermittent results then it can confuse the team more than providing a safety net and a confidence booster.

Don’t be afraid of removing automated checks that are always failing or give inconsistent results. Instead, aim for a clean and reliable suite of tests that can give correct indications of the health of the application.

Conclusion

Test Automation is a long term investment. It will take time and expertise in developing and maintaining test automation frameworks and automated scripts. Test automation is not a one-off effort where you deliver a solution and let it run. It needs constant monitoring and updating.

Rather than aiming to replace manual QAs or expecting the automated checks to find lots of defects, we should instead embrace the advantages it brings to the team, such as liberating QA’s time for more exploratory testing where chances of revealing defects is maximized, or using automated scripts to create test data that can be used for manual testing.

Understanding the limitations and setting realistic expectations is important in overcoming these myths of test automation.