You don’t need complex test automation

While there’re lots of information published on the internet about how one should develop/buy a Test Automation solution, there’s an elephant in the room — which is the fundamental question “Do I need to do it at all?”.

That is a good question. While I think that there’re some legitimate contexts where product/project/team can benefit from having a sophisticated test automation solution/framework, my general rule of thumb is “If you need a framework — you do it wrong”.

  • There’re at least several decent papers showing the benefit of unit-testing ([1],[2], [3]), while I could not find any research proving that system level tests could bring better results. Please let me know if you can prove me wrong.
  • There’s at least one half-scientific paper showing the benefits of unit tests in comparison to integration/system level tests [4]
  • There’s a good paper bringing a notion of running/writing and maintaining test vs. the benefit of having/running it at all [5]. If we know that, by definition, system level tests are more complex in maintenance, expensive in running and inspection, why do we insist on having so many of them?
  • Inability to write small and reliable unit tests is almost universally considered an indication of poor product architecture.

However, there’re legitimate contexts where you might want/need to invest in a complex test automation solution. My personal opinion is that it is when both of the following is true:

  • Product under test does not have enough reliable unit-test coverage;
  • The cost of writing/maintaining test automation solution is lower than the cost of refactoring/investing in product architecture and unit-test coverage for the expected planning horizon.

Or where this is true:

  • The quality is critical to the extent where we don’t really care about how expensive QA would be.

All things considered, at the time of writing I think we have only one working hypothesis, which is:

While all type of automated verification has its own benefits and areas of applicability, unit testing seem to be the most efficient way of automatic verification.

[1] https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html

[2] https://collaboration.csc.ncsu.edu/laurie/Papers/Unit_testing_cameraReady.pdf

[3] https://www.linkedin.com/pulse/unit-testing-software-quality-empirical-research-results-avteniev/

[4] https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam

[5] https://www.microsoft.com/en-us/research/wp-content/uploads/2015/05/The-Art-of-Testing-Less-without-Sacrificing-Quality.pdf

If you want to know more on the topic — feel free to have a look at my Test Automation hub: https://senpay.github.io/

Originally published at http://aqaguy.blogspot.com on April 18, 2020.