Subcutaneous tests: a faster and more stable alternative to Selenium-driven test automation

Effective Test Automation

Let’s assume a hypothetical situation (yet, so much real)— you were assigned to a project to help with the test automation initiative and/or test framework implementation. You have a huge test plan as an input, containing hundreds (if not thousands) of test cases and you need to do something about it, quick. It is too late for proper test-driven development as most of the code was written already. What would you do?

Problem statement

Usually, the first urge (if it is a Web application) may be to write some UI-level automated checks using tools like Selenium. It is almost and industry standard nowadays, but there’re more efficient approaches. Subcutaneous tests is one of them.

There’re many things which make UI-level automated checking the least desirable approach:

• UI-level automated checks will be slow. There’s just no way to avoid it. You can parallel them or do some other tweaks to speed them up somehow, but they still will be slow.
• UI-level automated checks will be flaky. Partly — because they’re slow. Partly, because Web browser and UI interfaces were never designed to be operated by a computer. (I have to admit, that is changing, and it is not necessarily good change).
• UI-level automated checks are, actually, the most difficult to write and maintain — because they check too much. (And because people usually try to automate “manual test cases” as is, making very ineffective automated checks)
• Even though some claim that UI-level automated checks emulate a real user, they don’t. A user won’t look for a field based on ID or XPath locator. A user won’t fill a form with a speed of light and then “fail” to submit because something didn’t appear in focus in the given timeframe. And now, when browsers are designed to be operated by a computer — what you automate is not a browser used by a user, but just a (very good) emulation.
• Some would argue that there’s functionality that is difficult to test otherwise. However, if there’s a functionality, which can be checked using UI-level automated checks only (provided it is not UI logic itself) — it may be a good sign of architectural problems in the product.

The only real benefit of UI-level automated checks is that it will allow writing some usable checks without actually diving deep into the product codebase. Which is hardly a benefit in the long run.

An alternative solution

So, let’s assume we have a user creation form which requires you to provide a valid user name. Once you give a user name that satisfies the rules — a new User object is created and stuffed into the database.

Test application front page

The application itself can be found here: https://github.com/senpay/login-form. Be warned, though — as you can guess from the screenshot, it is very buggy.

Let’s try to think about the list of “test cases” for this application:

Does it look simple? Yes, it does! Can we automate it using Selenium? Sure we can. Code with automated UI-level checks “covering” “test cases” can be found in LoginFormTest class if you check out uitests git tag:

$ git checkout uitests

A couple of things to note about those two tests would be:

Time to execute: ~12 seconds (12 seconds 956 milliseconds when I last checked)
Code coverage statistics:
Class: 100%
Method: 93.8% (30/32)
Line: 97.4% (75/77)

Now, let’s consider the possibility that Funtctional (System-level) automated check may be written without touching UI at all. This technique is called subcutaneous tests and was suggested by Martin Fowler quite a long time ago [1].

When people think about “non-UI” level Functional checks, they usually think about an API. And by API, they usually mean REST/SOAP or some other complicated stuff. However, an API (which stands for Application Programming iInterface) may be a way faster and simpler thing.

If we dig into the product code, we may find these interesting lines:

It turns out, that whenever we click something on the UI, those two methods are being invoked — a user is added, then the user list is presented. So, what if we use those methods directly? It does look like an API, after all!

Then, we can end up with something like:

This code can be found at subctests tag:

$ git checkout subctests

Unexpectedly, test methods themselves become slightly smaller. What happens if we run them?

Time to execute: ~21 milliseconds
Code coverage statistics:
Class: 77.8%
Method: 78.1 (30/32)
Line: 78.7 (75/77)

Well! The speed boost is impressive. It is more than 600 times faster!

On the other hand, we somewhat lost the coverage.

If this coverage loss is significant enough? That depends. What we lost was actually some of a glue code, which is (or may be) important (feel free to find what code is lost from the coverage yourself).

But if it has to be checked using extensive UI-level checks? That’s a different story.

At this point, we can:

• Add one UI-level check, to verify that the glue code is still working, or
• Provided that we don’t intend to change the glue code — leave it without automated checking, or
• Provided that there’s some human-driven testing involved — we accept that with quite big probability all possible glue-code related issues would be identified by human anyway, or
• Any other possible solution

What I usually recommend is to follow what I call the “Feature Tests Model” model, which I will describe in the next blog post.

Summary

• Functional automated check don’t have to be on UI or REST/SOAP API level. There’s an alternative approach, called subcutaneous tests, which allows checking the same functionality at much higher speed, ultimately allowing us to achieve an effective test automation.
• This approach has a caveat though — some of the code coverage gets lost in the way
• To overcome this limitation, one can use “Feature Tests Model”
• But in anycase, the speed boost (and check stability improvements) are so astonishingly significant, that I don’t really know why not everyone uses the approach yet.

PS

Did you enjoy this post? I think you might also find this video useful: https://www.youtube.com/watch?v=RVel7MZaSA0

It describes: 3 rules of test-driven development, TDD benefits, TDD drawbacks