Power Tester

e2e tests location (in or outside app repo?) : The decision that can make or break your automation efforts.

An e2e or system test can typically fail because of below reasons:

  1. Test code issue: Application code changed but the test isn't updated yet or other test related reasons.
  2. Real bug in the application code
  3. Environment issue
What we really want to capture are issues of type 2 and type 3 (i.e. real bugs in the application code and environment issues). However, when the e2e tests live in an external repository than the application repository, they typically fail due to type 1 reasons.

Lets explore some of the pros and cons of keeping e2e tests in the same repository vs keeping them in an external repository.

When e2e tests live in the same repository as the application repository

Pros

  1. Code and test changes go hand in hand.
  2. It is thus possible to make these tests as a part of Quality gate.
  3. Results in zero to little test failures due to code changes in higher environments.
  4. As a result, creates low noise and increased trust on tests.
  5. As a result, results in low maintenance efforts. Even when the code changes are frequent
  6. Tests are fixed when they are broken and only fail for right reasons (real bugs or env issues in higher environments).
  7. Tests are pushed at the same time when they fail in a PR. Caught typically on a developers local machine or in a pull request.
  8. Since these are typically created and maintained by developers, they scales really well as project grows.

cons

  1. Need initial investment in terms of workshops and mentoring developers to help them understand how to work with test frameworks.

When e2e tests live in an external repo than application repository

Pros

  1. I don't know if this should be called as a PRO but when QEs write and maintain the tests in their own external repo, team, typically they don't have to worry about educating and mentoring rest of teams and developers about using these frameworks. This saved time and effort although may seem like a pro, but in long run it is nothing compared to the maintenance efforts this small team have to spend fixing these perpetual, out-of-sync broken tests.

cons

  1. There is no good way to keep the tests in sync with application code changes.
  2. Not possible to make these tests as a part of Quality gate due to above reason.
  3. Results in a lot of false positives in tests.
  4. As a result, creates more noise, brings down trust on tests.
  5. As a result, results in high maintenance efforts. Specially if the code changes are frequent
  6. Tests are fixed after they are broken and due to wrong reasons.
  7. Tests are pushed after they have already failed in the pipeline.
  8. Typically e2e tests in outside repository are written and maintained by dedicated QA teams. Which means they dont scale well with QA capacity over time.

Credits

I would like to dedicate this article to @Amuthan Shakthivel. It was after one of our discussions that I thought of experimenting with this approach in our current project, which resulted into all these findings and eventually this article.

I have tried a similar approach in the past but had to prematuraly gave up on the idea due to infastructure pressure that our DevOps team felt at that moment, because of creating temporary profile environments, for each PR to run all those system tests. This time it was differnt since our projects DevOps setup was much more optimized and matured but that past experience was still hanging somewhere in my head.

I would not have thought about trying this approach again, if it wasn't for that discussion. So, thank you Amuthan :).