An e2e or system test can typically fail because of these reasons:
- Test code issue: Application code changed but the test was not updated yet, or other test-related reasons.
- Real bug in the application code
- Environment issue
What we really want to capture are issues of type 2 and type 3 (real bugs in the application code and environment issues). However, when e2e tests live in an external repository, they typically fail due to type 1 reasons.
Let us explore some of the pros and cons of keeping e2e tests in the same repository vs. keeping them in an external repository.
When e2e tests live in the same repository as the application
Pros
- Code and test changes go hand in hand.
- It is thus possible to make these tests a part of the quality gate.
- Results in zero to little test failures due to code changes in higher environments.
- Creates low noise and increased trust in tests.
- Results in low maintenance efforts, even when code changes are frequent.
- Tests are fixed when broken and only fail for the right reasons in higher environments.
- Tests are pushed at the same time when they fail in a PR — caught on a developer’s local machine or in a pull request.
- Since they are typically created and maintained by developers, they scale really well as the project grows.
Cons
- Requires initial investment in workshops and mentoring developers to help them work with test frameworks.