In the previous post, I listed some downsides of End-2-End test environments and briefly explained 2 of them. In this blog post, I want to explain the remaining items.

Test scope

Before creating a test it is good to think about the scope of the test. What do you want to achieve with the test? What part of the code/integration do you want to test and how are you going to test it? As a team you are responsible for the code you put in production, that is also where you should put the effort in testing. The code and the integration with the rest of the system should be part of the scope you want to cover with tests. Try to set up clear boundaries for your tests. Don’t try to help other teams by testing their code. But do test the requests that are sent to other teams systems. Mock/Stub all services/solutions that are touching your team solution, but don’t test the solutions behind those solutions.

Example

Let us assume the following situation:
Your team is responsible for the functionality offered to a customer to change its profile online. Parts of that profile information needs to be stored in the Customer Relation Management (CRM) system. This way you have an integration with that system. By sending an HTTP:Post request to a Customer endpoint offered by the CRM, you are able to change the information provided by the customer. The CRM stores the information in an Oracle Database Cluster.

What is the test scope in this situation? The team wants to test the integration, for the team offering the “change profile” functionality that stops by sending an HTTP:Post request to the customer endpoint of the CRM system. So what the team should test is the request sent. The team can test: - Is the request sent - Is the request valid? - Are the fields provided by the test sent - Is the request compliant with the contract offered by the Customer endpoint. - Are error responses handled properly - In case of an error, is the correct message displayed to the customer etc.

What the team shouldn’t test: - Is the provided customer information, written in the database - Is the information stored and provided by the next HTTP:Get? - Is the newsletter service using the new e-mail address provided? - Is all the business logic involved with changing an post address working? etc.

This seems very clear and straight forward but still, we see this mistake made a lot.

Unstable tests

Tests with a brought scope and other systems involved are always less stable than isolated tests. When multiple teams are deploying their software to the End-2-End test environment without any coordination, things might change while your tests are run. This means you might get false positives.

Slow feedback time

Let us look at the https://martinfowler.com/articles/practical-test-pyramid.html. In general: tests lower in the pyramid take less time than the higher ones. In the lower levels of the pyramid, you will find Unit tests etc. Tests ran in an End-2-End test environment are Integration tests and are clearly the higher ones in the pyramid. Both Integration-tests as Unit-tests are good tools to cover code quality. I don’t want to argue what’s better: Having a lot of unit tests covering smaller parts or having fewer integration tests that cover bigger parts. I think it depends on the project you are working on. But both ways the integration tests are more “expensive” to run, timewise. Running them on a flaky environment, and providing wrong information will not speed up the feedback time. Sometimes the End-2-End environments are so flaky that it is almost impossible to run automated tests on them. Which means somebody will have to do parts of the integration tests manually. This all will increase the feedback time of the tests and so the time it takes to deliver software.

Increase time-to-market

For all the managers and product owners reading this blog post…. All the points given in these blog posts will help increasing the time-to-market of the deliverables. So make sure what you demand, running tests at an End 2 End test environment will slow you down.

Costs

It is clear, the End-2-End test environment is a costly solution to cover quality. Although it might be a virtual environment, it still costs money to run it. Let’s not forget: - the support needs to be provided - the time spend to deploy the latest software - the effort to update the environment running everything - the network components - the security updates - the license cost (databases, OS, etc.) - the metal to run everything (Although it is virtual, it still runs on something) etc.

YBqopySaB9kYKhuKErJjaOMNzYSOQkaZ2fDltZB/