Getting your automated tests in the pipeline
According to the DevOps Research and Assessment (DORA), high-performing organisations release multiple times per day with a lower change failure rate; test automation can be a key component in achieving these twin aims. How do we go about introducing end-to-end automated tests to a CI/CD pipeline that will provide additional levels of test coverage while also not impacting another key metric, lead time for changes?
Two years ago, we were already releasing multiple times per day; however, we can have problems when trying to add these types of tests to a pipeline – including the potential for them to be hard to understand, slow to run, and untrusted by developers (due to flakiness). In fact, many teams end up only running them overnight for these reasons, but this reduces their value (and gives other problems)
Having experienced all of the problems mentioned in the past, I was very clear on the steps required to solve them, and the steps required to have end to end tests included in the pipeline, – required to pass before release, and providing valuable feedback that would not impact on how often developers can release.
* Why you don’t want to run tests overnight
Firstly I’d cover why it’s an antipattern to run end-to-end tests overnight in a continuous delivery environment. Apart from simply being too long to wait when developers can (and will!) release every day, it’s also introducing a clear separation between developers adding features and the writing of end-to-end tests. In addition, when tests fail overnight, it can be hard to trace the release back to the change that caused the failure.
* Gaining trust
Firstly, we built a small number of tests that would be included, without requiring them to pass before release. I then monitored these tests for a couple of weeks until I felt personally felt trust in these tests. Once I did, I could then make the passing of them a required step in the pipeline and feel confident about any release.
* Maintaining trust
After an initial set of tests were in the pipeline, it was important to maintain the trust initially gained. We built a basic automated process that would validate that the tests that were created and ran locally, would also run correctly 100% of the time in a Continuous Integration environment (as far as we could guarantee)
* Making tests easier to understand
We decided on creating abstractions and a Domain Specific Language that would make the understanding and maintenance of tests easier – ie, creating tests in the language of the application rather than the language of the technology.
* Making tests faster to run
**Firstly – Tooling that would scale with your tests
It’s important from the outset to have libraries and frameworks that will handle running tests in parallel. I will share how we built a solution based around WebdriverIO and Jest that allows for tests to be run in parallel. In addition, we used CI tooling that would allow tests to be grouped and run in parallel.
**Secondly – leveraging APIs
It’s important to not just test our APIs, but to use them to set up state and perform validation as part of browser tests, making the browser interactions tightly focused only on the parts we are interested in actually testing through the browser.
Join me as I discuss the steps and approaches above taken to get the tests included so that you too can have quick, reliable test automation in your pipeline