Debugging end-to-end user interface tests with Robot Framework and Playwright

Abstract

End-to-end tests on user interface (UI) are way to ensure that software works as it has been planned. These tests are slow to test manually, which can lead to adding test automation into the testing process. However, tests can break down due to a multitude reasons and proper maintenance is required. To ease solving process, frameworks are shipped with their own, inbuilt debugging tools, which eases the maintenance of the scripts. In this thesis, two different frameworks will be tested and analysed, what kind of debugging tools they bring and how easy they are to execute. The comparison was done through a case study, where a system under test was a demo of a coffee cart application. The system was tested using Robot Framework and Playwright, by writing same tests with both frameworks and then analysing, what common they have and how do they differ from one another when it comes to debugging faulty tests. The results showed that both have excellent post-test run logging and artefacts to debug faulty tests, while differences rise from features between the two tools. Robot Framework is simpler to set up and has inbuilt reporting with zero setup, while Playwright provides deeper debugging through videos, trace viewer and active debugging tool through inspector.