Using Dockerised Testcafe Scripts to measure the Reliability of Applications

Every industry solutionwe see today defines a certain level of user experience. The most appropriateway of measuring the user experience of an application is to feel its functionalities from an external point of view, by mimicking the actual user'sbehaviour. This is why we employ Quality Assurance Engineers to make sure thatthe application is tested from a "Real User's" perspective to seewhether the User Experience is as expected, before releasing the applicationinto production. UI (User Interface) Automation seems like the primary responsibility of a QA Engineer, not a Software Engineer. That may be true because UI automation scripts have helped QA Engineers to test the applicationas a real user, look for bugs and unexpected behaviour. But, I would like toshow another aspect of UI Automation scripts from a developer's perspective,where continuous UI automation can actually tell us whether a Production application is working as expected in real time.

When QA Engineers use UIautomation, it's more of a "readiness" check before pushing newfeatures into production. But when a Site Reliability Engineer uses such anapproach, that is to ensure the application is running reliably and providing the expected user experience to all its users, at all times. In one of my previous articles, I explained to you the role of a Site Reliability Engineer (SRE).

In this article, I wouldlike to show you how an SRE can leverage UI Automation Scripts to ensure thatan application is running reliably at all times. I will also be showing you how to run these UI Automation scripts in a Dockerised environment. (Something thatI was struggling to do, for quite some time).

This article will beunder following sections:

1. How UIAutomation can be used to measure the reliability of applications

2. Writing asimple UI Automation Script via Testcafe

3. Running a UIAutomation script on a GUI vs Headless

4. Dockerisingyour UI Automation Script

Using UIAutomation to Measure the Reliability of Applications

There are multiple ways to measure the reliability of a production application. This can be defined in different ways, based on the deployment structure, purpose of the application,and even the type of expected users. Some of the common ways of measuring the reliability of applications are given below.

· Pinging appURLs to see the liveliness

· The ratio ofsuccess to failure API calls from the application to the backends

· BrowserJavascript errors

· Load time ofPages

· Injecting Chaos and verifying that applications can function in various conditions like Network downtime etc.

Nevertheless, every application has its own important pages and user flows, that require carefulattention to ensure that applications work reliably and they provide the expected User Experience (UX). Aspects like whether the user can successfullyl og in, or whether the user can successfully add items to a cart and checkout,are user flows that are crucial, which are also highly dependent on many other components of the application ecosystem. Yes, it's true that QA Engineers verify these before deployment, but what if a certain dependency breaks in areal production environment? There will be no other way to verify this unless someone files a support ticket.

This is where Site Reliability Engineers, mimic the user's behaviour in these applications, as aperiodic job, which will check whether the major flows function as expected.This is where UI Automation can be leveraged to automatically and periodicallymeasure the reliability of certain parts of the application. Sometimes, with this approach, you will notice that the behavior changes with user load,network latencies, backend failures, etc. Even though with synthetic data, this ensures that certain aspects run as expected from a user's perspective.Furthermore, to enhance this, you can also run these scripts from different geographies, so that you can notice differences based on various parts of theworld.

Once you get the initial scripts and the structure ready, all you have to do is run these scripts in different environments which will give the required metrics at the end of theprocess.

One difficult decisionhere would be to figure out the major user flows which need to be monitored.Due to practical reasons, all the user flows in an application cannot bemonitored. For the purpose of reliability, you only need to measure the majoruser flows. To make this decision, you will have to speak to different stakeholders and identify the most important use cases or user flows of theapplications.

Writing asimple UI automation script using Testcafe

Testcafe is anApplication UI Automation framework, which runs on all popular environmentssuch as Windows, macOS, and Linux. It supports desktop, mobile, remote, and cloudbrowsers (UI or headless). Another good thing about it is, it's completelyopen-source.

In this section, let'swrite a simple UI automation script that will run both on your browser in a GUIand without a GUI (headless).

Please follow the stepsgiven in the below URL, to run your first Testcafe script. Afterward we willbuild a simple script and run it on our own.

https://devexpress.github.io/testcafe/documentation/getting-started/

The following script isa simple script that was taken from the above URL. You can write your ownscript for your application. Explaining this would be beyond the scope of thisarticle, but if you are really interested the documentation provided byTestcafe is quite efficient.

import { Selector }from 'testcafe';
fixture `Getting Started`
.page`http://devexpress.github.io/testcafe/example`;

test('My first test', async t => {
console.log("Startingto type the developer name")
await t
.typeText('#developer-name', 'Keet Malin')
.wait(2000);
console.log("Hitting th SUBMIT button afterentering the developer name")

await t
.click('#submit-button');

console.log("Waiting for the article header to beThank you, Keet Malin!")

// Use the assertion to check if the actual header textis equal to the expected one

await t.expect(Selector('#article-header').innerText).eql('Thankyou, Keet Malin!');
});

To run the above scriptvia Chrome, you can simply use the following command. It will bring up a newGoogle Chrome browser and show you step by step how it happens. I am usingGoogle Chrome as a simple example in this. You can use any browser you want.All supported browsers are given here.

testcafe chromescript.js

This approach ispreferred if you really want to see how the test simulation is working. Forexample, an engineer would be interested in seeing the flow of events.

Now let us talk about runningthis in the headless mode. Headless mode will not open up a browser, but itwill run in a browser and provide you a result. That means you can simply runthis in a place where there is no GUI, like a terminal or SSH, and it will giveyou the same result.

testcafe chrome:headlessscript.js

This approach ispreferred when you do not want to see the actual simulation, but you want tosee the end result. This is preferred when you want to run periodic jobs,without the user's intervention. In this article, I would be bringing forwardthe Headless approach, so we can dockerise and run it in a containerizedenvironment.

For both the above options, you will see a response like this.

Dockerizingyour Testcafe Execution

Now that you have thescript ready, it's about running this in a Docker container. Fortunately,Testcafe provides you with a DockerImage with Chrome and Firefoxinstalled.

The following articleexplains the features of the Docker Image.

https://devexpress.github.io/testcafe/documentation/guides/advanced-guides/use-testcafe-docker-image.html

TestCafe provides apreconfigured Docker image with Chromium and Firefox installed.

But, let us create ourown Dockerfile with this image and prepare it to be run.

FROM testcafe/testcafe
COPY . ./
CMD [ "chromium:headless","script.js" ]

Now run the followingcommand from the same directory as the Dockerfile and build the docker image.

docker build -t testcafe.

Now, run the dockerimage using the following command.

docker run testcafe

Now that the dockercontainer is ready, you can simply run this as a Kubernetes Cron Jobperiodically, to make sure that your UI flow is working as expected.

Make sure that you havea proper reporting mechanism applied during failures. I used Slack alerts forthis purpose. I had to write the code from scratch, and add it into the dockercontainer.

Key Takeaways

Testcafe is anapplication automation framework. The role of a QA Engineer is to ensure thatthe quality of the application is tested before it is sent to production.Ensuring the reliability of the application throughout is actually the role ofa Site Reliability Engineer (SRE).

Another point to makenote of is, that this is all synthetic data. It might not represent an actualuser's problems. But this can represent problems in general that occur due tothe Application's ecosystem.

In this article, I justexplained how to use this to validate some of the major flows. With thissynthetic monitoring, you can also measure the page load times, gather a lot ofdata on latencies between each page such as latencies to load data, and evenlatencies of components that take time to render on the browser. With this, youwill be able to define Service Level Objectives (SLOs) for applications.(Defining SLOs for applications is a separate topic and will not be discussedhere.)

The reason for me tohighlight the word docker is, because it can be easily run in Kubernetes, as acorn job to periodically monitor the user flows. As an SRE, you can also thinkof other ways to run this, but it all depends on the infrastructure used inyour organisation.

A simple cronjobmanifest for Kubernetes is given below.

apiVersion: batch/v1
kind: Job
metadata:
name:simple-test
spec:
backoffLimit: 3
template:
spec:
containers:
- image:<IMAGE_NAME>
name:simple-test
restartPolicy: Never

You can also see the successful completion of a job.