Test Automation

Overview

From this case study, you will learn about the advantages and disadvantages inherent in automatic tests and ways to benefit from them. Specific attention is paid to the difficulties and pitfalls of test automation in software development projects.

Industry

Test automation allows for the creation of high-quality applications with less effort in significantly less time. Nevertheless, many companies do not entirely understand how automated tests can be integrated into their daily development process, so they fall back on manual testing.

Automated systems use machines to perform operations previously performed by humans. They can work much faster, without fatigue or loss of concentration, which is common for manual testers.

Advantages & Disadvantages of Test Automation

Like many other IT practices, test automation is sometimes applied in the wrong ways, reducing its efficiency. To avoid the loss of performance, testing teams have to work around the shortcomings of test automation and make the most of its benefits.

Below, we will consider the main advantages and disadvantages of test automation.

Advantages of test automation:

• Reusability – once developed, test scripts can be reused an infinite number of times.
• High capacity – automation is the optimal solution for promptly testing a large number of desktop browsers and mobile devices.
• Uninterrupted testing – you can run tests at any time, set up your tools and programs to work at night, and analyze the results the next day. 
• Multitasking – automation engineers can attend to other tasks while testing is in process.
• Limited support – after automated tests are developed, their further support and results analysis take far less time than manual testing.
• Efficiency – once activated, the scripts of automated tests begin working  immediately, providing real-time results. There is no need to constantly verify compliance with test documentation, as is the case with manual testing.
• Faultless operation – tests are always performed correctly, since they are not subject to human error. Time-consuming and monotonous work increases the risk of manual testing errors.
• Payback – well-planned automation pays off in the long run and meets expectations

Disadvantages of test automation:

• Narrow view – the reusability of automatic tests can lead to the so-called “Pesticide Paradox,” where the repeatedly performed script stops detecting errors that go beyond its strictly formalized frameworks. In manual testing, an engineer can go beyond the general frameworks and perform several additional operations to find defects.
• Overlooks trivial errors – automatic tests may skip minor errors if the script is not programmed to check them. Most often, skipped errors relate to the positioning of certain elements, errors in inscriptions, objects, and forms with which the running script is not interacting.
• The need for constant updates – even though the development cost of automated tests is less than the wages of several manual testers, there are still significant expenses. Each update in the tested product entails updates in automatic tests, doubling developer expenses.
• Development costs – automated test development is a laborious, time-consuming and expensive process, since testers need to create an application to test another application. They use different frameworks, utilities, libraries, etc. to develop complex automated tests. All these elements also need to be tested and configured, which takes time.

In order to decide on the feasibility of automated tests, you need to evaluate whether the advantages outweigh the disadvantages in your particular case.

About the Client

Our client is a car leasing platform in the automotive industry that allows users to lease a vehicle online in a few simple steps. 

For this client, we decided to use automated tests since they provide reusability, uninterrupted testing, multitasking, efficiency and faultless operation.

Preparatory Process

Let’s consider a pyramid with manual test scripts at the top and Unit Tests / Component Tests are at the bottom:

• Manual Tests are used for manual testing of software development projects.
• GUI Tests are used to test for compliance of a product’s graphical user interface with previously adopted specifications. Testing is carried out by testers. 
• Acceptance Tests (API layer) rely on an interface or a protocol of communication between different parts of a computer program, designed to simplify the launch and maintenance of software. Testing is carried out by testers. 
• Unit Tests / Component Tests assess the performance of individual source code blocks, fragments of a single or multiple computer program modules, along with associated management data, usage and operating procedures. Testing is usually carried out by developers.

We already had detailed scripts for manual testing when the need for automation with GUI Tests arose. On average, complete manual testing took us between 2-3 hours. Our team needed to perform 2-3 functionality checks every day, which would have been an incredible chore with manual tests.

We needed to select tools to provide all the necessary features and meet the expectations of both autotest developers and managers who receive daily reports.

We decided on the following stack:

1. Python – a programming language with a relatively low entry threshold, accessible to anyone with minimal programming knowledge.
2. Robotframework – a framework that provides a wide range of features to speed up and simplify the writing of autotests.
3. Allure-report – one of the best options for working with reporting documentation.
4. Jenkins – the system used by our team for continuous integration of software, time settings, test activation, etc.
5. Cucumber – all test cases within the project are written in this language to ensure their understanding, even by those with a very basic idea of programming. For example, reports written in Cucumber are more reader-friendly for managers.

Development Process

The development of GUI tests starts with assembling a clear project structure in which tests, web elements, libraries, and individual scripts are all in place. Autotest developers usually call this structure Page Object Pattern and use it to perform smart automatic checks. It creates levels of abstraction, separates tests from test objects, and provides a simple interface for page elements.

Page Object simulates the pages of the application being tested as code objects, and provides separate classes for working with the HTML of each individual page. This approach significantly reduces the amount of repetitive code, since the same page objects can be used in different tests.

We used the Cucumber language to describe test cases. It brought tests to the following structure:

• (Given) some context
• (When) some action is carried out
• (Then) a particular set of observable consequences should be obtained

Each step of the test has a link to its corresponding section, which describes page elements with interaction steps for Selenium-webdriver:

AND As a new user i’m able to signup on homepage ->>

Home_page.Open home page and signup ->>

wait until page contains            Car Leasing

log to console                      INFO: Home page reached

click element                       xpath=.//*[@class=’’nav-bar’]

log to console                      INFO: Side menu opened

Since automated testing takes time, it is not always necessary to run all tests at once. When initializing a test run, you can use the [Tags] framework functionality to specify which tests you need at a given moment. You can also specify the environment in which the tests are to be performed by changing the variable in the test run initialization command:

Robot -d Results -i Smoke –variable EXAMPLE: ${Test} Tests/test.robot

Since the script was activated from a local machine, the final report was generated in the ‘Results’ folder that we previously specified in the script initialization line.

After the first successful run, we had to make sure that the tests would work as intended in all browsers. Checks were performed in Chrome, Firefox and Opera. 

After the successful completion of Desktop tests, we wrote tests for mobile view.

Test expansion presented the need for continuous automatic testing in Jenkins. In this system, we configured the launch time of tests and the amount of functionality to be tested. All the necessary parameters were specified in the pipeline settings.

As a final step, we installed Allure-report and configured reporting.

Results/Achievements

Not all projects require complete automation. It is sometimes enough to use auxiliary scripts to speed up teamwork. At the same time, if you plan long-term product development, automation could be a great asset to the success of your project.

Automated testing helped the Clever team perform laborious tasks more quickly than with manual testing, and to promptly release new products. While manual testing on average took a whole day, automation reduced testing time to a couple of hours. Saving time is especially beneficial for large, constantly developing projects like ours.