Automate regression testing for iOS and Android using Appium and Robot Framework

Before an app release, a quality assurance engineer must do a regression test to validate that the new release of the app won’t break any existing functionality. Performing a regression test may cost a lot of time if you want to validate every existing functionality in the app, especially if you want to validate the functionality for both iOS and Android on multiple devices and OS versions.

When doing a regression test, you’re always following a script with pre-defined test cases and check if those test cases succeed. Because the script for regression testing is always the same, it is smart to automate this process, so you can focus on other test methods like exploratory testing.

To automate a regression test for mobile apps, Robot Framework is the tool you want to use. Robot Framework is an open source automation framework to perform tests. The framework is using Appium to run the tests, which is a tool to run scripts for native apps using a webdriver.

Installation of Appium and Robot Framework

The installation of Appium and Robot Framework can be done using the terminal of macOS. The tools are also available for Windows and Linux, but when you want to write tests for iOS apps, you need Xcode, which is only available on a Mac. Because of this I only focus on the setup for macOS in the installation instructions.

To install Appium and RobotFramework, first install Homebrew, Node.js, Java Development Kit and Phyton 3. Some of them may already be installed on your mac if you’re developing apps.

If you don’t have Xcode and Android Studio installed yet, install them. Xcode can be downloaded in the App Store of macOS. Android Studio can be downloaded on the Android Developer website.

Install Robot Framework using the terminal:

pip3 install robotframework
pip3 install robotframework-appiumlibrary

After a successful install of Robot Framework, install Appium using the terminal:

npm install -g appium

After installing Appium and Robot Framework, it’s time to install the tools that help Appium to perform.

brew install ideviceinstaller
brew install carthage
brew install libimobiledevice —HEAD

npm install -g ios-deploy
npm install -g deviceconsole
gem install xcpretty

sudo npm install appium-doctor -g
sudo npm install appium-webdriveragent

After installing all the tools, you need to set the home path for Java and Android. If you run macOS Catalina, you need to set those paths in the .zprofile file. If you are using an older macOS operating system, you need to write this inside the .bash_profile file.

The paths can be different on each computer, In my case, my .zprofile is looking like this:

export JAVA_HOME=/usr/libexec/java_home
export ANDROID_HOME=/Users/j.zonneveld/Library/Android/sdk
export PATH=${JAVA_HOME}/bin:$PATH

After the setup, run the following command in the terminal to validate the installation of Appium is correctly:

appium-doctor

If all settings are correct, you can continue. If something is not set correctly, fix these issues first.

Now Appium is configured correctly, it is time to set up the certificates for Apple in Xcode. This is needed to install the XCTest runner on iOS devices, which is needed to trigger the test cases on the device.

Go to the terminal and fill in the following:

cd /usr/local/lib/node_modules/appium/node_modules/appium-xcuitest-driver/WebDriverAgent/
./Scripts/bootstrap.sh
open -a Xcode WebDriverAgent.xcodeproj

Xcode should open now. Open the target WebDriverAgentLib and sign this with your Developer Account. Do the same for WebDriverAgentRunner. Build the project to validate if the signing was successful.

Appium and Robot Framework are now successful installed and set up!

Start Appium using the terminal using the following command:

appium

Setup the project

Creating the first test can take some time if you’re not familiar with writing code or development tools. To make it a bit easier, download the startup project files here.

If you write your test cases smart, the script for automated testing can run for both iOS and Android apps without maintaining two different projects for those apps. A prerequisite for this is that both iOS and Android apps must have the same user interface.

Now, let’s edit the settings.robot file from the setup project to set the correct configuration. Inside this file, you need to set the ${APPLEDEVELOPERTEAMID} variable and configure the devices. The Apple Developer ID can be found in the Apple Developer Portal.

Also, change the settings of the devices to your own devices, by updating the data fields and adding the UDID. iOS devices that are used to run the test cases, must be registered in the Apple Developer Portal.

For Android, you need to specify the path to the APK file which should be used for the tests. For iOS, specify the bundle identifier of the app and pre-install the app on the device before starting the tests.

Writing the first test using Robot Framework

Now the project setup is done, it’s time to write the first test case. All test cases are written in a test suite, which is a robot file that contains all the test cases for a specific flow.

Each test suite contains three files, which are:

1. test cases, which can be found in the testsuites folder.
2. identifiers, which can be found in the testsuites/elements folder.
3. recurring actions to prevent writing duplicate scripts, which can be found in the testsuites/keywords folder.

Because each app has a different flow, it also requires different test cases. I will explain how to write your first test case using a invalid login example.

In the example, the first screen contains two buttons, which are register and log in. When the user presses on login, two fields for username and password appear and a button to log in. Let’s write a test case for this to validate if an alert will show the correct error if we log in with invalid credentials.

First, set the identifiers for the elements (which are the buttons and text fields) in the login_elements.robot file. For Android, the identifier is the ID the developer defines when setup a UI element in XML. For iOS, the identifier can be set using the property accessibilityLabel, which is explained in-depth in my article about the accessibility inspector.

In the identifiers file, the definition of the identifiers would look something similar to this:

${LOGIN_BUTTON}     xpath=//*[@resource-id='package.id:id/login_button' or @name="login_button"]
${TEXTFIELD_USERNAME}     xpath=//*[@resource-id='package.id:id/username_field' or @name="username_field"]
${TEXTFIELD_PASSWORD}     xpath=//*[@resource-id='package.id:id/password_field' or @name="password_field"]
${LOGIN_BUTTON_SUBMIT}     xpath=//*[@resource-id='package.id:id/login_button_submit' or @name="login_button_submit"]

The first identifier is the identifier on Android, the second is the identifier for iOS. By doing this, you can run the script both for iOS and Android apps.

Now, we are able to write our first test case. Open the login.robot file and you will find the first test case there, which looks like this:

*** Test Cases ***

Validate invalid password
    Go to login 
    
    Wait Until Page Contains Element    ${TEXTFIELD_USERNAME}

    Input Text                          ${TEXTFIELD_USERNAME}      j.zonneveld
    Input Text                          ${TEXTFIELD_PASSWORD}      fakep@ssword99
    Tap                                 ${LOGIN_BUTTON_SUBMIT}
    
    Wait Until Page Contains            Error
    Page Should Contain Text            Invalid password

The test does different things. First, the script will go to login. This is a defined step in the login_elements.robot file, because this step may recur when writing the next test case.

Then, the test will wait until the text field for the username is visible. When this is visible, we enter the username and invalid password and press the login button. Now an alert should be visible with the text “Invalid password”, so our text will succeed!

Running a test case using Appium and Robot Framework

Now our test case is done, so let’s validate how our test will perform. A test can run on both a simulator or a real device. I prefer to perform a test on real devices because this is the most ‘real’ situation.

Before you’re able to run the test on a real device, you must define the device in the settings.robot file. Here you must define the name of the device, the UDID of the device and the os version.

After the devices are defined, open the terminal and navigate to the root of the test project folder. Then, run the following command in the terminal:

robot --variable DEVICE:iPhoneXr --outputdir logs testsuites

Appium must be active. If you closed Appium, you can easily start the program in a separate terminal window using the command ‘appium’. To close Appium in the terminal, press control + c on the keyboard.

A test can run wireless on a device, but this may operate much slower than a situation where a device is connected by a cable.

After the tests are done, the output of the test can be found in the logs folder as an HTML file. Here you can read which test cases succeeded and failed.

Conclusion

Automating a regression test using Appium and Robot Framework is a great way to test mobile apps for iOS and Android. When defining the identifiers for both iOS and Android in one variable, the test script can run for apps on iOS and Android using the same script.

At the start of the project, it may require some extra time to set up the automation project, but this time will be won back after running the test multiple times on different devices and OS versions.

🚀 Like this article? Follow me on Twitter for more mobile development!