Parameterizing Your Robot Framework Tests

This is part 2 of 3 of the Getting Started with Robot Framework Series. You should read the first part, but also the companion article on why parameterizing test cases is important before continuing, as it contains the basis for work done in this article.

If you’re too impatient to read the whole thing (it is quite lengthy!) or just want the juice, here’s the code.

Goal of this installment

After you have worked through this part of the series, you will have

  • An improved test runner
  • A way to include new Resource Files in your tests
  • Some custom keywords that you can use in multiple test cases
  • A place to store grouped variables used in your tests
  • A test case that demonstrates various ways to parameterize the test cases

There is one goal that will not be achieved at this moment, and that is an actual Selenium-backed automation. This comes in the next part of the series.

A few definitions

  • Test Driver is a software component or test tool that can control the component or system under test.
  • Test Environment is the hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test.

In our setup, Robot Framework acts as the test driver which is placed in the test environment.

As described in the companion article we distinguish various types of data that our test may be using:

Example Variable Variable type Type description
browser, host operational data describe the context in which the tests will be executed: which environment, which proxy, what browser, what locale, etc.
login-page-url, main-site-url, username-field, password-field, login-button structural data describe the structure of application, usually used to describe User Interfaces
valid-username, valid-password actual test data describe the data that actually differs on case-to-case basis. This is what you think differentiates specific test cases and groups SUT behaviors

Test Case Definition

We had finished our last work with a plan to test this testers-friendly website and a rather unimpressive test case:

*** Test Cases ***
This test case does nothing
    #This is a comment.
    #Remember to indent the test case content with at least two spaces
    No operation

There are a couple of things that we should do

  • Convert it to a test case that at least describes an actual workflow (what to test)
  • Create scaffolds of keywords that describe how to achieve the result (how to test)
  • Extract parameters from this test case so that we can easily manage and extend it

In the end we will have implemented a (not yet fully functional) test case that does the following:

  1. Opens web page http://automationpractice.com/index.php
  2. Searches for fixed term dress
  3. Checks the name of the first search result item
  4. Opens the first result from the search results
  5. Checks that a result pages opens
  6. Checks that the breadcrumb contains the same name as the first search result

Let’s write this up as robot framework test case:

*** Test Cases ***
Search and verify result breadcrumb
    Open web page http://automationpractice.com/index.php
    Search for "dress"
    Remember title of 1st search result item
    Open 1st search result
    Verify that a product page is now open
    Verify that the breadcrumb contains the same name as the 1st search result title

This shows us a big advantage of Robot Framework - the test case is an actual plain English description of the operations to be performed. At this stage it fails to execute correctly, because we have instructed Robot Framework what is to be done, but not how to do it.

To run the test, type the following into your bash console:

./run.sh

Example results:

==============================================================================
Tests                                                                         
==============================================================================
Tests.Initial Test Suite                                                      
==============================================================================
Search and verify result breadcrumb                                   | FAIL |
No keyword with name 'Open web page http://automationpractice.com/index.php' found.
------------------------------------------------------------------------------
Tests.Initial Test Suite                                              | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Tests                                                                 | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html

Initial keywords definitions

Let’s fix it by providing (still empty) keyword definitions. A keyword is the plain English instruction that we want to describe in detail - we’ve used them to tell Robot Framework about our test case.

To define a new keyword we introduce a new section in our test file: *** Keywords ***. In this section, every line starting without indentation is a definition of a new keyword. Every indented line afterwards describes part of the keyword. For now, we can simply use the built-in placeholder: No operation.

*** Test Cases ***
Search and verify result breadcrumb
    Open web page http://automationpractice.com/index.php
    Search for "dress"
    Remember title of 1st search result item
    Open 1st search result
    Verify that a product page is now open
    Verify that the breadcrumb contains the same name as the 1st search result title
 
*** Keywords ***
Open web page http://automationpractice.com/index.php
    [Tags]  not yet ready
    No operation
 
Search for "dress"
    [Tags]  not yet ready
    No operation
 
Remember title of 1st search result item
    [Tags]  not yet ready
    [Documentation]  Extract a value from the webpage and make it available in the remainder of the test
    No operation
 
Open 1st search result
    [Tags]  not yet ready
    No operation
 
Verify that a product page is now open
    [Tags]  not yet ready
    No operation
 
Verify that the breadcrumb contains the same name as the 1st search result title
    [Tags]  not yet ready
    No operation

Of course, after this change, the “test” passes:

==============================================================================
Tests                                                                         
==============================================================================
Tests.Initial Test Suite                                                      
==============================================================================
Search and verify result breadcrumb                                   | PASS |
------------------------------------------------------------------------------
Tests.Initial Test Suite                                              | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html

(Remember: in this part of the series we’re only parameterizing the test and making sure that we can execute it in various ways)

Parameterizing simple keywords

A very nice feature of Robot Framework allows you to embed placeholders in the names of the keywords, which leads to dynamic discovery of applicable keywords. For example, consider the URL address that needs to be opened at the beginning of the test. It may seem fixed - at least in our example. Under usual circumstances you wouldn’t be testing a production system. Instead you would have one or more test systems, maybe even a local deployment, so this keyword simply cannot have the URL embedded in it. Let’s write it with a placeholder instead:

Open web page ${url}
    [Tags] not yet ready
    Log  ${url}

After running our test suite now, and investigating the results/log.html file, you will find that the value has indeed been placed correctly:

Robot log includes the variable value

This will allow us to call it with any URL. And as a matter of fact, we should also immediately tell Robot Framework, that the URL is not fixed:

*** Test Cases ***
Search and verify result breadcrumb
    Open web page ${SUT.baseurl}/index.php
    Search for "dress"
    Remember title of 1st search result item
    Open 1st search result
    Verify that a product page is now open
    Verify that the breadcrumb contains the same name as the 1st search result title

We could go even further, and use a placeholder to semantically describe the page we want to open:

*** Variables ***
${main page}  ${SUT.baseurl}/index.php

*** Test Cases ***
Search and verify result breadcrumb
    Open web page ${main page}
    Search for "dress"
    Remember title of 1st search result item
    Open 1st search result
    Verify that a product page is now open
    Verify that the breadcrumb contains the same name as the 1st search result title

How will we inform the Robot Framework what the ${SUT.baseurl} is? I’d wager that using an environment-specific config file is the way to go. We’ll create two: env/dev.yaml and env/prod.yaml, for our development and production environments correspondingly.

env/dev.yaml

SUT:
  baseurl: http://localhost:8080

env/prod.yaml

SUT:
  baseurl: http://automationpractice.com

Adjusting the runner file to include the environment description

Now that we have some environment-specific configuration files, we will slightly adjust the runner file. First of all, we’ll tell Docker to just use the whole working directory as the /home/robot directory, so that all future variable files will be automatically included. Second of all, we’ll tell Robot Framework to use a specific environment description file:

#!/usr/bin/env bash

docker run \
  --rm \
  --volume "$PWD":/home/robot/ \
  robotframework/rfdocker:latest \
  --variablefile "env/$ENV.yaml" \
  tests

To run it now, we need to pass the information about which environment the tests are running against. You can do it by either setting environment variable in your system (CI service usually does this to some capacity), or, when testing things manually, you can just prefix your command with the environment variable:

ENV=dev ./run.sh

This will now load the content of ./env/dev.yaml file, execute the tests and… fail:

[ ERROR ] Processing variable file '/home/robot/env/dev.yaml' failed: Using YAML variable files requires PyYAML module to be installed. Typically you can install it by running `pip install pyyaml`.
==============================================================================
Tests                                                                         
==============================================================================
[ ERROR ] Error in file '/home/robot/tests/initial_test_suite.robot': Setting variable '${main page}' failed: Resolving variable '${SUT.baseurl}' failed: Variable '${SUT}' not found.
Tests.Initial Test Suite                                                      
==============================================================================
Search and verify result breadcrumb                                   | FAIL |
Variable '${main page}' not found.
------------------------------------------------------------------------------
Tests.Initial Test Suite                                              | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Tests                                                                 | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html

We now lack support of the YAML format in our docker runner

Add support for YAML to Robot Framework

According to documentation, we need to somehow add custom module to our docker image. This is a good thing, we’ll need it anyway in the future, to add Selenium support. So let’s get down to it. We need to:

  • build off of the official robotframework image
  • install some additional requirements (this is usually done using a requirements.txt file and pip command)
  • build our docker image
  • use the newly created docker image instead of official one

We will create a Dockerfile to describe our changes, and requirements.txt to describe our - duh - requirements:

requirements.txt

To learn more about this format, refer to pip documentation:

pyyaml

Dockerfile

FROM robotframework/rfdocker:latest
COPY requirements.txt /requirements.txt
RUN pip install -r /requirements.txt

run.sh

Here we’re telling docker to first take instructions from Dockerfile and convert them into a functioning image under name customrf - and then we’re using this custom image to execute our tests. For more information on building Docker images, see here.

#!/usr/bin/env bash

docker build -t customrf .

docker run \
  --rm \
  --volume "$PWD":/home/robot/ \
  customrf:latest \
  --variablefile "env/$ENV.yaml" \
  tests

Execute tests

Nothing changes in our execution, but the results are completely different:

ENV=dev

ENV=dev ./run.sh

Sending build context to Docker daemon  612.4kB
Step 1/3 : FROM robotframework/rfdocker:latest
 ---> 78d04f0de2cd
Step 2/3 : COPY requirements.txt /requirements.txt
 ---> Using cache
 ---> 87a607257be2
Step 3/3 : RUN pip install -r /requirements.txt
 ---> Using cache
 ---> 40ef6e53a35e
Successfully built 40ef6e53a35e
Successfully tagged customrf:latest
==============================================================================
Tests                                                                         
==============================================================================
Tests.Initial Test Suite                                                      
==============================================================================
Search and verify result breadcrumb                                   | PASS |
------------------------------------------------------------------------------
Tests.Initial Test Suite                                              | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html
Variable resolving to localhost

ENV=prod

ENV=prod ./run.sh

Sending build context to Docker daemon  612.4kB
Step 1/3 : FROM robotframework/rfdocker:latest
 ---> 78d04f0de2cd
Step 2/3 : COPY requirements.txt /requirements.txt
 ---> Using cache
 ---> 87a607257be2
Step 3/3 : RUN pip install -r /requirements.txt
 ---> Using cache
 ---> 40ef6e53a35e
Successfully built 40ef6e53a35e
Successfully tagged customrf:latest
==============================================================================
Tests                                                                         
==============================================================================
Tests.Initial Test Suite                                                      
==============================================================================
Search and verify result breadcrumb                                   | PASS |
------------------------------------------------------------------------------
Tests.Initial Test Suite                                              | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Tests                                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html
Variable resolving to actual production value

Adding more semantic variables to keywords

We now can repeat the process of parameterizing keywords to enhance our other keyword: the one that enters a search criterion.

Search for "dress"
    [Tags] not yet ready
    No operation

What is “dress” in this case? It’s a search term, so we’ll call it that:

Search for "${search term}"
    [Tags]  not yet ready
    Log  ${search term}

The quote characters in this case are not required, but they are a useful trick. They provide a good visual (and programmatic) discriminator for the boundaries of the variable. The way Robot Framework tries to identify what’s a variable in the name, and what isn’t, it runs into risk of overshooting with the matching mechanism. Imagine, that you didn’t use the quotes, and later added a keyword Search for ${search term} using the search bar at the bottom of the page. Robot Framework will get confused about which keyword to use, hence the quotes that help him decide.

As for now, that’s all that we’ll do with this keyword.

Parameterizing keywords with complex placeholders

Take notice of the keyword Remember title of 1st search result item. Number 1 is an obvious candidate for parameterization, but it is not that simple. You cannot just skip -st, because you run into risk of ambiguity: Remember title of 1 search result item - is it “one search result” or “first search result”?

We can change it to Remember title of search result item number 1. It will increase readability of the test case, but there’s also more that we can achieve: we can use initial input data validation to tell Robot Framework “Hey, only numbers here, please”:

...

*** Test Cases ***
Search and verify result breadcrumb
    Open web page ${main page}
    Search for "dress"
    Remember title of search result item number 1
    Open 1. search result
    Verify that a product page is now open
    Verify that the breadcrumb contains the same product name as the search result no. 1 title    
*** Keywords ***

...

Remember title of search result item number ${number:\d+}
    [Tags]  not yet ready
    [Documentation]  Extract a value from the webpage and make it available in the remainder of the test
    Log  ${number}
 
Open search result number ${number:\d+}
    [Tags]  not yet ready
    Log  ${number}
 
Verify that the breadcrumb contains the same product name as the title of search result number ${number:\d+}
    [Tags]  not yet ready
    Log  ${number}
...

This should still run, and we should nicely retrieve ${number} from the test case:

Structuring of resources

Robot Framework has a method of structuring the keywords for easy reuse. Namely, you can provide a resource file which contains keyword definitions, and include it in multiple test suites. It is a good idea to follow modularization principles and provide rather specifically tailored resources.

We will now split the operations available in certain pages to page-oriented resources. First, let’s create a proper directory structure (don’t mind the components subdirectory, we’ll get there with time):

#> tree resources
resources
├── components
└── pages
    ├── main_page.robot
    ├── product_page.robot
    └── search_results.robot

2 directories, 3 files

Resource pages are simply robot framework files that lack the Test Cases section. Now we will split our existing keywords into specific resources:

resources/pages/main_page.robot

*** Keywords ***
Search for "${search term}"
    [Tags]  not yet ready
    Log  ${search term}

resources/pages/search_results.robot

*** Keywords ***
Remember title of search result item number ${number:\d+}
    [Tags]  not yet ready
    [Documentation]  Extract a value from the webpage and make it available in the remainder of the test
    Log  ${number}

Open search result number ${number:\d+}
    [Tags]  not yet ready
    Log  ${number}

resources/pages/product_page.robot

*** Keywords ***
Verify that the breadcrumb contains the same product name as the title of search result number ${number:\d+}
    [Tags]  not yet ready
    Log  ${number}

Verify that a product page is now open
    [Tags]  not yet ready
    No operation

Putting it together

In order to tell Robot Framework that he has to include additional directories in Library search path, we need to include additional line --pythonpath "resources/" in our runner script:

#!/usr/bin/env bash

docker build -t customrf .

docker run \
  --rm \
  --volume "$PWD":/home/robot/ \
  customrf:latest \
  --variablefile "env/$ENV.yaml" \
  --pythonpath "resources/" \
  tests

And we have to tell the test suite where to look for all the keywords that we have just removed from our test suite. We’ll do it by providing a Settings section in the test suite file:

*** Settings ***
Resource  pages/main_page.robot
Resource  pages/product_page.robot
Resource  pages/search_results.robot
...

This is enough for our test suite to be passable, but there’s one trick that I had learned from one of the Robot Framework experts, which definitely improves the readability of the test suite. The trick is to include information about the source of the keyword in the line where it is used. If we include this trick, we’ll get the following test suite file:

*** Settings ***
Resource  pages/main_page.robot
Resource  pages/product_page.robot
Resource  pages/search_results.robot

*** Variables ***
${main page}  ${SUT.baseurl}/index.php

*** Test Cases ***
Search and verify result breadcrumb
    Open web page ${main page}
    main_page.Search for "dress"
    search_results.Remember title of search result item number 1
    search_results.Open search result number 1
    product_page.Verify that a product page is now open
    product_page.Verify that the breadcrumb contains the same product name as the title of search result number 1

*** Keywords ***
Open web page ${url}
    [Tags]  not yet ready
    Log  ${url}

After running the test suite now, we’ll get an additional perk in the log.html - all the resources will be included in the log:

Setting test metadata from environment variable

Since we’re already at defining Settings of the test suite, how about learning how to access environment variables directly in Robot Framework? We can do this to actually provide a bit of useful information for our test, namely shorten the time needed to find out in which environment the test had been executed. Let’s add the following line to our settings:

*** Settings ***
Metadata  Environment  %{ENV}
...

Note the % instead of $ before in the variable name. It means that the variable comes not from Robot Framework, but from the environment. We do however run into a pickle here: since our runner is a Docker container, it does not automatically inherit the shell’s environment. We have to explicitly tell Docker to set specific variables, and then Robot Framework will be able to access them.

This was not the case when we were telling it which environment configuration file to use - that had been resolved at the run.sh runtime, so Docker got told to use specifically --variablefile "env/prod.yaml".

We can tell Docker the value of environment variable to be passed into the container, by adding the following line in the run.sh:

-e ENV="$ENV"

The run.sh now looks like this:

#!/usr/bin/env bash

docker build -t customrf .

docker run \
  --rm \
  --volume "$PWD":/home/robot/ \
  -e ENV="$ENV" \
  customrf:latest \
  --variablefile "env/$ENV.yaml" \
  --pythonpath "resources/" \
  tests

This produces the following bit of metadata in the log file:

Test Suite metadata example

Summary

At this moment we have the following:

  • Custom Docker-based test runner (Dockerfile)
  • Way to install additional modules to the test runner (requirements.txt)
  • Specific test case
  • Environment-specific operational parameterization via a combination of environment variables and yaml-files (env/*.yaml)
  • Custom keywords and parameterized keywords
  • Resources for keywords reuse
  • Reporting of metadata and access of environment variables

What’s coming next

In the next part of the series, we’ll:

  • Finally get some actual verification done!
  • Bind Selenium to our test script
  • Add Setup and Teardown sequences to the test data
  • Finalize the page objects by using additional yaml files
If you've enjoyed this content, how about