A Synthetic User Monitoring (SUM) Selenium test is a Selenium IDE (SIDE) script that is potentially enriched with SUM-specific annotations.
This page does not intend to explain in detail how to create a Selenium IDE test. (Plenty of documentation can be found on the internet about the ways to create an efficient Selenium IDE test.)
It intends to explain how such a test is integrated with the SUM data model.
The easiest method of creating a SUM Selenium test scripts is:
Please note: Websites containing Shadow DOM structures are not supported by the Selenium IDE. Since we need the Selenium IDE to create a SIDE file for SUM, we also do not support the monitoring of websites containing Shadow DOM structures. However, in some cases, workarounds to create Selenium IDE scripts exist.
When you create a Selenium script, you may want to have some Selenium commands value/target set from SUM variables. This is typically the case for user names or sensitive data like passwords.
For that, you can add a SUM variable annotation in the "Description" field of the Selenium command.
Such a description is ignored by Selenium. However, when the SIDE file is imported into SUM, the content is parsed and the annotations are detected.
SUM supports four types of variable annotations:
The following character can be used for variable names:
During the SIDE file import, SUM will create the corresponding variables in the SUM script.
The created variable will be of the following types:
The SUM variable default value is the value of the "value" field of the Selenium command in Selenium IDE.
IMPORTANT: For the secured variables, do not keep the sensitive value in the command value field. (It would be stored in clear.) Replace it with a dummy value.
If the same variable annotation is declared several times, only one variable is created. Its default value are those of the last declaration. You cannot use the same variable name for both a standard and a secured variable. If your SIDE contains such a situation, its import will fail.
If the Selenium command is disabled, the annotation is ignored.
Please note: If the SIDE file is re-imported and a variable annotation is changed or removed, the SUM script will be updated but the deprecated variable will not be removed from the Script variables.
When a scenario is created for the SUM script, it will inherit all the variables of the script. These variables are listed in the SUM Configuration, in the Variable section of the scenario details.
IMPORTANT: It is required to set the value of the secured variables. The default value should not be the correct one.
If a command has a variable annotation, SUM replaces its value with the value of the SUM variable before the command execution.
You can change the variable value during the scenario configuration. This is described on page Synthetic User Monitoring - Setup & Configuration
SUM steps are a concept that only exists in the SUM scripts but not in the Selenium test script. The reason to create SUM steps is to gain more insight into the execution and performance of your SUM script.
SUM captures its monitoring metrics at the step level.
By defining steps, you can "group" the Selenium test script commands that are related to the same activity. This way, you can better evaluate which activity creates a problem.
For instance, imagine a monitoring script executing the following commands:
With SUM steps you can group these commands in logical steps:
There are two ways to define SUM steps in a Selenium test script:
Adding SUM Steps as Selenium Tests in a Test Suite
During the SIDE import, a SUM step is created for each test added to the test suite.
The step starts with the first command of the test and ends with the first of the following events:
The created step name is the name of the test.
The created step is identified by the identifier of the test script plus the GUID of the Selenium test.
Adding SUM Steps using the SUM step annotation
To explicitly define a new step, you can add a SUM step annotation in the "Description" field of a Selenium command.
Such a description is ignored by Selenium. However, when the SIDE file is imported into SUM, the content is parsed and the annotations are detected.
Syntax
You can use the following characters for the step name:
Behavior
The step starts with the command containing the step annotation.
It ends with the first of the following events:
The created step is identified by the identifier of the test script plus the GUID of the Selenium command with the step annotation.
When you create a Selenium script, you may want to instruct SUM to take a screenshot after a specific Selenium command. This is an "explicit screenshot".
For that, you can add a SUM screenshot annotation in the "Description" field of the Selenium command.
Such a description is ignored by Selenium.
@sap.sum.screenshot
This annotation does not take parameter.
During the SIDE file import, the screenshot annotations have no impact.
When a scenario is created for the SUM script, the screenshot annotations have no direct impact. However, in the "Selenium" section of the created scenario, a dedicated property gives the possibility to enable/disable the explicit screenshots capture.
After the execution of a command with the explicit screenshot annotation, if the explicit screenshot capture is enabled in the Scenario Selenium configuration, SUM takes a screenshot of the monitored application UI.
When you create a Selenium script, some commands may perform sensitive actions for which you want to ensure that SUM:
Even in case of Scenario execution error.
For that you can add a SUM sensitive annotation in the "Description" field of the Selenium command.
@sap.sum.sensitive.action → considers the current command only.
@sap.sum.sensitive.step → considers all the commands in the current step (the annotation can be declared on any active command of the step)
These annotations do not take parameters.
During the SIDE file import, the sensitive annotations have no impact.
When a scenario is created for the SUM script, the sensitive annotations have no impact.
When SUM executes a Selenium command considered sensitive:
Here is a set of recommendations to keep in mind when you design an SAP Cloud ALM – Synthetic User Monitoring (SUM) scenario.
SUM is not intended for application testing or for task automation.
Complex scenarios are costly to create and maintain. It is better to have 2–3 simple scenarios rather than 1 complex one.
The execution of the SUM scenario should not negatively impact the monitored application.
Try to anticipate what could be the impact of not fully executing a scenario through to the end (e.g. in the event of availability issues).
For instance, monitoring a procurement application, a scenario booking actual items, and saving a booking could jeopardize the production system.
For such monitoring, we recommend that you manipulate fake elements.
The SUM infrastructure is not designed to manipulate sensitive or personal user information within the created scenarios.
For instance, when you monitor a sales application, you should not navigate through real sales operations. Especially if this implies navigating through personal or sensitive information related to the sales operations (such as names, credit card numbers, or account balances).
If the scenario requires authentication when accessing the application to be monitored, do not connect with an actual end-user account.
We strongly recommend that you use specifically created monitoring users with no associated personal or sensitive data.
In the context of Synthetic User Monitoring (SUM) CRUN, the following Selenium IDE commands are supported:
Please note:
If an imported Selenium IDE script contains unsupported commands, no error is displayed at the time of import.
However, at the time of execution, SUM will return a "command XXX not supported" error.
| Command Name | Description |
|---|---|
| add selection | Adds a selection to the set of options in a multi-select element. |
| select frame | Selects a frame within the current window. |
| select window | Selects a popup window using a window locator. Once a pop-up window has been selected, all commands will go to that window. |
| remove selection | Remove a selection from the set of selected options in a multi-selected element using an option locator. |
| select | Select an element from a drop-down menu using an option locator. Option locators provide different ways of specifying a select element. If no option locator prefix is provided, a match on the label will be attempted. |
| Command Name | Description |
|---|---|
| assert alert | Confirm that an alert has been rendered with the provided text. The test will stop if the assert fails. |
| assert | Check that a variable is an expected value. The variable's value will be converted to a string for comparison. The test will stop if the assert fails. |
| assert checked | Confirm that the target element has been checked. The test will stop if the assert fails. |
| assert confirmation | Confirm that a confirmation has been rendered. The test will stop if the assert fails. |
| assert editable | Confirm that the target element is editable. The test will stop if the assert fails. |
| assert element present | Confirm that the target element is present somewhere on the page. The test will stop if the assert fails. |
| assert element not present | Confirm that the target element is not present somewhere on the page. The test will stop if the assert fails. |
| assert not checked | Confirm that the target element has not been checked. The test will stop if the assert fails. |
| assert not editable | Confirm that the target element is not editable. The test will stop if the assert fails. |
| assert not selected value | Confirm that the value attribute of the selected option in a dropdown element does not contain the provided value. The test will stop if the assert fails. |
| assert not text | Confirm that the text of an element does not contain the provided value. The test will stop if the assert fails. |
| assert selected value | Confirm that the value attribute of the selected option in a dropdown element contains the provided value. The test will stop if the assert fails. |
| assert selected label | Confirm that the label attribute of the selected option in a dropdown element contains the provided value. The test will stop if the assert fails. |
| assert text | Confirm that the text of an element contains the provided value. The test will stop if the assert fails. |
| assert value | Confirm the value of an input field (or anything else with a value parameter). For checkbox/radio elements, the value will be "on" or "off" depending on whether the element is checked or not. The test will stop if the assert fails. |
| verify | Soft assert that a variable is an expected value. The variable's value will be converted to a string for comparison. The test will continue even if the verify fails. |
| verify checked | Soft asserts that a toggle button has been checked. The test will continue even if the verify fails. |
| verify editable | Soft assert whether the specified input element is editable. The test will continue even if the verify fails. |
| verify element present | Soft asserts that the specified element is somewhere on the page. The test will continue even if the verify fails. |
| verify element not present | Soft asserts that the specified element is not somewhere on the page. The test will continue even if the verify fails. |
| verify not checked | Confirm that the target element has not been checked. The test will stop if the assert fails. |
| verify not editable | Soft assert whether the specified input element is not editable. The test will continue even if the verify fails. |
| verify not selected value | Soft asserts that the expected element has not been chosen in a select menu by its option attribute. The test will continue even if the verify fails. |
| verify not text | Soft asserts the text of an element is not present. The test will continue even if the verify fails. |
| verify selected label | Soft assert the visible text for a selected option in the specified select element. The test will continue even if the verify fails. |
| verify selected value | Soft asserts that the expected element has been chosen in a select menu by its option attribute. The test will continue even if the verify fails. |
| verify text | Soft assert the text of an element is present. The test will continue even if the verify fails. |
| verify title | Soft asserts the title of the current page contains the provided text. The test will continue even if the verify fails. |
| verify value | Soft assert the value of an input field. For checkbox/radio elements the value will be "on" or "off" depending on whether the element is checked or not. The test will continue even if the verify fails. |
| assert prompt | Confirm that a JavaScript prompt has been rendered. The test will stop if the assert fails. |
| assert title | Confirm the title of the current page contains the provided text. The test will stop if the assert fails. |
| Command Name | Description |
|---|---|
| click | Click on a target element. |
| check | Check a toggle-button (checkbox/radio). |
| uncheck | Uncheck a toggle-button (checkbox/radio). |
| Command Name | Description |
|---|---|
| store | Save a target string as a variable for easy re-use. |
| store text | Gets the text of an element and stores it for later use. This works for any element that contains text. |
| store window handle | Store the handle of the current page. |
| store value | Store the value of an element. |
| store title | Store the title of the current page. |
| store attribute | Store and get the value of an element attribute. |
| store xpath count | Store the number of nodes that match the specified xpath. |
| Command Name | Description |
|---|---|
| wait for element present | Wait for a target element to be present on the page. |
| wait for element editable | Wait for an element to be editable. |
| wait for element not editable | Wait for an element to not be editable. |
| wait for element not present | Wait for a target element to be not present on the page. |
| wait for element not visible | Wait for a target element to not be visible on the page. |
| wait for element visible | Wait for a target element to be visible on the page. |
| wait for text | Wait for a target text to be visible on a specified element on the page. |
| pause | Wait for the specified amount of time. |
| set speed | Set execution speed. By default, there is no such delay, e.g., the delay is 0 milliseconds. This setting is global and will affect all test runs until changed. |
| Command Name | Description |
|---|---|
| open | Opens a URL and waits for the page to load before proceeding. This accept both absolute and relative URLs. |
| send keys | Simulates keystroke events on the specified element, as though you typed the value key-by-key. This simulates a real user typing every character in the specified string; it is also bound by the limitations of a real user, like not being able to type into a invisible or read only elements. |
| type | Sets the value of an input field, as though you typed it in. Can also be used to set the value of combo boxes, check boxes, etc. In these cases, value should be the value of the option selected, not the visible text. |
| set window size | Set the browser's window size, including the browser's interface. |
| run | Run a test case from the current project. |
| run script | Creates a new "script" tag in the body of the current test window, and adds the specified text into the body of the command. Beware that JS exceptions thrown in these script tags aren't managed by Selenium, so you should probably wrap your script in try/catch blocks if there is any chance that the script will throw an exception. |
| execute script | Executes a snippet of JavaScript in the context of the currently selected frame or window. The script fragment will be executed as the body of an anonymous function. To store the return value, use the 'return' keyword and provide a variable name in the value input field. |
| execute async script | Executes an async snippet of JavaScript in the context of the currently selected frame or window. The script fragment will be executed as the body of an anonymous function and must return a Promise. The Promise result will be saved on the variable if you use the 'return' keyword. |
| edit content | Sets the value of a content editable element as if you typed in it. |
| echo | Prints the specified message into the third table cell in your Selenese tables. |
| webdriver answer on visible prompt | Affects a currently showing alert prompt. This command instructs Selenium to provide the specified answer to it. |
| webdriver choose cancel on visible confirmation | Affects a currently showing confirmation alert. This command instructs Selenium to cancel it. |
| webdriver choose cancel on visible prompt | Affects a currently showing alert prompt. This command instructs Selenium to cancel it. |
| webdriver choose ok on visible confirmation | Affects a currently showing confirmation alert. This command instructs Selenium to accept it. |
| Command Name | Description |
|---|---|
| if | This is the opening command for a conditional block. Along with it, you provide a JavaScript expression you would like to evaluate. This can consist of variables created from prior JavaScript expressions in your test. This all goes in the target input field of the if command. If the expression evaluates to true then the test will execute the commands that follow it up until the next conditional control flow command (e.g., else if, else, or end). If the expression evaluates to false it will skip the commands that follow and jump to the next relevant conditional control flow command (e.g., else if, else, or end). |
| elseIf | This command is used within an if command block. Just like with if it takes a JavaScript expression in the target input field to evaluate, executing either the command branch that follows it, or skips to the next relevant control flow command (e.g., else or end). |
| else | Else is the final condition you can have in an if block. When none of the prior conditions are met, this command branch will be executed. After it's done it will jump to the end command. |
| end | This command terminates the conditional command block. Without it the command block is incomplete and you'll receive a helpful error message letting you know when trying to run your test. |