Creating Synthetic User Monitoring Scripts

A Synthetic User Monitoring (SUM) Selenium test is a Selenium IDE (SIDE) script that is potentially enriched with SUM-specific annotations.

This page does not intend to explain in detail how to create a Selenium IDE test. (Plenty of documentation can be found on the internet about the ways to create an efficient Selenium IDE test.)

It intends to explain how such a test is integrated with the SUM data model.

Create Selenium Test Scripts for Synthetic User Monitoring

The easiest method of creating a SUM Selenium test scripts is:

  1. Use Selenium IDE to record the main sequence of your scenario. The IDE is a browser plug-in available for numerous different browsers. You can find the documentation for the Selenium IDE here.
  2. In Selenium IDE, replay the created script and make the necessary adjustments. For instance:
    • Adjust the "Target" fields of commands to ensure that they properly identify the targeted elements. 
    • Add "wait for xxx" commands to wait for the UI rendering.
    • Add "assert xxx" and "verify xxx" commands to ensure that the monitored application displays the expected information.
  3. Make sure your Selenium IDE script only contains commands that are supported by SUM. You find the full list at the end of this page.
  4. When your Selenium IDE script is ready, add the SUM-specific annotations (as described in the next sections).
    For instance:
    • Add SUM secured variable annotations to store sensitive command values (e.g. a password) in SUM secured variables.
    • Add SUM variable annotations to store command values (not sensitive) in SUM variables.
    • Add SUM step annotations to group script commands in SUM steps.
  5. For each command with a sensitive value (e.g. password):
    1. Replace the values with a dummy text (e.g. "dummy").
    2. Make sure that you added a SUM secured variable. At execution time, SUM will replace the command value with the value of the SUM secured variable.
  6. Save your Selenium IDE script. This will create a SIDE file that you can import into SUM to create the SUM Selenium script.

Please note: Websites containing Shadow DOM structures are not supported by the Selenium IDE. Since we need the Selenium IDE to create a SIDE file for SUM, we also do not support the monitoring of websites containing Shadow DOM structures. However, in some cases, workarounds to create Selenium IDE scripts exist.

Define SUM Variables in Selenium Tests

When you create a Selenium script, you may want to have some Selenium commands value/target set from SUM variables. This is typically the case for user names or sensitive data like passwords.

For that, you can add a SUM variable annotation in the "Description" field of the Selenium command.

Such a description is ignored by Selenium. However, when the SIDE file is imported into SUM, the content is parsed and the annotations are detected.

Syntax

SUM supports four types of variable annotations:

  • @sap.sum.variable=<variable_name> – to have the command value field replaced by the specified SUM standard variable value.
  • @sap.sum.secure.variable=<variable_name> – to have the command value field replaced by the specified SUM secured variable value.
  • @sap.sum.variable.target=<variable_name> – to have the command target field replaced by the specified SUM standard variable value.
  • @sap.sum.secure.variable.target=<variable_name> – to have the command target field replaced by the specified SUM secured variable value.

The following character can be used for variable names:

  • Alphanumeric ([a-z], [A-Z], [0-9])
  • Underscore ('_')

Behavior during SIDE file import

During the SIDE file import, SUM will create the corresponding variables in the SUM script.

The created variable will be of the following types:

  • String for the @sap.sum.variable and @sap.sum.variable.target annotations.
  • Secured for the @sap.sum.secure.variable and @sap.sum.secure.variable.target annotations.

The SUM variable default value is the value of the "value" field of the Selenium command in Selenium IDE.

IMPORTANT: For the secured variables, do not keep the sensitive value in the command value field. (It would be stored in clear.) Replace it with a dummy value.

If the same variable annotation is declared several times, only one variable is created. Its default value are those of the last declaration. You cannot use the same variable name for both a standard and a secured variable. If your SIDE contains such a situation, its import will fail. 

If the Selenium command is disabled, the annotation is ignored.

Please note: If the SIDE file is re-imported and a variable annotation is changed or removed, the SUM script will be updated but the deprecated variable will not be removed from the Script variables.

Behavior during the SUM Scenario Configuration

When a scenario is created for the SUM script, it will inherit all the variables of the script. These variables are listed in the SUM Configuration, in the Variable section of the scenario details.

IMPORTANT: It is required to set the value of the secured variables. The default value should not be the correct one.

Behavior at Runtime

If a command has a variable annotation, SUM replaces its value with the value of the SUM variable before the command execution.

You can change the variable value during the scenario configuration. This is described on page Synthetic User Monitoring - Setup & Configuration

Add SUM Steps to Selenium Tests

Motivation

SUM steps are a concept that only exists in the SUM scripts but not in the Selenium test script. The reason to create SUM steps is to gain more insight into the execution and performance of your SUM script.

SUM captures its monitoring metrics at the step level.

By defining steps, you can "group" the Selenium test script commands that are related to the same activity. This way, you can better evaluate which activity creates a problem.

For instance, imagine a monitoring script executing the following commands:

  1. Open an application URL
  2. Wait for the UI to display the logon screen
  3. Enter a user credentials to log in the application
  4. Wait for the UI to load and display a list of elements
  5. Use a search functionality to find something
  6. Wait for the search to return a result
  7. Click the search result
  8. Wait for the UI to display the result details
  9. Logout
  10. Wait for the confirmation that the session is now closed

With SUM steps you can group these commands in logical steps:

  • Open Website = grouping all the Selenium commands 1 and 2
  • Logon =  grouping the Selenium commands 3 and 4
  • Search = grouping the Selenium commands 5 and 6
  • Show Details = grouping the Selenium commands 7 and 8
  • Logout = grouping the Selenium commands 9 and 10

Create SUM Steps

There are two ways to define SUM steps in a Selenium test script:

  • By adding a Selenium test in the Selenium test suite
  • By adding the SUM step annotation in the description field of a command in a Selenium test script

Adding SUM Steps as Selenium Tests in a Test Suite

During the SIDE import, a SUM step is created for each test added to the test suite.

The step starts with the first command of the test and ends with the first of the following events:

  • A SUM step annotation creates a new step.
  • The end of the Selenium test.

The created step name is the name of the test.

The created step is identified by the identifier of the test script plus the GUID of the Selenium test.

Adding SUM Steps using the SUM step annotation

To explicitly define a new step, you can add a SUM step annotation in the "Description" field of a Selenium command.

Such a description is ignored by Selenium. However, when the SIDE file is imported into SUM, the content is parsed and the annotations are detected.

  • If the Selenium command is disabled, the annotation is ignored.
  • If the Selenium command is in a nested test (a test called via the "run" command), the annotation is ignored.
  • If the Selenium command is in a conditional block (IF, ELSEIF, ELSE), the annotation is ignored.

Syntax

  • @sap.sum.step=<step_name>
  • @sap.sum.step="<step name with space>"

You can use the following characters for the step name:

  • Alphanumeric ([a-z], [A-Z], [0-9])
  • Underscore ('_')
  • Space if name is enclosed with " "

Behavior

The step starts with the command containing the step annotation.

It ends with the first of the following events:

  • A SUM step annotation creates a new step.
  • The end of the test.

The created step is identified by the identifier of the test script plus the GUID of the Selenium command with the step annotation.

Add an Explicit Screenshot

When you create a Selenium script, you may want to instruct SUM to take a screenshot after a specific Selenium command. This is an "explicit screenshot".

For that, you can add a SUM screenshot annotation in the "Description" field of the Selenium command.

Such a description is ignored by Selenium.
 

Syntax

@sap.sum.screenshot

This annotation does not take parameter.


Behavior during the import

During the SIDE file import, the screenshot annotations have no impact.


Behavior during the SUM Scenario configuration

When a scenario is created for the SUM script, the screenshot annotations have no direct impact. However, in the "Selenium" section of the created scenario, a dedicated property gives the possibility to enable/disable the explicit screenshots capture.


Behavior at runtime

After the execution of a command with the explicit screenshot annotation, if the explicit screenshot capture is enabled in the Scenario Selenium configuration, SUM takes a screenshot of the monitored application UI.

Indicate a Sensitive Scenario Command or Step

When you create a Selenium script, some commands may perform sensitive actions for which you want to ensure that SUM:

  • Never takes a screenshot of the monitored UI
  • Never displays the command "target" and "value"

Even in case of Scenario execution error.

For that you can add a SUM sensitive annotation in the "Description" field of the Selenium command.


Syntax

@sap.sum.sensitive.action → considers the current command only.

@sap.sum.sensitive.step → considers all the commands in the current step (the annotation can be declared on any active command of the step)

These annotations do not take parameters.


Behavior During the Import

During the SIDE file import, the sensitive annotations have no impact.


Behavior During the SUM Scenario Configuration

When a scenario is created for the SUM script, the sensitive annotations have no impact.


Behavior at Runtime

When SUM executes a Selenium command considered sensitive:

  • No screenshot is taken after the command, e.g.
    • With the scenario configured to automatically take screenshots on error, if the command fails, no automatic screenshot will be taken for the command.
    • With the scenario configured to automatically take screenshots after each step, if the command is the last of the step, no screenshot will be taken at the end of the step.
  • The command "target" and "Value" are obfuscated in the execution results. E.g. if the command execution fails, the SUM UI will not indicate the "target" or the "value" of the command

 

Best Practices for Synthetic User Monitoring Script Creation

Here is a set of recommendations to keep in mind when you design an SAP Cloud ALM – Synthetic User Monitoring (SUM) scenario.

Synthetic User Monitoring is a monitoring application

SUM is not intended for application testing or for task automation.

Keep your scenario simple

Complex scenarios are costly to create and maintain. It is better to have 2–3 simple scenarios rather than 1 complex one.

Do not create "disruptive" scenarios

The execution of the SUM scenario should not negatively impact the monitored application.

Try to anticipate what could be the impact of not fully executing a scenario through to the end (e.g. in the event of availability issues).

For instance, monitoring a procurement application, a scenario booking actual items, and saving a booking could jeopardize the production system.

  • What happens if the scenario ends unexpectedly, leaving a fake booking record?
  • What happens if the scenario is executed too often, creating a shortage of booked items for the actual consumers?

For such monitoring, we recommend that you manipulate fake elements.

Do not manipulate actual user data

The SUM infrastructure is not designed to manipulate sensitive or personal user information within the created scenarios.

For instance, when you monitor a sales application, you should not navigate through real sales operations. Especially if this implies navigating through personal or sensitive information related to the sales operations (such as names, credit card numbers, or account balances).

Do use monitoring users

If the scenario requires authentication when accessing the application to be monitored, do not connect with an actual end-user account. 

We strongly recommend that you use specifically created monitoring users with no associated personal or sensitive data.

Selenium Commands Supported by Synthetic User Monitoring

In the context of Synthetic User Monitoring (SUM) CRUN, the following Selenium IDE commands are supported:

Please note:

If an imported Selenium IDE script contains unsupported commands, no error is displayed at the time of import.
However, at the time of execution, SUM will return a "command XXX not supported" error.

Command NameDescription
add selection Adds a selection to the set of options in a multi-select element.
select frameSelects a frame within the current window.
select windowSelects a popup window using a window locator. Once a pop-up window has been selected, all commands will go to that window.
remove selectionRemove a selection from the set of selected options in a multi-selected element using an option locator.
selectSelect an element from a drop-down menu using an option locator. Option locators provide different ways of specifying a select element. If no option locator prefix is provided, a match on the label will be attempted.

Command NameDescription
assert alert Confirm that an alert has been rendered with the provided text. The test will stop if the assert fails.
assertCheck that a variable is an expected value. The variable's value will be converted to a string for comparison. The test will stop if the assert fails.
assert checked Confirm that the target element has been checked. The test will stop if the assert fails.
assert confirmation Confirm that a confirmation has been rendered. The test will stop if the assert fails.
assert editableConfirm that the target element is editable. The test will stop if the assert fails.
assert element presentConfirm that the target element is present somewhere on the page. The test will stop if the assert fails.
assert element not presentConfirm that the target element is not present somewhere on the page. The test will stop if the assert fails.
assert not checked Confirm that the target element has not been checked. The test will stop if the assert fails.
assert not editableConfirm that the target element is not editable. The test will stop if the assert fails.
assert not selected valueConfirm that the value attribute of the selected option in a dropdown element does not contain the provided value. The test will stop if the assert fails.
assert not textConfirm that the text of an element does not contain the provided value. The test will stop if the assert fails.
assert selected valueConfirm that the value attribute of the selected option in a dropdown element contains the provided value. The test will stop if the assert fails.
assert selected labelConfirm that the label attribute of the selected option in a dropdown element contains the provided value. The test will stop if the assert fails.
assert textConfirm that the text of an element contains the provided value. The test will stop if the assert fails.
assert valueConfirm the value of an input field (or anything else with a value parameter). For checkbox/radio elements, the value will be "on" or "off" depending on whether the element is checked or not. The test will stop if the assert fails.
verifySoft assert that a variable is an expected value. The variable's value will be converted to a string for comparison. The test will continue even if the verify fails.
verify checkedSoft asserts that a toggle button has been checked. The test will continue even if the verify fails.
verify editableSoft assert whether the specified input element is editable. The test will continue even if the verify fails.
verify element presentSoft asserts that the specified element is somewhere on the page. The test will continue even if the verify fails.
verify element not presentSoft asserts that the specified element is not somewhere on the page. The test will continue even if the verify fails.
verify not checkedConfirm that the target element has not been checked. The test will stop if the assert fails.
verify not editableSoft assert whether the specified input element is not editable. The test will continue even if the verify fails.
verify not selected valueSoft asserts that the expected element has not been chosen in a select menu by its option attribute. The test will continue even if the verify fails.
verify not textSoft asserts the text of an element is not present. The test will continue even if the verify fails.
verify selected labelSoft assert the visible text for a selected option in the specified select element. The test will continue even if the verify fails.
verify selected valueSoft asserts that the expected element has been chosen in a select menu by its option attribute. The test will continue even if the verify fails.
verify textSoft assert the text of an element is present. The test will continue even if the verify fails.
verify titleSoft asserts the title of the current page contains the provided text. The test will continue even if the verify fails.
verify valueSoft assert the value of an input field. For checkbox/radio elements the value will be "on" or "off" depending on whether the element is checked or not. The test will continue even if the verify fails.
assert promptConfirm that a JavaScript prompt has been rendered. The test will stop if the assert fails.
assert titleConfirm the title of the current page contains the provided text. The test will stop if the assert fails.

 

Command NameDescription
clickClick on a target element.
checkCheck a toggle-button (checkbox/radio).
uncheckUncheck a toggle-button (checkbox/radio).

Command NameDescription
storeSave a target string as a variable for easy re-use.
store textGets the text of an element and stores it for later use. This works for any element that contains text.
store window handleStore the handle of the current page.
store valueStore the value of an element.
store titleStore the title of the current page.
store attributeStore and get the value of an element attribute.
store xpath countStore the number of nodes that match the specified xpath.

Command NameDescription
wait for element presentWait for a target element to be present on the page.
wait for element editableWait for an element to be editable.
wait for element not editableWait for an element to not be editable.
wait for element not presentWait for a target element to be not present on the page.
wait for element not visibleWait for a target element to not be visible on the page.
wait for element visibleWait for a target element to be visible on the page.
wait for textWait for a target text to be visible on a specified element on the page.
pauseWait for the specified amount of time.
set speedSet execution speed. By default, there is no such delay, e.g., the delay is 0 milliseconds. This setting is global and will affect all test runs until changed.

Command NameDescription
openOpens a URL and waits for the page to load before proceeding. This accept both absolute and relative URLs.
send keysSimulates keystroke events on the specified element, as though you typed the value key-by-key. This simulates a real user typing every character in the specified string; it is also bound by the limitations of a real user, like not being able to type into a invisible or read only elements.
typeSets the value of an input field, as though you typed it in. Can also be used to set the value of combo boxes, check boxes, etc. In these cases, value should be the value of the option selected, not the visible text. 
set window sizeSet the browser's window size, including the browser's interface.
runRun a test case from the current project.
run scriptCreates a new "script" tag in the body of the current test window, and adds the specified text into the body of the command. Beware that JS exceptions thrown in these script tags aren't managed by Selenium, so you should probably wrap your script in try/catch blocks if there is any chance that the script will throw an exception.
execute scriptExecutes a snippet of JavaScript in the context of the currently selected frame or window. The script fragment will be executed as the body of an anonymous function. To store the return value, use the 'return' keyword and provide a variable name in the value input field.
execute async scriptExecutes an async snippet of JavaScript in the context of the currently selected frame or window. The script fragment will be executed as the body of an anonymous function and must return a Promise. The Promise result will be saved on the variable if you use the 'return' keyword.
edit contentSets the value of a content editable element as if you typed in it.
echoPrints the specified message into the third table cell in your Selenese tables.
webdriver answer on visible promptAffects a currently showing alert prompt. This command instructs Selenium to provide the specified answer to it.
webdriver choose cancel on visible confirmationAffects a currently showing confirmation alert. This command instructs Selenium to cancel it.
webdriver choose cancel on visible promptAffects a currently showing alert prompt. This command instructs Selenium to cancel it.
webdriver choose ok on visible confirmationAffects a currently showing confirmation alert. This command instructs Selenium to accept it.

Command NameDescription
if

This is the opening command for a conditional block.

Along with it, you provide a JavaScript expression you would like to evaluate. This can consist of variables created from prior JavaScript expressions in your test. This all goes in the target input field of the if command.

If the expression evaluates to true then the test will execute the commands that follow it up until the next conditional control flow command (e.g., else if, else, or end).

If the expression evaluates to false it will skip the commands that follow and jump to the next relevant conditional control flow command (e.g., else if, else, or end).

elseIf

This command is used within an if command block.

Just like with if it takes a JavaScript expression in the target input field to evaluate, executing either the command branch that follows it, or skips to the next relevant control flow command (e.g., else or end).

else

Else is the final condition you can have in an if block. When none of the prior conditions are met, this command branch will be executed.

After it's done it will jump to the end command.

endThis command terminates the conditional command block. Without it the command block is incomplete and you'll receive a helpful error message letting you know when trying to run your test.