Contact Us
×
How you can contact us:

  • Technical Assistance

    Request product support from SAP

  • Non-Technical Assistance

    Request non-product support or provide feedback on SAP Support Portal site

  1. SAP Support Portal Home
  2. Application Lifecycle Management
  3. SAP Cloud ALM
  4. SAP Cloud ALM for Operations
  5. SAP Cloud ALM for Operations Expert Portal
  6. Synthetic User Monitoring

Synthetic User Monitoring - Setup & Configuration

Synthetic User Monitoring - Setup & Configuration

Synthetic User Monitoring (SUM) configuration is to be performed via the application itself.

The configuration involves three components:

  • Runners - to define and configure where the monitoring scenarios are executed.
  • Resources - to upload and handle the Selenium IDE files
  • Scenarios - to define and configure the monitoring scenarios

Information

  • Several monitoring scenarios can be created from a given Selenium IDE file.
  • Several scenarios can be associated to one runner.
  • One scenario can be associated to several runners.

Prerequisites

The set up Synthetic User Monitoring you need:

  • At least one Selenium remote webdriver infrastructure to configure the runners (You can find more information about Selenium remote webdriver here)
  • At least one Selenium SIDE file to configure the scenarios. (refer to Creating Synthetic User Monitoring Scripts)

Typical Setup Flow

The typical setup consists of four steps:
  1. Configure a new Runner: configure its endpoint to connect to your Selenium remote webdriver infrastructure
    (see Configure Runners section below).
  2. Import the SIDE file into SUM. This creates a new resource listing the scripts defined in the imported SIDE file
    (see Import Resources section below).
  3. Create and configure a new scenario (choose its name, indicate the corresponding script, ...,)
    (see Scenario Maintenance section below).
  4. Associate and activate the Scenario to your Runner. The SUM schedules the Scenario executions
    (see Activate Monitoring section below).

Configure Runners

The first step during the setup is the creation of a SUM Runner. A runner is the component in charge of the execution of the monitoring scenarios.

Every runner has a Runner Endpoint. A runner endpoint is the Selenium remote webdriver infrastructure that Synthetic User Monitoring (SUM) accesses to pass the scenario commands to execute.

It also hosts the web browser used to access the monitored applications.

To access the endpoint, SUM uses the connection information defined in the runner configuration. Two types of configuration are possible:

InternetThe endpoint is access via the HTTPS URL specified in the configuration.
On-premise
The endpoint is access via the SAP Cloud Connector information specified in the configuration
Important

For Internet type, only HTTPS URLs are supported.

A runner endpoint can be located in the cloud or on-premises.

To get a Selenium remote webdriver infrastructure, you have several possibilities:

  • Some companies are specialized in providing such infrastructure (SauceLabs, BrowserStack, LambdaTest, ...). 
    This is often the easiest way to obtain a reliable and efficient execution infrastructure.

    Please note: SAP does not recommend any specific Selenium remote webdriver infrastructure provider.

  • You can also create your own infrastructure.
    For instance, Selenium provides samples docker images containing the minimum required elements for such an infrastructure.

You can find more information about Selenium remote webdriver here.

Important

The runner endpoint URL must be a trusted infrastructure. SUM will send the scenario commands to this URL, including sensitive data maintained in the scenario configuration. 

Declare a SUM Runner

  1. Open the SUM Configuration tray
  2. Expand the Runners section
  3. Click the pen icon to open the Configure Runners dialog.
  4. Click the "Add" button on top of the table. 
  5. Fill-in the General and Endpoint fields
  6. Click the "Add" button.

Edit a Runner Endpoint (Optional)

If necessary you can adjust the endpoint of the SUM Runner later.

  1. Open the SUM Configuration tray
  2. Expand the Runners section
  3. Click the pen icon to open the Configure Runners application
  4. Select the runner you want to edit in the table
  5. Navigate to the Endpoint tab.

  6. Perform the required changes

    Once you enter changes the "Save" and a "Discard" button will appear. Changed fields will be highlighted.

  7. Click the "Save" button to save your changes.

Import Resources

The second step is to import the Selenium SIDE files that contain your scripts. For more information on how to create scripts, please read Creating Synthetic User Monitoring Scripts
  1. Open the SUM Configuration tray
  2. Expand the Resources section
  3. Click the pen icon to open the Configure Resources dialog.
  4. Click the "Upload" button on top of the table. 
  5. Select your SIDE file using the "Browse" button
  6. Click the "Upload" button.

When a new SIDE file is imported, it becomes a new Resource entry in the SUM configuration.

During the import process, the SIDE file content is analyzed and:

  • Each test suite becomes a SUM Selenium script.
  • Each test in the test suite becomes a step of the SUM script.
  • The enclosed SUM annotations are evaluated to:
    • Create the additional SUM steps.
    • Create the declared variables.
  • If the SIDE file already existed in SUM, it is updated. For more information refer to the FAQs section.

Relationship between Elements in the SIDE File and SUM Scripts Elements

Script Names and IDs

Names and IDs come directly from the SIDE file.

Script ElementSIDE ElementInternal Identifier
ScriptName of the test suiteIdentifier of the resource (SIDE file) + GUID of the Selenium test suite
Step coming from a testName of the testIdentifier of the Script + GUID of the Selenium test 
Step coming from an annotationAnnotation textIdentifier of the Script + GUID of the Selenium command

Nested Tests

Via the "run" Selenium command, a test can call another Selenium test (in the SIDE file).

In this case, the commands inside the called test belong to the current SUM step. Moreover, if a called test contains SUM step annotations, they are ignored.

Step Thresholds

When the steps are created, they are automatically configured to:

  • Monitor the availability
  • Temporarily ignore the performance (because the thresholds values cannot be provided by the SIDE file)

You can set up thresholds later as described in the section "Scenario Maintenance"

SUM Variables

When the SIDE file is imported, SUM detects the SUM variable annotations and automatically creates the corresponding SUM script variables.

By default, the value of the created variable will be the "value" field of the corresponding Selenium command.

If during the SUM scenario creation good practice was followed, the secured variables don't have their correct value. It will be necessary to appropriately set them via the SUM configuration in the parameters section of the Scenario details as described in the section "Scenario Maintenance".

Scenario Maintenance

The third step is to create a scenario for your SUM scripts.

  • You can create several scenarios from the same Selenium IDE script

  • One runner can run several scenarios

  • Each scenario can only be assigned to one runner

Create a new Scenario

  1. Open the SUM Configuration tray
  2. Expand the Scenarios section.
  3. Click the pen icon to open the Configure Scenarios dialog.
  4. Click the "Add" button.
  5. Enter a meaningful name for the scenario
  6. Select the SUM script you want to run
  7. Click the "Add" button

The creation will automatically open the Scenario Details view. You can also access this later by clicking on the line of the scenario in the scenario table. 

Maintain Thresholds

At scenario creation, all thresholds are temporarily disabled, because SUM doesn't know the appropriate values for the thresholds.

There are two ways to configure the thresholds:

  • Automatically based on the first execution of the SUM script
  • Manually

Automatic setting of the performance thresholds

If you don't adjust the threshold values manually, they will be automatically evaluated after the first successful execution of the scenario. The automatically set thresholds will be:

  • Green-to-yellow = step execution response time * 1.5 (i.e. response time + 50%)
  • Yellow-to-red = step execution response time * 3 (i.e. response time + 200%)

This automatic setting will also activate the monitoring of the performance. 

Please note: If any threshold is manually adjusted before the first successful execution of the scenario, SUM will not automatically set the thresholds.

Reset automatic setting of the performance thresholds

After their first adjustment (manual or automatic) SUM will not longer try to automatically re-adjust the scenario thresholds.

You can however indicate to SUM that you want a new automatically re-adjustment of the scenario thresholds:

  1. Access the corresponding Scenario Details
  2. Select the Thresholds tab.
  3. In the Performance Ratings section, change the Enable mode to: "Automatically on next successful execution"
  4. Click the [Save] button to persist your changes

The next successful execution of the scenario will be use as a benchmark to define new values for the thresholds.

Manual setting of the performance thresholds

You can also configure manually the scenario performance thresholds in the "Thresholds" section of the Scenario details.

  1. Open the SUM Configuration tray
  2. Expand the Scenarios section.
  3. Click the pen icon to open the Configure Scenarios dialog.
  4. Access the corresponding Scenario Details by clicking on the line of the scenario
  5. Select the Performance tab.
  6. The field "Enable mode" informs you is the thresholds were set automatically
  7. You can turn on an overall threshold for the overall script execution using the "Scenario Thresholds" section
  8. You can also set thresholds on step level using the "Step Thresholds" section.
  9. All changed fields will be highlighted.
  10. Click the "Save" button to save your changes

Adjust the Execution Period

By default, SUM executes the scenarios every 5 minutes (300 seconds).

You can adjust the time interval with which a scenario is executed.

  1. Open the SUM Configuration tray
  2. Expand the Scenarios section.
  3. Click the pen icon to open the Configure Scenarios dialog.
  4. Access the corresponding Scenario Details by clicking on the line of the scenario
  5. Select the Scheduling tab.
  6. Adjust the scheduling settings
    1. If the default value is currently used, uncheck the "Use default" flag
    2. Change the "Execution Period".
    3. To revert to the default value, check the "Use default" flag
  7. Click the "Save" button to save your changes

Adjust SUM Variables in the Script

If you defined SUM variables in your Selenium test script SUM uses the default values defined in the script during the execution.

If you use variables for passwords or user names, you most likely will have to change these default values. 

  1. Open the SUM Configuration tray
  2. Expand the Scenarios section.
  3. Click the pen icon to open the Configure Scenarios dialog.
  4. Access the corresponding Scenario Details by clicking on the line of the scenario
  5. Select the Variables tab.
  6. Here you find all the variables defined by the script, along with their default value.
    1. If a wrench icon is displayed next to the Value, it indicates that the default value was changed for the scenario.
  7. Enter the new value in the Value field.
    1. A green "+" icon indicates that the previous value was the default value
    2. A wrench icon indicates that the changed value was not the default value
    3. Save your changes
  8. To revert a variable value to its default, check the checkbox in front of the variable row and click the "Restore to default"
    1. A red "-" icon will indicate that the variable will be set back to default
  9. Click the "Save" button to save your changes

Adjust Selenium Execution Parameters

SUM allows you to adjust some properties related to the execution of the Selenium script associated with the scenario. For instance, the web browser to use, the selenium IDE "execution speed", etc.

You can adjust the following properties:

PropertyDescription
Browser > BrowserThe web browser to use.
Currently, SUM supports Chrome (default choice) and Edge Chromium. 
Browser > Webdriver argumentsThe webdriver arguments to pass to the runner endpoint. (e.g., --headless, --incognito, etc.)
Execution > Execution speed

This is similar to the Selenium IDE "Test Execution Speed". It corresponds to a delay introduced after the execution of each command.
Possible values are:

  • Fastest (default) = zero delay
  • Fast = 0.5 seconds delay
  • Medium = 1.5 seconds delay
  • Slow = 2.5 seconds delay
  • Slowest = 3 seconds delay
Execution > Element Wait timeout [s]When a command has several targets, this is the maximum wait timeout for each of the targets.
Does not apply to the commands with an explicit timeout (e.g. the wait commands)
Execution > Global Wait timeout [s]Maximum time wait timeout to find the target of a command.
Does not apply to the commands with an explicit timeout (e.g. the wait commands)
Screenshots > Automatic screenshots

Possible options are:

  • None (default) = no automatic screenshot is taken.
  • At each step = a screenshot is automatically taken at the end of each step execution.
  • On error only = in case of execution error, a screenshot is taken when the error is encountered.
    A screenshot is also taken at the end of each step execution but, if the execution is successful, screenshots are not stored.
Screenshots > Enable explicit screenshotsWhen enabled, a screenshot is taken after each command with the @sap.sum.screenshot annotation.

  1. Open the SUM Configuration tray
  2. Expand the Scenarios section.
  3. Click the pen icon to open the Configure Scenarios dialog.
  4. Access the corresponding Scenario Details by clicking on the line of the scenario
  5. Select the Selenium tab.
  6. For the properties you want to change:
    1. If the default value is currently used, uncheck the "Use default" flag
    2. Change the property value.
    3. To revert to the default value, check the "Use default" flag
  7. Click the "Save" button to save your changes

Activate Monitoring

The last step is to activate the scenario on a runner. Please note that you can assign several runners to a scenario. Also, you can use a same runner for several scenarios.

  1. Open the SUM Configuration tray
  2. Expand the Scenarios section.
  3. Click the pen icon to open the Configure Scenarios dialog.
  4. Access the corresponding Scenario Details by clicking on the line of the scenario
  5. Select the General tab.
  6. Select the runner you want to use from the drop-down box
  7. Click the "Assign" button. (This could take a few seconds)
  8. Set the toggle button to ON to activate the script execution.
  9. Click the "Save" button

Housekeeping

By default, SUM keeps the scenarios executions metrics for 14 days.
This retention period is configurable. the minimal value is 1 day. The maximum is 30 days.

For the screenshots captured during the scenarios executions, SUM has a specific retention period.
By default, it is 3 days. It's configurable with a minimum of 1 day and a maximum of 10 days.

Screenshots can consume a lot of space. This is something to consider when configuring their retention period.


Procedure

  1. Open the Configuration and access the Application Configuration section.
  2. Click the icon to open the Application Configuration popup.
  3. If checked, uncheck the "Use default" flag for the retention period to adjust.
  4. Adjust the retention period and save.

The change will be taken into account the next time the housekeeping job is executed. This happens once a day.

 


Be aware that SAP provides you a fair usage concept with 8GB HANA memory in total. If you extend the given memory you might need to subscribe additional memory blocks with our BTP service SAP Cloud ALM memory extension.

Frequently Asked Questions

TermDescription
ScenarioMonitoring scenario - A Selenium Script scheduled to be regularly executed by a Runner
Step

Monitoring scenario step - Sub-part of a monitoring scenario focusing on a specific action.
e.g., Logon

Script

Selenium IDE test suite describing a sequence of actions (a.k.a. commands) to perform on the application to monitor.

This is what simulates the end-user actions on the monitored application.

SIDE file

Selenium IDE file containing one or several test suites that Synthetic User Monitoring will use as Scripts.

SIDE files are produced by the Selenium IDE plugin when a Selenium test project is saved.

ResourceExternal file uploaded in the SUM database.
Currently, the only resource SUM supports is Selenium IDE (SIDE) files.
Selenium

Selenium is an open-source framework dedicated to Web application tests.

AnnotationSynthetic User Monitoring specific keyword added into the comment section of a Selenium command.
RunnerThe component in charge of the execution of the monitoring scenarios.
Runner endpointSelenium remote web driver infrastructure: the entity hosting the web browser and the web driver used to access the monitored applications

Selenium is an open-source framework dedicated to web application tests.

It consists of several components.

In the context of the Synthetic User Monitoring, the involved Selenium components are mainly:

  • Selenium IDE – Integrated Development Environment for Selenium. This is the tool that is used to create the Selenium scripts.
  • Selenium WebDriver – This is the component that "operates" the Web browser. It interprets the Selenium commands and converts them into action on the Web browser. The WebDriver is Web-browser-specific.

Important: Selenium is not developed by SAP.

Usually designed with the Selenium IDE component, a Selenium IDE script (or SIDE file) corresponds to a Selenium IDE project.

It describes the sequences of actions (also referred to as commands) to be performed on a web browser, such as clicks, setting texts, waiting for elements, etc.

These sequences are organized into:

  • Tests: a sequence of actions to be executed one after the other.
  • Test suites: a sequence of tests to be executed one after the other.

Each action has a set of attributes to indicate:

  • Command: the action to be performed.
  • Target: the element on which the action has to be performed.
  • Value: the value to be applied for the command. Some commands have no value.
  • Comment: a free text to describe the expected action.

Selenium IDE proposes several different commands, most of which are supported by Synthetic User Monitoring. Please refer to the complete list of Selenium commands supported by SUM for details.

To help with the script creation, the Selenium IDE proposes a recorder.
It also gives the possibility to play back the script.

When such a script is saved via the Selenium IDE, it generates a ".side" file.
This file is the JSON representation of the sequence of actions.

The scenario availability depends on the success of the  Selenium IDE commands during the scenario execution.

If a command fails during execution, the scenario availability will be rated as failure (Red).

A command failure can be caused by:

  • A command target not found preventing the command execution,
  • A failed assert or verify command,
  • An invalid evaluation of a property, ...

If the script defines steps, then SUM will also evaluate the availability at the step level.

For a given execution, the step availability depends on the success of the commands in the step.

Info: Generally, when a failure happens, the SUM scenario execution is interrupted.
Except for the verify commands. In this case, the scenario availability is rated as failed but the execution continues.

The scenario performance depends on the response time (i.e. duration) of the scenario and scenario steps execution.

The response time is compared to a pair of thresholds: Green-to-Yellow and Yellow-to-Red.

For each scenario, performance thresholds can be defined at:

  • Scenario level,
  • Step level (for each scenario step).

For each execution, SUM provides performance metrics for:

  • The scenario level
  • The step level
  • The overall scenario (considers both the scenario level and the step level performance)

Scenario-level Performance

It considers the total response time of the execution (i.e. all the executed commands, in all the steps).

The scenario level performance is evaluated only if both:

  • The scenario thresholds are enabled in the scenario configuration
  • The scenario availability is successful

The scenario level performance is:

  • Red - If the total response time of the execution exceeds the Yellow-to-Red threshold.
  • Yellow - If the total response time of the execution exceeds the Green-to-Yellow threshold.
  • Green - if the total response time of the execution is below the Green-to-Yellow threshold.
  • Grey - if the scenario performance monitoring is not enabled
  • Blue - if the performance cannot be evaluated because the scenario availability has failed (red).

Step-level Performance

Considers the response time of the step execution (i.e. all the executed commands in the step).

For a given step, the step level performance is evaluated only if both:

  • The step thresholds are enabled in the scenario configuration
  • The step availability is successful

The step level performance is:

  • Red - If the step response time exceeds its Yellow-to-Red threshold.
  • Yellow - If the step response time exceeds its Green-to-Yellow threshold.
  • Green -If the step response time is below the Green-to-Yellow threshold.
  • Grey - if the step performance monitoring is not enabled
  • Blue - if the performance cannot be evaluated because the step availability has failed (red).

Overall Scenario Performance

For an execution of the scenario the overall performance will be:

  • Critical (Red), if at least one Yellow-to-Red threshold is crossed (for either the scenario level or one of the step level)
  • Poor (Yellow), if at least one Green-to-Yellow threshold is crossed and no Yellow-to-Red threshold is crossed.
  • Good (Green), if no Green-to-Yellow or Yellow-to-Red threshold is crossed.
  • Not Enabled (Grey), if all the scenario thresholds are disabled.
  • Not Rated (Blue), if the overall availability has failed.

SUM identifies a SIDE file via both its Project name and the main ID (GUID) in the .side file.

If a SIDE file with the same project name and GUID is re-imported, SUM will not create a new resource entry. It will update the existing one.

Please note: Only the resource owner can update it.

During the import, the following changes are performed:

At Script level

  • Scripts no longer in the SIDE file are deleted. If SUM scenarios exist for the script, then the update is rejected. I.e. only scripts without scenarios can be removed.

  • New scripts in the SIDE file are added to the resource.
  • Modified scripts (e.g. change of the test suite name) are updated.

At Step level

  • Deleted steps are removed. Their threshold configuration is deleted. The corresponding monitoring metrics remain until deleted by the housekeeping.
  • New steps are created. Their threshold configuration is created with empty values and a disabled state.
  • Modified steps (e.g. change of the step name is the annotation) are updated.

Please note: If a step annotation is moved from one command to another - i.e. with the same step name - SUM considers that it is a new step. (Steps coming from an annotation are identified by the Script identifier plus the GUID of the Selenium command)

At SUM Variable level

  • Deleted variables are currently not removed.
  • New variables are added.
  • Changed variables values are updated.

SUM identifies a SIDE file via both its Project name and the main ID (GUID) in the .side file.

If you re-import a SIDE file after changing it and you want to create a new resource then you need to change one of the two above elements.

The easiest is to change the project name:

  1. Open the SIDE file in a Selenium IDE
  2. Click on the Project name field (on top of the IDE UI) to change the name
  3. Save the project. By default, the proposed name for the .side file will be the new project name.
  4. Then, you can import this new SIDE file in SUM. This will create a new resource containing a copy of the previously imported SIDE file.