How to Import and Configure SUM Selenium Scenarios

How to Import a Selenium Scenario in SUM

Prerequisites

  • You have a SIDE script that is compatible with SUM.

Very Important

We strongly recommend that you remove all sensitive data (e.g. passwords) from the SIDE file before it is imported. SUM is not designed to securely store sensitive data in the script. (Use secured variables for sensitive data.)

Procedure

  1. Open the SUM configuration popup.
  2. Navigate to the Scenarios section.
  3. Above the scenarios table, click the [Import] button.
    This opens the Import Scenario popup.
  4. In the Import Scenario popup, specify:
    1. The Customer Network the scenario will belong to.
    2. The .side file to import. (Use the [Browse...] button.)
    3. Click the [Import] button.

The new Selenium scenario is imported.

Good to Know

The Selenium scenario ID comes from the Selenium-IDE-generated GUID (in the .side file).

If you reimport the same SIDE file, it will "update" the previously imported Selenium scenario.
This also means that you won't have the possibility to reimport it in another Customer Network.

What Happens when a SIDE file is Imported in SUM

See What is a Selenium IDE script

When a SIDE file is imported, its content is analyzed and:

  • Each test suite becomes a SUM Selenium scenario.
  • Each test in the test suite becomes a step of the SUM scenario.
  • The enclosed SUM annotations are evaluated to:
    • Create the additional SUM steps.
    • Create the declared variables.


About the Names and IDs

Names and IDs come directly from the SIDE file.

 
 
 
Step coming from an annotationAnnotation textGUID of the Selenium command
Step coming from a testName of the testGUID of the Selenium test 
ScenarioName of the test suiteGUID of the Selenium test suite

As a consequence, if the same SIDE file is imported several times, this will update the existing SUM scenarios because the Selenium GUIDs will be the same.

See How to add SUM steps in a Selenium script


About Nested Tests

Via the "run" Selenium command, a test can call another test.

In this case, the "run" command creates a new SUM step.

However, only the first level calls are considered.
In other words, if a called test (1st level) calls other tests (2nd level), these subsequent calls do not create SUM steps.


About the Step Thresholds

When the steps are created, they are automatically configured to:

  • Monitor the availability
  • Temporarily ignore the performance (because the thresholds values cannot be provided by the SIDE file)

See How to configure SUM Selenium scenario thresholds


About the Variables

When the SIDE file is imported, SUM detects the SUM variable annotations and automatically creates the corresponding scenario variables.

By default, the value of the created variable will be the "value" field of the corresponding Selenium command.

If the Selenium command has no value, the variable is not created.

If the SUM scenario creation good practice were followed, the secured variables don't have their correct value.
It will be necessary to appropriately set them via the SUM configuration in the Parameters section of the Scenario details.

See How to define SUM variables and SUM secured variables

How to Define the Selenium Scenario Execution Type: Local or Remote

Prerequisite

  • The Selenium scenario has been imported in SUM.
  • You have the URL and credential for the remote execution.


Procedure

By default, when a SUM robot (location) triggers the execution of a Selenium scenario, it tries to execute it locally (on the robot host).
(See How to configure the location host for local Selenium scenario execution)

However, this behavior can be controlled in the scenario configuration.

SUM configuration lets you decide between:

  • Configuring the remote execution globally for the scenario (for all its locations)
  • Configuring the remote execution only for one specific location (scenario-on-location level)


Configure Remote Execution Globally

  1. Open the SUM Configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Selenium tab.
  5. In the Browser section, set the Remote Enabled setting to "true".
    This expands the Remote Configuration section.
  6. In it, specify the default (shared for all the locations):
    1. Remote URL – only HTTPS URLs are supported.
    2. The logon User/Password to the Remote URL.
  7. If some locations have different remote settings:
    1. Select the Deployment tab.
    2. Repeat for each of the locations:
      1. In the Location Name column, click on the location to be configured.
        It is only possible for the locations on which the scenario is deployed.
        This drills down to the scenario-on-location details.
      2. Select the Selenium tab.
      3. Overwrite the settings specific to this location.
  8. [Save] your changes.


Configure Remotely for a Specific Scenario-on-Location

  1. Open the SUM Configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Deployment tab.
  5. In the Location Name column, click on the location to be configured.
    It is only possible for the locations on which the scenario is deployed.
    This drills down to the scenario-on-location details.
  6. Select the Selenium tab.
  7. In the Browser section, set the Remote Enabled setting to "true".
    This expands the Remote Configuration section.
  8. In it, specify the:
    1. Remote URL – only HTTPS URLs are supported.
    2. The logon User/Password to the Remote URL.
  9. [Save] your changes.

How to Deploy a Selenium Scenario on a Location

Prerequisites

Procedure

The deployment of a Selenium scenario on a location is similar to the other types of scenario:

  1. Open the SUM configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Deployment tab.
    This lists all the locations that are able to execute a Selenium scenario (i.e. with the Selenium capability).
  5. In the Deploy column, switch to "ON" to deploy the scenario onto the corresponding location.
    You can repeat the operation to deploy on several locations.
  6. [Save] your changes.

Configuration Recommendations

Here are configuration recommendations to prevent Selenium Webdriver or web-browser issues.

Error Message (in Agent log)Error Message (in UI)Possible Cause
org.openqa.selenium.TimeoutException: timeout: Timed out receiving message from renderer: 30.000timeout: Timed out receiving message from renderer: 30.000Likely the ChromeDriver was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.
org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)		unknown error: Chrome failed to start: exited abnormally.The robot OS is probably Linux. The ChromeDriver can only be executed in headless mode.
org.openqa.selenium.WebDriverException: unknown error: cannot find Chrome binaryunknown error: cannot find Chrome binaryChrome is not installed on this robot or the binary path given does not contain chrome executable.
org.openqa.selenium.WebDriverException: unknown error: unable to discover open pagesunknown error: unable to discover open pages

The ChromeDriver has crashed.

This problem often occurs on Windows Server OS.

org.openqa.selenium.SessionNotCreatedException: session not created from tab crashedsession not created
from timeout: Timed out receiving message from renderer: 600.000		This problem often occurs on Windows Server OS and when headless mode is enabled.

For SUM locations running on a Linux host, the chromedriver can only be executed in "headless" mode.

For SUM locations running on a Windows host, it's also highly recommended to execute the chromedriver in "headless" mode (to prevent locking the sapadm user).


Procedure for Configuring Headless Mode for a Location

  1. Open the SUM Configuration pop-up.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click on one of the scenario names that is deployed on your Linux host.
    This drills down to the Scenario details.
  4. In the Location name column, click on the Linux Location to be configured.
  5. Select the Selenium tab.
  6. In the Browser section, set the argument parameter to "--headless".

How to Configure the SUM Selenium Scenario Screenshots

Important

  • Screenshots must not be taken for screens containing sensitive data. SUM does not store screenshots in a secured storage.
  • Screenshot storage could quickly consume a considerable amount of space in your Focused Run HANA database. See How long are screenshots kept

There are three situations in which SUM can take screenshots:

  • Automatically, for each execution
  • Automatically, when an availability error occurs
  • Explicitly, on commands identified by a SUM screenshot annotation

See How to take UI screenshots during the SUM Selenium scenario execution

The Scenario configuration defines how SUM will capture the screenshots:

  1. Open the SUM configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Selenium tab.
  5. In the Screenshots section, configure the scenario:
    • To take Automatic Screenshots:
      • None – (by default) to disable the automatic screenshots
      • At each step – to take a screenshot at the end of each step for every execution
      • On Error only – to take automatic screenshots only when the execution finishes with an availability error
    • To Enable Explicit Screenshots:
      • True – (by default) to take the screenshots defined via the SUM screenshot annotations
      • False – to deactivate the annotation screenshots
  6. [Save] your changes.

You can also define specific screenshot configuration for a given location (at scenario-on-location level).
In this case, the SUM considers the scenario-on-location configuration instead of the configuration at the scenario level.

  1. Open the SUM configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Deployment tab.
    This lists all the locations where the Selenium scenario is deployed. (The Deploy column is set to "ON".)
  5. In the Location Name column, click on the location to be configured.
    This displays the Scenario-on-location details.
  6. Select the Selenium tab.
  7. In the Screenshots section, configure the scenario:
    • To take Automatic Screenshots:
      • None – (by default) to disable the automatic screenshots
      • At each step – to take a screenshot at the end of each step for every execution
      • On Error only – to take automatic screenshots only when the execution finishes with an availability error
    • To Enable Explicit Screenshots:
      • True – (by default) to take the screenshots defined via the SUM screenshot annotations
      • False – to deactivate the annotation screenshots
  8. [Save] your changes.

How Long are the Screenshots Kept (Housekeeping)

Screenshot storage could quickly consume a considerable amount of space in your Focused Run HANA database.

An average screenshot occupies about 170 KB of disk space.
A scenario configured to be executed every 10 minutes on 10 locations will lead to 1,440 executions per day.
If the scenario has 10 steps and is configured to capture screenshots for each step of each execution, this represents about 2.4 GB of screenshots every day.

To prevent exhausting the Focused Run HANA space, SUM has specific housekeeping settings for the screenshots. By default, these are as follows:

  • 2 days for screenshots captured during an execution that finished with a successful availability status. (This includes the poor and critical performance statuses.)
  • 4 days for screenshots captured during an execution that finished with an availability error.

Note: Housekeeping is performed when the housekeeping engine is executed – once per day.

In the SUM monitoring UI, you can also "protect" the screenshot of a specific execution from the housekeeping.
In this case, the corresponding screenshots are protected until the housekeeping of the execution (i.e. the SUM standard housekeeping) is performed.

How to Change the Housekeeping Configuration?

You can change the housekeeping configuration in the Infrastructure section of the SUM configuration.
This requires a SUM super-administrator user.

  1. Open the SUM configuration popup.
  2. Navigate to the Infrastructure section.
  3. In the Housekeeping Engine section, adjust the:
    • Lifetime of Screenshots
    • Lifetime of Error Screenshots
  4. [Save] your changes.

How to Configure SUM Selenium Scenario Thresholds

Unlike other SUM scenario types, SUM Selenium scenarios are not created with the User Monitoring Script Editor.
Therefore, the scenario thresholds cannot be predefined (before the scenario import).

As a consequence, when a SUM Selenium scenario is first imported, the monitoring of the performance is temporarily disabled.


Automatic Setting of the Performance Thresholds

If you don't adjust the threshold values manually, they will be automatically valuated after the first successful execution of the scenario.
The automatically set thresholds will be:

  • Green-to-yellow = step execution response time * 1.5 (i.e. response time + 50%)
  • Yellow-to-red = step execution response time * 3 (i.e. response time + 200%)

This automatic setting will also reactivate the monitoring of the performance. 

Note: The threshold values will be set at scenario level (for all the locations).

Manual Setting of the Performance Thresholds

As with the other SUM scenario types, you can configure the Selenium scenario performance thresholds in the Thresholds section of the Scenario details:

  1. Open the SUM configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Thresholds tab.
    This lists all the scenario steps.
  5. In the Performance column, switch the selector to "ON" to activate monitoring of the performance for the corresponding step.
  6. In the G2Y [ms] and Y2R [ms] columns, you can set the values (in milliseconds) for the following respectively:
    • The green-to-yellow threshold (to detect the poor performance status)
    • The yellow-to-red threshold (to detect the critical performance status)
  7. [Save] your changes.

You can also define specific thresholds for a given location (at scenario-on-location level).
In this case, the monitoring considers these scenario-on-location thresholds instead of the thresholds defined at the scenario level.

  1. Open the SUM configuration popup.
  2. Navigate to the Scenarios section.
  3. In the Scenario Name column, click the Selenium scenario to be configured.
    This drills down to the Scenario details.
  4. Select the Deployment tab.
    This lists all the locations on which the Selenium scenario is deployed. (The Deploy column is set to "ON".)
  5. In the Location Name column, click on the location to be configured.
    This drills down to scenario-on-location details.
  6. Select the Thresholds tab.
    This lists all the scenario steps.
  7. In the Performance column, switch the selector to "ON" to activate monitoring of the performance for the corresponding step.
  8. In the G2Y [ms] and Y2R [ms] columns, you can set the values (in milliseconds) for the following respectively:
    • The green-to-yellow threshold (to detect the poor performance status)
    • The yellow-to-red threshold (to detect the critical performance status)
  9. [Save] your changes.

 

Additional Information