User Experience Monitoring

Support Package	SAP Note
SP03-SP06	2464104 - Corrections for UXMon 7.2 2604141 - Extractor for User Experience Monitoring (UXMon) creates too many log entries

Troubleshooting

Design Time

Symptom
Your RFC content check fails even if you have correctly mapped the XML structure shown under tab Export Parameters in a Content Check path.

Export Parameters

<item>

<SCENARIO_ID>2</SCENARIO_ID>

</item>

</SCRIPTS>

</OUTPUT>

The current implementation of EEM is not operating on the displayed XML structure which is only dumped out for the end-user in the EEM Editor. During runtime it is operating on predefined classes for RFC handling with a different hierarchy:

Be Careful

Wrong path, but what you would expect from XML: EXPORT.OUTPUT.SCRIPTS.item.Scenario_ID

Correct Path

To get a correct path, generally ignore OUTPUT and metatag item, which just represents an ABAP table unknown to java.:

Correct path: EXPORT.SCRIPTS.Scenario_ID

Symptom

The EEM Editor might crash with an EXCEPTION_ACCESS_VIOLATION when executing an RFC script.

Reason

The problem is located in the JCO native libraries.

Solution

Configure the EemEditor explicitly to launch with a specified 32bit JDK. For this purpose, add this parameter to EemEditor.ini:

-vm c:/full/path/to/javaw.exe

Symptom
When executing an http script in EEM Editor some requests fail with error message
Variable inplaceEditForm:reference:c0-e6, inplaceEditForm_accordionMenu_expanded_state:reference:c0-e7 is not defined, e.g., because a preceding request did not deliver the expected result.

Reason
The string is a JSON object but handled as a variable. This might happen for example when creating a script for SuccessFactors.

Solution
Create a variable with exactly the same name and value.
Example:

Name

inplaceEditForm:reference:c0-e6, inplaceEditForm_accordionMenu_expanded_state:reference:c0-e7

Value

{inplaceEditForm:reference:c0-e6, inplaceEditForm_accordionMenu_expanded_state:reference:c0-e7}

Symptom
You are recording SAPGUI Script under usage of a F4 help dialog and action dumps with
"Runtime error MESSAGE_TYPE_X has occurred occurs"

Solution

Open the SAPGUI menu HELP => SETTINGS => tab F4 Help

Set Display radio button to "Dialog(modal)"

Symptom

Editor finished execution half way of a replay without any error information. Go to workspace\.metadata\.log,
there is some OOM identified.

Solution

Open EemEditor.ini file and add the following parameters to increase the Java heap size: -XX:MaxPermSize=64m, -Xmx128m

Symptom
You imported a SAPGUI recording into EEM Editor and provided correct logon credentials. A script execution in EEM Editor fails and you get error messages like: "The control could not be found by id. (SAP Frontend Server ):619: wnd1/usr/ctxt2.text"

Reason
There might be several reasons for this behaviour and a step by step analysis is required.

Procedure

Re-record the script in SAPGUI and ensure that you do not use value history function of SAPGUI to provide values to text fields even if they are provided. Type it character by character again.
Stop the recorder and try to execute the created VBS file directly in the SAPGUI.
1. If the VBS file shows a exceptional behavior by showing control errors the issue is related to your SAPGUI installation not to EEMon
2. In this case provide the version/buildnumber of your SAPGUI to support together with the recorded file.
3. In most cases a different version of SAPGUI solves the issue.
If the script was executed successfully in SAPGUI alone but fails after import to EEM Editor please check the User credentials again.
Execute the script step by step to identify the EEM message which causes the issue by addressing an unknown UI element.
Execute the script again till the point where the script would during next action. Use the mouse to click on the UI element which should be addressed during a correct execution. The red square helps you to navigate in the UI components. For example
Proceed with script execution to exit the script. The selection of the correct UI element with the mouse has created an availability check for this element. We do not need this check but it shows the component ID of the important UI element that is not pressed or activated during the exceptional script runs.
Compare the component ID from check with the component ID mentioned in the next message. In some cases the IDs differ. Try to execute the script with a corrected component ID in the message.
Run the script again with hardcopy and component trace options:
1. Go to script configuration, SAPGUI settings.
2. Activate "hardcopy" and "Component Trace" and hit "Apply".
3. Perform another replay of the script. During the replay the editor now captures screen-shots for every message and in addition a dump of the UI element hierarchy.
4. You can review this by clicking at single messages in the replay result.
If it is still not working prepare a WTS connection to the recording PC to an account where the SAP support can use the EEM Editor and execute and analyze the script in detail.

Reason
The client plugin (EEM recorder) has a defect.

Solution
Manually eliminate the duplication as long as no official fix is available.
Example:

Recorded parameter

cmd=get_esidcmd=get_esid

Corrected parameter

cmd=get_esid

Configuration Time

Symptom

In the SOLMAN_SETUP step 'Configure EEM Robots' terminates with the error Operation "getMatchingAgentInfo{urn:EemAdminWsd/EemAdminVi/document}" not supported (interface: "CO_EEM_EEM_ADMIN_VI_DOCUMENT" ...} when clicking on button 'Check Agents'.

Reason

The web service in Java has been enhanced to provide a filter option when searching for robot candidates.

Solution

Execute in SOLMAN_SETUP the activity Create Logical Ports in step Configure Automatically:

Symptom
When deploying or uploading a script that cannot be started on the robot because of missing properties or an error the deployment throws the exception
SOAP:1.001 CX_SY_CONVERSION_OVERFLOW:.An error occurred when deserializing in the simple transformation program /1SAI/....Overflow converting from <java timestamp>.
Reason
The long value of the java timestamp is mapped to an INT4 data element.
Solution
Check in transaction SRT_UTIL in the payload for the real error (e.g. property not found in SecureStore:...) correct it and deploy the script again. The mapping is corrected with 7.1 SP08.

Symptom

Error Message after trying to deploy scripts to robots:

Resources cannot be uploaded: Script= ... SoapFaultCode 4: Deserialization fails: Nested Message: XML Deserialization Error. XML Parser has thrown exception while parsing input
Resources cannot be uploaded.;Script= ... SoapFaultCode:5 Reader is not positioned on start element.Probably incorrect usage of stream parsing methods.
ICF Error when receiving the response: HTTPIO_PLG_CANCELED

Resolution

This is a known bug in the http service of the SAP J2EE server. SAP Note 1638655 lists the patch levels for J2EE software components that fix the problem.

Work-around: Point the consumer proxy directly to the J2EE dispatcher instead of pointing to the ICM. Procedure:

Start SOAMANAGER in SAPGUI via transaction /nSOAMANAGER
Select Web Service Configuration
Search by Consumer Proxy, search pattern *EemAdmin*. Select CO_EEM_EEM_ADMIN_VI_DOCUMENT and click Apply Selection.
Select tab Configurations in the lower pane.
Select the logical port SMD and click Edit.
Select tab Transport Settings
Update the field Port Number of Access URL to contain the J2EE dispatch port (e.g. 50000).
Select tab Messaging
for Message ID (Synchronous), select _Suppress ID Transfer
Click Save

Symptom
The SOLMAN_SETUP task 'Create Endpoints for communication' fails with message Error during End Point creation.
Reason
The configuration For service definition AI_EEM_LIST_ALL_SCENARIOS could not be generated. There was a problem in after import method when applying the SP.
Solution
Execute the report REPAIR_AFTER_IMPORT to solve the issue and execute the task again.

Run Time

The context menu in the monitoring UI can contain the following entries:

Go to E2E Trace Analysis
This entry is only selectable if system data or traces have been collected and if user has the authority to jump to the E2E Trace.
View Collector Log
This entry provides a jump-in to ABAP application log (SLG1) to display the result of the extractor collecting the system data or traces of the script executions. This entry is only selectable if the system data is not available.
Copy TransactionID to Clipboard
The entry is only selectable for a script execution (time stamp). It provides the possibility to collect the logs from the robot and analyze it in the EEM editor. See also
Run Script with Trace
This entry is only possible for a selected script and robot combination. It triggers immediately an additional execution with provided trace level.
Set Temporary Configuration
The selected script on the selected robot is executed for the configured time frame with the specified trace level.
Filter to
Opens a new tab with just the selected script.
Assign Steps Filter
Opens configuration menu to assign a filter on step level.
Jump to Alert Inbox
Provides jump in to Unified Alert Inbox.
Collect Trace
Triggers a trace collection immediately for the selected script execution.

Symptom
You cannot see any system data in the Monitoring UI or jump-in to E2E Trace is not available.

Background
Executions (Timestamp and steps below) are written with bold letters if system data or a trace is available. If the execution is triggered with a trace level > 0 the text is written in italic letters.
Currently there are 3 types of scripts where system data or traces can be collected/displayed:

1. HTTP Script for SAP J2EE based applications.

System data/traces are only available if tracing has been enabled (Run Script with trace or Set temporary configuration). It is possible to configure so that HTTP log and DSR records are always written. See also Trace Enabling for EEM.

2. HTTP Script for ABAP based application.

System data/statistical records are always available. Detail traces (SQL, ABAP Trace,..) are only available when tracing has been enabled.

3. SAPGUI Script

System data/statistical records are not for all steps available. Detail traces (SQL, ABAP Trace,..) are not possible at the moment.

Solution

When system data/traces are missing (see description above) you can check the following steps:

1 . The parameter "Expire in seconds" of temporary configuration should have a value higher than 60 seconds (default 1800)

2. Is the script assigned to a technical scenario? In setup step 3.3. check that a technical scenario is filled in for the script.

3. Is the extractor running?
Go to work center SAP Solution Manager Administration, in Infrastructure launch the Extractor Framework view. For each technical scenario there is an extractor EEM system data of TechScen

4. Check the log of the extractor.

If there is an error message like

Current SMD Upgrade status does not allow trace collection

the SMD Upgrader must be executed after applying an LM-Service patch.
This can be done in SOLMAN_SETUP -> Basic Configuration -> Step 2.4 (SAP Solution Manager Internal Connectivity) -> Activity: Run Java Upgrader

If there is an error message like

Trace collection failed: 1/0 exceptions occurred during synchronous/asynchronous trace collection.

an error occurred during trace collection in SAP Solution Manager J2EE and you need to check the log viewer of the SAP Solution Manager Java Stack.

- Start the NWA log viewer ( /NWA -> Monitoring -> Logs and Traces -> Show Default trace.

- open the filter and enter: location contains com.sap.sup.admin.e2e.trace and click on 'Apply filter '

Manual Trace Collection

- Get the BusinessTransaction.xml from robot (click the link if you want to analyze the BusinessTransaction.xml within the EEM Editor)

Select the interesting execution in the EEM Monitoring UI
Right-click with mouse and select "Copy Transaction ID to clipboard"

- Open the E2E Trace application from the Root Cause Analysis work center for the Technical System(s)/Scenario you want to collect the trace and paste this id there

- Trigger the server-side trace collection and check the progress and in parallel the default.trc as mentioned above.

Known Problems
1. HTTP Script for SAP J2EE based applications
-HTTP log or DSR records could not be found.

2. HTTP Script for ABAP based application
-ICM HTTP log does not have the correct format.

3. SAPGUI Script
-ABAP Statistic records could not be found because the time difference of Robot and called system is too high.

Symptom
The Monitoring UI does not display any script executions. You just see the deployed scripts with grey icons and status 509 or 109.
Solution
The system does not get any notifications from the robots. There could be several reasons:

The user for sending the notifications is locked
The user for sending the notifications is stored on each robot. If the password is changed in user master record (SU01) and not in the setup application the robots use still the old password. But even if the password is changed in the the setup application it might happen that there is a small overlap when robots still using the old password until they receive the new password. The user can be unlocked in SU01. The name of the user is specified with parameter wsnotify.user .
Accepting notifications is disabled
For 7.1 in the setup application (SOLMAN_SETUP) and for 7.01 in eem_admin accepting notifications can be disabled globally.
Robots are not available
If robots are not reachable they are usually displayed at disconnected and they cannot send any notification.
URL for sending the notification has been changed or is not correct
The URL (including protocol, host and port) has been changed or is not correct. This might happen if the URL is using a host name and port for the message server that is not reachable from the robot location. The URL can be adapted by adjusting the parameter wsnotify.url .
Notification is using https but iaik files are not installed. In SMDAgentApplication.log you can find this exception
java.lang.Exception: Could not create https handler:java.lang.ClassNotFoundException iaik.protocol.https.Handler
See also http://wiki.sdn.sap.com/wiki/display/EEM/SSL+For+Wsnotify
After upgrading to SAP Solution Manager 7.2 some robots cannot send notifications.
See also AiEemBulkNotificationSt is not found

In general you should check the SMDAgentApplication.log of the robot if there is an error when sending the notification.

There might be several reasons why the monitoring UI does not start.

You get error messages like 'Cannot retrieve language bundle' or 'Cannot retrieve severity rules'.

Endpoints were not generated successfully
The endpoints the monitoring UI is calling are created in SOLMAN_SETUP for EEM in step 2.4 (Configure Automatically)

When the WSDL has been loaded successfully but further WS calls are failing like in the screen shot then the following points could be the reason: Cannot Retrieve language bundle or severity rules information.

Missing authorization.
The user does not have the correct role and authorization for this WS is missing. To verify this an authorization trace for the user in ST01 can be recorded when calling the monitoring UI.
Wrong WS configuration
During SOLMAN_SETUP you explicitly allow in
-> System Preparation -> Step 5.2 Web Service infrastructure Configuration
which authentication types all web services can use:
- Basic Authentication
- X.509 Certificate Authentication
- Ticket Authentication
If you don't allow for example Ticket Authentication it is not possible to logon first via SAPGUI to SAP Solution Manager and start from there in the workcenter the EEM Monitoring UI. The following screen shot shows the required settings when all authentication types should be used:

Symptom

Http script runs in error (Editor or Robot) because of an ssl handshake failure

Message in the Editor:
SSL handshake failed, probably due to timeout, please restart the script: Received fatal alert: handshake_failure

In the Log you will see:
handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Solution

The SSL handshake failure is typically caused because client side (EEM) and web server could not agree on a common https protocol (e.g. SSL3, TLSV1, TLSV1.1, TLSV1.2) or on a cipher suite. There are multiple possible reasons for this:

There was some kind of timeout

Just try again ...

The Java VM running the EEM script (in the editor / on the robot) misses the "unlimited strength jurisdiction policy" files.

Please install Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. Information on where to get these files can be found in SAP Note 1240081.

JDK used by the Editor
JDK used by the robots (Sample path: ...\usr\sap\<Agent-SID>\SMDA97\exe\sapjvm_6\jre\lib\security)

The Java VM running the EEM script does not support the SSL protocol implementation / cipher suite at all

Java 6 has only limited support for crypto algorithms, whereas Java 8 comes with a more complete choice. Example: Java 6 does not support TLSV1.1 or TLSV1.2, while Java 8 does. So if the web server requires to use TLSV1.2, the EEM script will fail to execute running on Java 6. Since January 2018 Diagnostics Agents connecting against SAP Solution Manager 7.2 are also supported on SAP JVM 8.1.

As a cross check try if it makes a difference to run the EEM editor with Java 8. See Configure the Java VM for the Editor.

You can also activate a detailed SSL trace by setting the Java VM parameter -Djavax.net.debug=all (in EemEditor.ini). Output will be sent to the console. To make the console visible run the editor with java.exe (instead of the default javaw.exe). See again Configure the Java VM for the Editor.

For Diagnostics Agent communication issues having to do with TLS V1.2 please see the SAP Note 2463712.

Symptom

When executing a script the following exception is thrown:

EXCEPTION
com.sap.smd.eem.executor.MessageException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target[
LinkedList0:
Issuer: CN=Company CA 5, OU=Trust Center Company, O=Company Services GmbH, C=DE
Subject: CN=selfservice-test.portal.Company.de, C=DE, ST=NRW, L=Anywhere, Company GmbH
SerialNum: 8166
Expires: Thu Nov 25 08:25:04 CET 2010
....
Index: 1

LinkedList1:
Issuer: C=BE, O=GlobalSign nv-sa, OU=RootSign Partners CA, CN=GlobalSign RootSign Partners CA
Subject: CN=CompanyCA 5, OU=Trust Center Company, O=Company Services GmbH, C=DE
SerialNum: 40000000001094550f4da
Expires: Thu Feb 07 12:00:00 CET 2013
...
Exception: null
Index: 2

LinkedList2:
Issuer: CN=GlobalSign Root CA, OU=Root CA, O=GlobalSign nv-sa, C=BE
Subject: C=BE, O=GlobalSign nv-sa, OU=RootSign Partners CA, CN=GlobalSign RootSign Partners CA

Solution

At least one of the bold written certificates above must be entered to the truststore (either global truststore or script specific truststore).

Please check 'How to Import HTTP Trust Certificate' to import the certificate.

How to identify which truststores are used and what is contained?
To get information which certificates are already part of which truststores you have to increase the log level of SMDAgentApplication.log of an SMD Agent to Debug and then stop and start the script. This is an extract of the output:
Info com.sap.smd.eem.executor.http.CertificateContext - loadKeyStore: Return the cached key store for file <Agent_path>\exe\sapjvm_5\jre\lib\security/cacerts
Debug com.sap.smd.eem.executor.http.CertificateContext - dumpKeys: TrustedCertificateEntry Subject:CN=VeriSign Class 1 Public Primary Certification Authority ...
Debug com.sap.smd.eem.executor.http.CertificateContext - dumpKeys: TrustedCertificateEntry Subject:CN=Entrust.net...
Info com.sap.smd.eem.executor.http.CertificateContext - loadKeyStore: Return the cached key store for file <Agent_path>\SMDAgent\applications.config\com.sap.smd.agent.application.eem\scripts/<Script_Name>/trustStore.jks

Debug com.sap.smd.eem.executor.http.CertificateContext - dumpKeys: TrustedCertificateEntry Subject:CN=portal-qas.company.de,C=DE,ST=...
Info com.sap.smd.eem.executor.http.CertificateContext - loadKeyStore: Return the cached key store for file <Agent_path>\SMDAgent\applications.config\com.sap.smd.agent.application.eem/trustStore.jks
Debug com.sap.smd.eem.executor.http.CertificateContext - dumpKeys: TrustedCertificateEntry Subject:CN=portal-qas.company.de,C=DE,ST=...

Symptom
You are running a https script which aborts with "javax.net.ssl.SSLException: java.lang.UnsupportedOperationException"

Solution

Check if the script has a valid truststore or if the password for the global password is set.
When the password for the global truststore is not set you can add in eem_admin for the scope global the property
global.truststore.password and set it to eemstore.

Symptom
You are running a script which aborts with "java.net.ConnectException: Connection timed out: connect"

Solution

Check if a proxy is required to reach the url.
Set the properties http.proxy.host and http.proxy.port to the corresponding values and set http.proxy.enable to true.

Depending of the called system it might be required to set properties in different scopes:

If the called system and therefore the script always require these proxy properties you need to set them in the editor in the script properties (in the scope of the Script): HTTP / Network Settings. Then Proxy Enable / Proxy Host and Port
If the location/robot always require a proxy to reach called system(s) you need to set these proxy settings in SOLMAN_SETUP in the scope of the Robot:
- http.proxy.enable = true
- http.proxy.host = proxy_location
- http.proxy.port = 8888
If the called system requires requires these proxy settings only from some locations you need to set them in SOLMAN_SETUP in the scope of the Script on Robot: Select the Script Name, Select the Robot Name:
- http.proxy.enable = true
- http.proxy.host = proxy_location2
- http.proxy.port = 8088

Symptom
When executing a https script you get the exception"java.lang.NoClassDefFoundError: javax/crypto/SunJCE_b"

Background
Jce-policy file is not extracted to the lib\security folder of the SAP JVM during the installation of agent 7.11 (SAP Note 1234387), which makes the https doesn't work for EEM.

Solution

Download the JCE jurisdiction policy files as described in SAP Note 1240081.
Unzip it and copy the two JAR files, local_policy.jar and US_export_policy.jar, in the following folder:
/usr/sap/<DASID>/SYS/exe/jvm/<PLATFORM>/sapjvm_<xxx>/sapjvm_5/jre/lib/security
On Unix platform do not forget to set the owner of the files to the Diagnostics Agent user (<dasid>adm:sapsys) and to set the permissions mode to 777.
Restart the agent.
Goto the folder \usr\sap\<AGENT_SID>\<AGENT_INSTANCE>\exe\sapjvm\jre\lib\security and check if the local_policy.jar and US_export_policy.jar exist. If not, please also copy the local_policy.jar and US_export_policy.jar to this folder.

Symptom
You created a SAPGUI script which is running correctly in the editor and uploaded it to an EEM Robot where it fails. There might be different kind of errors like:

- RESULT_FAILED Message ... Current window id is wnd[0] while expected window id is wnd[1], please check for popups

Reason
SAPGUI scripts are executed in background on the robot with operating user SAPSERVICE<SID> which cannot interact with desktop. There might be several reasons for an error and a step by step analysis is required.

Procedure

You can change temporally the settings of the service to interact with desktop. Every SAPGUI script execution is then visible when you log on to the robot with the SAPService user.

Start services.exe
Select the service e.g SAPDAA_96 and open the properties.
Open tab 'Log On' and select 'Local System Account' and 'Allow service to interact with desktop' as shown on the screen shot below:
Wait until the script will be executed and display the result.

After the analysis you should change the setting back to its original value.

Symptom
SAPGUI script aborts at step 3 with exception The control could not be found by id
Background

The issue is caused by the configuration of the backend ABAP system in respect to multiple user logons.

A SAPGUI script is recorded after the login procedure; the login procedure is not part of the recording. To handle the logon the EEM Editor automatically adds the necessary steps. During the import it inserts 4 specific steps bevor the “main” the script logic.

From those 4 the steps 1 and 2 handle the regular login. The step 3 and 4 are conditional steps which are automatically inserted by the EEM Editor to confirm the multi login pop-up. That is, to handle the situation, when an active user session already exists and the backend system prompts the user on which actions he wants to take: “terminate the current logon”, “terminate other logons” or the “continue with this logon, without ending any other logon”. The steps 3 and 4 will attempt to select the “continue with this logon” checkbox in the corresponding pop-up window and press the “login”.

However, in some systems multiple login is not allowed (setting profile parameter login/disable_multi_gui_login). In this case the step 3 will always fail. It will try to find that “continue with this logon” checkbox which simple won’t be presented in the backend system. If the system does not allow multiple logons this pop-up has only two checkboxes, and the login will not succeed. All further steps of the script will probably fail either. The error message “FAILED: control could not be found by id” means that the particular “third” checkbox could not be found.

The issue may not occur in the EEM editor and the SAPGUI script work fine. The situation is typically different in the SAP Solution Manager system, if multiple robots are login on the ABAP system with the same user credentials and if by chance two robots/executions happen to run in parallel or shortly one after another.

Solution
To get the script running successfully you have the following options:

Configure different users per robot. Use for each script execution another user. This can be reached by setting the properties sapgui.user and sapgui.password in scope script_on_robot (different for each script instance on the robots).
Change the type of the user to SERVICE. This will prevent the pop-up for this user and as the advantage that the password will not expire.
Allow multiple logons with same user in the backend systems. Add the user to the profile parameter login/multi_login_users
Avoid parallel executions of the script. Synchronize executions in a way that a robot always logs out before another one tries a logon. See also Script Synchronization and Serialization

Symptom

After upgrading to SAP Solution Manager 7.2 sending a notification fails with the following error message:

[Thread[EEM WsNotifyQueue,5,main]] Error
com.sap.smd.eem.executor.wsnotify.WsNotifierST.doNotification notification call failed for URL https://<solman_host>:<port>/sap/bc/srt/scs/sap/ai_eem_bulk_notification_st?sap-client=XXX (user: SM_EXTERN_WS)

[EXCEPTION]

com.sun.xml.internal.ws.model.RuntimeModelerException: runtime modeler error: Wrapper class com.sap.smd.eem.wsnotifyst.soapfunctions.mc_style.AiEemBulkNotificationSt is not found. Have you run APT to generate them?

Background

The JVM version of the SMD Agent might be too old.

Solution

Check the JVM version of the SMD Agents a version like 6.1.010 is too old. Apply a newer JVM version (e.g. 6.1.076 ) as described in note 1774669 (Diagnostics Agent - How To Switch or Update SAP JVM) or in the wiki page Diagnostics Agent Maintenance Procedures.

Symptom

After upgrading to SAP Solution Manager 7.2 sending a notification fails with the following error message:

[Thread[EEM WsNotifyQueue,5,main]] Error com.sap.smd.eem.executor.wsnotify.WsNotifierST.doNotification notification call failed for URL https://<solman_host>:<port>/sap/bc/srt/scs/sap/ai_eem_bulk_notification_st?sap-client=XXX (user: SM_EXTERN_WS)

[EXCEPTION]

javax.xml.ws.soap.SOAPFaultException: Web service processing error; more details in the web service error log on provider side (UTC timestamp XXX; Transaction ID ...)

                at com.sun.xml.internal.ws.fault.SOAP12Fault.getProtocolException(SOAP12Fault.java:214)
                at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:117)
                at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:108)
                at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
                at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:135)
                at com.sun.proxy.$Proxy39.aiEemBulkNotificationSt(Unknown Source)

javax.xml.ws.soap.SOAPFaultException: SRT: Serialization / Deserialization failed
at com.sun.xml.internal.ws.fault.SOAP12Fault.getProtocolException(SOAP12Fault.java:210)
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:125)

Background

With older versions of the EEM Editor it was possible to create scripts with step names longer than 40 characters. In 7.20 the SOAP runtime throws an exception if the step name is too long.

In SRT_UTIL you can find an error like:

Data loss occurred when converting Step with a very long name 123456789012345678901234567890

Solution

Import the script in a new EEM Editor and reduce the length of the step names.

Symptom
Sending a notification fails with the following error message:
com.sap.smd.eem.executor.wsnotify.WsNotifier - doNotification: notification call failed for URL http://../sap/bc/srt/scs/sap/ai_eem_bulk_notification?
EXCEPTION
javax.xml.rpc.soap.SOAPFaultException: Processing Error. More details in WS Error Log (transaction SRT_UTIL) by selection with UTC timestamp
at com.sap.engine.services.webservices.jaxrpc.wsdl2java.soapbinding.MimeHttpBinding.buildFaultException(MimeHttpBinding.java:747)

In SRT_UTIL you see this error message:
<EXCEPTION_INFO>
<TYPE>CX_INVALID_TRANSFORMATION</TYPE>
<ERROR_TEXT>The transformation /1BCDWB/WSSBED2302F1E3963004DA could not be executed</ERROR_TEXT>
<CX_SOAP_CORE>
<E_LOCATION>
<CLASS>CL_SXMLP_DATA_ST==============CP</CLASS>
<METHOD>CL_SXMLP_DATA_ST==============CM004</METHOD>
</E_LOCATION>
<E_TEXT>CX_INVALID_TRANSFORMATION:An exception with the type CX_INVALID_TRANSFORMATION occurred, but was neither handled locally, nor declared in a RAISING clause.The transformation /1BCDWB/WSS<...> could not be executed
</E_TEXT>

Background
It can happen that, e.g. after an upgrade, some transformations seem to have disappeared:
Meta data still exists (and the contents of the transformation can be displayed in Transaction STRANS) but the TRDIR entry is missing.

Solution
Apply note 1390832. The note contains report RS_RETRIEVE_XSLT which must be run afterwards in order to re-create some missing transformation programs.

Symptom
java.lang.RuntimeException: No implementation for SHA or MD5:
java.security.NoSuchProviderException: no such provider: IAIK
#at
com.sap.smd.eem.wsnotify.bulk.AI_EEM_BULK_NOTIFICATION_Stub

Solution

Follow the description to enable SSL for WSnotify.

Workaround

Revert to http.

Symptom
EEM Robot does not show sapgui as supported protocol

Solution
The EEM robot tries to access the SAPGUI scripting object as test whether SAPGUI scripts can be supported on the robot. Possible reasons for not supporting SAPGUI scripts:

SAPGUI is not installed on the robot PC: SAPGUI for Windows must be installed
The scripting functionality of sapgui is not installed. Open SAPlogon -> options -> Scripting. There must be a message like "Scripting is installed".
EEM Robot fails to execute SAPGUI script due to missing privileges of MS Windows service-user "SAPService<DASID>".
Add the "SAPService<DASID>" to group Administrators and restart the EEM Robot

Reporting

Missing Data in Reporting
Even there are executions in the monitoring UI there are no entries available in reporting.
There might be several reasons for this behaviour.

The script or the steps of the scripts are not reporting relevant
Check in SOLMAN_SETUP step 6 if the flags for the Script/steps are set.
he extractor is not running
Check in extractor framework for the SAP Solution manager SID if the extractor EEM Reporting Data is running:

If the extractor EEM Reporting Data is not scheduled, then execute again the step BW Basic Settings of the Technical Monitoring -> End-User-Experience configuration in SOLMAN_SETUP.
- Go to step 'BW Basic Settings';
- Unselect the check box 'step BW Basic Settings';
- Save;
- Select the check box 'step BW Basic Settings' again;
- Save;
To prevent that scripts executed with trace falsify the normal response time reporting all executions with trace level > 0 are not reporting relevant. Check that property trace.e2etracelevel is not set permanently (neither in scope global nor in any other scope). If the property is set as in the screen shot below all executions are triggered with a trace and are therefore not reporting relevant.

FAQ's

General

As all other technical monitoring use cases EEM has three different kind of roles for configuration, first level support (L1) and second level support (L2). More details about these roles can be found in the Security Guide

SAP_SM_EEM_CONF

This role is required for all configuration tasks of EEM in SOLMAN_SETUP as well as uploading roles from the EEM Editor to the EEM Repository.
The role can be assigned or a dedicated user with this role can be created in SOLMAN_SETUP -> Basic Configuration -> Create Configuration user (Step 8).

SAP_SM_EEM_LEVEL01

This role is required for using the EEM Monitoring UI or the Alert Inbox. With this role the user has just display authorizations. It is not possible to trigger any trace in the Monitoring UI or confirm an alert in the alert inbox.

The role can be assigned or a dedicated user can be created in SOLMAN_SETUP -> Technical Monitoring -> End-User Experience -> Standard Users (Step 7).

SAP_SM_EEM_LEVEL02.

This role is required for using the EEM Monitoring UI or the Alert Inbox. With this role the user can trigger an extra execution (with trace) in the Monitoring UI or confirm an alert in the alert inbox.

The role can be assigned or a dedicated user can be created in SOLMAN_SETUP -> Technical Monitoring -> End-User Experience -> Standard Users (Step 7).

The network traffic depends on the type and number of scripts deployed to the robot and the frequency of script execution:

In general, every script execution triggers a result notification to SAP Solution Manager. Data volume for this is in the range of 1kB.

SAPGUI and RFC scripts trigger the same amount of traffic as in "real" execution.

HTTP scripts may be very sensitive to modifications. In particular, the following two aspects are relevant:
- Compression: Double check that the responses are compressed by looking at the response header content-encoding
- Static resources: Make sure that static resources (images, javascript, style sheets) are handled in the same way as for an end user in the browser. Typically these resources are cached in the browser and thus should be deactivated in the EEM script.

You can check the exact data volume in the EEM Editor looking at the replay. For each message bytes sent/received is reported.

Yes, to replay SAPgui-based EEM Scripts the scripting parameter has to be activated on the system. In some cases it is not possible to enable SAP GUI Scripting for lack of a dedicated application server. This implies that users who are allowed to use SAP GUI Scripting work on the same server as others, so the support cannot be enabled for the server. The problem can be solved by setting the rights to run SAP GUI Scripting per user. As in the previous chapter the profile parameter sapgui/user_scripting needs to be set to “TRUE”. The new profile parameter sapgui/user_scripting_per_user allows the administrator then to enable SAP GUI Scripting support for specific users. Unless the administrator explicitly changes the value, this parameter is set to “FALSE”. If the profile parameter is set to “TRUE” the following happens:

On the login screen SAP GUI Scripting is available for every user.
After login SAP GUI Scripting only remains available for those users that have the authorization for the Execute(16) action of the authorization object S_SCR in class BC_A.

Basically you need two users in SAP Solution Manager 7.01:

A dialog user is used for WebDynpro application and for communication from EEM Editor to EEM Repository. The User needs the Role Z_EEM_ADMIN or Z_EEM_DISP or a copy of it.

A communication user is used by EEM Robots to send the notification about a performed EEM script execution to SAP Solution Manager. The communication user needs role SAP_BC_WEBSERVICE_SERVICE_USER. You can create the user before EEM Setup and assign the role via CUA.

Be Careful

You need to set the password during EEM Setup to be able to forward the logon credentials to EEM Robots

With SAP Solution Manager 7.1 all required users can be created out of guided procedure for EEM setup. If you need to build up own roles or use CUA please refer SP corresponding SAP Solution Manager Security Guide which will provide all information what's inside SM_EEM_CONF_COMP and L1 and L2 role.

DB space depends on number of executions and if system data or traces are collected. The required space for each script execution can be estimated as 250 bytes + #Steps * 300 bytes. A script with 10 steps needs in average 3 Kbytes.

When system time is collected you can double the size. Additionally in JAVA schema each execution needs approximately 20Kbytes per execution. When the script is executed with trace flags (e.g. by setting temporary configuration) the size can increase depending on trace level and persisted traces up to Mbytes.

Therefore it is really recommended to check that housekeeping jobs are running correctly. See also the chapter about Housekeeping.

The EEM/UXMon robots act in the system landscape like real users.

To get a realistic end-user experience the user's roles and profiles mustn't differ from real human users.

The only exception is the licensing aspect: As the users used by EEM/UXMon have a technical character they are free of charge in all cases (but for Cloud applications).
Flag them as "Test".

However if possible it is recommended to use service users to avoid regularly password changes. To ensure that script replay with a service user is working correctly it should be recorded already with a service user.

Important: Be aware that Dialog Users and Service Users have a different behavior. Sometimes scripts will not work properly if you use a Service User. SAP Note 1322944 could be interesting in case of issues with SSO ticket.

No, it is designed to monitor a landscape from end-user perspective with a minimized impact on the landscape itself.

As background please consider the image below: The end-user runs the Citrix client on his desktop. This client communicates via the TRDP protocol to the Citrix server / Windows Terminal Server. On this server the actual application like SAP Gui or the browser is executed. The communication protocol between the application running on the Citrix server (and being displayed in the Citrix/WTS client) is then typically http or the SAP Gui protocol.

EEM does not support the Citrix/RDP protocol and an EEM Script cannot execute actions that are accessed via the Citrix protocol. But you can put an EEM Robot close to the Citrix Farm to monitor if the farm is able to access the SAP backends with a good performance therefore regard Citrix as a "long monitor cable". On one hand a performance issue between the end user and the Citrix farm cannot be monitored with this setup. On the other hand it is helpful to know that the issue is neither between Citrix farm and SAP nor inside the SAP landscape.

A technical scenario of type End User Experience (END_USR_EX) is required for full integration of interactive reporting, Root Cause Analysis and Workmode management. See also Technical_Scenarios.

It is the other way round. SAP End-User Experience Monitoring benefits from the integration of the E2E trace Analysis and its SAP Passport Technology in the EEM work flow.
To point it out shortly: EEM uses the Root Cause Analysis but is much more than that due to the pro-active character of EEM.

Yes, a single Robot can execute several different EEM Scripts against different productive SAP Systems. Of course the network configuration and firewall settings have to allow the contact between EEMRobot and Server.

EEM Robots are a re-use of the well introduced SMD Agent technology. From a technical point of view a EEM Robot is an Application (Aglet) running on an ordinary SMD Agent. A difference maybe the location of the EEM Robot. While a SMD Agent used for "Diagnostics in SAP Solution Manager" is situated close to the back-end, the EEM Robot runs close to the client-side/end-user in a SAP landscape.

The SMD Agents in the use-case of a Diagnostics Agent for root cause analysis could do the job of a EEM Robot, too. This setup may only be useful for test purpose. For a productive End User Experience Monitoring Infrastructure it is recommended to install a standalone SMD Agent as EEM Robot close to the real end users to be able to get monitoring data from their perspective.

HTTP scripts can be created by recording from a browser session by using the "EEM recorder mode" of the SAP Client Plugin (called EEM Recorder in this mode). Only Microsoft Internet Explorer is supported for EEM recording. More details on the currently supported browser versions and operating systems are mentioned in SAP Note 1435190. Note that other clients supported by the SAP Client Plugin for E2E Trace are typically not supported for EEM recording.

RFC scripts can be recorded from the "Business Explorer" (BEx) via the EEM Recorder. See also HowTo section EEM_Scripts_for_RFC.

SAPGUI scripts are recorded using the built-in recording functionality of SAPGUI for Windows. See also HowTo section SAPGUI Scripting.

In addition all types of scripts can be created manually from scratch, e.g. using wizards in the EEM Editor.

Currently http/https, SAPGUI, RFC, and Webservice EEM Scripts are supported. A mixture of different protocol types in one script is not possible.

EEM supports HTTP basic authentication, certificate based authentications as well as NTLM authentication.

www-authenticate: Negotiate (SPNEGO) is supported for both Kerberos and NTLM as of SAP Solution Manager 7.2 SP03.

Infrastructure

Direction EEM Robot -> SolMan

- P4 Port of Solmans Java Stack (5xx04; with xx=Systen Number)
- HTTP Port of solmans ABAP Stack
- optional RMI or HTTP Port of a Wily Enterprise Manager (default 6001 + 8081)

Direction SolMan -> EEM Robot

- no connections in this direction

It is recommended to install one EEM Robot per important business hotspot.

The prerequisites are derived from the underlying Diagnostics Agent. Diagnostics Agent installation guidance is available in SAP Note 1365123.

Some special requirements for EEM are:

To execute SAPGUI scripts an installed SAPGUI is needed on the EEM Robot's host.

Monitoring

There a several possible reasons for this behavior:

The EEM script may still contain static requests to download graphics or stylesheets while the browser has these objects in the local cache. If browser caching is enabled on end user PCs we recommend to disable those cached requests in the EEM script. Hint: Execute the complete business transaction once with the same browser before you record it in the EEM recorder. During recording the cache will then be filled as in real live. When importing the recording to the EEM editor it will activate exactly those messages which were not handled by the browser cache.

The EEM robot uses one HTTP connection, some browsers like IE8 up to 8 parallel HTTP connections per default.

Some browsers start rendering before all data is transferred completely. The EEM robots exactly measures the start and end point of the data transfer.

An effect in the opposite direction (lower response times in EEM) is caused by the fact that EEM does not render the http responses and does not process the Javascript code.

In general it is not a good idea to compare manual tasks with EEM Robot activities. The good news is that EEM results are comparable under each other, showing extreme low deviations.

In general the prerequisites as mentioned above in What is required to get E2E Traces from EEM Monitoring UI? must be fulfilled.

For all scripts executed against a SAP J2EE engine tracing must be enabled for a given time frame. Either or with 'Run script with trace' or 'Set temporary Configuration'. The time frame itself can be maintained in the eem_admin application on tab 'UI configuration' (for SAP Solution Manager 7.01) or via the monitoring UI -> Button Administration -> Temporary Configuration. See also Troubleshooting: Missing System data or traces

For ABAP and SAP J2EE systems

There are two possibilities to get E2E Traces from the EEM Monitoring UI of a tuple of EEM Script and Robot as shown in the image below:

Run Script with trace

This option triggers an additional immediate execution with the trace level configured in the Admin UI under the tab UI configuration.

Set Temporary configuration

This option will set the trace level for the scheduled executions in the time frame that is configured in the Admin UI.

Both options will enable the trace if user has authority before executing the script.

To get the E2E Trace integration working you need:

Ensure that E2E Trace is working for the recorded script(i.e. Root Cause Analysis and Managed System setup must be executed before) .
The system(s) you want to collect the traces for must be connected to SAP Solution Manager and setup must be executed.
The involved systems must be associated with the script
- Solution Manager 7.01: The involved systems must be entered in the admin UI under the scripts tab in the column Involved Systems as a comma separated list (e.g. CRP,BWP).
- Solution Manager >=7.1: The involved systems must be assigned to a technical scenario. This scenario is then assigned to the script. Both steps are performed in the EEM setup guided procedure.
The extractor for these involved systems must be scheduled. This is automatically scheduled when the scenario is created or can be done manually in the SOLMAN_SETUP step 3.2 (Maintain Scenario).

Hint: The term Involved Systems is just a group of systems and starting with Solution Manager 7.10 it is named Technical Scenario.

Remark: For SAPGUI scripts you just get statistical data but no detailed traces like SQL Trace.

For other components writing Introscope transaction traces

Other components that are writing Introscope transaction traces (like Tomcat, BOBJ, SMP,...) do not need a trace enabling. They react on flags in the SAP Passport. When flag/bit 7 is set (representing the decimal value 128) the transaction trace is send to the Introscope Enterprise Manager from where the E2E Trace application can collect them.

As a consequence if transaction traces should be collected permanently for the EEM script the property trace.e2etracelevel should have the value 128.

The style how script executions are displayed indicates whether detailed system data or traces are available. Bold written executions means system data has been collected. System data is read by E2E Trace application in form of statistic records. There is no overhead during script execution. Statistic records are written be default for each dialog step for ABAP systems. For SAP J2EE engine only if tracing has been enabled. See also What is required to get detailed times of system components in EEM Monitoring UI?

Italic written script executions means the script execution contains trace flags (>0) which might have an impact on response time. See also What is required to get E2E Traces from EEM Monitoring UI?

Bold Italic written script execution means script execution contains trace flags (>0) and Trace data has been collected.