Simple Diagnostics Agent Troubleshooting

Installation Issues

Unknown Operation DownloadSimpleDA

  • Issue: After setting up the SHA, adding the DownloadSimpleDA operation, and triggering the Outside Discovery, the SDA Installation fails with errors like "Unknown Operation DownloadSimpleDA"
  • Solution: Check the access rights and ownerships of the operation and its directory to look like that:

# ls -la /usr/sap/hostctrl/exe/operations.d total 24

drwxr-x--- 2 root sapsys 4096 May 24 16:30 .

drwxr-x--- 7 root sapsys 4096 May 24 16:30 ..

-rw------- 1 root root   1286 May 24 15:06 DownloadSimpleDA.conf

 

If it is not like this, change it accordingly and restart the Host Agent:

/usr/sap/hostctrl/exe/saphostexec -restart


Failed to start /usr/sap/..../SimpleDA.lmsldesc 

  • Issue: File /usr/sap/hostctrl/exe/descriptors.d/SimpleDA.lmsldesc is ignored by the SHA because permission rules are violated.
  • Solution:
    • Remove the file /usr/sap/hostctrl/exe/descriptors.d/SimpleDA.lmsldesc
    • Stop any SDA related OS processes
    • Start the installation procedure again


Illegal Permissions on Master Password File

  • Issue: Error message: http 500: Internal Server Error; The following error occurred in the Simple Diagnostics Agent: com.sap.solman.agent.core.error.SystemException illegal permissions on master password file
  • Reason: File /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password has wrong permissions. It seems that the OWNER_READ bit is not set for the master password file
  • Solution:
    • Ensure that /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password has the correct permissions, e.g. -r-------- 1 sapadm sapsys 256 Nov 12  2017 master_password
    • Restart the agent using the Agent Action "ping" in the Agent Administration UI


Configuration of the SDA HTTP Port: java.net.BindException: Address Already in Use

  • Symptom: On the monitored host, the default Simple Diagnostics Agent HTTP inbound port '3005' might be occupied by another system process. If so, the Simple Diagnostics Agent cannot start and the SAP Host Agent returns the HTTP request with the response "HTTP 500"
  • Issue: The SDA cannot be started because the reserved tcp port '3005' is already bound by another process
  • Solution: The tcp port '3005' is usually reserved for the http port of the SDA. So, first try to switch the process that blocks the SDA to another port. If this is not possible, follow SAP Note 2549424 - Configure the Simple Diagnostics Agent's HTTP Port


Cannot Execute Java VM

  • Issue: java.exe cannot be executed in the SDA installation directory (e.g. SMDAgent/default/sapjvm/bin/java -version)

  • Reason: The msvcr100.dll is missing on the operation system
  • Solution: Apply the SAP Note 1837221 - SAP JVM requires the Microsoft Visual C++ 2010 runtime

SDA Checks General

All these checks can be done once the SDA has been deployed.


Check SDA Connectivity

  • At the Agent Action select Ping Agent and press Go.
  • You should get a pop-up containing some JSON formatted information.
  • if there is no pop-up
    • have a look at the Info Area at the top of the screen for some error messages
    • if you suspect connectivity issues
      • check the connectivity in transaction SM59


Check SDA Installation

  • in the details pane have a look at the Installation Log Messages. If you like to update the SDA, select the host and choose update from the selection menu.
  • !!! Unfortunately the window won't be refreshed, you have to search again to get the current information displayed !!!

Quick checks (Unix, adapt accordingly for Windows)

clear ; \

echo '=== SHA installed? ===' ; ls -l /usr/sap/hostctrl/ ; cd /usr/sap/hostctrl/ ; \

echo '=== SLD Destination configured? ===' ; ls -l exe/slddest.cfg* exe/config.d/slddest* ; \

echo '=== SLDREG run? ===' ; ls -l work/outsidediscovery.log ; \

echo '=== SDA installed? Version? ===' ; grep version SMDAgent/default/agent/configuration/agent/default.properties ; \

echo '=== Java Symlink present? ===' ; ls -l SMDAgent/default/sapjvm ; \

echo '=== Java startable? Version? ===' ; SMDAgent/default/sapjvm/bin/java -version ; \

echo '=== SDA running? ===' ; ps -fu sapadm ; \

echo '=== SDA pid correct? ===' ; cat SMDAgent/default/agent/pid ; \

echo '=== SDA password file permissions correct? ===' ; ls -l /usr/sap/hostctrl/SMDAgent/default/configuration/agent/#/master_password ; \

echo '=== port file present? ===' ; cat SMDAgent/default/agent/port

 

Expected Output

=== SHA installed? ===

total 12

drwxr-x---  3 sapadm sapsys 4096 Mar  3 13:41 SMDAgent

drwxr-x--x  6 root   sapsys 4096 May  4 18:32 exe

drwxr-x--- 11 sapadm sapsys 4096 May  4 18:32 work

=== SLD Destination configured? ===

-rw-r----- 1 root root 114 Jan 24 15:57 exe/slddest.cfg

=== SLDREG run? ===

-rw------- 1 root root 1049136 May  4 18:08 work/outsidediscovery.log

=== SDA installed? Version? ===

version=1.10.0.201705041226

=== Java Symlink present? ===

lrwxrwxrwx 1 sapadm sapsys 47 May  4 16:39 SMDAgent/default/sapjvm -> /usr/sap/hostctrl/SMDAgent/default/sapjvm_8_jre

=== Java startable? Version? ===

java version "1.8.0_112"

Java(TM) SE Runtime Environment (build 8.1.028)

SAP Java Server VM (build 8.1.028 25.51-b02, Dec 14 2016 16:11:21 - 81_REL - optU - linux amd64 - 6 - bas2:282279 (mixed mode))

=== SDA running? ===

UID        PID  PPID  C STIME TTY          TIME CMD

sapadm   12082 29457  0 17:44 ?        00:00:00 /bin/sh ./go.sh

sapadm   12140 12082  2 17:44 ?        00:01:31 ../sapjvm/bin/java -server -XtraceFile=../log/vm-trace.log -XX:OnOutOfMemoryError=./oom.sh -Xmx512m -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../heap-dump -XX:HeapDumpOnDemandPath=../heap-dump-OnDemand -XX:ThreadDumpPath=thread-dump -Declipse.ignoreApp=true -Djava.awt.headless=true -Dj

sapadm   22143     1  0 18:32 ?        00:00:01 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D

=== SDA pid correct? ===

12140

=== SDA password file permissions correct? ===

-r-------- 1 sapadm sapsys 256 Nov 12  2017 master_password

=== port file present? ===

loopback:3005

 

Check SDA Configuration

  • in the details pane have a look at the Configuration Information
  • find the last configuration you are looking and check the content by clicking the link in the Status, Version column to see the content of a particular configuration message
  • click the link in the Error Message column to see the full text of it.
  • The Configuration status is:
    • White if the Diagnostics Agent is not configured. This should not be happen because there is always a default configuration at installation time. If you see a white configuration status anyway, do the following check
      • Open the 'LMDB' application
      • Search the host and open it by click on 'Display' after selecting the host in the result table
      • Get the timestamp of 'Last Change by Data Supplier'
      • Open SM37
      • Find Enter in the simple job selection
        • 'Job name': SAP_LMDB_NOTIFY_LDB*
        • 'User name': FRN_BTC_LDB
        • 'Job Start Condition': enter a time around the timestamp (-/+ 1h) you found in the LMDB
      • Execute the search
      • Check of all jobs were 'Finished'
      • Otherwise find the root cause in the job log and create a ticket for SV-FRN-INF-LDB
      • It is possible to execute the installation manually by using the 'Agent Administration'
    • Green means that the metrics are sent successfully from the Solution Manager to the Diagnostics Agent. The status can have two different  characteristic Confirmed or up-to-date, if the status is Confirmed than the agent has the configuration versions that it has expected. If the status is up-to-date, the confirmation by metrics is still missing. 
    • Yellow pending: the SRSM can't send any metrics to the Diagnostics Agent but it is assumed that it will be successfully send later. For that reason the SRSM tries to send the configuration in an interval and the status is “pending”. This happens e.g. if network problems occur, the authorization fails or the agent is not available for the moment.
    • Yellow Confirmation outdated means that the SRSM has sent metrics to the Diagnostics Agent but the metric version is older than expected. The SRSM tries it later again (“self-healing”). You can identify this in the configuration information, looking for the Send Count > 1.
    • Red if an error occurred on Diagnostics Agent side and it cannot manage the configuration. 

SDA Checks via Agent Administration

  • Start the Agent Administration form FIORI launchpad.
  • Search the corresponding host.
    • If the host name is known, go to Host Search
    • otherwise, go to Technical System Search.
  • In the result list, select the corresponding row.
  • The Diagnostics Agent's state is
    • White if the self-monitoring is not yet configured (currently part of the MAI configuration which is in Managed System Setup/Simple System Integration)
    • Green if the heartbeat is received regularly at the SRSM
    • Red if the heartbeat is configured, but is not received


SDA Availability State 'GREEN'

Configuration Status "Confirmed"

  • Issue:  No issue. The SDA is working perfectly.
  • Solution:  If still something is missing, re-execute the Simple System Integration configuration.

Configuration Status "Confirmation outdated"

  • Issue:  No issue. The SDA is working perfectly.
  • Solution:  Wait some time until the confirmation is calculated. This is only a timing issue.

Configuration Status "Up-to-date"

  • Issue: No issue. The SDA is working perfectly.
  • Solution: Wait some time until the confirmation is calculated. This is only a timing issue.

Configuration Status "Error"

  • Issue: The SDA is still able to send metrics but is not able to retrieve new configurations or updates. Typically the SDA is in an inconsistent state due to update issues but the OS process is still running. 
  • Solution: Click on "Error" and follow up with SDA Error Messages

Configuration Status "Pending"

  • Issue: The SDA is still able to send metrics but is not able to retrieve new configurations or updates. Typically the SHA (SAP Host Agent) is not available/reachable.
  • Solution: Execute the PING agent action for the specific host and follow up with SDA Error Messages    


SDA Availability State 'GREY'

Configuration Status "Pending"

  • Issue: The SDA did not receive its self monitoring configuration. Typically the SHA (SAP Host Agent) is not available/reachable.
  • Solution: Execute the Agent Action "PING" and follow up with SDA Error Messages regarding the ping result.


SDA  Availability State 'RED'

Configuration Status "Error"

  • Issue: The SDA is not able to send metrics and not able to retrieve new configurations nor updates. The SDA is in a bad shape.
  • Solution: Click on "Error", select the first line in Configuration tab with a red configuration status and follow up with SDA Error Messages regarding the displayed error.

Configuration Status "Pending"

  • Issue: The SDA is not able to send metrics and is not able to retrieve new configurations or updates.
  • Solution: Execute the Agent Action "PING" and follow up with SDA Error Messages regarding the ping result.    

Configuration Status "Confirmation outdated"

  • Issue: The SDA is not able to send metrics but was able to retrieve new configurations or updates. The server is working on a configuration confirmation
  • Solution: Wait some time and press "Refresh".  This is only a timing issue.

Configuration Status "Up-to-date"

  • Issue: The SDA is not able to send metrics but was able to retrieve new configurations or updates. The server could not confirm the configuration so far.
  • Solution: Open the Agent Internal application from Agent Action drop down. Check in Application "agent", if the value for URL and e2e.mai.user are correct and on your SAP Focused Run server, if the e2e.mai.user is not locked. If the "agent" application is not configured correctly, re-configure the SDA by starting "Configure Agent" from Mass Action menu entry. If the user is locked, unlock the user. If the unlock still occurs, check, if the SDA's that should connect against the SAP Focused Run system have the correct credentials.

SDA Error Messages

Bad credentials in SM59 destination xxx

  • Reason: The destination that is used to create a connection between SAP Focused Run and a SDA has wrong credentials for the sapadm user of SAP Host Agent.
  • Solution: Adjust the credentials in the corresponding destination on the SAP Focused Run system. In case you need to reset the credentials, consult the SAP Host Agent administrator.


Connect to <hostname>: 1128 failed: NIECONN_REFUSED(-10)

  • Reason: On the host, there is no SAP Host Agent installed. 
  • Solution: The solution depends on the result that is expected. If the host is
    • still valid
      • start with the (re-)installation of a SAP Host Agent
    • invalid
      • start the cleanup of the host. The cleanup is a complex procedure and documented in Decommissioning


Connect to <hostname>: 1128 failed: NIEHOST_UNKNOWN(-2)

  • Reason: The SAP Focused Run system is not able to create a connection from the SAP Focused Run system to the SDA. The network is not able to find the host. 
  • Solution: The solution depends on the result that is expected. If the host is
    • still valid
      • start with the (re-)installation of a SAP Host Agent
    • invalid
      • start the cleanup of the host. The cleanup is a complex procedure and documented in Decommissioning


Connection to partner timed out after 180s

  • Reason: The SAP Host Agent is not able to connect to the SDA via HTTP, but the SDA process is still up and running. Usually the root cause is an issue with the SDA update process. 
  • Solution: Stop the SDA process on the host.
    • Windows
      • Start the Task Manager, find the java process and stop it.

  • Linux
    • execute 'ps -fu sapadm' to find the running process (<pid>). Stop the process by 'kill <pid>'.

  • In the Agent Administration, select the host and execute the Agent Action 'PING'. This operation should restart the agent. It cannot be precluded that this results aftereffects.


Error: HTTP 404 Not Found

  • Reason: The SAP Host Agent is not able to create a connection to the Simple DIagnostics Agent. In the directory of the SDA a port file is missing. Usually the file will be created, when the SAP Host Agent starts the SDA for the first time. But the start of the SDA might fail if on the host, several processes of the SDA were started.

  • Solution: Stop all java processes of 'sapadm' user, Stop all /bin/sh ./go.sh processes of user 'sapadm'. Re-Install the SDA by executing the Mass Action "Install/Update Agent" in the Agent Management Application (Agent Admin UI).

Error in SDA Configuration

The issue can be seen, if the Agent Action "Configure Agent" ended with "Error". In this case, click on the error message link in the configuration tab. If you see the error details like:

 

http 500: Internal Server Error; Internal Server Error

Operation [id=00505694209A1ED7A8D674965B8A62D9] terminated with errors : 127 ()

description:Start SDA

Error: could not open `C:\Program Files\SAP\hostctrl\SMDAgent\default\sapjvm\lib\amd64\jvm.cfg'

exitcode:1

 

  • Reason: The SDA is not installed or in a corrupt state.
  • Solution: Re-Install the SDA by using the Agent Action "Install/Upgrade Agent". Re-trigger the "Configure Agent" action afterwards. Ensure that no old SDA process is running (otherwise kill the process) and there is enough disc space left to install and operate the SDA before re-installation will be triggered. Refer to SAP Note 2501820 - Simple Diagnostics Agents needed Disk Space.


Unable to Redeploy an SDA (Install After Manual Deletion)

It is not possible to install/update a SDA after manual deletion of the SDA on the host. The installation procedure skips the installation with the notice "SDA and JRE are already up-to-date, skip updating binaries"

  • Reason: During the installation procedure it is checked, if the version, that will be installed is newer then the existing one. If they are identical, the installation will be skipped to avoid unnecessary load in the system. The existing version of the SDA was recognized in the LMDB with outside discovery data upload.
  • Solution: The version in the LMDB must be changed so that the comparison returns a difference in the existing and the update version. Please follow the process
  1. Start the LMDB by selecting the host in the Agent Admin UI and press "Display Host" button.
  2. Change into the "Edit" mode.
  3. Press the "Details" button
  4. Select the tab "Associated Instances"
  5. Click on "Diagnostics Agent default on ...." in the column "Associated Instance"
  6. Select tab "Properties", press "Edit" (confirm the pop up) and scroll to the Version attribute
  7. Change the value to something different than the current value and press "Read Only"
  8. Close the pop up and press "Save"
  9. Restart the update of the agent in the Agent Admin UI


DPCConsumer Could Not Push Metric Data

  • Reason 1: The Simple Diagnostics Agent uses an outdated HTTP-client implementation.
  • Solution 1: Update the Simple Diagnostics Agent to version 1.19 or higher.
  • Reason 2: The Solution Manager or the network is under load and processing or transmitting is delayed.
  • Solution 2: Set the property servlet-time-out to 120 (default 60) (time in s) in the agent's configuration file (hostctrl/SMDAgent/default/configuration/agent/[SM]/custom.properties).
  • Component: SV-FRN-INF-SDA


A Sync Point Is Missing in the Result from the ABAP System (Solution Manager XYZ). Check If the DPC Sync Point Functionality Is Available

  • Reason 1: The ST-PI version might be outdated. This can be checked at SAP Logon → System → Status ... → SAP System Data → Details.
  • Solution 1: To make the DPC_SYNCPOINT running, the ST-PI version must be at least 4.
  • Reason 2: Another reason could be a problem with time zone settings of the system. Check the TZCUSTHELP report in the statistic records in STAD. If it reads "System time zone or UTC timer is incorrect"
  • Solution 2: Correct the time zone settings.
  • Component: SV-FRN-INF-PRV or SV-FRN-INF-SDA

Additional helpful information for support:

  • Access right issue: Transaction ST01 → Authorization Trace for user SDAGENT (or the one that is configured) → activate → wait 5 minutes → deactivate → analyze → attach result to the case
  • Time zone issue: Transaction SA38 in managed system → TZCUSTHELP → Execute → enter current date and time → attach result to the case


Web Service Invocation Error Occurred on Binding Provider

  • Reason: The SDA does not have appropriate privileges to call WS-methods at sap control.
  • Solution: Add the following line to the profile DEFAULT.PFL:  service/admin_users = sapadm. This is mentioned in the SAP Focused Run Expert Portal.

Troubleshooting Resource Consumption

Troubleshooting Resource Consumption and Booting Issues of the SDA

Troubleshoot Out-of-Memory Issues

In case of out-of-memory issue the SAP support might need heap dumps to analyze the issue. Those can be generated as follows:

  • If possible, the SDA automatically creates a heap dump in case of an out-of-memory error. If you have OS access to the affected SDA, check if the file /usr/sap/hostctrl/SMDAgent/default/agent/default/heap-dump exists and if so, save it. Additionally, save /usr/sap/hostctrl/SMDAgent/default/agent/error if existing.
  • Alternatively, you can manually trigger and download a heap dump of the SDA using the Agent Action Get Heap Dump in the Agent Administration. Mind that this only works if the SDA is running and connected to the SAP Focused Run server. If the out-of-memory reproducibly occurs after a dedicated amount of up-time, this functionality can be used to get a heap dump some time before the actual out-of-memory error happens.
  • If the Get Heap Dump function cannot be used because the SDA does not connect to the SAP Focused Run server, you can locally generate a heap dump by using the jvmmon tool:
    • Execute /usr/sap/hostctrl/SMDAgent/default/sapjvm/bin/jvmmon.
    • If there is more then one Java process running, enter the process ID of the SDA process (to be identified comparing the process path). Otherwise, the process is selected automatically
    • Enter dump heap.
    • Enter exit.
    • The heap dump files heap-dump-OnDemand.hprof and heap-dump-OnDemand.addons can be found at /usr/sap/hostctrl/SMDAgent/default/.

Troubleshooting SAP Host Agent

In some cases it is necessary to increase the log level of the SAP Host Agent without restarting the agent itself. If you are hosting the SAP Host Agent on a Linux machine, you are able to change the log level in the following way:

  • User-defined signal 1 (SIGUSR1, USR1) → decrease the tracelevel
  • User-defined signal 2 (SIGUSR2, USR2) → increase the tracelevel

Example:

kill -USR2 <pid> 

Where <pid> is the process id of the saphostctrl process.

Running SDA and Diagnostics Agent on the Same Host

It is possible to run an SDA and a Solution Manager Diagnostics Agent (SMDA, Solution Manager 7.2) on the same host. But for flawless operations the SMDA needs to be configured for Host Agent outside discovery which is also used for the SAP Focused Run scenario. Therefore follow SAP Note 2556432 - Switch Outside Discovery from Diagnostics Agent to SAP Host Agent, and use the local System Landscape Data Router (SLDR) for the SLD parameters in step 6.1. See also Preparing System Landscape Data Router for more details on the SLDR and its usage in mixed SAP Focused Run and SAP Solution Manager operations.

Report SRSM_AMA_UPLOAD_BINARY

For SAP Focused Run versions 2.0 FP01 and FP02

If you think, an error message is incomplete or cut-off, apply SAP note 2886763 - Report SRSM_AMA_UPLOAD_BINARY error message is cut



For all SAP Focused Run versions

If the report generates an error message which starts with:

     "Error when analyzing *.SAR: error during execution of SAPCAR extract command, error_code=0, exit_code=28"

Delete the following folders in the tmp-directory of the application server:

  • agent
  • sapjvm_8_jre

The real name of the tmp-directory can be identified in transaction AL11, parameter DIR_TEMP.

 

Advanced Configuration

The SDA can be customized. All known options are documented in a dedicated page. Please check Simple Diagnostics Agent - Advanced Configuration.