CA APM Introscope(R) is an application performance management solution created to manage Java Application performance. Unlike development tools, Introscope(R) is designed to scale with minimal performance impact. This allows you to monitor and manage your application performance in live production environments.
The Right to View (RTV) version of CA APM Introscope is a restricted, read-only form of the full product and is bundled with SAP Solution Manager. With the RTV version, support is limited to products that are licensed and supported by SAP. The instrumentation, dashboards, Probe Builder Directives (PBDs), management modules, and Smartstor data contained within the RTV version of CA APM Introscope as provided by SAP is the intellectual property of SAP. Use of these functions is restricted by SAP and may only be used in an unrestricted manner by purchasing a full license from Broadcom.

Scope

• Monitoring of non-ABAP applications and infrastructures
• Tracing of non-ABAP applications and infrastructures

Setup

• SAP Setup Guide for Introscope 10.7

SAP Solution Manager Basic Configuration → Configure CA Introscope

In this SAP Solution Manager setup step, you connect existing CA Introscope Enterprise Managers to SAP Solution Manager.

When discovering EMs these are made known to SAP Solution Manager.

For this setup action some files of Enterprise Manager have to be read:

• <EM_HOME>/Introscope_Enterprise_Manager.lax
• <EM_HOME>/config/IntroscopeEnterpriseManager.properties
• <EM_HOME>/config/users.xml
• <EM_HOME>/config/domains.xml
• <EM_HOME>/product/enterprisemanager/plugins/com.wily.introscope.em_<version>.jar
• <EM_HOME>/bin/EMService.conf (Windows only) 

Also some files of the Enterprise Manager get edited by different SAP Solution Manager setup steps:

• <EM_HOME>/config/IntroscopeEnterpriseManager.properties (Basic Configuration → Configure CA Introscope => Discover Introscope EM)
• <EM_HOME>/config/users.xml (Basic Configuration → Configure CA Introscope → User Management)
• <EM_HOME>/config/domains.xml (Basic Configuration → Configure CA Introscope → User Management)
• <EM_HOME>/sap/<SolMan_SID>.e2emai.properties (Basic Configuration → Configure Automatically → Push DPC Configuration to CA Introscope)
• <EM_HOME>/config/agentcluster.xml (Managed Systems Configuration → Configure Automatically → Generate System-level Metrics)

Accordingly the user running the Diagnostics Agent assigned to the Enterprise Manager host has to have read/write permissions to the above listed folders and files.

Basic Configuration → Configure Automatically → Push DPC Configuration to CA Introscope

In this automatic activity, you send the Data Provider Connector configuration to the CA Introscope Enterprise Managers. This information is required to enable CA Introscope Enterprise Managers to send data via the DPC push functionality to the SAP Solution Manager system. In other words the Enterprise Manger then knows to which SAP Solution Manager(s) it has to send metrics to Monitoring and Alerting Infrastructure.

You may also see  SAP Note 1751225 Introscope Push.

If the Diagnostics Agent assigned to the Enterprise Manager host got changed this activity has to be executed again.

Managed Systems Configuration → Enter System Parameters → Common Parameters

In this step you can choose the Enterprise Manager which will be used by the managed system agents to report performance data.

The Enterprise Managers of type MoM (Manager of Managers) and Standalone are available to be chosen. The Enterprise Managers defined as Collectors will not be shown in this step. If a MoM scenario is configured, it is required to choose the MoM EM. In this case, the agents will connect firstly to the MoM EM and then they will be automatically redirected to a collector node.

Managed Systems Configuration → Configure Automatically → Generate System-level Metrics

In this automatic activity, you enable CA Introscope Enterprise Manager to aggregate performance metrics on system and instance level. System and instance level aggregates of selected metrics are used in system monitoring.

Introscope agents report metrics per Java node to the Enterprise Manager. By adapting the agentclusters.xml file, the Enterprise Manager is enabled to build instance and system level aggregates for selected metrics.

The Introscope Host Adapter (aka WilyHost agent) is an application which runs inside the SMDAgent. One of the tasks of the Introscope Host Adapter is to collect data from different configurable destinations and push them to the Introscope Enterprise Manager. The following picture provide a big picture of the architecture of the Introscope Host Adapter.

All applications running inside the SMDAgent are available in the Agent Administration UI. The Introscope Host Adapter can be found as shown in the picture below.

1. Agent Administration UI
2. Application Configuration
3. com.sap.smd.agent.application.wilyhost

The resources used by the Introscope Host Adapter application can be found under the 'Application Resources' node:

Note: When using the application configuration take care when performing changes. Changes should not be performed in the scope GLOBAL unless they are strictly required. The changes in scope global affect ALL SMDAgents connected to the SAP Solution Manager system.

In SAP Solution Manager, the Introscope Host Adapter is configured during the activity 'Introscope Host Adapter' of the Managed System Configuration for the target system: Step 7 "Finalize Configuration".

When the activity Introscope Host Adapter is performed, the application is configured to collect data from the managed system that will be monitored. The main files used by the agent are the following:

• Files create/modified by setup:
  • SAPAgentConfig.xml
    • File created during the Introscope Host Adapter setup activity. It contains the destinations configuration and the link to actions to be performed for each destination
  • IntroscopeSAPAgent.profile
    • File is created from the IntroscopeSapAgent.profile.template. The host and port of the Enterprise Manager are configured in this file.
• Files used by application:
  • WilyHostAgentTemplate.xml
    • The is a global template file that defines all the possible actions possible for collecting all the Introscope Host Adapter data.
  • WilyHost.xsd
    • The is the XML schema file that defines the rules of validation for the WilyHostAgentTemplate.xml file. If you modify the action template it needs to conform to the schema.

Note: No manual configuration must be performed to the above files. The activity Introscope Host Adapter will configure the application automatically.

The file SAPAgentConfig.xml holds the configuration of the Destinations and the Actions that will be used/performed to collect data from the managed system.

• Destinations: The destinations will point out where the information will be collected. The destinations are configurable, for example a destination might be a Jco connection to the managed system or the location of a specific file.

For a typical double stack system you will have configurations generated for the following destinations:

•  ABAP instance
• JAVA instance
• SAPStartSrv
• SAPHOSTAGENT
• File (for GC)

The following picture shows an example of a Jco destination. As you can see, the Jco destination has the connection information to the managed system, client, user, password, system number and hostname of the application server.

• Actions: The actions indicates to the Introscope Host Adapter what are the metrics to be collected from the managed system. For all destinations in the SAPAgentConfig.xml a pool of actions are configured. The actions point to an action templates that are defined in the WilyHostAgentTemplate.xml file.

The following picture shows an example of some actions defined in a SAPAgentConfig.xml.

The SAPAgentConfig.xml fully configured looks as follows:

The following picture describes how the destinations and actions work in the managed system. As you can see, the destinations are the location where the data will be collected and the actions are the collections of data.

The file IntroscopeSAPAgent.profile holds the connection configuration to the Introscope Enterprise Manager. This file will tell the Introscope Host Adapter application what is the Enterprise Manager that the application must connect. The activity Introscope Host Adapter in the Managed System Configuration uses the IntroscopeSapAgent.profile.template file and then creates the IntroscopeSapAgent.profile. The following picture shows an example of IntroscopeSapAgent.profile.

Troubleshooting

If the Introscope Host Adapter applications shows issues, for example: agent is not connected to the Enterprise Manager or managed system metrics are not being collected. Then the SMDAgent logs must be checked to find out what is the root cause of the issue. The Introscope Host Adapter application may not work correctly due to distinct issues and the SMDAgent logs will show the root cause.

The Introscope Java Agent is a third party tool created by CA Technologies and is used in combination with the CA Introscope Enterprise Manager. The Introscope Java Agents and Introscope Enterprise Manager is used in the context of the SAP Solution Manager to collect performance information of Java applications, its main purpose is to collect data from the applications and systems running on Java Virtual Machines (JVMs), the data collected is pushed to the Introscope Enterprise Manager where it is stored.

The following picture shows the architecture of the Introscope Java Agents in SAP NetWeaver, but the same Introscope Java Agent is also used to monitor others SAP systems, for example: SAP Mobile Platform and SAP BusinessObjects Enterprise.

Introscope Java Agent Configuration and Download Instructions

Each Introscope Java Agent release supports certain JVM versions. The following table provides an overview about the supported JVMs depending on the Introscope Java Agent release. The download location of the Introscope Java Agent is stated in the Introscope release notes.

It is not possible to connect Introscope Java Agents to an Introscope Enterprise Manager with a version lower than the agent's, i.e. do not choose an Introscope Java Agent with higher release than the Enterprise Manager for any Managed System.

Configuration Steps

The configuration of the Introscope Java Agent consists in the following steps:

1. Deployment of the Introscope Java Agent files in the managed system server(s)
2. Configuration of the Introscope Java Agent profile;
3. Configuration of the Java parameters of the Introscope Java Agent in the managed system;

Introscope Java Agent Profile

The configuration of the Introscope Java Agent is stored in the IntroscopeAgent.profile. The main properties configured in this file are as follows:

• introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=<EM_host>: The host of the Enterprise Manager that the Introscope Java Agent will connect.
• introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT=<EM_port>: The port of the Enterprise Manager that will be used by the Introscope Java Agent.
• introscope.autoprobe.directivesFile=<Directive_Files>: The Auto Probe Directive files that will be used to instrument the JVM.

Auto Probe Directive Files

The Introscope Java Agent instruments the SAP JVM inserting probes (measurement points) on the Java byte codes during their executions. The measurement points are configured by the Directive Files (*.pbd), which are added in the IntroscopeAgent.profile. SAP provides standard Directive Files that should be used depending on the managed system type. The available Directives Files are mentioned in the IntroscopeAgent.profile of each release of Introscope Java Agent (check the section Directives Files of the profile file).

For NetWeaver systems, the configuration of the Auto Probe files is usually done automatically during the Managed System Configuration (activity Byte Code Adapter Installation). For non-NetWeaver systems, the required files are mentioned in the setup guides for each system type. Take care when changing the Auto Probe files configuration, as some Auto Probe files might cause a performance impact.

Java Parameters

The Java parameters required to start the Introscope Java Agent are as follows:

Configuration for Distinct Managed System Types

SAP NetWeaver Java Systems

The configuration of the Introscope Java Agent of SAP NetWeaver systems is performed automatically by the activity "Byte Code Adapter Installation"  in the step "Finalize Configuration of the Managed System Configuration". This activity extracts the Introscope Java Agent files to the server, configures the IntrosocopeAgent.profile and adds the required Java parameters to the Java server nodes.

Non-NetWeaver Java Systems

The Introscope Java Agent is also used to monitor Java systems that are not based on the SAP NetWeaver platform. The support and the configuration steps of these systems vary depending on the system type. Refer to the configuration guide of each system type in the Maintenance of Product in the System Landscape Wiki.

Introscope Agent Log Files

When the Introscope Java Agents are configured and started for the managed system, the Introscope Java Agent will create log files which will provide runtime information. The Introscope Java Agent log files are:

• IntroscopeAgent * .log: Provides runtime Information about the Introscope Java Agent and connection to the Enterprise Manager.
• AutoProbe.log: Provides information about the monitored classes/methods of the Java Application. Limit or disable auto probe logging as described here: https://support.ca.com/us/knowledge-base-articles.TEC595870.html

The location of the log files may vary depending on the configuration and system type, some example are below:

SAP NetWeaver Java Systems:

When the automatic setup is used for SAP NetWeaver Java systems (activity Byte Code Adapter Installation), the Introscope Java Agent log files will be available at \usr\sap\SMD\SMDA97\SMDAgent\temp.

If the Diagnostics Agents on-the-fly feature is used in the managed system, the flag "Deploy Byte Code Adapter under Managed System Instance(s)" must be selected during the Managed System Configuration. In this case the Introscope Java Agent log files will be located at /usr/sap/<ManagedSID>/<InstanceID>/ByteCodeAdapter/logs.

If the manual installation has been performed, the Introscope Java Agent logs are available by default in the following location \usr\sap\ccms\wily\logs.

SAP BusinessObjects Enterprise:

• <BOE_PATH>\SAP BusinessObjects Enterprise XI 4.0\Java\wily\logs

How to Read the Log

If the connection has been established successfully, the following entry is shown in the IntroscopeAgent* .log:

Connected controllable Agent to the Introscope Enterprise Manager at myhostname.domain.com:6001 [...]
Host = "myhostname", Process = "SAP Netweaver", Agent Name = "SMJ_J002_server0", Active = "true".

The above log entry shows the Introscope Enterprise Manager host name and port as well as the host name, the process name and the agent name used by the Introscope Java Agent to report the performance information. This information is useful in order to find the Introscope Java Agent in the Web View or Workstation Investigator. The following picture shows the Introscope Java Agent node in the Introscope Investigator:

Troubleshooting

Byte Code Adapter Does not Start

After performing the configuration of the Introscope Java Agent it is not possible to see the Java metrics in the Introscope Enterprise Manager. Refer to the Byte Code Adapter Does not Start wiki page.

Managed System Does not Start When Byte Code Adapter is Active

In case you need to disable the Introscope Agent in an emergency case, follow SAP Note 2246406.

JMX Metrics Missing in the Introscope Web View

If the JMX metrics are missing, the Early Watch Alert will be rated gray as metrics "APPLICATION THREADS" and "ACTIVETHREADSCOUNT" will be missing. Refer to SAP Note 2461965 (Missing metrics: Java performance data missing).

High Availability Setup for Introscope Enterprise Manager

High availability is typically requested for the MoM of an Enterprise Manager (EM) cluster, but can also be implemented for a stand-alone EM. HA for collectors is less common because the assumption is that other collectors take over the load of a failing collector.

There are two approaches to achieve high availability of the EM:

1. EM Hot Failover
2. "standard" approach based on HA software

EM Hot Failover

This solution is defined by Introscope and relies on features built into the Enterprise Manager. Two EMs are active at the same time and they negotiate which of them should be responsible for alerts etc. The setup is described in the Config and Admin Guide. Switching between the EMs relies on DNS changes. Unfortunately this approach cannot be used in combination with SAP Solution Manager since the SMDAgent responsible for administrating the EM are not capable of supporting this.

Standard Approach based on HA Software

The standard approach is to install services on a logical host and to switch this logical host between different physical hosts using some kind of HA software. This is supported by diagnostics agents and can also be used for Enterprise Managers. In comparison to the EM Hot Failover approach this approach causes the additional delay of starting the EM.

Prerequisites

1. A clustering / high availability solution is installed with physical hosts phostA, phostB
2. A switch over group that should control the Introscope EM is defined that contains a logical server name emhost and an associated IP address.
3. Diagnostics agents are installed on both physical hosts phostA, phostB
4. The agents on-the-fly feature is enabled for both agents on phostA and phostB. emhost is one of the logical host names that are covered by the agents. See diagnostics agents and HA for more details.

Introscope EM Setup

The following steps refer to Microsoft Cluster Services (MSCS) as HA solution. If needed adapt the procedure to your solution.

1. Create a Cluster Disc (shared volume) in the switch over group in MSCS. Let's assume this is Windows drive S: and we plan to install into folder S:\introscope.
2. Ensure that the diagnostics agent has read and write permission to this file system / folder. Typically this means that user SapServiceDAA (for diagnostics agent sid DAA) needs this permission. The share /saploc is not needed.
3. Install the EM on phostA into folder S:\introscope. To change the target installation folder edit installer.properties and set USER_INSTALL_DIR=S:\introscope. Do not forget to install the management modules.
4. Check that the installation was successful and that the Windows service Introscope Enterprise Manager can be started. Check that the EM can be reached via http://emhost:8081/webview.
5. Add the Windows service Introscope Enterprise Manager to the switch over group cluster resource type service
6. Take the cluster resource offline
7. Switch the cluster resource to the other node phostB. The S: drive should now be available on phostB.
8. On host phostB create the Windows Service for the EM:
   1. Open a command prompt with Administrator permission
   2. go to folder S:\introscope\bin and issue the command EMCtrl64 register (for the 64 bit version; for 32bit use EMCtrl32 register)
9. Take the cluster resource online
10. Check that the Introscope EM can be reached via http://emhost:8081/webview no matter whether emhost is currently on phostA or phostB.

Connect EM to SAP Solution Manager

1. Go to Solution Manager Infrastructure Preparation, Step 4: Define CA Introscope. Activate Edit mode.
2. Click Discover Introscope EM.
   1. As installation path enter the location on the shared drive.
      S:\introscope.
   2. As diagnostics agent choose the agent with the logical host name emhost.
   3. Click Discover installation.
   4. As a result the new EM should show up as line in the table.
3. Edit the EM host name in the table such that the logical host name emhost is displayed here.
4. Change the Client Host to also contain the logical host name emhost
5. Click User Management to set the connection user to the EM.

1. Go to Solution Manager Infrastructure Preparation, Step 4: Define CA Introscope. Activate Edit mode.
2. Click Discover Introscope EM.
   1. As installation path enter the location on the shared drive.
      S:\introscope.
   2. As diagnostics agent choose the agent with the logical host name emhost.
   3. Click Discover installation.
   4. As a result the new EM should show up as line in the table.
3. Edit the EM host name in the table such that the logical host name emhost is displayed here.
4. Change the Client Host to also contain the logical host name emhost

Click User Management to set the connection user to the EM.

Upgrading the Introscope Enterprise Manager

This section summarizes hints on upgrading the Introscope Enterprise Manager (EM). The focus is to run the upgrade in a way to minimize the effort on SAP Solution Manager side.

Traps

1. The default installation folder for Introscope EM 8.x is /usr/sap/ccms/wilyintroscope. The default installation folder for Introscope EM 9.x and 10.x is /usr/sap/ccms/apmintroscope.
2. Installing a new EM on top of an existing installation is not supported.
3. Prior to SAP Solution Manager 7.1 SP12: Changing the installation folder of the EM causes follow-up work on SAP Solution Manager side: The installation path is part of the internal ID, and changing this ID breaks the association of managed systems to the EM.
4. both procedures described below assume that you continue to use the same host name and port for the upgraded EM. If you plan to change this there is no way to avoid touching the managed system setup. Furthermore, the Introscope agents accept updates to the EM connection data only with a restart: A restart of the managed systems will be needed.

Procedures for SAP Solution Manager 7.2

With SAP Solution Manager 7.2 you can install the EM into the same folder (after removing the old installation in that folder) or into a new folder. In any case the procedure will be the same:

1. Stop the existing EM
2. Windows: Remove the Windows service e.g. via bin/EMCtrl32 unregister
3. Move the existing installation to a backup folder
4. Perform the installation. Make sure that the same RMI port is used as for the previous installation that should be replaced in SAP Solution Manager.
5. Remember to install the management module package and to copy any other files that you want to keep from old EM (see Copy files from the old EM after an upgrade).
6. In SAP Solution Manager Infrastructure Preparation Step 4 (Define CA Introscope), Discover the new installation. If the same host and port are used the previous version is replaced with the new one. Installation path is updated. Managed systems are connected and assigned to the new EM installation.
7. In Basic Configuration step "Configure Automatically" perform the activity Push DPC Configuration to CA Introscope. See SAP Note 1751225.

Copy files from the old EM after an upgrade

1. SmartStor: The Introscope SmartStor contains the collected metrics: The location of SmartStor on disk is specified by properties
   1. introscope.enterprisemanager.smartstor.directory
   2. introscope.enterprisemanager.smartstor.directory.archive
      in file <EM_HOME>/config/IntroscopeEnterpriseManager.properties. If you want to keep the already collected metrics move these folders from the old to the new EM.
2. The Introscope traces are stored in a folder that is configured with properties introscope.enterprisemanager.transactionevents.storage.dir in <EM_HOME>/config/IntroscopeEnterpriseManager.properties.  If you want to keep the already collected traces move this folders from the old to the new EM.
3. Manually transfer any custom management modules (located in <EM_HOME>/config/modules) from the old to the new location. But do not overwrite newly installed standard SAP modules with older versions.
4. Manually transfer any customization (e.g Java VM parameters in file Introscope Enterprise Manager.lax or EMService.conf, other modified properties in files <EM_HOME>/config/IntroscopeEnterpriseManager.properties,  <EM_HOME>/config/em-jetty-config.xml, <EM_HOME>/config/users.xml, <EM_HOME>/config/users-sap.xml, <EM_HOME>/config/domains.xml, <EM_HOME>/config/apm-events-thresholds-config.xml, <EM_HOME>/config/loadbalancing.xml ).
   Do not replace the new configuration files with the old copies since you then may loose any newly introduced properties. Instead, copy and paste any customization from your old files.
5. Copy all certificate files used for single sign-on from SolutionManager. You find them in <EM_HOME>/sap/TrustedCerts.
6. For monitoring capabilities of SAP Solution Manager copy file <EM_HOME>/config/agentclusters.xml from the old to the new location.
7. Also for monitoring capabilities of SAP Solution Manager copy files <EM_HOME>/sap/<SolMan_SIDs>.e2emai.properties (see SAP Note 1751225).
8. If you have a full Introscope license copy this too from folder <EM_HOME>/license.
9. If Emergency Monitoring feature available starting with Introscope version 9.7 is activated copy files <EM_HOME>/sap/emon/*.

DO NOT copy the contents of folder sap/plugins. This might cause versioning conflicts. If you copy the complete sap/ folder of your EM installation then make sure to delete the sub folder sap/plugins. It will be re-generated with current plugin versions after restarting the Enterprise Manager.

Setup SSL Communication for Introscope

• Enable HTTPS for Introscope Enterprise Manager
• SSL Communication for Introscope Enterprise Manager

How to Change the Enterprise Manager Host Name

How to change the CA Enterprise Manager host name

Documentation and Help

Starting with Introscope 9 the CA Introscope documentation is delivered with the management modules and can be called on the EM via URL http://<EM_HOST>:<EM_HTTP_PORT>/IntroscopeHelp.

Sizing and Performance Guide

The Sizing and Performance Guide is part of the CA APM help (e.g. http://<EM_HOST>:<EM_HTTP_PORT>/IntroscopeHelp → Planning Information → Sizing and Performance Guide).

Manager of Managers Scenario (MOM)

When the Introscope start to present Agent disconnection, missing metrics, Introscope stopping due memory issues or the connection limit is reached, the best option is install the Introscope on a cluster scenario. More details on this scenario can be found on:

http://<EM_HOST>:<EM_HTTP_PORT>/IntroscopeHelp → Planning Information → Sizing and Performance Guide → CA APM Environment Options →  Clustered Environment

The installation and configuration details for Clustered Scenario, can be found in the Introscope Installation Guide.

Displaying results of ABAP Hotspot Analysis

ABAP kernel 7.20 and higher offer a functionality to collect performance data per function module, form, and method. This can be configured and viewed on ABAP side, and also be monitored with the Introscope Host Adapter. The steps to use it are described here.

Requirements and Setup on ABAP Side

1. Instrumentation

The hot spots must be defined in table HOTSPOT_DEF. The entries can either be created  with SE16 or via an XML file which can be uploaded with function module SHSTI_UPLOAD_XML.
Example of the xml file:

Be careful with class names. They needs to be entered with runtime name, which means the name must end with the pattern ====CP. The number of '=' depends on the class name and must be filled until the complete length has 32 character.

2. Activation

After each change of instrumentation the kernel needs to know that a change of instrumentation happened. This can be done with function module SHSTI_START_TRACE( per Instance). Once the instrumentation is ready the hot spot measurement can be started automatically with each restart by setting profile parameter abap/hotspot_trace to auto.

3. Authorization

The user SMDAGENT_<SOLMAN_SID> needs the authorization to execute function modules in function group SHSTI. In case the role template SAP_IS_MONITOR does not contain the authorization you need to enhance role ZSAP_IS_MONITOR (or the customer specific role) for authorization object S_RFC with following values:

ACTVT           = 16 
RFC_NAME   = SHSTI 
RFC_TYPE    = FUGR

Setup on Introscope Host Adapter

Currently the corresponding action is not active by default. It must be configured manually. For this purpose, two changes must be done:

1. An ActionTemplate must be defined in the file WilyHostAgentTemplate.xml. This should be done by uploading a modified file into scope Global via agent administration. Here are the lines to be added:

2. On each instance where the hotspot monitoring should be enabled an action must be defined in file SapAgentConfig.xml. This file exist in scope of the SMDAgent. The example snippet re-uses a destination SD7|ldai1sd7_SD7_89 that was already defined by the managed system setup. The RFC connection is then shared by the hotspot action and the standard Abap polling actions.

<action prefix="SD7|ldai1sd7_SD7_89|AbapHotspot" name="SD7|ldai1sd7_SD7_89 AbapHotspot" destination="SD7|ldai1sd7_SD7_89" template="AbapHotspot" />
Example:

Result

The Introscope Investigator tree of the Introscope host agent contains an additional tree AbapHotsport under the instance. When selecting the instance node you get three additional tabs in the right area of the Investigator: Function Modules, Form Routines, Methods. Each tab displays a table of metrics for the corresponding category. Selecting a sub-node in the tree on the left automatically filters the graph/table to the selected subset.

Example

• SAP Note 797147 Introscope Central Note
• SAP Note 2909673 Introscope 10.7 Release Notes
• SAP Note 2451758 SAP Extended Diagnostics by CA, foundation 10.5
  
• SAP Note 1280961 SAP Extended Diagnostics by CA Wily
  
• SAP Note 2246406 Disabling Introscope Java Agent in emergency case

General

Information to be Provided for Support Messages

The following items are required for support messages:

• Version of software component LM-SERVICE on SAP Solution Manager Java stack
• Version of component ST on SAP Solution Manager ABAP stack
• http Remote connection (if not conflicting with company policy) to
  • SAP Solution Manager Java stack
  • SAP Solution Manager ABAP stack
  • Introscope Webview
• Credentials (in Secure Area) for a user with admin permission
• Log files from component with problems (EM, agent, ...)
• Screenshots of the problem (if this makes sense)

Versioning

The mapping between Introscope versions and SAP versions of the Introscope delivery is as follows:

• Introscope follows a versioning scheme consisting of four parts: major.minor.servicepack.patch level, e.g. 9.1.0.2.
• SAP has a versioning scheme consisting of three parts: version SP support package Patch patch level.
• The mapping is defined as follows:
  • Introscope major is mapped to SAP version
  • Concatenation of Introscope minor and service pack is mapped to SAP support package
  • Introscope patch level is mapped to SAP patch level if possible: If multiple SAP patches for the same Introscope patch level are required the mapping for the patch level is broken and the SAP patch level is increased
• Examples
  • Introscope 8.2.4.0 maps to 8 SP24 P0.
  • 9.1.0.2 maps to 9 SP10 Patch 2
  • an update of the Java agent shipment using the same Introscope patch level 9.1.0.2 will result in 9 SP10 P3.

Enterprise Manager (EM)

Additional Information to be Provided for EM Issues

1. Description of the problem
   • What kind of problem (slow Workstation / Webview, gaps in line chart, MoM losing connection to collectors, ...)?
   • Screenshot of the problem (e.g. from Workstation / Webview screens)
   • Re-occurrence: periodically, occasionally,
   • Reproducible? If yes, how to reproduce?
2. Time range
   • When did the problem start occurring? 
   • Did the problem disappear without any action? If yes, when did it disappear?
3. Thread dumps
   • if you observe any kind of hang / crash situation trigger a series of thread dumps for the MoM at least, if possible also for the collectors. See below for details on triggering thread dumps.
4. Log files
   • Provide a complete set of log files (full contents of /logs folder) for MoM and all collectors. The log files must cover the time range when the problem occurred. Thread dumps will also arrive in the logs folder (em.log / EMService.log)
5. Heap dumps
   • In case of out of memory situations heap dumps may have been generated (java_*.hprof)  in the EM installation folder. They might be needed as well
6. Contents of the /data folder of the MoM installation
   • This contains important self-monitoring metrics for post-mortem analysis

EM Additional Diagnostics Options

To enhance the diagnostics options the following configuration changes are recommended (at least in case problems occur)

• Run the Enterprise Manager on SAP JVM instead of the bundled Oracle / IBM JVM. This will provide more details in thread dumps, address some heap dump problems of Oracle JVM and enable a bunch of other diagnostics facilities. Make sure to use the same major Java version as present in the bundled JVM:
  • Java 6 for EM <= 9.6,
  • Java 7 for EM 9.7 and 10
  • Java 8 for EM 10.1 and higher
• Oracle / SAP JVM: Set the JVM parameter -XX:+HeapDumpOnOutOfMemoryError 

Enterprise Manager Log Files

Most relevant log files for problems with EM in sub directory logs are:

• IntroscopeEnterpriseManager.log
• IntroscopeEnterpriseManagerSupport.log

To get log out put with log level DEBUG in file IntroscopeEnterpriseManager.log replace line

log4j.logger.Manager=INFO, logfile

with

log4j.logger.Manager=DEBUG, logfile

in file file <EM_Home>/config/IntroscopeEnterpriseManager.properties.

On Unix platforms the file em.log in the EM home directory contains helpful information. All console output of the Java VM, including full thread dumps, goes here.

On Windows platforms the logs folder contains an additional file EMService.log. This is written by the Windows service wrapper and may contain important information in case the EM fails to start as Windows service.

Enterprise Manager Version

There are multiple ways to determine the EM version:

1. The log file IntroscopeEnterpriseManager.log contains the full version string at each startup of the EM. Example:

   8/23/12 09:49:47.289 AM MESZ [INFO] [WrapperSimpleAppMain] [Manager] Introscope Enterprise Manager Release 9.1.0.2(Build 581100)

   
   SAP Solution Manager 7.1 discovers the version and displays it in Basic Configuration.

Adapt Enterprise Manager Java VM Parameters

Depending on the OS there are different locations where you adapt the Java VM parameters.

Windows Service
Adapt the file <EM_HOME>/bin/EMService.conf. The properties wrapper.java.additional.<number> contain the Java parameters. Java heap size is specified in parameters wrapper.java.maxmemory and wrapper.java.initmemory.

Unix
Adapt the file Introscope_Enterprise_Manager.lax in the EM home directory. Property lax.nl.java.option.additional contains all JVM parameters. For Java heap size see Java parameter -Xmx and -Xms.

Recommended Java VM parameters for Enterprise Manager

In case the EM crashes with OutOfMemory (OOM) it makes sense to analyze the Garbage Collections (GC) in more detail. So setting additional java parameters 
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails is recommended. 
With java parameter -Xloggc:gc.log the GC output is redirected from file em.log to a dedicated file gc.log.

The parameter -XX:+HeapDumpOnOutOfMemoryError is useful to trigger automatically a heap dump in case of OOM.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.

The bigger the Java heap size is configured the more it makes sense to work with parallel GCs by
setting additional java parameters -XX:+UseParNewGC -XX:+UseConcMarkSweepGC  or    -XX:+UseG1GC.

Start/Stop the Enterprise Manager

1. Windows Service: Default Windows Service display name is "Introscope Enterprise Manager"
2. Unix: use the script EMCtrl.sh in folder bin:
   • EMCtrl.sh start
   • EMCtrl.sh
   • stop

Register/Unregister Windows Service

The Windows Service to start/stop the Enterprise Manager usually gets registered by default during EM installation. In case you want to register or unregister this Windows Service manually you can do that with the script <EM_HOME>/bin/EMCtrl32.bat respectively EMCtrl64.bat. The command to register the service is EMCtrl64.bat register and EMCtrl64.bat unregister to remove the Windows Service.

The Windows Service display name and other details can be set in file  <EM_HOME>\bin\EMService.conf. Changes to e.g. property wrapper.displayname modifying the default service display name "Introscope Enterprise Manager" of an already registered Windows Service only take effect after unregistering and registering the Windows Service.

Other properties you may need to change e.g. if you run several EMs on the same Windows host are wrapper.name and wrapper.description.

Collect Thread Dumps from the Enterprise Manager

If the Enterprise Manager appears stuck it may make sense to trigger a thread dump. The procedure differs depending on the operating system and the mode it is started.

Windows 
If the EM is running as service (which is normal): Open a command prompt as administrator and issue the command sc control IScopeEM 255 (IScopeEM is the default name of the service). The thread dump will be appended to the service log file logs/EMService.log. For processing in any thread dump analysis tools you will have to remove the wrapper prefix added to every line.

Example output of the command:

Using jmxremote or SAP JVM is an alternative, but requires extra configuration.

If the EM is launched interactively (Introscope_Enterprise_Manager.exe) you can hit Ctrl-Break to trigger a dump. Unfortunately you get it on the console, difficult to capture.

Unix 
On OS level identify the process ID of the Enterprise Manager, e.g. via the command ps -ef | grep java. 
Send signal 3 to this process: kill -3 pid. 
The thread dump will be written to the file em.log in the EM installation directory

Collect Heap Dumps from the Enterprise Manager

If the EM crashes and you find errors like "JAVA heap space out of memory" in the IntroscopeEnterpriseManager.log may have either configured the EM with low Java heap size (then see "Adapt Enterprise Manager Java VM parameters" above to increase the heap size) or there could be a memory leak. For the latter heap dumps are needed for analysis. To get heap dumps in case of an OutOfMemory set the additional java parameter -XX:+HeapDumpOnOutOfMemoryError (see "Adapt Enterprise Manager Java VM parameters" above; a restart of the EM is needed). Setting this parameter automatically a heap dump is triggered if an OutOfMemory occurs. The heap dump could consist of more than one file and is usually named java_pid<pid>.*.

In case you want to trigger heap dumps manually set java parameter -XX:+HeapDumpOnCtrlBreak. With every triggered thread dump (see "Collect Thread Dumps from the Enterprise Manager") you then also get a heap dump.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.

Performance / Scalability / Capacity

The EM provides several self-monitoring metrics about its internal status. The metric Overall Capacity (%) is also reported for each EM in the SAP Solution Manager Administration work center. A value of 100% means that it has reached its capacity limit. Short peaks beyond 100% are not critical. Only if the capacity is permanently substantially beyond 100% you will experience problems like missing data points, poor performance, etc. You can find this Overall Capacity (%) metric in the Introscope Investigator via this path: Superdomain -> Custom Metric Host -> Custom Metric Process -> Custom Metric Agent -> Enterprise Manager -> Overall Capacity (%).

A more detailed view is available in the Introscope Investigator tree via the following path: Superdomain → Custom Metric Host → Custom Metric Process → Custom Metric Agent → Enterprise Manager. If you select the node Enterprise Manager in the tree, the tab Overview on the right side displays eight charts on EM sanity, as shown in the screenshot. It reveals heap utilization, number of metrics and agents.

Perflog Analysis

The EM collects Metrics about itself and reports them to a file called perflog.txt under 15 second time periods. By analyzing the entries in this file, it is possible identify beforehand if the EM is becoming overloaded and avoid future issues. The EM Perflog Analyzer performs this analysis automatically and provides the user with recommendations on each point checked.

EM Perflog Analyzer Tool

Workstation webstart

Release Info

All known Webstart problems documented here were resolved with Introscope version 9.7 and subsequent versions. It might be easier to upgrade the Introscope Enterprise Manager and get a Webstart Workstation that works out of the box than to tweak the webstart for older versions. See SAP Note 2138309 and SAP Note 2285189.

The Java WebStart for the Introscope Workstation fails for higher patch levels of Java 7 and for Java 8 due to changes in the security checks of Java WebStart. In some cases the launch of the Workstation fails silently, in other cases an exception is shown.

In any case the following generic workarounds exist:

• Use a Workstation that is explicitly installed on the client PC Note that Enterprise Manager and Workstation version must match
• Use the HTML-based Webview via a URL like http://xxxxx.yyy.corp:8081/webview

In addition, for the following Java patch levels workarounds are available:

• Java 7 Update 45 (Java 1.7.0_45) and newer: Introscope Workstation 9.1.5.0 and older fail to start due to security related changes. Please see SAP Note 565954. No action needed for Introscope 9.5.6
• Java 7 Update 51 (Java 1.7.0_51) and newer: Additional security changes get introduced causing the Workstation to fail again. As Workaround launch the Java Control Panel (Start->All Programs->Java->Configure Java). On the Security tab leave the Security Level as “High” and add the WebStart URL to the “Exception Site List”
  Example for WebStart URL: http://xxxxx.yyy.corp:8081
• Java 7 Update 71 (Java 1.7.0_71) and newer: Workstation WebStart client fails again. Workaround for Introscope 9.5.6: add line <property name="jnlp.eclipse.security" value="osgi"/> to file  <EM_HOME>/product/enterprisemanager/plugins/com.wily.introscope.workstation.webstart_<version>/WebContent/jnlp/workstation.jsp. The only workaround for Introscope 9.1.5 is to explicitly use Java 6, as described below in Section "Webstart Additional Hints".

Workstation Webstart: Expired Certificates as of March 1, 2015

Furthermore as of March 1, 2015, the CA APM Java JRE certificate has expired. This will impact all the customers who use web start client, version 9.7 and lower. Customers who initiate the workstation through the downloaded JNLP will be blocked if their security is set beyond medium. This issue is independent from issues related to individual Java Updates as listed above.

WORKAROUND:

Below are options to work around this issue:

• Option 1: If the security level is set to high or above, the level should be brought down to medium. Once the security level is set to medium, the web start client can be initiated to start the workstation.

In order to do this → Go to Control Panel → Java → Security and adjust the security level.

• Option 2: Add the web start URL to the exception site list. Once this site is added re-initiating the web start client starts the workstation.

In order to do this → Go to Control Panel → Java → Security → Click on Edit Site List and add the URL in the location box.

Webstart Additional Hints

1. We have not found a way to launch the Workstation 9.1.5 with recent Java 7 and Java 8 versions. You can, however, have multiple JRE versions installed on the same host. In particular, you could have Java 6 installed to launch the Workstation, and at the same time use a recent Java 8 version for webstart of other applications.

   1. Use an absolute path like "C:\Program Files\Java\jre6\bin\javacpl.exe" to launch the Java Control Panel. Disable other Java versions in tab "Java".

   2. Use an absolute path like "C:\Program Files\Java\jre6\bin\javaws" http://emhost:8081/workstation to launch the Workstation.

2. In some cases having multiple JRE/JDK versions installed might cause problems. Consider uninstalling JRE versions that you do not really need.

3. Try launching the Webstart bypassing the browser In a command window. This isolates any potential browser add-in issues and allows to use webstart even if the browser add-in is disabled: Use a call like this:

   "C:\Program Files\Java\jre6\bin\javaws" http://emhost:8081/workstation

Java Bytecode Agent (BCA)

Introscope agents of a certain version can connect to Introscope Enterprise Managers of the same or higher version. It is not possible to connect them to an EM with a lower version than the agent is. For more information, refer to the Introscope JAVA Agent WIKI.

Log Files

Relevant log files for problems with the BCA are:

• from directory /usr/sap/<AgentSID>/<AgentNo>/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.<Release>/wily
  • IntroscopeA
  • gent.profile
• from directory /usr/sap/<AgentSID>/<AgentNr>/SMDAgent/temp
  • IntroscopeAgent.<SID_Instance>_server*.log
  • AutoProbe.<SID_Instance>_server*.log
• from directory /usr/sap/SID>/<inst>/work of the managed system if it's a Netweaver java node:
  • dev_server0 (contains the actually used JVM parameters)
  • std_server*.out

 If requested by SAP Development set in IntroscopeAgent.profile the property  log4j.logger.IntroscopeAgent from

to

A restart is not required. Do not forget to reset the value back to INFO after investigating the issue.

Agent Version

The agent version can be determined via one of the following procedures:

• Check the log file IntroscopeAgent*.log for the right agent. The agent version is dumped at every restart of the Java VM.
• Check the Introscope Investigator tree. For each agent, the node Agent Stats contains
  • Agent build and release
  • SAP build timestamp
  • location of the agent profile that is used

Byte Code Adapter does not start

Byte Code Adapter does not start

Missing CPU Metrics

The CPU node in the Investigator tree of the byte code instrumentation agent (e.g. Netweaver agent) is populated via a native library, the so-called platform monitor. There are multiple possible reasons why this node might be missing:

• There is no platform monitor extension for all supported Netweaver platforms. As fallback you can use the OS metrics as provided by the Introscope host adapter (under node SAP Host Agent)
• Windows: If the user running the Java VM is not in the administrators group it must be added to the group Performance Monitor Users to obtain the privileges required to read performance data.

Missing JMX Metrics

For troubleshooting JMX metric collection in the agent you can increase logging of the JMX service by setting these two additional properties:

JMX debug logging

In general polling of particular MBeans will be switched off until the next restart of the Java VM as soon as any exception occurs for polling that MBean.

For Netweaver there is a known problem that reporting of http sessions ("Current Http Sessions") stops reporting. See SAP Note 2067705 for a fix.

Troubleshoot Netweaver Jmx Metrics

Java Managed System does not start due to missing Agent.jar

If a Java Managed System has been instrumented for CA Introscope via the Java VM parameter -javaagent:/some/path/to/Agent.jar and the file Agent.jar is not accessible by the instrumented Java VM then it fails to start. Possible reasons why the file Agent.jar is not accessible:

1. The corresponding Diagnostics Agent has been deleted before the CA Introscope instrumentation has been removed from the Java Managed System. The Java Managed System does not start anymore because the required Agent.jar in the Diagnostics Agent file system has been deleted.
2. The managed Java VM does not have read access to the file system tree where the Introscope agent is located (e.g. user SAPService<SID> on Windows starting the SAP Netweaver system)

In the first case the instructions for uninstalling the Introscope Byte Code Agent were not followed correctly. 
You can find those instructions in the Introscope Installation or Setup Guide for your SAP Solution Manager Release.

=> Alphabetical Index => S
=> SAP Solution Manager
=> "your solution manager release"
=> "one of the sections on installations"
=> "respective Wily Introscope guide".

In these guides, please refer to the section titled "Uninstalling Introscope Agents". It is recommended to use the SAP Solution Manager provided mechanism to remove the problematic properties. 

In case the Diagnostics Agent was already deleted, it may be necessary to delete the properties by hand. In case of problems please create a ticket at component SV-SMG-DIA-WLY-BCA.

Known problems for Bytecode Agent

Known problems for Bytecode Agent

Introscope Host Adapter (wilyhost)

Check if Introscope Host Adapter is connected to the Enterprise Manager

Check Introscope Host Adapter connection to EM

Missing ABAP metrics

Missing ABAP metrics

Log Files and Log Levels

Relevant logfiles for problems with Introscope Host Adapter are:

• SMDAgentApplication*.log (most content here)
• SMDSystem*.log (in case wilyhost does not start up at all)
  They are persisted in directory /usr/sap/<SMDAgentID>/<InstanceNo>/SMDAgent/log

The log level for SMDAgentApplication.log might have to be increased to DEBUG for troubleshooting. This is done via diagnostics agent administration.

If the Introscope host adapter does not show up in the Investigator the file jvm_SMDAgent.out may also be relevant. It corresponds to IntroscopeAgent*.log of the BCA. To increase the log level for the Introscope runtime of the host adapter (in jvm_SMDAgent.out) edit the application resource "IntroscopeSapAgent.profile" and change the property

Thread Dumps

Thread dumps for the Introscope host adapter means thread dumps for the whole diagnostics agent. They can be triggered in various ways, e.g.:

1. Via the SAP Management Console / MMC: Right-Click on the java process for the agent and click Dump Stack Trace
2. On OS level (Unix): find out the process ID of the agent process and issue the command kill -3 <pid>

In any case the thread dump is written to the file std_smdagent.out in the work directory of the diagnostics agent.