CA APM Introscope(R) is an application performance management solution created to manage Java Application performance. Unlike development tools, Introscope(R) is designed to scale with minimal performance impact. This allows you to monitor and manage your application performance in live production environments.
The Right to View (RTV) version of CA APM Introscope is a restricted, read-only form of the full product and is bundled with SAP Solution Manager. With the RTV version, support is limited to products that are licensed and supported by SAP. The instrumentation, dashboards, Probe Builder Directives (PBDs), management modules, and Smartstor data contained within the RTV version of CA APM Introscope as provided by SAP is the intellectual property of SAP. Use of these functions is restricted by SAP and may only be used in an unrestricted manner by purchasing a full license from Broadcom.
Introscope consists of the Enterprise Manager central server component, Byte Code Adapters running as part of managed Java systems, and Host Adapters running as part of the diagnostics agent. These components and the integration with SAP Solution Manager are described in the following.
The Introscope Enterprise Manager (EM) is the central Introscope server component providing user interfaces, persisting metric and trace data that is delivered by agents, and acting as integration point for SAP Solution Manager. Depending on the expected number of agents and metric load the EM can be run as stand-alone installation or as a more scalable cluster consisting of several collector EMs and a single Manager of Managers (MoM EM).
SAP provides a rich set of default content for common managed product types like SAP Netweaver Portal, SAP Process Orchestration, and a set of extensions that support the integration with SAP Solution Manager. Both is contained in the software component management modules.
An Enterprise Manager can be connected to multiple SAP Solution Manager and SAP Focused Run systems. Since the SID of the Solution Manager is used as part of the file name for configuration files every Solution Manager SID and every Focused Run SID must be unique.
To connect the EM to a SAP Solution Manager a diagnostics agent must be installed on the EM host and the agent needs access to files in the EM installation. Reason for required write access is given where needed:
After this you can discover the EM and set credentials: In the launchpad click Configuration (All Scenarios) and then Infrastructure Preparation → Define CA Introscope.
If you do not grant write access to Introscope configuration files to the diagnostics agent then you can still connect the Introscope Enterprise Manager to Solution Manager: Credentials and configuration files are then not updated by the diagnostics agent. Instead you have to manually maintain credentials in users.xml and domains.xml and then fill these credentials at least for the connection user in the Define CA Introscope → User Management dialog. You may get a warning here due to missing write permission, but the connection will succeed. Write access to sap/<SolMan_SID>.e2emai.properties and config/agentcluster.xml is still mandatory.
Additional details are described in the help.
The extractor framework periodically extracts key metrics from Introscope and persists them in SAP Solution Manager's BI.
Metrics for System Monitoring are selected via templates in the Monitoring and Alerting Infrastructure. SAP Solution Manager triggers subscriptions for metrics in Introscope and the EM periodically pushes metric updates to SAP Solution Manager.
The Introscope Host Adapter (aka WilyHost agent) is an application which runs inside the SMDAgent. One of the tasks of the Introscope Host Adapter is to collect data from different configurable destinations and push them to the Introscope Enterprise Manager. The following picture provide a big picture of the architecture of the Introscope Host Adapter.
All applications running inside the SMDAgent are available in the Agent Administration UI. The Introscope Host Adapter can be found as shown in the picture below.
The resources used by the Introscope Host Adapter application can be found under the 'Application Resources' node:
Note: When using the application configuration take care when performing changes. Changes should not be performed in the scope GLOBAL unless they are strictly required. The changes in scope global affect ALL SMDAgents connected to the SAP Solution Manager system.
In SAP Solution Manager, the Introscope Host Adapter is configured during the activity 'Introscope Host Adapter' of the Managed System Configuration for the target system: Step 7 "Finalize Configuration".
When the activity Introscope Host Adapter is performed, the application is configured to collect data from the managed system that will be monitored. The main files used by the agent are the following:
Note: No manual configuration must be performed to the above files. The activity Introscope Host Adapter will configure the application automatically.
The file SAPAgentConfig.xml holds the configuration of the Destinations and the Actions that will be used/performed to collect data from the managed system.
For a typical double stack system you will have configurations generated for the following destinations:
The following picture shows an example of a Jco destination. As you can see, the Jco destination has the connection information to the managed system, client, user, password, system number and hostname of the application server.
The following picture shows an example of some actions defined in a SAPAgentConfig.xml.
The SAPAgentConfig.xml fully configured looks as follows:
The following picture describes how the destinations and actions work in the managed system. As you can see, the destinations are the location where the data will be collected and the actions are the collections of data.
The file IntroscopeSAPAgent.profile holds the connection configuration to the Introscope Enterprise Manager. This file will tell the Introscope Host Adapter application what is the Enterprise Manager that the application must connect. The activity Introscope Host Adapter in the Managed System Configuration uses the IntroscopeSapAgent.profile.template file and then creates the IntroscopeSapAgent.profile. The following picture shows an example of IntroscopeSapAgent.profile.
If the Introscope Host Adapter applications shows issues, for example: agent is not connected to the Enterprise Manager or managed system metrics are not being collected. Then the SMDAgent logs must be checked to find out what is the root cause of the issue. The Introscope Host Adapter application may not work correctly due to distinct issues and the SMDAgent logs will show the root cause.
|Logging Information||Log File|
|Host Agent Information when trying to connect to the Enterprise Manage||/usr/sap/SMD/SMDA<xx>/work/jvm_SMDAgent.out|
|SMD Agent Application log file. All of the interesting logging for the Wily Host Agent will be found here||SMDAgentApplication.X.log|
|SMD Agent System log file||SMDSystem.X.log|
The Byte Code Adapter, also known as Introscope Java Agent, is part of the CA Introscope APM solution and is used in combination with the CA Introscope Enterprise Manager. The main purpose is to collect data from the applications and systems running on Java Virtual Machines (JVMs). The data collected is pushed from the agents to the Introscope Enterprise Manager where it is persisted.
The byte code adapter is used to monitor SAP NetWeaver AS Java, but the same byte code adapter is also used to monitor other Java-based SAP systems, for example SAP Mobile Platform and SAP BusinessObjects Enterprise.
For some reasons, it may be required to change the host name used by the Enterprise Manager service. The steps below show how to do it:
Edit/add property introscope.enterprisemanager.ipaddress in file IntroscopeEntepriseManager.properties of the Enterprise Manager. The property introscope.enterprisemanager.ipaddress accepts hostname and IP address as values. Then restart the Enterprise Manager service to the changes take effect.
In order to also adjust the hostname of the navigation links in the CA Introscope WebView, set the property: introscope.enterprisemanager.fullyQualifiedHostName=<fqn of EM host>
Note: Introscope does not allow to generate navigation links with protocol HTTPS, even if you set property introscope.enterprisemanager.webserver.port with an HTTPS port.
In the Solution Manager Infrastructure preparation navigate to step 4 Define CA Introscope. Here you can rename the server name of the EM host in edit mode in field Host.
For ALL managed systems, run again the activities Introscope Host Adapter and Byte Code Adapter in the Managed System Configuration. These activities will reconfigure the agents to use the new Enterprise Manager host.
Also, for the agents that have been configured manually it is also required to update the IntroscopeAgent.profile of these agents to use the new Enterprise Manager host name.
Information to be Provided for Support Messages
The following items are required for support messages:
Additional Information to be Provided for EM Issues
EM Additional Diagnostics Options
To enhance the diagnostics options the following configuration changes are recommended (at least in case problems occur)
Enterprise Manager Log Files
Most relevant log files for problems with EM in sub directory logs are:
To get log out put with log level DEBUG in file IntroscopeEnterpriseManager.log replace line
in file file <EM_Home>/config/IntroscopeEnterpriseManager.properties.
On Unix platforms the file em.log in the EM home directory contains helpful information. All console output of the Java VM, including full thread dumps, goes here.
On Windows platforms the logs folder contains an additional file EMService.log. This is written by the Windows service wrapper and may contain important information in case the EM fails to start as Windows service.
Enterprise Manager Version
There are multiple ways to determine the EM version:
1 The log file IntroscopeEnterpriseManager.log contains the full version string at each startup of the EM. Example:
8/23/12 09:49:47.289 AM MESZ [INFO] [WrapperSimpleAppMain] [Manager] Introscope Enterprise Manager Release 18.104.22.168(Build 581100)
2 SAP Solution Manager discovers the version and displays it in Basic Configuration.
Adapt Enterprise Manager Java VM Parameters
Depending on the OS there are different locations where you adapt the Java VM parameters.
Adapt the file <EM_HOME>/bin/EMService.conf. The properties wrapper.java.additional.<number> contain the Java parameters. Java heap size is specified in parameters wrapper.java.maxmemory and wrapper.java.initmemory.
Adapt the file Introscope_Enterprise_Manager.lax in the EM home directory. Property lax.nl.java.option.additional contains all JVM parameters. For Java heap size see Java parameter -Xmx and -Xms.
Recommended Java VM parameters for Enterprise Manager
In case the EM crashes with OutOfMemory (OOM) it makes sense to analyze the Garbage Collections (GC) in more detail. So setting additional java parameters
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails is recommended.
With java parameter -Xloggc:gc.log the GC output is redirected from file em.log to a dedicated file gc.log.
The parameter -XX:+HeapDumpOnOutOfMemoryError is useful to trigger automatically a heap dump in case of OOM.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.
The bigger the Java heap size is configured the more it makes sense to work with parallel GCs by
setting additional java parameters -XX:+UseParNewGC -XX:+UseConcMarkSweepGC or -XX:+UseG1GC.
Start/Stop the Enterprise Manager
Register/Unregister Windows Service
The Windows Service to start/stop the Enterprise Manager usually gets registered by default during EM installation. In case you want to register or unregister this Windows Service manually you can do that with the script <EM_HOME>/bin/EMCtrl64.bat. The command to register the service is EMCtrl64.bat register and EMCtrl64.bat unregister to remove the Windows Service.
The Windows Service display name and other details can be set in file <EM_HOME>\bin\EMService.conf. Changes to e.g. property wrapper.displayname modifying the default service display name "Introscope Enterprise Manager" of an already registered Windows Service only take effect after unregistering and registering the Windows Service.
Other properties you may need to change e.g. if you run several EMs on the same Windows host are wrapper.name and wrapper.description.
Collect Thread Dumps from the Enterprise Manager
If the Enterprise Manager appears stuck it may make sense to trigger a thread dump. The procedure differs depending on the operating system and the mode it is started.
If the EM is running as service (which is normal): Open a command prompt as administrator and issue the command sc control IScopeEM 255 (IScopeEM is the default name of the service). The thread dump will be appended to the service log file logs/EMService.log. For processing in any thread dump analysis tools you will have to remove the wrapper prefix added to every line.
Example output of the command:
C:\usr\sap\ccms\apmintroscope>sc control IScopeEM 255
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
Using jmxremote or SAP JVM is an alternative, but requires extra configuration.
If the EM is launched interactively (Introscope_Enterprise_Manager.exe) you can hit Ctrl-Break to trigger a dump. Unfortunately you get it on the console, difficult to capture.
On OS level identify the process ID of the Enterprise Manager, e.g. via the command ps -ef | grep java.
Send signal 3 to this process: kill -3 pid.
The thread dump will be written to the file em.log in the EM installation directory
Collect Heap Dumps from the Enterprise Manager
If the EM crashes and you find errors like "JAVA heap space out of memory" in the IntroscopeEnterpriseManager.log may have either configured the EM with low Java heap size (then see "Adapt Enterprise Manager Java VM parameters" above to increase the heap size) or there could be a memory leak. For the latter heap dumps are needed for analysis. To get heap dumps in case of an OutOfMemory set the additional java parameter -XX:+HeapDumpOnOutOfMemoryError (see "Adapt Enterprise Manager Java VM parameters" above; a restart of the EM is needed). Setting this parameter automatically a heap dump is triggered if an OutOfMemory occurs. The heap dump could consist of more than one file and is usually named java_pid<pid>.*.
In case you want to trigger heap dumps manually set java parameter -XX:+HeapDumpOnCtrlBreak. With every triggered thread dump (see "Collect Thread Dumps from the Enterprise Manager") you then also get a heap dump.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.
Performance / Scalability / Capacity
General Enterprise Manager performance tuning recommendations are collected in SAP Note 3128329.
The EM provides several self-monitoring metrics about its internal status. The metric Overall Capacity (%) is also reported for each EM in the SAP Solution Manager Administration work center. A value of 100% means that it has reached its capacity limit. Short peaks beyond 100% are not critical. Only if the capacity is permanently substantially beyond 100% you will experience problems like missing data points, poor performance, etc. You can find this Overall Capacity (%) metric in the Introscope Investigator via this path: Superdomain -> Custom Metric Host -> Custom Metric Process -> Custom Metric Agent -> Enterprise Manager -> Overall Capacity (%).
A more detailed view is available in the Introscope Investigator tree via the following path: Superdomain → Custom Metric Host → Custom Metric Process → Custom Metric Agent → Enterprise Manager. If you select the node Enterprise Manager in the tree, the tab Overview on the right side displays eight charts on EM sanity, as shown in the screenshot. It reveals heap utilization, number of metrics and agents.
The EM collects Metrics about itself and reports them to a file called perflog.txt under 15 second time periods. By analyzing the entries in this file, it is possible identify beforehand if the EM is becoming overloaded and avoid future issues.
It is possible to check if the wily host application is connected to the EM directly in the Introscope Investigator. By default the Introscope Host Adapter application connects to the EM using the process name = 'SAP HostAgent Process' and agent name 'SAP HostAgent <SMDAgent_instancename>'. The following picture shows the same agent of the above log entry connected to the EM.
To find information about the connection between the Introscope Host Adapter to the Enterprise Manager it is required to check the jmv_smdagent.log (or jvm_smdagent_<node name> in case agents on-the-fly is used). This file shows the information about the connection between the Introscope Host Adapter application and the Enterprise Manager.
In case of connections problems between the Introscope Host Adapter and the Enterprise Manager, check the following:
Log Files and Log Levels
Relevant logfiles for problems with Introscope Host Adapter are:
The log level for SMDAgentApplication.log might have to be increased to DEBUG for troubleshooting. This is done via diagnostics agent administration.
If the Introscope host adapter does not show up in the Investigator the file jvm_SMDAgent.out may also be relevant. It corresponds to IntroscopeAgent*.log of the BCA. To increase the log level for the Introscope runtime of the host adapter (in jvm_SMDAgent.out) edit the application resource "IntroscopeSapAgent.profile" and change the property
Thread dumps for the Introscope host adapter means thread dumps for the whole diagnostics agent. They can be triggered in various ways, e.g.:
In any case the thread dump is written to the file std_smdagent.out in the work directory of the diagnostics agent.