Byte Code Adapter

Overview

The byte code adapter, also known as Introscope Java Agent, is part of the CA Introscope APM solution and is used in combination with the CA Introscope Enterprise Manager. The Introscope Java Agents and Introscope Enterprise Manager are used in the context of the SAP Solution Manager to collect performance data from Java applications. The main purpose is to collect data from the applications and systems running on Java Virtual Machines (JVMs). The data collected is pushed from the agents to the Introscope Enterprise Manager where it is persisted.

The following picture shows the architecture of the Introscope Java Agents in SAP NetWeaver, but the same Introscope Java Agent is also used to monitor other Java-based SAP systems, for example SAP Mobile Platform and SAP BusinessObjects Enterprise.

Configuration

Download

Check SAP Note 2909673 for the exact download locations and references to the setup guide.

Note that it is not possible to connect Introscope Java Agents to an Introscope Enterprise Manager with a version lower than the agent's, i.e. do not choose an Introscope Java Agent with higher release than the Enterprise Manager for any Managed System.


Setup

The setup procedure is described in the corresponding Introscope setup guide.


Automated configuration for Netweaver AS Java

When installing the Byte Code Adapter for a Netweaver Java system make sure to check the Diagnostics Agent Wiki.


Manual Configuration Steps

The configuration of the Introscope Java Agent consists in the following steps:

  1. Deployment of the Introscope Java Agent files in the managed system server(s)
  2. Configuration of the Introscope Java Agent profile;

    3. Configuration of the Java parameters of the Introscope Java Agent in the managed system;

    Deployment

Deployment basically means to extract the ZIP archive into a file system folder on the managed system. Aspects to consider here:

  • The process of the managed system needs read access to all files and folders of the byte code adapter. Furthermore, write access is needed to the logs folder, typically wily/logs.
  • The folder for the byte code adapter should be on a local file system, not on a network share. The managed system will fail to start if the Agent.jar is not available.
  • The folder should not be in a folder under the Solution Manager diagnostics agent if "agent on the fly" is used: Agent on the fly may delete all byte code adapter files. And again, the managed system will fail to start if the Agent.jar is not available.
  • To simplify byte code adapter updates it is recommended to install the byte code adapter into a folder containing the agent version: This allows to decouple the installation of a new version from the actual downtime of the managed system to activate the changes. Deploying the files and setting the Java VM parameters could be done while the managed system is up, and the new Java VM parameters  take effect then in the next maintenance window. Typically the managed system keeps the byte code adapter files locked while it is up and running. Trying to install the byte code adapter on top of an existing installation would require a sequence of 
    1. Stopping the managed system to unlock files
    2. Perform the actual update of the byte code adapter
    3. Start the managed system again.

Introscope Java Agent Profile

The configuration of the Introscope Java Agent is stored in the IntroscopeAgent.profile. The main properties configured in this file are as follows:

  • introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=<EM_host>: The host of the Enterprise Manager that the Introscope Java Agent will connect.
  • introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT=<EM_port>: The port of the Enterprise Manager that will be used by the Introscope Java Agent.
  • introscope.autoprobe.directivesFile=<Directive_Files>: The Auto Probe Directive files that will be used to instrument the JVM.


Auto Probe Directive Files

The Introscope Java Agent instruments the SAP JVM inserting probes (measurement points) on the Java byte codes during their executions. The measurement points are configured by the Directive Files (*.pbd), which are added in the IntroscopeAgent.profile. SAP provides standard Directive Files that should be used depending on the managed system type. The available Directives Files are mentioned in the IntroscopeAgent.profile of each release of Introscope Java Agent (check the section Directives Files of the profile file).

For NetWeaver systems, the configuration of the Auto Probe files is usually done automatically during the Managed System Configuration (activity Byte Code Adapter Installation). For non-NetWeaver systems, the required files are mentioned in the setup guides for each system type. Take care when changing the Auto Probe files configuration, as some Auto Probe files might cause a performance impact.


Java Parameters

The Java parameters required to start the Introscope Java Agent are as follows:

Java ParameterDescription
-javaagent:<path to the agent.jar file>Location of the Agent.jar file
-Dcom.wily.introscope.agentProfile=<path to the IntroscopeAgent.profile>Location of the IntroscopeAgent.profile which holds the agent configuration
-Dcom.wily.introscope.agent.agentName=<AgentName>Defines the name used by the agent to connect to the Enterprise Manager.

Configuration for Distinct Managed System Types

SAP NetWeaver Java systems

The configuration of the Introscope Java Agent of SAP NetWeaver systems is performed automatically by the activity "Byte Code Adapter Installation"  in the step "Finalize Configuration of the Managed System Configuration". This activity extracts the Introscope Java Agent files to the server, configures the IntrosocpeAgent.profile and adds the required Java parameters to the Java server nodes.

Non-NetWeaver Java systems

The Introscope Java Agent is also used to monitor Java systems that are not based on the SAP NetWeaver platform. The support and the configuration steps of these systems vary depending on the system type. Refer to the configuration guide of each system type in the Maintenance of Product in the System Landscape Wiki.

Troubleshooting

General Hints

  • Introscope agents of a certain version can connect to Introscope Enterprise Managers of the same or higher version. It is not possible to connect them to an EM with a lower version than the agent is. 

Log and Configuration Files

When the Introscope Java Agents are configured and started for the managed system, the Introscope Java Agent will create log files which will provide runtime information. The Introscope Java Agent log files are:

  • IntroscopeAgent*.log: Provides runtime Information about the Introscope Java Agent and connection to the Enterprise Manager.
  • AutoProbe.log: Provides information about the monitored classes/methods of the Java Application.
  • IntroscopeAgent.profile: Main configuration file for the byte code adapter. 

The location of the log files may vary depending on the configuration and system type, some example are below:


SAP NetWeaver Java Systems

When the automatic setup is used for SAP NetWeaver Java systems (activity Byte Code Adapter Installation), the Introscope Java Agent log files will be available at /usr/sap/DAA/SMDA97/SMDAgent/temp. The IntroscopeAgent.profile will be located in /usr/sap/<AgentSID>/<AgentNo>/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.<Release>/wily/

If the Diagnostics Agents on-the-fly feature is used in the managed system, the flag "Deploy Byte Code Adapter under Managed System Instance(s)" must be selected during the Managed System Configuration. In this case the Introscope Java Agent log files will be located at /usr/sap/<ManagedSID>/<InstanceID>/ByteCodeAdapter/logs.

If the manual installation has been performed, the Introscope Java Agent logs are available by default in the following location \usr\sap\ccms\wily\logs.

Additional logs of interest from SAP Netweaver Java are the following from directory /usr/sap/SID>/<inst>/work of the managed system if it's a Netweaver java node:

  • dev_server0 (contains the actually used JVM parameters)
  • std_server*.out
SAP BusinessObjects Enterprise
  • <BOE_PATH>\SAP BusinessObjects Enterprise XI 4.0\Java\wily\logs


Using the Log

Version Information

The agent version can be determined via one of the following procedures:

  • Check the log file IntroscopeAgent*.log for the right agent. The agent version is dumped at every restart of the Java VM.
  • Check the Introscope Investigator tree. For each agent, the node Agent Stats contains
    • Agent build and release
    • SAP build timestamp
    • location of the agent profile that is used
EM Connection Information

If the connection has been established successfully, the following entry is shown in the IntroscopeAgent* .log:

IntroscopeAgent.log: successful connection 
Connected controllable Agent to the Introscope Enterprise Manager at myhostname.domain.com:6001 [...]

Host = "myhostname", Process = "SAP Netweaver", Agent Name = "SMJ_J002_server0", Active = "true".

The above log entry shows the Introscope Enterprise Manager host name and port as well as the host name, the process name and the agent name used by the Introscope Java Agent to report the performance information. This information is useful in order to find the Introscope Java Agent in the Web View or Workstation Investigator. The following picture shows the Introscope Java Agent node in the Introscope Investigator:


Log Level

If requested by SAP Development increase the log level by adjusting the property log4j.logger.IntroscopeAgent in IntroscopeAgent.profile:

IntroscopeAgent.profile: Log Level 
# default:


log4j.logger.IntroscopeAgent=INFO, logfile


# increased log level:


log4j.logger.IntroscopeAgent=DEBUG, logfile

A restart is not required. Do not forget to reset the value back to INFO after investigating the issue.

Common Issues

Java System Does not Start

If your Managed Java System does not start due to missing Agent.jar or any other issue related to the byte code adapter check SAP Note 2246406 for an emergency work-around to bring your system up again. 

If a Java Managed System has been instrumented for CA Introscope via the Java VM parameter -javaagent:/some/path/to/Agent.jar and the file Agent.jar is not accessible by the instrumented Java VM then it fails to start. Possible reasons why the file Agent.jar is not accessible:

  1. The corresponding Diagnostics Agent has been deleted before the CA Introscope instrumentation has been removed from the Java Managed System. The Java Managed System does not start anymore because the required Agent.jar in the Diagnostics Agent file system has been deleted.
  2. The managed Java VM does not have read access to the file system tree where the Introscope agent is located (e.g. user SAPService<SID> on Windows starting the SAP Netweaver system)

In the first case the instructions for uninstalling the Introscope Byte Code Agent were not followed correctly. 
You can find those instructions in the Introscope Installation or Setup Guide for your SAP Solution Manager Release.

=> Alphabetical Index => S
=> SAP Solution Manager
=> "your solution manager release"
=> "one of the sections on installations"
=> "respective Wily Introscope guide".

In these guides, please refer to the section titled "Uninstalling Introscope Agents". It is recommended to use the SAP Solution Manager provided mechanism to remove the problematic properties. 

In case the Diagnostics Agent was already deleted, it may be necessary to delete the properties by hand. In case of problems please create a ticket at component SV-SMG-DIA-WLY-BCA.


Byte Code Adapter Does not Start

If the Byte Code Adapter does not appear in the Introscope Investigator use this checklist:

  • Check the IntroscopeAgent.log (see above for expected location): f the IntroscopeAgent.log does not exist or is not being updated for a long period, this means that the Byte Code Adapter (ISAgent) is not running.
  • Check the results of activity Byte Code Adapter Installation in the Managed System Configuration in SAP Solution Manager for the target system
  • OS Permissions: If the managed system runs on a UNIX server, ensure that the OS permissions are correctly configured as described in SAP Note 1163751. In most of the cases the Byte Code Adapter cannot start due to wrong permissions at OS level.  The easiest configuration to make it work is to give 775 (e.g: chmod 775) permissions to the Managed System OS user in the Byte Code Adapter folder.
  • SAP NetWeaver Java: Check if the mandatory Java VM parameters -javaagent -Dcom.wily* are present in dev_server0. Search  dev_server* and std_server* for issues related to the Byte Code Adapter.
  • SAP NetWeaver Java: Check if the mandatory Java VM parameters are enabled in Netweaver Administrator. They may have been disabled on purpose by some administrator.
  • Other APM Tools: If the file IntroscopeAgent.log is not created or updated during startup and there are no error messages in std_server*.out you should also check if probably another APM tool (like DynaTrace) is activated. See also SAP Note 2934510.


Missing JMX Metrics

For troubleshooting JMX metric collection in the agent you can increase logging of the JMX service by setting these two additional properties:

JMX debug logging

log4j.logger.IntroscopeAgent.JMX\ Service=ALL, logfile

log4j.logger.IntroscopeAgent.JMXService=ALL, logfile


In general polling of particular MBeans will be switched off until the next restart of the Java VM as soon as any exception occurs for polling that MBean. To avoid this you can change the behavior via Netweaver Administrator:

  • Go to NWA > Availability and Performance Management > Resource Monitoring > Java System Reports or History Reports
  • Choose correct instance and server nodes, tab "Reports", choose "Monitor Browser" in the combo box. Then we can navigate to the monitor and check the counters.
  • If any counter is not active here, click Configuration --> Edit, React on Resource Failure = ignore, set Enable to true, then save. After this, the counter should become active.

For NetWeaver there is a known problem that reporting of http sessions ("Current Http Sessions") stops reporting. See SAP Note 2067705 for a fix.


Missing CPU Metrics

The CPU node in the Investigator tree of the byte code instrumentation agent (e.g. Netweaver agent) is populated via a native library, the so-called platform monitor. There are multiple possible reasons why this node might be missing:

  • There is no platform monitor extension for all supported SAP Netweaver platforms. As fallback you can use the OS metrics as provided by the Introscope host adapter (under node SAP Host Agent)
  • Windows: If the user running the Java VM is not in the administrators group it must be added to the group Performance Monitor Users to obtain the privileges required to read performance data.