Enable HTTPS for Introscope Enterprise Manager

Goal

The goal of the procedure described on this page is to convert all http-based communication to the Introscope Enterprise Manager from HTTP to HTTPS.


Overview

The procedure consists of up to four parts which must be executed for standalone EM, MoM and the collectors:

  • Enable https port - this will switch to https, but still use the default certificates
  • (optional) Use your own certificate - update keystore and em-jetty-config.xml to use your own certificate
  • (optional) update https client in SAP Solution Manager Java stack to check certificates
  • Check status in SAP Solution Manager

Enable HTTPS Port

1 Under the EM installation directory, open the file IntroscopeEnterpriseManager.properties which is located under the folder /config.

2 In this file, activate the property introscope.enterprisemanager.webserver.jetty.configurationFile to enable customizing of the Jetty server. This is done by setting the following parameter:

IntroscopeEnterpriseManager.properties

introscope.enterprisemanager.webserver.jetty.configurationFile=em-jetty-config.xml

This line is present already, but commented out. With this parameter set, the customization present in the file em-jetty-config.xml will take action. 


3
Open the file em-jetty-config.xml and ensure the correct ports are set for http and/or https. By default, this file will come with the port 8444 set for HTTPS and port 8081 for HTTP.

4 Ensure also that the desired section is not commented. By default the HTTPS settings will come active and the HTTP settings are commented.

5 Again in the file IntroscopeEnterpriseManager.properties set the following parameter with the HTTPS port:

IntroscopeEnterpriseManager.properties

introscope.enterprisemanager.webserver.port=8444

Even though this property is ignored by Introscope after enabling the Jetty customization, it will still be used by SAP Solution Manager to generate URLs. To enable HTTPS after changing the port here the flag for "HTTPS" must be set in the SAP Solution Manager Infrastructure Preparation Configuration afterwards.

 

6 In the SAP Solution Manager system, access the Infrastructure Preparation under section 'Define CA Introscope". Here, use the discover button to reload the EM information.

7 Set the 'HTTPS' flag for the Enterprise Manager entry. Here the 'detail' section will show the new information:


8
Restart the Enterprise Manager. This will ensure it is using the newly set port. Afterwards, the EM and SAP Solution Manager will connect correctly through the new port. Also, from this point on, the WebView and Workstation will also use the HTTPS port.

9 Note – when using the HTTPS port 8444 for the EM WebView the complete SSL/TLS support in Introscope including the set of supported ciphers is standard Java functionality – it is not anything Introscope specific. This means the SSL/TLS Support depends on the underlying JRE/JDK being used.

If you encounter errors using a “weak” cipher please check that the JRE installed as part of the EM is fully equipped with the “unlimited strength jurisdiction policy”.

The policy files can be downloaded from the link:

http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

In case of further issues always ensure that you are using a current and supported JRE for running with the EM. If necessary add the unlimited strength jurisdiction policy files again.

 

Use your Own Certificate

This section describes an example sequence of commands to generate a server key/certificate for HTTPS usage. Other tools and procedures can most likely be used, but SAP can only provide support when exactly following this sequence. Since the procedure is not Introscope-specific common Internet resources can provide additional information.

In the examples the default store type "jks" is used. You can also use the store type "pkcs12" as advised by warnings in the keytool output.

Change the values used here:

  • myca is the certificate authority signing your keys.
  • fully qualified domain name of the EM host: myhost.my.domain
  • keystore password and key password: caapmsec
  • truststore password: caapmsec

The procedure has changed starting with Introscope 10.7.0.358 (10.7 SP0 Patch 6): The certificate is now validated and the root certificate must be stored in a separate trust store. See also https://knowledge.broadcom.com/external/article/231695.

Generate Key Pair

Create a keystore file to store the server's private key and self-signed certificate.

You may want to extend the validity and adjust other cert attributes. Alternatively to the self-contained command you can also fill the information interactively by omitting most of the parameters.

Keytool: Generate Key

keytool -keystore emhttps.jks -storepass caapmsec -alias emhttps -genkeypair -keyalg RSA -keypass caapmsec -dname "CN=myhost.my.domain, OU=AGS, O=SAP AG, L=Walldorf , ST=Baden-Wuerttemberg, C=DE"

This command creates a keystore emhttps.jks with a single entry. Check the contents:

Keytool: Check Keystore

keytool -keystore emhttps.jks -storepass caapmsec -list -v

Your keystore contains 1 entry

Alias name: emhttps

Creation date: 21.04.2021

Entry type: PrivateKeyEntry

Certificate chain length: 1

Certificate[1]:

Owner: CN=myhost.my.domain, OU=AGS, O=SAP AG, L=Walldorf, ST=Baden-Wuerttemberg, C=DE

Issuer: CN=myhost.my.domain, OU=AGS, O=SAP AG, L=Walldorf, ST=Baden-Wuerttemberg, C=DE


Generate a Certificate Signing Request (CSR)

This command generates a certificate signing request into the file myhost.csr.

Keytool: Generate a Certificate Signing Request (CSR)

keytool -keystore emhttps.jks -storepass caapmsec -alias emhttps -certreq -file myhost.csr


Submit the CSR

Submit the certificate signing request (myhost.csr) to your certificate authority. This procedure is specific to your company / environment and cannot be documented further. We assume that the resulting signed certificate is stored in myhost.cer.

Create a Backup of the Keystore

Just to enable recovering to this point in case of errors in the following steps create a copy of the current keystore:

Backup of the Keystore

copy emhttps.jks emhttps_backup.jks


Import any Intermediate Certificate

If there is any intermediate certificate then import this intermediate certificate of your CA into the keystore. Depending on your environment multiple certificates may be needed.

Do not import the root certificate - this must be imported into a separate trust store (see below).

Keytool: Import the CA Certificate

keytool -keystore emhttps.jks -storepass caapmsec -importcert -file myca.cer -alias myca -noprompt


Import the Signed Certificate Reply

After importing the CA certificates import the signed certificate that you received from your CA (myhost.cer):

Keytool: Import the Signed Certificate Reply

keytool -keystore emhttps.jks -storepass caapmsec -importcert -file myhost.cer -noprompt -alias emhttps -trustcacerts


Check the Keystore

Keytool: Check the Keystore

keytool -keystore emhttps.jks -storepass caapmsec -list

 

Keystore type: jks

Keystore provider: SUN

 

Your keystore contains 2 entries

 

emhttps, 21.04.2021, PrivateKeyEntry,

Certificate fingerprint (SHA1): D9:41:E9:0F:0E:9F:07:A7:CF:48:01:DD:BD:F7:2A:F0:57:9A:06:3A

myca, 21.04.2021, trustedCertEntry,

Certificate fingerprint (SHA1): B1:9A:E6:DF:6E:76:24:94:ED:FC:18:FA:6D:5A:9E:12:52:35:BE:93

You can also display the details by adding the parameter -v (truncated in the output below)

Keytool: Details of the Keystore

keytool -keystore emhttps.jks -storepass caapmsec -list -v

Keystore type: jks

Keystore provider: SUN

 

Your keystore contains 2 entries

 

Alias name: emhttps

Creation date: 21.04.2021

Entry type: PrivateKeyEntry

Certificate chain length: 3

Certificate[1]:

Owner: CN=dmyhost.my.domain, OU=AGS, O=SAP, C=DE

Issuer: CN=MyCA, O=SAP, L=Walldorf, C=DE

Serial number: 663a10f9c533c56e496eee6f5c61f5aa

Valid from: Wed Apr 21 12:46:50 CEST 2021 until: Thu Apr 21 12:46:50 CEST 2022

Certificate fingerprints: ...

Signature algorithm name: SHA256withRSA

Subject Public Key Algorithm: 2048-bit RSA key

Version: 3

 

Certificate[2]:

...

Certificate[3]:

..

*******************************************

Alias name: myca

Creation date: 21.04.2021

Entry type: trustedCertEntry

 

Owner: CN=myCA, O=SAP, L=Walldorf, C=DE

Issuer: CN=myRootCA, O=SAP AG, L=Walldorf, C=DE

Trust Store: Import Root Certificate

Import the root certificate of your CA into a separate trust store.

For illustration purpose this example uses the store type "pkcs12". You can choose between the default "jks" and "pkcs12".

Import Root CA into Trust Store

keytool -keystore emtrust.pkcs12 -storetype pkcs12 -storepass caapmsec -importcert -file "My Global Root CA.crt" -alias global -noprompt

keytool -keystore emtrust.pkcs12 -storetype pkcs12 -storepass caapmsec -list -v


Adjust em-jetty-config.xml

Now your keystore (emhttps.jks) is ready for use. The examples below assume that you put it into the /config folder of your Enterprise Manager installation.

Open em-jetty-config.xml in an editor and update the following properties to specify your keystore and key. 

Differences between 10.5 and 10.7

For Introscope 10.7 the format of the file em-jetty-config.xml has changed substantially due to the upgrade of the embedded Jetty to version 9.4.x. You cannot copy the file from Introscope version 10.5 or earlier! When transitioning from 10.5 or earlier to 10.7. Table below provides a mapping between <= 10.5 and 10.7

Property (10.7) Property (<= 10.5)Description Example
KeyStorePathkeystorefile name of the keystore (relative to config folder)emhttps.jks
KeyStorePasswordpasswordpassword of the key storecaapmsec
KeyManagerPasswordkeyPasswordpassword of the key entry (typically same as keystore password)caapmsec
certAliascertAliasalias of the privateKeyEntry in the storeemhttps
em-jetty-config.xml (10.7)

<Set name="KeyStorePath">

  <SystemProperty name="introscope.config" default="./config" />/emhttps.jks

</Set>

<Set name="KeyStorePassword">caapmsec</Set>

<!-- Typically, key store password and private key password are identical. When private key

     password is different than key store password, uncomment and set the following setting to

     private key password. -->

<!-- Set name="KeyManagerPassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set -->

<Set name="certAlias">emhttps</Set>

<Set name="TrustStorePath">

  <SystemProperty name="introscope.config" default="./config" />/emtrust.pkcs12

</Set>

<Set name="TrustStorePassword">caapmsec</Set>

em-jetty-config.xml (<=10.5)

<Call name="addConnector">

<Arg>

<New class="com.wily.webserver.TrustingSslSocketConnector">

<Set name="validateCertificates">false</Set>

<Set name="HeaderBufferSize">8192</Set>

<Set name="RequestBufferSize">16384</Set>

<Set name="verifyHostnames">false</Set>

<Set name="port">8444</Set>

<Set name="keystore"><SystemProperty name="introscope.config" default="./config" />/emhttps.jks</Set>

<Set name="password">caapmsec</Set>

<Set name="keyPassword">caapmsec</Set>

<Set name="truststore"><SystemProperty name="introscope.config" default="./config" />/internal/server/keystore</Set>

<Set name="trustPassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>

<Set name="needClientAuth">false</Set>

<Set name="certAlias">emhttps</Set>

</Arg

</Call>


Check HTTPS in EM

After saving the file em-jetty-config.xml restart the Enterprise Manager and check that the HTTPS configuration is active:

1 The log file logs/IntroscopeEnterpriseManager.log should contain log entries as given in the example:

a ServerConnector should explicitly mention SSL and the HTTPS port (line 1)

b Directly below you should see details about the certificate that is selected via certAlias above. (line 2-4). If you do not see at least one or two of these lines this means your HTTPS configuration is NOT VALID!

2 Access the Enterprise Manager via HTTPS: https://myhost.my.domain:8444/. If you decide to stay with the default certificate you may get certificate warnings. Using your own company-signed certificate no warnings should occur.

IntroscopeEnterpriseManager.log

[INFO] [main] [Manager.EMWebServer] EM[ServerConnector@3292d91a{SSL, (ssl, http/1.1)}{0.0.0.0:8444}] Request Header Size is :16384

[INFO] [main] [Manager.EMWebServer] Certificate 'emssl' subject: CN=myhost.my.domain,O=My company ltd,L=Frankfurt,C=DE

[INFO] [main] [Manager.EMWebServer] Certificate 'emssl' issuer: CN=Some CA,O=My company ltd

[INFO] [main] [Manager.EMWebServer] Certificate 'emssl' subject alternative DNS name: myhost.my.domain


Update HTTPS Client in SAP Solution Manager

In SAP Solution Manager Infrastructure Preparation, Step "Define CA Introscope", set the "https" flag for the EM.

You can configure the destinations towards Introscope in the SAP Solution Manager Java stack to check certificates:

Launch Netweaver Administrator (http://host:port/nwa) on the Java stack of SAP Solution Manager, go to System Management --> Configuration --> Destinations. There is one http destination entry for every Enterprise Manager following the naming scheme IntroscopeEM_emhost@port (port is the RMI port). The url specifies the http/https URL prefix that is used to access the Enterprise Manager via http. You can configure use of certificates here.

Check Status in SAP Solution Manager

In SAP Solution Manager Infrastructure Preparation, Step "Define CA Introscope": Make sure that all changes are saved and that all Enterprise Managers show status "green" after a refresh. 

HTTPS configuration is completed.

Troubleshooting

Jetty Configuration Dump

For the https configuration you can activate a dump of the configuration: If em-jetty-config.xml is activated as described above add the property dumpAfterStart=true, immediately after the <Configure id="Server" ...> tag:

em-jetty-config.xml

<Configure id="Server" class="org.eclipse.jetty.server.Server">

<Set name="dumpAfterStart">true</Set>

This will trigger a dump of the Jetty configuration to the stdout channel: EMService.log on Windows, em.log on Unix.

HTTP Access Log

To check if http requests arrive at the EM at all you can activate logging of all http requests. For this purpose add the following section to em-jetty-config.xml. All http requests will arrive in a file IntroscopeHttp-*.request.log.

em-jetty-config.xml

<Call name="insertHandler">

  <Arg>

    <New id="RequestLog" class="org.eclipse.jetty.server.handler.RequestLogHandler">

      <Set name="requestLog">

        <New id="RequestLogImpl" class="org.eclipse.jetty.server.NCSARequestLog">

          <Set name="filename"><Property name="jetty.logs" default="./logs"/>/IntroscopeHttp-yyyy_mm_dd.request.log</Set>

          <Set name="filenameDateFormat">yyyy_MM_dd</Set>

          <Set name="LogTimeZone">GMT</Set>

          <Set name="retainDays">10</Set>

          <Set name="append">true</Set>

          <Set name="LogLatency">true</Set>

        </New>

      </Set>

    </New>

  </Arg>

</Call>

SSL Trace

To get more details on TLS connection negotiation you can activate the standard Java SSL tracing. For this purpose add the Java VM parameter -Djavax.net.debug like below  

-Djavax.net.debug=ssl:handshake:verbose:keymanager:trustmanager

or

-Djavax.net.debug=all:

On Windows add a new line to bin\EMService.conf:

EMService.conf

wrapper.java.additional.8=-Djavax.net.debug=ssl:handshake:verbose:keymanager:trustmanager

.8 is the next free number for the parameter group wrapper.java.additional. Depending on your configuration you may have to choose a different number.

On Unix change the property lax.nl.java.option.additional in Introscope_Enterprise_Manager.lax.

References & Reservations

References

The official CA/Broadcom Introscope documentation applies for the configuration procedure. Use the link
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7.html
to get to the overview page of all guides.

  1. CA APM Configuration and Administration Guide, Chapter Configuring Enterprise Manager Communications
  2. CA APM Java Agent Implementation Guide, Chapter Installing and Configuring the Java Agent, Configuring the connection to the Enterprise Manager
  3. CA APM Security Guide
  4. Official Jetty documentation: https://www.eclipse.org/jetty/documentation/ (9.4.x is embedded in Introscope 10.7)

Reservations

  • Help links from SAP dashboards to the help content are currently always generates as HTTP links. Change to https manually to get the help content displayed.
  • Only one of the channels for each HTTP and RMI can be used in SAP Solution Manager: either secure or non-secure.
  • Currently no support for certificate and keystore handling in SAP Solution Manager setup UIs. You can use the destination service of the J2EE server (via Netweaver Administrator) to configure certificates.