お問い合わせ

Next Generation – Data Collection Infrastructure

This guide describes how a cloud application can be enabled for SAP Cloud ALM via the OpenTelemetry (OTEL) based Next Generation – Data Collection Infrastructure (NG-DCI). Specifically, how the OTEL data collection for Real User Monitoring in SAP Cloud ALM can be achieved. For SAP Cloud ALM Real User Monitoring both frontend and backend measurements are leveraged.

The Next Generation – Data Collection Infrastructure consists of the Data Collection Instrumentation (DCI), Data Collector Runtime (DCR) a BTP service, and SAP Cloud ALM (including the Data Routing Infrastructure, DRI). The actual instrumentation is done with the DCI.

Below you can find the broader concept and architecture diagram:

 

Currently, Real User Monitoring, Health Monitoring, and Exception Monitoring are able to consume data provided by NG-DCI. However, several other use cases of SAP Cloud ALM for operations plan to use NG_DCI in the future.

We start with the integration of BTP Cloud Foundry with SAP Cloud ALM as the target platform. However, in the future, we will also enable SAP Focused Run to consume data provided by NG-DCI. So the instrumentation at the managed service side needs to be done only once. Please be aware, that all functionalities in yellow boxes are future plans. 
 

High Level Architecture

The graphic below depicts the high-level architecture of the NG-DCI data collection. In this simple scenario, we assume, that we want to monitor a full-stack application built on SAP BTP. The front end consists of SAP UI5. The UI requests will be routed to the application backend via the Application Router. Here, the application backend consists of three different microservices. In order to monitor the backend with SAP Cloud ALM all three microservices are instrumented with DCI. To gather the client performance we leverage SAP UI5's ability to measure and export these Frontend Statistics Records (FESR). One of the three microservices acts as a receiver for these FESRs. This receiver is instrumented with both the DCI as well as the FESR receiver library. DCI will always export the data to the SAP Cloud ALM routing infrastructure. This routing infrastructure will route the data to the correct SAP Cloud ALM tenant.

The numbers in the picture correspond to the sections described below.

Not all steps are required for all use cases.

  • If you want to get the end-user response times to be displayed in Real User Monitoring you need to go through all steps. 
  • If you only want to get data from your back-end services for Health Monitoring or Integration & Exception Monitoring you can skip section 1 and can start with section 2.  

Prerequisites

The cloud application to be monitored may consist of several microservices and run on SAP BTP Cloud Foundry. In order to execute the steps described below you need some development knowledge and deployment authorization in SAP BTP Cloud Foundry.

Configure Settings with Download Link

The OTEL ALM Libraries are shipped by SAP Repository Based Shipment Channel (RSBC). Request the technical user name and password in the RBSC UI. Please also check the corresponding documentation for User Authentication, for a full documentation of the RBSC please see the RBSC documentation.

Java

At least one Java application handling Front-end Statistical Records (FESR) requests must exist and be implemented using Spring Boot.

Required Java version:

  • Runtime needs to be Java 11

Required library versions:

  • otel-agent-ext-java: Version 1.5.4
  • calm-crun-client-api-java*: Version 1.5.3
  • fesr-to-otel-java: Version 1.5.4

* only relevant for Integration Monitoring

For complete version history, please check the following Version Update for Data Collection Instrumentation Libraries (SAP Help).

For the Java OTEL ALM Libraries, an integration in the SAP CF Java build pack is available. In that case, no integration into the Maven build is required as the libraries are already included in the build pack itself. If you do not use the Java build pack, you have to include the library as a Maven dependency:

  • Add artifactory as repository in pom.xml

<repositories>

<repository>

<id>***your-artifactory-id***</id>

<url>https://73555000100200018064.mavensrv.base.repositories.cloud.sap</url>

</repository>

</repositories>

  • Update settings.xml pointing to technical user credentials for the artifactory

    Note - It is recommended to store the password in a vault and reference it in the settings, so that it can be read during maven build and pipeline deployment!

<?xml version="1.0" encoding="UTF-8"?>

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">

<servers>

<server>

<id>**your-artifactory-id**</id>

<username>***technical-user-name read from vault***</username>

<password>***password read from vault***</password>

</server>

</servers>

</settings>

Node.js or CAP (JavaScript) based

At least one node.js application handling Front-end Statistical Records (FESR) requests must exist and use express.js.

Required library versions:

  • xotel-agent-ext-js: Version 1.5.2
  • fesr-to-otel-js: Version 1.5.0

For complete version history, please check the following Version Update for Data Collection Instrumentation Libraries (SAP Help).

In your .npmrc file, specify the credentials for the repository containing the OTEL ALM Libraries:

//73555000100200018064.npmsrv.cdn.repositories.cloud.sap/:_auth=<base64 encoded version of username:password>

The value for the _auth property is Base64 encoding of the combination of 'username:password' where the user name is the user name of the technical user. Respectively, the same holds for password.

Note - It is recommended to store the password in a vault and reference it in the settings, so that it can be read during build and pipeline deployment!

Step 1: Enabling Frontend Data Collection

You only need to perform the steps described in this section if you want to get the end-user response times to be displayed in Real User Monitoring. 

As of now, the only offered frontend measurements come from SAP UI5. For this, we leverage the Front End Subrecords measurement provided by the UI5 framework. Currently, we only support "standalone" UI5 applications for front-end measurements.

To instrument your own application with Frontend measuring there are three steps:

  • Configure your UI to gather frontend statistics and send them to a receiving endpoint
  • Configure your reverse proxy (e.g. approuter) to forward the FESR beacons from UI5
  • Provide an FESR receiver endpoint that will transfer the data to SAP Cloud ALM


1a Configure UI5

The first point can be achieved with a simple meta tag in your root index.html of your UI5 application. We will send the beacon to the /fesr endpoint (Note: depending on your project and routing setup this may contain an application specific prefix like in this example):

<meta name="sap-ui-fesr" content="/<optional app-specific-prefix>/fesr">

Here, the prefix is /<optional app-specific-prefix> is an application specific prefix example. It may or may not be necessary in your case. The endpoint /fesr is mandatory.


1b Configure Application Router (Reverse Proxy)

Since your UI application will likely use a reverse proxy to communicate with your backend you need to configure the /fesr route to be routed to the specific endpoint which will be configured in the next step. Usually, this is done by a central approuter. You must configure your central approuter to route the /fesr routes:

xs-app.json 
{

  "routes": [

    {

      "source": "^/fesr$",

      "target": "/fesr",

      "destination": "<destination pointing to your fesr receiver backend>",

      "csrfProtection": false,

      "authenticationType": "xsuaa"

    }

  ]

}

Note that csrfProtection must be set to false.


1c Instrument Receiver with FESR Receiver Library and DCI

Now, that UI5 is sending the FESR beacons to /fesr they need to be received and transmitted to SAP Cloud ALM. For this, you need to include both the FESR receiver library as well as the OpenTelemetry data collection instrumentation:

FESR Receiver

Step 2: Enabling Backend Data Collection

To configure the data collection instrumentation for backend services you only need the respective autoinstrumentation:

This concludes the NG-DCI configuration in your SAP BTP application, afterwards, you can check the supported monitoring applications in SAP Cloud ALM.

Step 3: Authentication

In order to export towards SAP Cloud ALM you have to setup the destination via OAuth based authentication.

  • Bind to Destination Service
  • Add the below lines to mta.yaml

mta.yaml 
modules:

- name: <your application srv module>

  requires:

   - name: <destination-service-name>

resources:

 - name: <destination-service-name>

   parameters:

     service-plan: lite

     service: destination

   type: org.cloudfoundry.managed-service

  • Create a destination configuration using Destination Service for persisting the baseUrl and retrieving it.
  • The Destination endpoint should be your SAP Cloud ALM endpoint "Api" url taken from the API service key.
    E.g. https://eu10.alm.cloud.sap/api
  • The Destination name must be CALM_datacollector
  • Proxy Type: Internet
  • Authentication Type: OAuth2ClientCredentials
  • Enter the client ID and secret from the API service key.
  • Token Service URL: API service key parameter "url"
  • How to retrieve the API service key, can be found here.

The destination should look like this:

When clicking on button Check Connection you might receive http status code 404.

Customer Support

In case you encounter problems using or setting up the Next Generation – Data Collection Infrastructure in SAP Cloud ALM, please report a case using the following components:

  • SV-CLM-INF-DCR - Data Collection & Routing

Please select your SAP Cloud ALM tenant when reporting the case.