Operations Automation for Alert Management

How you can contact us:

Technical Assistance

Request product support from SAP
Non-Technical Assistance

Request non-product support or provide feedback on SAP Support Portal site

Operations Automation for Alert Management

Operations Automation for Alert Management allows to define and automate alert reaction flows using the guided procedure framework. They help to reduce the time from detection to correction via

Automated problem context collection
- Problem context collection activities are triggered automatically as soon as an alert was raised. The results will be available in alert inbox/ alert details as soon as alert processor starts processing the alert.
- Manual system login or OS level login can be avoided or at least minimized
Standardization of alert resolution process and user guidance
Remote read-only access and download of relevant log and trace files

Alert Reaction procedures consist of problem context collection activities and manual activities that guide you through the resolution of the alert.

The problem context collection activities can be triggered automatically as soon as the alert was raised. They collect information like alert status, system and database status details. You can easily add custom operations that check for example database connectivity, the availability of certain processes on OS level or execute SQL statements for analysis. The results of the problem context collection are available in a result report directly accessible from the related alert in alert inbox.

The manual activities guide you through the analysis and resolution of the problem. For example, they provide a navigation link to the file system browser that allows to analyze the relevant log and trace files of the managed system without OS level access.

Alert Reaction procedures can be defined for

System Monitoring alerts
Configuration & Security Analytics alerts as of SAP Focused Run 3.0 FP02
Integration & Cloud Monitoring alerts as of SAP Focused Run 3.0 FP02

How it Works

Alert Reaction Process with Guided Procedures

As soon as an alert is triggered, SAP Focused Run checks if suitable guided procedures exist for this alert. If suitable alert reaction procedures are found, the automatic activities inside the guided procedure are triggered automatically and collect different problem context collection data. The guided procedure and its result report are accessible from the alert details.

You receive a notification about a new critical alert. From the notification, you open the alert screen and select the result report. You review the result report to understand the problem context and the current state:

Is the alert still red?
Which other alerts are open for same system?
Is database down?
Is database connect possible?
Is instance down?
Status of ABAP work processes?
Full file systems?
…

Afterward open the guided procedures and follow the verification and solution activities. Read and follow the activity documentation and update the status of this activity. Optionally add a comment which is visible in the HTML result report. Some activities provide context-sensitive navigation links to other analysis tools (for example the file system browser to analyze log files). Follow the documentation to perform the activities in the analysis tool. After the problem was solved trigger the verification activities.

Optionally you can also search and trigger additional alert reaction procedures from the alert details screen as follows:

Select actions ->Search guided procedure. It shows you all guided procedures that are mapped to the existing alerts. If no guided procedure is mapped to the existing alert, the list remains empty. Optionally you can switch to all guided procedures. Now all guided procedures for alert management are shown and not only the ones that are mapped to the alert.
Select the guided procedure that you want to execute.
Press button “Execute automatically”. It will be executed in a new batch job.

Automatic Execution of Guided Procedures

As soon as a new alert is triggered, the Outbound Adapter “GP alert handling” is called. The outbound adapter checks if suitable alert reaction procedures exist for this alert by comparing the following alert attributes to the search attributes of the existing guided procedures:

“Managed Object Name”
“Category” and
“Technical name”

If suitable alert reaction procedures are found a new execution instance of the guided procedure is registered for the alert.

After the registration links to the relevant guided procedure and if executed to their result reports become visible in alert details screen.

The outbound adapter also checks if the guided procedure can be executed automatically. A guided procedure will only be executed automatically, if the flag “Can be executed automatically in background” is set.

In this case a new background job with the name “FRN_ARP_<GP Instance ID> running under the user FRN_BTC_GPA will be created.

The job executes the automatic activities in the guided procedure starting in the first step.

If a guided procedure consists only of automatic activities the job will execute the guided procedure from the beginning to the end.
If the guided procedure consists of automatic activities and manual activities, it will stop at the first manual activity. Automatic activities behind the first manual activities will not be processed by the job but are intended to be processed by user processing the guided procedure

Configuration

Create user FRN_BTC_GPA as per security guide
The alert processing users need to have authorization for Alert Management and Guided procedures per security guide
Activate the outbound connector for GP alert handling
The outbound connector for “GP alert handling” needs to be activated if alert reaction procedures shall be triggered automatically for new alerts. Proceed as follows
1. Open the SAP Focused Run launchpad.
2. Select tile “Alert Management “Alert Consumer Variant Configuration”.
3. Select page Outbound Integration
4. Create a variant that includes outbound connector “GP alert handling” and eventually other outbound connectors required. Save.
5. Select tile “Alert Management “.
6. Select Configuration --> Default Outbound Integrations --> Edit.
7. Select the previously created outbound integration variant for monitoring use case "Technical System Monitoring" if you would like to trigger alert reaction procedures for system monitoring alerts. The new default outbound integration variant will become active for new systems as soon as simple system integration is performed. If you want to activate it for existing systems, you need to repeat the SSI configuration.
8. Select the previously created variant for monitoring use case "Configuration & Security Analysis" if you would like to trigger alert reaction procedures for Configuration & Security Analysis alerts.
9. Select the previously created variant for monitoring use case " Integration & Cloud Monitoring" if you would like to trigger alert reaction procedures for Integration & Cloud Monitoring alerts.
10. Select save.
Define the batch user for executing Alert Reaction Procedure Batch jobs.
1. Logon to SAP Focused Run via SAPGUI.
2. Execute ABAP report “PR_CONF_GP_ARP_BTC” in transaction SA38.
3. If the batch user is different than “FRN_BTC_GPA” enter the other user and execute the report.

Define Own Operation Flows for Alerts

In this section you find how to guides how the create a custom guided procedure for your alerts by either copying an existing SAP delivered guided procedure or by created a complete new one. The guided procedures for Event & Alert Management can be found in the Guided Procedures Catalog via the tile "Alert Management Guided Procedures"

How to create own operation flows for alerts

Step 1: Determine alert attributes (required only for "Guided Procedures for System Monitoring")

The alert reaction procedure needs to be mapped to one or several alerts. This is required so that the outbound connector can decide which guided procedure to trigger for a certain alert and that the guided procedure search functionality can find the right guided procedure for the alert. To map the guided procedure to an alert you need to determine the alert attributes as follows:

Open SAP Focused Run Launchpad
Select tile “System Monitoring Template Management”
Open the relevant template
Select tab alerts
Select the alert for which you want to define an alert reaction procedure
Select the overview tab
Write down the values defined for Managed Object Type, Category and Technical Name

Step 2: Create the new alert reaction procedure.

You can either create a new alert reaction procedure new or by copy using an existing procedure as template:

Open SAP Focused Run Launchpad
Select Tile “Alert Management - Guided Procedures”
Select the right application area depending whether you wan to define guided procedures for System Monitoring, Integration monitoring or Configuration & Security Analytics.
In case an alert reaction procedure already exists which should serve as a template for your new alert reaction procedure , select it.
Select “Create Guided Procedure
Fill the data as follows:
- Specify a meaningful name and description for the GP
- Specify a customer-specific ABAP package with the right transport layer if you want to transport the guided procedure from Focused Run development system to QA and production system. The package is required to know in which transport route the GP will be released. If you don't want to transport the guided procedure, you can choose "Temporary Package".
- Set "Productive" flag, if the guided procedure is to be transported to a production system. If the GP is for testing purposes on the local system, do not set the flag.
- Keep Flag “Enforce Step sequence” unchecked.
Select tab “Search Attributes”.
Enter the search attributes as follows:
1. For "Guided Procedures for System Monitoring":
  - Enter the managed object type corresponding to the managed object type of the alert.
  - Enter the category corresponding to the category of the alert
  - Enter technical name of the alert into field “alert name”
  - Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background. Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background.
2. For Guided Procedures for Integration Monitoring or Configuration & Security Analytics:
  - Select the F4 help besides alert name. Enter some filter criteria into "Display Name" or "Technical name" fields or leave them blank. Select Go. Select the right alert from the list.
  - Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background. Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background. In this case an alert processor can trigger it manually from an alert.
Select OK

For information on how to maintain the content of the new guided procedure please refer to the guides below.

How to ensure that SAP delivered templates are not triggered any more

If you have created a customer specific alert reaction procedure for an alert for which an alert reaction procedure has been delivered by SAP, both guided procedures will be triggered. The desired behavior is mostly that only the customer specific procedure is triggered. Proceed as follows to achieve this:

Open SAP Focused Run Launchpad
Select Tile “Alert Management - Guided Procedures”
Select the SAP delivered alert reaction procedure template which should no longer be triggered.
Set the "Hidden" flag.

How to edit the content of own operation flows for alerts

Add a step

To add a step, select New --> Step --> Standard Step. You can also reuse an existing step. A new step will be inserted below the selected step, in the Steps tree.

Define a help text

The step help text provides detailed information about each step, such as what is checked, and what will happen in the background.

The "New" button in the help text section opens the Create New Document pop up, which allows you to define documentation for a step or an activity.

You can create a new HTML document or upload a PDF. If the HTML result report needs to show also the documentation in line, choose HTMt.

Remove a step

Select the step and click on Remove to remove obsolete steps.

Add an activity

Each step consists out of one or more activities.

For alert reaction procedures, you can use manual or plug-in activities.

Type	Description
Manual	Simple manual activities consist of a description and either no navigation link, or a navigation link to a UI in SAP Focused Run, in the managed system, or an external link. The navigation link can be configured.
Custom UIs	Deprecated. Do not use.
Automatic	Automatic activities are activities that can be carried out in a guided procedure without user interaction or manual steps. If you have developed a suitable automatic activity you can include it in the alert reaction procedure.
Plug-in	Plug-ins are automatic activities that can be customized during guided procedure creation time. SAP delivers many plug-ins that can be easily customized and included into the alert reaction procedure.

Add a plug-in activity

Proceed as follows to include a new plug-in activity in your alert reaction procedure:

Select the relevant step.
Select Automatic Activity --> New
Choose "Select a Plug-in" and select the required plug-in from the value help.
Rename the activity
Maintain the plug-in attributes per your requirements
Select OK
While the new activity is selected press "Help text" --> New
Enter a name for the documentation
Enter a suitable documentation text. Press save.
Use the buttons "Move up or Move down " to change the order of activities in the step

Add a Manual Activity

Select Manual activities – new
Enter a name for the activity
If required, select New to define a navigation link.
Select ok
Select help next new and define an appropriate instruction text

Example: Manual activities to analyze log and trace files
Very often it is required to analyze critical errors in log and trace files. To support the analysis of relevant log and trace files without manual log on to the managed system or operating system level SAP Focused Run provides the following:

The file system browser that allows to browse the file system of managed hosts and display or download files.
Expert navigation links in the guided procedure provide the possibility to open the file system browser context sensitively in the relevant log or trace file directory of the hosts of the managed system or database

The following expert navigation link classes are available:

CL_GPCR_FSB_ABAP_JAVA_WORKDIR
provides navigation links to open file system browser in the work directory for all instances of an ABAP or JAVA system
CL_GPCR_FSB_JAVA_SERVERLOGDIR
provides navigation links to open File System Browser in the server log directory for all servers of a JAVA system
CL_GPCR_FSB_BOBJLOGDIR
provides navigation links to open File System Browser in the logging directory for all servers of a BOBJ system
CL_GPCR_FSB_BOBJSERVERPIDDIR
provides navigation links to open File System Browser in the serverpids directory for all servers of a BOBJ system
CL_GPCR_FSB_HANA_TRACEDIR
provides navigation links to open File System Browser in the trace directory for all instances of a HANA Database

You can define a manual activity to check relevant log and trace files of the managed system as follows:

Select Manual activities – new
Enter a name for the activity
Select New to define a navigation link.
Select Expert
Choose the expert link class according your requirements (see above)
Select ok

Remove a Activity

Select the relevant activity and press button "Remove" to remove the relevant activity if it is obsolete.

Change the contents and appearance of the alert reaction result report

Select button "Report Configuration".
Set flag "Include help documentation" if the alert reaction report should also contain step and activity documentation. If not keep this flag disabled.
Some activities provide context sensitive navigation link into the relevant application in FRUN or the managed system. With the flag "Include Navigation Link" you can decide whether the navigation links should be included in the HTML report or not.
Choose "CSS class name"CL_GPCR_CSS_STYLE_MIME_REP" to choose a more suitable style sheet for the HTML report than the standard one. Alternatively, you could define you own style sheet class for the html report to adapt it to your needs.
Select include navigation links if it the report should include the activity specific navigation links
Select include log messages and select all and all messages
Select Ok
Select Save

Activate the alert reaction guided procedure

Select button "Check". If the check reports inconsistencies, fix them.
Select button. "Activate"

Test the alert reaction guided procedure

Try to generate the alert which is mapped to the guided procedure for the relevant system , instance or database.
Open Alert Management. Search for the relevant alert.
If the outbound adapter “GP alert handling” is activated, the guided procedure should have been triggered automatically in background and should be visible in the alert details in the guided procedure section. If not: Select actions à search guided procedure. Check that you find the alert reaction procedure under “Filtered guided procedures”. Select Execute in background or Execute Manually.
Click on the report link to review the report
Click on the GP link and process the guided procedure

If the alert reaction guided procedure needs to be transported:

Press button "Transport".
Include the guided procedure into a new transport request or choose an existing.
Release and import the requests per your transport rules.

Useful plug-in activities for Alert Reaction Procedures

The following plug-in activities are especially useful for alert reaction procedures

Alert Handling: Check Current Rating of Alert

This Plug-in activity displays the status of the alert and the rating of its related metrics in a table.

It provides a context sensitive navigation link into the alert details for the alert.

It can be included at the beginning of the guided procedure to show the metrics at the time when the alert was raised and additionally at the end in the verification step to show whether the alert is resolved and related metrics have changed to green.

Alert Handling: Check Related Alerts for Current System

This Plug-in activity displays all currently open alerts for the system, it's instances, hosts and database in a table. It provides a navigation link into the alert inbox. This plug-in supports alert correlation. For example: If the system is down and the system database is down at the same time you can conclude that database unavailability might be root cause for system unavailability.

Alert Handling: Check SDA availability

This plug-in activity checks if the simple diagnostics agents related to the system or database in scope of the guided procedure are reachable from SAP Focused Run. It provides a context sensitive navigation link to the agent framework.

Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of ABAP or JAVA systems. It executes SAP Control functions for the ABAP or JAVA system in the scope of the guided procedure and displays the result in the table. The following sapcontrol functions are supported:

Function	Description
ABAPGetWPTable	Display a snapshot of the current work process load of an ABAP system in the Alert reaction procedure similar to SM66 - Global Work Process Overview.
ABAPGetSysWPTable	Display a snapshot of the currently running work processes of an ABAP system in the Alert reaction procedure.
GetProcessList	Shows the running status of all processes of all system instances of an ABAP or Java system
J2EEGetProcessList2	Shows the running status of all server nodes and of all system instances of an ABAP or Java system

Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of databases. It executes SAP host control operations for databases. If the guided procedure is executed for a technical system, it executes the operations for the system database of the technical system. If the guided procedure is executed in the scope of a database it executes the operations for the database. of the system in the scope of the guided procedure and displays the result in the table. The following SAP host control operations are supported:

Operation	Description
GetDatabase(System)Status	Display the running status of all database components
GetDatabaseProperties	Display selected properties of the database .

Alert Handling: Execute Custom Operation New

This plug-in activity triggers custom operations which are deployed on the SAP host agents of the managed systems host. Results of the custom operation are displayed in the guided procedure HTML result report.

A custom operation is a set of commands, executed in the configured order until the first failure happens, or the complete set was successfully executed. The result of every single command is returned in the result of the operation itself. The configuration of the operations is expected in the sub-directory operations.d of the exe directory where SAPHostAgent was installed (generally on UNIX /usr/sap/hostctl/exe, and on windows C:\Program Files\SAP\hostctrl\exe).

It is intended that custom operations are defined by the customer and distributed to the system landscape by customer specific deployment means. SAP Focused Run does not deliver custom operations for alert reactions or alert reaction and does not provide a deployment and distribution mechanism for custom operations.

To learn how to set up the Custom Operations Plug-In see the next section

Additional plug-ins that you can reuse in your alert reaction guided procedure are described here.

How to set up the Custom Operations Plug-In

Step 1: Define and test custom operations

You would like to define a custom operation that checks if the database is accessible from an ABAP instance by executing R3trans -d under the sidadm user. Required input parameter is the system identifier. The custom operation shall work for Unix systems

Log on to your test server where an ABAP instance is installed.
Navigate to /usr/sap/hostctrl/exe/operations.d
Create a file with the extension .conf (e.g. GP_Check_R3trans.conf ) and the following contents:
- Command: R3trans -d
- Username:$[SID:#required#tolower]adm
- Name: GP_Check_R3trans
- Description: DB connectivity test via R3trans.
- ResultConverter: flat
- Platform: Unix
Test the custom operation as follows:
Navigate to /usr/sap/hostctrl/exe. Execute ./saphostcontrol -function ExecuteOperation -name GP_Check_R3trans SID=<3 character identifier of the ABAP System>

Step 2: Configure the custom operation plug-in

Select Automatic Activities – New
Select Plugin “Alert Handling: Execute Custom Operation New”
Rename the activity
Enter the operation name (without .conf file extension) at the end.
Enter the arguments required by the custom operation. If several arguments are required, separate them by “;” without blanks). Some arguments can be replaced at runtime:
- ${SID} will be replaced by 3 letter short SID of system
- ${DBID} will be replaced by 3 letter id of DB
- ${INSNR} will be replaced by instance number of ABAP, JAVA System or HANA DB
- Example: the custom operation expects an input parameter “SID” with the system short SID. Enter SID=${SID}
Select the execution scope to define on which hosts of the managed system or database the custom operation shall be executed
- All hosts Custom operation will be executed on all hosts agents of technical system and on host agent of system database
- Technical System: Custom operation will be executed on all hosts agents of technical system but not on host agent of system database (if system database is on separate host)
- HANA Master Node: Custom operation will be executed on the host agent where the HANA Master node of the database is installed
- Database hosts: Custom operation will be executed on all hosts agents of the database but not on host agents of the technical system
- Central host: Custom operation will be executed on the host agent where the ABAP or JAVA Central Server is installed
Enter timeout until how long the activity will wait to retrieve the result in seconds (recommended: 20)
Set Wait for result:
- Yes: Custom operation will wait for result and log a success message whether custom operation has been executed successfully and display the result in the GP HTML result report
- No: fire and forget
Define helper class (optional)
In some cases it is required to do some evaluation if the result was ok or not and to display the result of the custom operation in a better readable format. This can be done by a helper class. This is a customer specific ABAP class that needs to be implemented using the interface “IF_CUST_OPERATION_HELPER_IMPL”. It must be developed especially for the defined custom operation. If you have developed such a helper class, enter it here. Otherwise leave the field blank.

Result:

When the guided procedure is executed manually or automatically, the custom operation will be triggered on the specified hosts agents. If wait for result=yes, the returning data will be displayed. If the custom operation has a helper class, the result can be semantically evaluated and a corresponding log message will be written.

Example:

Configure an activity to automatically check connectivity from ABAP central instance to database via R3trans-d. Prerequisite is that the custom operation “GP_Check_R3trans” has been defined and deployed to the host agents
Select Automatic Activities – New
- Select Plugin “Alert Handling: Execute Custom Operation New”
- Rename the activity to “Check DB connectivity”
- Enter the operation name GP_Check_R3trans.
- Enter the argument SID=${SID}
- Select the execution scope to Central host:
- Enter timeout 20
- Set Wait for result to yes
Configure an activity to automatically check connectivity to host agents via operation SAPHostAgent.Ping. The operation SAPHostAgent.Ping is available in host agents by default.
Select Plugin “Alert Handling: Execute Custom Operation New”
- Rename the activity to “Check Host Agent Availability”
- Enter the operation name SAPHostAgent.Ping.
- Leave arguments empty.
- Select the execution scope to All Hosts
- Enter timeout 20
- Set Wait for result to yes

Release Notes

SAP Focused Run 4.0 SP00

Guided Procedure tiles were renamed to Operations Automation

SAP Focused Run 3.0 FP02

Alert Reaction Operation Flows can be triggered for Integration & Exception Monitoring Alerts
Alert Reaction Operation Flows can be triggered for Configuration & Security Alerts