Operations Automation for Alert Management

Operations Automation for Alert Management allows to define and automate alert reaction flows  using the guided procedure framework. They help to reduce the time from detection to correction via

  • Automated problem context collection
    • Problem context collection activities are triggered automatically as soon as an alert was raised. The results will be available in alert inbox/ alert details as soon as alert processor starts processing the alert.
    • Manual system login or OS level login can be avoided or at least minimized
  • Standardization of alert resolution process and user guidance
  • Remote read-only access and download of relevant log and trace files

Alert Reaction procedures consist of problem context collection activities and manual activities that guide you through the resolution of the alert.

The problem context collection activities can be triggered automatically as soon as the alert was raised. They collect information like alert status, system and database status details. You can easily add custom operations that check for example database connectivity, the availability of certain processes on OS level or execute SQL statements for analysis. The results of the problem context collection are available in a result report directly accessible from the related alert in alert inbox.

The manual activities guide you through the analysis and resolution of the problem. For example, they provide a navigation link to the file system browser that allows to analyze the relevant log and trace files of the managed system without OS level access. 

Alert Reaction procedures can be defined for

  • System Monitoring alerts
  • Configuration & Security Analytics alerts 
  • Integration & Cloud Monitoring alerts 

How it Works

Alert Reaction Process with Guided Procedures

As soon as an alert is triggered, SAP Focused Run checks if suitable guided procedures exist for this alert. If suitable alert reaction procedures are found, the automatic activities inside the guided procedure are triggered automatically and collect different problem context collection data. The guided procedure and its result report are accessible from the alert details.

You receive a notification about a new critical alert. From the notification, you open the alert screen and select the result report. You review the result report to understand the problem context and the current state:

  • Is the alert still red?
  • Which other alerts are open for same system?
  • Is database down?
  • Is database connect possible?
  • Is instance down?
  • Status of ABAP work processes?
  • Full file systems?
  • …

Afterward open the guided procedures and follow the verification and solution activities. Read and follow the activity documentation and update the status of this activity. Optionally add a comment which is visible in the HTML result report. Some activities provide context-sensitive navigation links to other analysis tools (for example the file system browser to analyze log files). Follow the documentation to perform the activities in the analysis tool. After the problem was solved trigger the verification activities.

Optionally you can also search and trigger additional alert reaction procedures from the alert details screen as follows:

  1. Select actions ->Search guided procedure. It shows you all guided procedures that are mapped to the existing alerts.  If no guided procedure is mapped to the existing alert, the list remains empty. Optionally you can switch to all guided procedures. Now all guided procedures for alert management are shown and not only the ones that are mapped to the alert.
  2. Select the guided procedure that you want to execute.
  3. Press button “Execute automatically”. It will be executed in a new batch job.

Automatic Execution of Guided Procedures

As soon as a new alert is triggered, the Outbound Adapter “GP alert handling” is called. The outbound adapter checks if suitable alert reaction procedures exist for this alert by comparing the following alert attributes to the search attributes of the existing guided procedures:

  • “Managed Object Name”
  • “Category” and 
  • “Technical name”

If suitable alert reaction procedures are found a new execution instance of the guided procedure is registered for the alert.

After the registration links to the relevant guided procedure and if executed to their result reports become visible in alert details screen.

The outbound adapter also checks if the guided procedure can be executed automatically. A guided procedure will only be executed automatically, if the flag “Can be executed automatically in background” is set.

In this case a new background job with the name “FRN_ARP_<GP Instance ID> running under the user FRN_BTC_GPA will be created.

The job executes the automatic activities in the guided procedure starting in the first step.

  • If a guided procedure consists only of automatic activities the job will execute the guided procedure from the beginning to the end.
  • If the guided procedure consists of automatic activities and manual activities, it will stop at the first manual activity. Automatic activities behind the first manual activities will not be processed by the job but are intended to be processed by user processing the guided procedure

Configuration

  • Create user FRN_BTC_GPA as per security guide
  • The alert processing users need to have authorization for Alert Management and Guided procedures per security guide
  • Activate the outbound connector for GP alert handling
    The outbound connector for “GP alert handling” needs to be activated if alert reaction procedures shall be triggered automatically for new alerts. Proceed as follows
    1. Open the SAP Focused Run launchpad.
    2. Select tile “Alert Management “Alert Consumer Variant Configuration”.
    3. Select page Outbound Integration
    4. Create a variant that includes outbound connector “GP alert handling” and eventually other outbound connectors required.  Save.
    5. Select tile “Alert Management “.
    6. Select Configuration --> Default Outbound Integrations --> Edit.
    7. Select the previously created outbound integration variant for monitoring use case "Technical System Monitoring" if you would like to trigger alert reaction procedures for system monitoring alerts. The new default outbound integration variant will become active for new systems as soon as simple system integration is performed. If you want to activate it for existing systems, you need to repeat the SSI configuration.
    8. Select the previously created variant for monitoring use case "Configuration & Security Analysis" if you would like to trigger alert reaction procedures for Configuration & Security Analysis alerts.
    9. Select the previously created variant for monitoring use case " Integration & Cloud Monitoring" if you would like to trigger alert reaction procedures for Integration & Cloud Monitoring alerts.
    10. Select save.
  • Define the batch user for executing Alert Reaction Procedure Batch jobs.
    1. Logon to SAP Focused Run via SAPGUI.
    2. Execute ABAP report “PR_CONF_GP_ARP_BTC” in transaction SA38.
    3. If the batch user is different than “FRN_BTC_GPA” enter the other user and execute the report.

Define Own Operation Flows for Alerts

In this section you find how to guides how the create a custom guided procedure for your alerts by either copying an existing SAP delivered guided procedure or by created a complete new one. The guided procedures for Event & Alert Management can be found in the Guided Procedures Catalog via the tile "Alert Management Guided Procedures"

Step 1: Determine alert attributes (required only for "Guided Procedures for System Monitoring Alerts")

The alert reaction procedure needs to be mapped to one or several alerts. This is required so that the outbound connector can decide which guided procedure to trigger for a certain alert and that the guided procedure search functionality can find the right guided procedure for the alert. To map the guided procedure to an alert you need to determine the alert attributes as follows:

  1. Open SAP Focused Run Launchpad
  2. Select tile “System Monitoring Template Management”
  3. Open the relevant template
  4. Select tab alerts
  5. Select the alert for which you want to define an alert reaction procedure
  6. Select the overview tab
  7. Write down the values defined for Managed Object Type, Category and Technical Name

Step 2: Create the new alert reaction procedure.

You can either create a new alert reaction procedure new or by copy using an existing procedure as template:

  1. Open SAP Focused Run Launchpad
  2. Select Tile “Alert Management - Guided Procedures”
  3. Select the right application area.
    If the guided procedure shall be triggered for System Monitoring or Health Monitoring alerts then select "Guided Procedures for System Monitoring Alerts" .
    If the guided procedure shall be triggered for Integration  Monitoring  alerts then select "Guided Procedures for Integration  Monitoring Alerts".
    If the guided procedure shall be triggered for Configuration & Security Analysis Alerts then select "Guided Procedures for Configuration & Security Analysis Alerts" .
  1.  In case an alert reaction procedure already exists  which should serve as a template for your new  alert reaction procedure , select it.
  2. Select “Create Guided Procedure
  3. Fill the data as follows:
    • Specify a meaningful name and description for the GP
    • Specify a customer-specific ABAP package with the right transport layer if you want to transport the guided procedure from a Focused Run development system to QA and production system. The package is required to know in which transport route the GP will be released. If you don't want to transport the guided procedure, you can choose "Temporary Package".
    • Set "Productive" flag, if the guided procedure is to be transported to a production system. If the GP is for testing purposes on the local system, do not set the flag.
    • Keep Flag “Enforce Step sequence” unchecked.
  4. Select tab “Search Attributes”.
  5. Enter the search attributes as follows:
    1.  For "Guided Procedures for System Monitoring Alerts":
      • Enter the managed object type corresponding to the managed object type of the alert.
      • Enter the category corresponding to the category of the alert
      • Enter technical name of the alert into field “alert name”
      • Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background.  Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background.
    2. For Guided Procedures for Integration Monitoring or Configuration & Security Analytics Alerts:
      • Select the F4 help besides alert name. Enter some filter criteria into   "Display Name" or "Technical name" fields or leave them blank. Select Go. Select the right alert from the list.  
      • Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background.  Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background. In this case an alert processor can trigger it manually from an alert.
  6.  Select OK

For information on how to maintain the content of the new guided procedure please refer to the guides below.

If you have created a customer specific alert reaction procedure for an alert for which an  alert reaction  procedure  has been delivered by SAP, both guided procedures will be triggered. The desired behavior is mostly that only the customer specific procedure is triggered.  Proceed as follows to achieve this:

  1. Open SAP Focused Run Launchpad
  2. Select Tile “Alert Management - Guided Procedures”
  3. Select the SAP delivered  alert reaction procedure template which should no longer be triggered. 
  4. Set the "Hidden" flag.

Add a step

To add a step, select  New --> Step --> Standard Step. Enter a name and a description for the step.
You can also reuse an existing step. A new step will be inserted below the selected step, in the Steps tree. 

Define a help text

The step help text provides detailed information about each step, such as what is checked, and what will happen in the background.

The "New" button in the help text section opens the Create New Document pop up, which allows you to define documentation for a step or an activity.

You can create a new HTML document or upload a PDF. If the HTML result report needs to show also the documentation in line, choose HTML.

Remove a step

Select the step and click on Remove to remove obsolete steps.

Add an activity

Each step consists out of one or more activities. 

For alert reaction procedures, you can use manual or plug-in activities.

TypeDescription
Manual

Simple manual activities consist of a description and either no navigation link, or a navigation link to a UI in SAP Focused Run, in the managed system, or an external link. The navigation link can be configured.

Custom UIsDeprecated. Do not use.
Automatic

Automatic activities are activities that can be carried out in a guided procedure without user interaction or manual steps. If you have developed a suitable automatic activity you can include it in the alert reaction procedure.

Plug-in

Plug-ins are automatic activities that can be customized during guided procedure creation time. SAP delivers many plug-ins that can be easily customized and included into the alert reaction procedure.

As of Focused Run 5.0 SP00 you can optionally configure that steps and  activities become only visible and executable if certain conditions are met.

Add a plug-in activity

Proceed as follows to include a new plug-in activity in your alert reaction procedure:

  1. Select the relevant step.
  2. Select Automatic Activity --> New
  3. Choose "Select a Plug-in" and select the required plug-in from the value help.
  4. Rename the activity
  5. Maintain the plug-in attributes per your requirements
  6. Select OK
  7. While the new activity is selected press "Help text" --> New
  8. Enter a name for the documentation
  9. Enter a suitable documentation text. Press save.
  10.  Use the buttons "Move up or Move down " to change the order of activities in the step

Add a Manual Activity

  1. Select Manual activities – new
  2. Enter a name for the activity
  3. If required, select New to define a navigation link.
  4. Select ok
  5. Select help next new and define an appropriate instruction text
  6. As of SAP Focused Run 4.0 FP03 you can optionally add a questionnaire that a user can fill out when processing the activity.
    1. Select Input "Simple"
    2. Select "Add" 
    3. Specify an Input ID. This is the id under which the answer will be stored in the "GP Instance Context".
    4. Specify the Input Display Name. This is the text which is displayed during runtime.
    5. Select whether the input is mandatory or not.
    6. Select the data type: Either check box, text, drop down, multi value drop down. For dropd own or multi value drop down you have to define the items to be shown: Add a key and a value per item. The value is shown in the runtime Ui. The key will be stored in "GP Instance Context" if it was selected by the user filling out the questionnaire.
    7. Enter a description. The description can contain additional hints how to define the input. During runtime the description will be displayed when hovering over the Input Display name.
    8. Select Ok.

Example: Manual activities to analyze log and trace files
Very often it is required to analyze critical errors in log and trace files. To support the analysis of relevant log and trace files without manual log on to the managed system or operating system level SAP Focused Run provides the following:

  • The file system browser that allows to browse the file system of managed hosts and display or download files.
  • Expert navigation links in the guided procedure provide the possibility to open the file system browser context sensitively in the relevant log or trace file directory of the hosts of the managed system or database

The following expert navigation link classes are available:

  • CL_GPCR_FSB_ABAP_JAVA_WORKDIR
    provides navigation links to open file system browser in the work directory for all instances of an ABAP or JAVA system
  • CL_GPCR_FSB_JAVA_SERVERLOGDIR
    provides navigation links to open File System Browser in the server log  directory for all servers of a JAVA  system
  • CL_GPCR_FSB_BOBJLOGDIR
    provides navigation links to open File System Browser in the logging directory for all servers of a BOBJ system
  • CL_GPCR_FSB_BOBJSERVERPIDDIR
    provides navigation links to open File System Browser in the serverpids directory for all servers of a BOBJ system
  • CL_GPCR_FSB_HANA_TRACEDIR
    provides navigation links to open File System Browser in the trace  directory for all instances of a HANA Database

You can define a manual activity to check relevant log and trace files of the managed system as follows:

  1. Select Manual activities – new
  2. Enter a name for the activity
  3. Select New to define a navigation link.
  4. Select Expert
  5. Choose the expert link class according your requirements (see above)
  6. Select ok

Remove a Activity

Select the relevant activity and press button "Remove" to remove the relevant activity if it is obsolete.

Change the contents and appearance of the alert reaction result report

  1. Select button "Report Configuration".
  2. Set flag "Include help documentation" if the alert reaction report should also contain step and activity documentation. If not keep this flag disabled.
  3. Some activities provide context sensitive navigation link into the relevant application in FRUN or the managed system. With the flag "Include Navigation Link" you can decide whether the navigation links should be included in the HTML report or not.
  4. Choose "CSS class name"CL_GPCR_CSS_STYLE_MIME_REP" to choose a more suitable style sheet for the HTML report than the standard one. Alternatively, you could define you own style sheet class for the html report to adapt it to your needs.
  5. Select include navigation links if it the report should include the activity specific navigation links
  6. Select include log messages and select all and all messages
    Select Ok
  7. Select Save

Activate the alert reaction guided procedure

  1. Select button "Check". If the check reports inconsistencies, fix them.
  2. Select button. "Activate"

Test the alert reaction guided procedure

  1. Try to generate the alert which is mapped to the guided procedure for the relevant system , instance or database.
  2. Open Alert Management. Search for the relevant alert.
  3. If the outbound adapter “GP alert handling” is activated, the guided procedure should have been triggered automatically in background and should be visible in the alert details in the guided procedure section. If not:  Select actions à search guided procedure. Check that you find the alert reaction procedure under “Filtered guided procedures”. Select Execute in background or Execute Manually.
  4. Click on the report link to review the report
  5. Click on the GP link and process the guided procedure

If the alert reaction guided procedure needs to be transported:

  1. Press button "Transport".
  2. Include the guided procedure into a new transport request or choose an existing.
  3. Release and import the requests per your transport rules.

The following plug-in activities are especially useful for alert reaction procedures

  • Alert Handling: Check Current Rating of Alert

This Plug-in activity displays the status of the alert and the rating of its related metrics in a table.

It provides a context sensitive navigation link into the alert details for the alert.

It can be included at the beginning of the guided procedure to show the metrics at the time when the alert was raised and additionally at the end in the verification step to show whether the alert is resolved and related metrics have changed to green.

  • Alert Handling: Check Related Alerts for Current System

This Plug-in activity displays all currently open alerts for the system, it's instances, hosts and database in a table. It provides a navigation link into the alert inbox. This plug-in supports alert correlation. For example: If the system is down and the system database is down at the same time you can conclude that database unavailability might be root cause for system unavailability.

  • Alert Handling: Check SDA availability

This plug-in activity checks if the simple diagnostics agents related to the system or database in scope of the guided procedure are reachable from SAP Focused Run. It provides a context sensitive navigation link to the agent framework.

  • Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of ABAP or JAVA systems. It executes SAP Control functions for the ABAP or JAVA system in the scope of the guided procedure and displays the result in the table. The following sapcontrol functions are supported:

Function Description

ABAPGetWPTable

Display a snapshot of the current work process load of an ABAP system in the Alert reaction procedure similar to SM66 - Global Work Process Overview.

ABAPGetSysWPTable

Display a snapshot of the currently running work processes of an ABAP system in the Alert reaction procedure.

GetProcessList

Shows the running status of all processes of all system instances of an ABAP or Java system

J2EEGetProcessList2

Shows the running status of all server nodes and of all system instances of an ABAP or Java system

  • Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of databases. It executes SAP host control operations for databases. If the guided procedure is executed for a technical system, it executes the operations for the system database of the technical system. If the guided procedure is executed in the scope of a database it executes the operations for the database. of the system in the scope of the guided procedure and displays the result in the table. The following SAP host control operations are supported:

Operation

Description

GetDatabase(System)Status

Display the running status of all database components

GetDatabaseProperties

Display selected properties of the database . 

  • Alert Handling: Execute Custom Operation New

This plug-in activity triggers custom operations which are deployed on the SAP host agents of the managed systems host. Results of the custom operation are displayed in the guided procedure HTML result report.

A custom operation is a set of commands, executed in the configured order until the first failure happens, or the complete set was successfully executed. The result of every single command is returned in the result of the operation itself. The configuration of the operations is expected in the sub-directory operations.d of the exe directory where SAPHostAgent was installed (generally on UNIX /usr/sap/hostctl/exe, and on windows C:\Program Files\SAP\hostctrl\exe).

It is intended that custom operations are defined by the customer and distributed to the system landscape by customer specific deployment means. SAP Focused Run does not deliver custom operations for alert reactions or alert reaction and does not provide a deployment and distribution mechanism for custom operations.

To learn how to set up the Custom Operations Plug-In see the next section

Additional plug-ins that you can reuse in your alert reaction guided procedure are described here.

Step 1: Define and test custom operations

You would like to define a custom operation that checks if the database is accessible from an ABAP instance by executing R3trans -d under the sidadm user. Required input parameter is the system identifier. The custom operation shall work for Unix systems

  • Log on to your test server where an ABAP instance is installed.
  • Navigate to /usr/sap/hostctrl/exe/operations.d
  • Create a file with the extension .conf  (e.g. GP_Check_R3trans.conf ) and the following contents:
    • Command: R3trans -d
    • Username:$[SID:#required#tolower]adm
    • Name: GP_Check_R3trans
    • Description: DB connectivity test via R3trans.
    • ResultConverter: flat
    • Platform: Unix
  • Test the custom operation as follows:
  • Navigate to /usr/sap/hostctrl/exe. Execute ./saphostcontrol -function ExecuteOperation -name GP_Check_R3trans SID=<3 character identifier of the ABAP System>

Step 2: Configure the custom operation plug-in

  1. Select Automatic Activities – New
  2. Select Plugin “Alert Handling: Execute Custom Operation New”
  3. Rename the activity
  4. Enter the operation name (without .conf file extension) at the end.
  5. Enter the arguments required by the custom operation. If several arguments are required, separate them by “;” without blanks). Some arguments can be replaced at runtime:
    • ${SID} will be replaced by 3 letter short SID of system
    • ${DBID} will be replaced by 3 letter id of DB
    • ${INSNR} will be replaced by instance number of ABAP, JAVA System or HANA DB
    • Example: the custom operation expects an input parameter “SID” with the system short SID.  Enter SID=${SID}
  6. Select the execution scope to define on which hosts of the managed system or database the custom operation shall be executed
    • All hosts Custom operation will be executed on all hosts agents   of technical system and on host agent of system database
    • Technical System: Custom operation will be executed on all hosts agents   of technical system but not on host agent of system database (if system database is on separate host)
    • HANA Master Node: Custom operation will be executed on the host agent where the HANA Master node of the database is installed
    • Database hosts: Custom operation will be executed on all hosts agents   of the database but not on host agents of the technical system
    • Central host: Custom operation will be executed on the host agent where the ABAP or JAVA Central Server is installed
  7. Enter timeout until how long the activity will wait to retrieve the result in seconds (recommended: 20)
  8. Set Wait for result:
    • Yes: Custom operation will wait for result and log a success message whether custom operation has been executed successfully and display the result in the GP HTML result report
    • No: fire and forget
  9. Define helper class (optional)
    In some cases it is required to do some evaluation if the result was ok or not and to display the result of the custom operation in a better readable format. This can be done by a helper class. This is a customer specific ABAP class that needs to be implemented using the interface “IF_CUST_OPERATION_HELPER_IMPL”. It must be developed especially for the defined custom operation. If you have developed such a helper class, enter it here. Otherwise leave the field blank.

Result:

When the guided procedure is executed manually or automatically, the custom operation will be triggered on the specified hosts agents. If wait for result=yes, the returning data will be displayed. If the custom operation has a helper class, the result can be semantically evaluated and a corresponding log message will be written.

Example:

  1. Configure an activity to automatically check connectivity from ABAP central instance to database via R3trans-d. Prerequisite is that the custom operation “GP_Check_R3trans” has been defined and deployed to the host agents
    Select Automatic Activities – New
    • Select Plugin “Alert Handling: Execute Custom Operation New”
    • Rename the activity to “Check DB connectivity”
    • Enter the operation name GP_Check_R3trans.
    • Enter the argument SID=${SID}
    • Select the execution scope to Central host:
    • Enter timeout 20
    • Set Wait for result to yes
  2. Configure an activity to automatically check connectivity to host agents via operation SAPHostAgent.Ping. The operation SAPHostAgent.Ping is available in host agents by default.
    Select Plugin “Alert Handling: Execute Custom Operation New”
    • Rename the activity to “Check Host Agent Availability”
    • Enter the operation name SAPHostAgent.Ping.
    • Leave arguments empty.
    • Select the execution scope to All Hosts
    • Enter timeout 20
    • Set Wait for result to yes

Release Notes

SAP Focused Run 5.0 SP00

  • Information about the managed object system type,  database type as well as alert context information are written into the "Guided Procedure Context" at the start of the guided procedure.
  • Introduced "Execute Conditions" for steps and activities so that they become only visible and can be executed if certain conditions are met.

SAP Focused Run 4.0 FP03

  • Introduced guided procedure instance specific storage so-called "Guided Procedure Context" where guided procedure activities can store data and other activities can read the data and react accordingly.
  • Provided APIs so that customer developed activities or plug-ins can store data in the "Guided Procedure Context"  or read data from "Guided Procedure Context".
  • Introduced manual activities with predefined input similar to a questionnaire. During design time the questions  can be defined. During runtime, a user executing the activity can answer these questions. The answers are stored  in the "Guided Procedure Context".

SAP Focused Run 4.0 SP00

  • Guided Procedure tiles were renamed to Operations Automation

SAP Focused Run 3.0 FP03

  • From GP planning you can suppress the sending of e mails for Guided Procedure Reports if all the activities inside the guided procedure have a green rating.
  • The table of contents in the guided procedure result report shows a rating per step/chapter.

SAP Focused Run 3.0 FP02

  • Alert Reaction Operation Flows can be triggered for Integration & Exception Monitoring Alerts
  • Alert Reaction Operation Flows can be triggered for Configuration  & Security Alerts