Step 1: Determine alert attributes (required only for "Guided Procedures for Advanced System Monitoring")

The alert reaction procedure needs to be mapped to one or several alerts. This is required so that the outbound connector can decide which guided procedure to trigger for a certain alert and that the guided procedure search functionality can find the right guided procedure for the alert. To map the guided procedure to an alert you need to determine the alert attributes as follows:

1. Open SAP Focused Run Launchpad
2. Select Advanced System Management.
3. Select tile “System Monitoring Template Management”
4. Open the relevant template
5. Select tab alerts
6. Select the alert for which you want to define an alert reaction procedure
7. Select the overview tab
8. Write down the values defined for Managed Object Type, Category and Technical Name

Step 2: Create the new alert reaction procedure.

You can either create a new alert reaction procedure new or by copy using an existing procedure as template:

1. Open SAP Focused Run Launchpad
2. Select Advanced Event & Alert Management
3. Select Tile “Alert Management - Guided Procedures”
4. Select the right application area depending whether you wan to define guided procedures for System Monitoring, Integration monitoring or Configuration & Security Analytics.
5.  In case an alert reaction procedure already exists  which should serve as a template for your new  alert reaction procedure , select it.
6. Select “Create Guided Procedure
7. Fill the data as follows:
   • Specify a meaningful name and description for the GP
   • Specify a customer-specific ABAP package with the right transport layer if you want to transport the guided procedure from Focused Run development system to QA and production system. The package is required to know in which transport route the GP will be released. If you don't want to transport the guided procedure, you can choose "Temporary Package".
   • Set "Productive" flag, if the guided procedure is to be transported to a production system. If the GP is for testing purposes on the local system, do not set the flag.
   • Keep Flag “Enforce Step sequence” unchecked.
8. Select tab “Search Attributes”.
9. Enter the search attributes as follows:
   1.  For "Guided Procedures for System Monitoring":
      • Enter the managed object type corresponding to the managed object type of the alert.
      • Enter the category corresponding to the category of the alert
      • Enter technical name of the alert into field “alert name”
      • Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background.  Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background.
   2. For Guided Procedures for Integration Monitoring or Configuration & Security Analytics:
      • Select the F4 help besides alert name. Enter some filter criteria into   "Display Name" or "Technical name" fields or leave them blank. Select Go. Select the right alert from the list.  
      • Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background.  Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background. In this case an alert processor can trigger it manually from an alert.
        
10.  Select OK

For information on how to maintain the content of the new guided procedure please refer to the guides below.

If you have created a customer specific alert reaction procedure for an alert for which an  alert reaction  procedure  has been delivered by SAP, both guided procedures will be triggered. The desired behavior is mostly that only the customer specific procedure is triggered.  Proceed as follows to achieve this:

1. Open SAP Focused Run Launchpad
2. Select Advanced Event & Alert Management
3. Select Tile “Alert Management - Guided Procedures”
4. Select the SAP delivered  alert reaction procedure template which should no longer be triggered. 
5. Set the "Hidden" flag.

Add a step

To add a step, select  New--> Step-->Standard Step. You can also reuse an existing step. A new step will be inserted below the selected step, in the Steps tree. 

Define a help text

The step help text provides detailed information about each step, such as what is checked, and what will happen in the background.

The "New" button in the help text section opens the Create New Document pop up, which allows you to define documentation for a step or an activity.

You can create a new HTML document or upload a PDF. If the HTML result report needs to show also the documentation in line, choose HTMt.

Remove a step

Select the step and click on Remove to remove obsolete steps.

Add an activity

Each step consists out of one or more activities. 

For alert reaction procedures, you can use manual or plug-in activities.

Add a plug-in activity

Proceed as follows to include a new plug-in activity in your alert reaction procedure:

1. Select the relevant step.
2. Select Automatic Activity --> New
3. Choose "Select a Plug-in" and select the required plug-in from the value help.
4. Rename the activity
5. Maintain the plug-in attributes per your requirements
6. Select OK
7. While the new activity is selected press "Help text" --> New
8. Enter a name for the documentation
9. Enter a suitable documentation text. Press save.
10.  Use the buttons "Move up or Move down " to change the order of activities in the step

Add a Manual Activity

1. Select Manual activities – new
2. Enter a name for the activity
3. If required, select New to define a navigation link.
4. Select ok
5. Select help next new and define an appropriate instruction text

Example: Manual activities to analyze log and trace files
Very often it is required to analyze critical errors in log and trace files. To support the analysis of relevant log and trace files without manual log on to the managed system or operating system level SAP Focused Run provides the following:

• The file system browser that allows to browse the file system of managed hosts and display or download files.
• Expert navigation links in the guided procedure provide the possibility to open the file system browser context sensitively in the relevant log or trace file directory of the hosts of the managed system or database

The following expert navigation link classes are available:

• CL_GPCR_FSB_ABAP_JAVA_WORKDIR
  provides navigation links to open file system browser in the work directory for all instances of an ABAP or JAVA system
• CL_GPCR_FSB_JAVA_SERVERLOGDIR
  provides navigation links to open File System Browser in the server log  directory for all servers of a JAVA  system
• CL_GPCR_FSB_BOBJLOGDIR
  provides navigation links to open File System Browser in the logging directory for all servers of a BOBJ system
• CL_GPCR_FSB_BOBJSERVERPIDDIR
  provides navigation links to open File System Browser in the serverpids directory for all servers of a BOBJ system
• CL_GPCR_FSB_HANA_TRACEDIR
  provides navigation links to open File System Browser in the trace  directory for all instances of a HANA Database

You can define a manual activity to check relevant log and trace files of the managed system as follows:

1. Select Manual activities – new
2. Enter a name for the activity
3. Select New to define a navigation link.
4. Select Expert
5. Choose the expert link class according your requirements (see above)
6. Select ok

Remove a Activity

Select the relevant activity and press button "Remove" to remove the relevant activity if it is obsolete.

Change the contents and appearance of the alert reaction result report

1. Select button "Report Configuration".
2. Set flag "Include help documentation" if the alert reaction report should also contain step and activity documentation. If not keep this flag disabled.
3. Some activities provide context sensitive navigation link into the relevant application in FRUN or the managed system. With the flag "Include Navigation Link" you can decide whether the navigation links should be included in the HTML report or not.
4. Choose "CSS class name"CL_GPCR_CSS_STYLE_MIME_REP" to choose a more suitable style sheet for the HTML report than the standard one. Alternatively, you could define you own style sheet class for the html report to adapt it to your needs.
5. Select include navigation links if it the report should include the activity specific navigation links
6. Select include log messages and select all and all messages
   Select Ok
7. Select Save

Activate the alert reaction guided procedure

1. Select button "Check". If the check reports inconsistencies, fix them.
2. Select button. "Activate"

Test the alert reaction guided procedure

1. Try to generate the alert which is mapped to the guided procedure for the relevant system , instance or database.
2. Open Alert Management. Search for the relevant alert.
3. If the outbound adapter “GP alert handling” is activated, the guided procedure should have been triggered automatically in background and should be visible in the alert details in the guided procedure section. If not:  Select actions à search guided procedure. Check that you find the alert reaction procedure under “Filtered guided procedures”. Select Execute in background or Execute Manually.
4. Click on the report link to review the report
5. Click on the GP link and process the guided procedure

If the alert reaction guided procedure needs to be transported:

1. Press button "Transport".
2. Include the guided procedure into a new transport request or choose an existing.
3. Release and import the requests per your transport rules.

The following plug-in activities are especially useful for alert reaction procedures

• Alert Handling: Check Current Rating of Alert

This Plug-in activity displays the status of the alert and the rating of its related metrics in a table.

It provides a context sensitive navigation link into the alert details for the alert.

It can be included at the beginning of the guided procedure to show the metrics at the time when the alert was raised and additionally at the end in the verification step to show whether the alert is resolved and related metrics have changed to green.

• Alert Handling: Check Related Alerts for Current System

This Plug-in activity displays all currently open alerts for the system, it's instances, hosts and database in a table. It provides a navigation link into the alert inbox. This plug-in supports alert correlation. For example: If the system is down and the system database is down at the same time you can conclude that database unavailability might be root cause for system unavailability.

• Alert Handling: Check SDA availability

This plug-in activity checks if the simple diagnostics agents related to the system or database in scope of the guided procedure are reachable from SAP Focused Run. It provides a context sensitive navigation link to the agent framework.

• Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of ABAP or JAVA systems. It executes SAP Control functions for the ABAP or JAVA system in the scope of the guided procedure and displays the result in the table. The following sapcontrol functions are supported:

• Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of databases. It executes SAP host control operations for databases. If the guided procedure is executed for a technical system, it executes the operations for the system database of the technical system. If the guided procedure is executed in the scope of a database it executes the operations for the database. of the system in the scope of the guided procedure and displays the result in the table. The following SAP host control operations are supported:

• Alert Handling: Execute Custom Operation New

This plug-in activity triggers custom operations which are deployed on the SAP host agents of the managed systems host. Results of the custom operation are displayed in the guided procedure HTML result report.

A custom operation is a set of commands, executed in the configured order until the first failure happens, or the complete set was successfully executed. The result of every single command is returned in the result of the operation itself. The configuration of the operations is expected in the sub-directory operations.d of the exe directory where SAPHostAgent was installed (generally on UNIX /usr/sap/hostctl/exe, and on windows C:\Program Files\SAP\hostctrl\exe).

It is intended that custom operations are defined by the customer and distributed to the system landscape by customer specific deployment means. SAP Focused Run does not deliver custom operations for alert reactions or alert reaction and does not provide a deployment and distribution mechanism for custom operations.

To learn how to set up the Custom Operations Plug-In see the next section

Additional plug-ins that you can reuse in your alert reaction guided procedure are described here.

Step 1: Define and test custom operations

You would like to define a custom operation that checks if the database is accessible from an ABAP instance by executing R3trans -d under the sidadm user. Required input parameter is the system identifier. The custom operation shall work for Unix systems

• Log on to your test server where an ABAP instance is installed.
• Navigate to /usr/sap/hostctrl/exe/operations.d
• Create a file with the extension .conf  (e.g. GP_Check_R3trans.conf ) and the following contents:
  • Command: R3trans -d
  • Username:$[SID:#required#tolower]adm
  • Name: GP_Check_R3trans
  • Description: DB connectivity test via R3trans.
  • ResultConverter: flat
  • Platform: Unix
• Test the custom operation as follows:
• Navigate to /usr/sap/hostctrl/exe. Execute ./saphostcontrol -function ExecuteOperation -name GP_Check_R3trans SID=<3 character identifier of the ABAP System>

Step 2: Configure the custom operation plug-in

1. Select Automatic Activities – New
2. Select Plugin “Alert Handling: Execute Custom Operation New”
3. Rename the activity
4. Enter the operation name (without .conf file extension) at the end.
5. Enter the arguments required by the custom operation. If several arguments are required, separate them by “;” without blanks). Some arguments can be replaced at runtime:
   • ${SID} will be replaced by 3 letter short SID of system
   • ${DBID} will be replaced by 3 letter id of DB
   • ${INSNR} will be replaced by instance number of ABAP, JAVA System or HANA DB
   • Example: the custom operation expects an input parameter “SID” with the system short SID.  Enter SID=${SID}
6. Select the execution scope to define on which hosts of the managed system or database the custom operation shall be executed
   • All hosts Custom operation will be executed on all hosts agents   of technical system and on host agent of system database
   • Technical System: Custom operation will be executed on all hosts agents   of technical system but not on host agent of system database (if system database is on separate host)
   • HANA Master Node: Custom operation will be executed on the host agent where the HANA Master node of the database is installed
   • Database hosts: Custom operation will be executed on all hosts agents   of the database but not on host agents of the technical system
   • Central host: Custom operation will be executed on the host agent where the ABAP or JAVA Central Server is installed
7. Enter timeout until how long the activity will wait to retrieve the result in seconds (recommended: 20)
8. Set Wait for result:
   • Yes: Custom operation will wait for result and log a success message whether custom operation has been executed successfully and display the result in the GP HTML result report
   • No: fire and forget
9. Define helper class (optional)
   In some cases it is required to do some evaluation if the result was ok or not and to display the result of the custom operation in a better readable format. This can be done by a helper class. This is a customer specific ABAP class that needs to be implemented using the interface “IF_CUST_OPERATION_HELPER_IMPL”. It must be developed especially for the defined custom operation. If you have developed such a helper class, enter it here. Otherwise leave the field blank.

Result:

When the guided procedure is executed manually or automatically, the custom operation will be triggered on the specified hosts agents. If wait for result=yes, the returning data will be displayed. If the custom operation has a helper class, the result can be semantically evaluated and a corresponding log message will be written.

Example:

1. Configure an activity to automatically check connectivity from ABAP central instance to database via R3trans-d. Prerequisite is that the custom operation “GP_Check_R3trans” has been defined and deployed to the host agents
   Select Automatic Activities – New
   • Select Plugin “Alert Handling: Execute Custom Operation New”
   • Rename the activity to “Check DB connectivity”
   • Enter the operation name GP_Check_R3trans.
   • Enter the argument SID=${SID}
   • Select the execution scope to Central host:
   • Enter timeout 20
   • Set Wait for result to yes
2. Configure an activity to automatically check connectivity to host agents via operation SAPHostAgent.Ping. The operation SAPHostAgent.Ping is available in host agents by default.
   Select Plugin “Alert Handling: Execute Custom Operation New”
   • Rename the activity to “Check Host Agent Availability”
   • Enter the operation name SAPHostAgent.Ping.
   • Leave arguments empty.
   • Select the execution scope to All Hosts
   • Enter timeout 20
   • Set Wait for result to yes