Guided Procedures for Alert Management allow to define and automate alert reactions using the guided procedure framework. They help reduce the time from detection to correction via

• Automated problem context collection
  Problem context collection activities are triggered automatically as soon as an alert was raised. The results will be available in alert inbox / alert details as soon as alert processor starts processing the alert. Manual system logon or OS level logon can be avoided or at least minimized
• Standardization of alert resolution process and user guidance
  The guided procedure framework guides the user through the analysis and resolution of the alert
• Remote read -only access and download of relevant log and trace files
  

Alert Reaction procedures consist out of problem context collection activities and out of manual activities that guide you through the resolution of the alert. The problem context collection activities can be triggered automatically as soon as the alert was raised. They collect information like alert status, system and database status details. You can easily add custom operations that check for example database connectivity, the availability of certain processes on OS level or execute SQL statements for analysis. The results of the problem context collection are available in a result report directly accessible from the related alert in alert inbox. The manual activities guide you through the analysis and resolution of the problem. For example, they provide a navigation link to the file system browser that allows to analyze the relevant log and trace files of the managed system without OS level access. SAP Focused Run provides alert reaction procedures for ABAP and Java instance availability alerts. They can be used out of the box or enhanced with customer specific activities.  
Alert Reaction procedures are currently only available for system monitoring alerts.

As soon as an alert is triggered, SAP Focused Run checks if suitable guided procedures exist for this alert. If suitable alert reaction procedures are found, the automatic activities inside the guided procedure are triggered automatically and collect different problem context collection data. The guided procedure and it's result report are accessible from the alert details.

You receive a notification about a new critical alert. From the notification, you open the alert screen and select the result report. You review the result report to understand the problem context and the current state:

• Is the alert still red?
• Which other alerts are open for same system?
• Is database down?
• Is database connect possible?
• Is instance down?
• Status of ABAP work processes?
• Full file systems?
• …

Afterward open the guided procedures and follow the verification and solution activities. Read and follow the activity documentation and update the status of this activity. Optionally add a comment which is visible in the HTML result report. Some activities provide context-sensitive navigation links to other analysis tools (for example the file system browser to analyze log files). Follow the documentation to perform the activities in the analysis tool. After the problem was solved trigger the verification activities.

Optionally you can also search and trigger additional alert reaction procedures from the alert details screen as follows:

1. Select actions ->Search guided procedure. It shows you all guided procedures that are mapped to the existing alerts.  If no guided procedure is mapped to the existing alert, the list remains empty. Optionally you can switch to all guided procedures. Now all guided procedures for alert management are shown and not only the ones that are mapped to the alert.
2. Select the guided procedure that you want to execute.
3. Press button “Execute automatically”. It will be executed in a new batch job.

• Create user FRN_BTC_GPA as per security guide
• The alert processing users need to have authorization for Advanced Event & Alert Management and Guided procedures per security guide
• Activate the outbound connector for GP alert handling
  The outbound connector for “GP alert handling” needs to be activated if alert reaction procedures shall be triggered automatically for new alerts. Proceed as follows
  1. Open the SAP Focused Run launchpad.
  2. Select Advanced Event & Alert Management.
  3. Select Tile “Alert Management “Alert Consumer Connector”
  4. Select tab Outbound Connector
  5. Select Maintain Outbound Connector Variants
  6. Create a variant that includes outbound connector “GP alert handling” and eventually other outbound connectors required.  Save.
  7. Switch back to Alert Management - Alert Consumer Configuration. Set “Outbound Connector”: “Use Variant”
  8. Select the Outbound Connector Variant that includes outbound connector “GP alert handling”. Save
• Define the batch user for executing Alert Reaction Procedure Batch jobs.
  1. Logon to SAP Focused Run via SAPGUI.
  2. Execute ABAP report “PR_CONF_GP_ARP_BTC” in transaction SA38.
  3. If the batch user is different than “FRN_BTC_GPA” enter the other user and execute the report.

As soon as a new alert is triggered, the Outbound Adapter “GP alert handling” is called. The outbound adapter checks if suitable alert reaction procedures exist for this alert by comparing the alert attributes “Managed Object Name”, “Category” and “Technical name” to the corresponding guided procedure search attributes. If suitable alert reaction procedures are found, new execution instances of these guided procedures are triggered and registered for the alert. After registration, links to the relevant guided procedure and to their result reports become visible in alert details screen. The Outbound Adapter checks if the guided procedure search attribute “Can be executed automatically in background” is set to true. If yes, a new background job is created with the name “FRN_ARP_<GP Instance ID> running under user FRN_BTC_GPA. The job executes from the beginning of the guided procedure all automatic activities until the end if the guided procedure consists only out of automatic activities. If the guided procedure consists out of automatic activities and manual activities, it will stop at the first manual activity. Automatic activities behind the first manual activities will not be processed by the job but are intended to be processed by user processing the guided procedure.

How to create customer specific alert reaction procedures for alerts

Determine alert attributes

The alert reaction procedure needs to be mapped to one or several alerts. This is required so that the outbound connector can decide which guided procedure to trigger for a certain alert and that the guided procedure search functionality can find the right guided procedure for the alert. To map the guided procedure to an alert you need to determine the alert attributes as follows:

1. Open SAP Focused Run Launchpad
2. Select Advanced System Management.
3. Select tile “System Monitoring Template Management”
4. Open the relevant template
5. Select tab alerts
6. Select the alert for which you want to define an alert reaction procedure
7. Select the overview tab
8. Write down the values defined for Managed Object Type, Category and Technical Name

Create the new customer specific alert reaction procedure.

You can either create a new alert reaction procedure new or by copy using an existing procedure as template. We recommend to create it by copy as below:

1. Open SAP Focused Run Launchpad
2. Select Advanced Event & Alert Management
3. Select Tile “Alert Management - Guided Procedure Catalog”
4. Select the alert reaction procedure template which should serve as a template for the customer specific alert reaction procedure.
5. Select “Create -By copy”
6. Fill the data as follows:
   • Specify a meaningful name and description for the GP
   • Specify a customer-specific ABAP package with the right transport layer if you want to transport the guided procedure from Focused Run development system to QA and production system. The package is required to know in which transport route the GP will be released. If you don't want to transport the guided procedure, you can choose "Temporary Package".
   • Set "Productive" flag, if the guided procedure is to be transported to a production system. If the GP is for testing purposes on the local system, do not set the flag.
   • Keep Flag “Enforce Step sequence” unchecked.
7. Select tab “Search Attributes”.
8. Enter the search attributes as follows:
   • Enter the managed object type corresponding to the managed object type of the alert.
   • Enter the category corresponding to the category of the alert
   • Enter technical name of the alert into field “alert name”
   • Select “true” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert and executed automatically in background.  Select “false” for “Can be executed automatically in background” if the guided procedure shall be registered for a new alert but not executed automatically in background.
9. Select OK

How to ensure that SAP delivered Guided Procedure Templates are not triggered any more

If you have created a customer specific alert reaction procedure for an alert for which an  alert reaction  procedure  has been delivered by SAP, both guided procedures will be triggered. The desired behavior is mostly that only the customer specific procedure is triggered.  Proceed as follows to achieve this:

1. Open SAP Focused Run Launchpad
2. Select Advanced Event & Alert Management
3. Select Tile “Alert Management - Guided Procedure Catalog”
4. Select the SAP delivered  alert reaction procedure template which should no longer be triggered. 
5. Set the "Hidden" flag.

How to edit the content of customer - specific alert reaction procedures

Add a step

To add a step, click on New, and select the step type in the menu. You can also reuse an existing step. A new step will be inserted below the selected step, in the Steps tree. Step types can be standard, parent, plug-in or complete.

How to define a help text

The step help text provides detailed information about each step, such as what is checked, and what will happen in the background.

The "New" button in the help text section opens the Create New Document pop up, which allows you to define documentation for a step or an activity.

You can create a new HTML document or upload a PDF. If the HTML result report needs to show also the documentation in line, choose HTML

Add a complete step

The Complete step is the last step in the guided procedure. It provides an overview of the execution status of the previous steps. Additionally, the HTML result can be generated from the complete step. If your guided procedure does not have a complete step yet, add it.

Remove a step

Select the step and click on Remove to remove obsolete steps.

Add activity

Each step consists out of one or more activities. The following activity types are supported by Guided Procedure Framework.

For alert reaction procedures, we recommend to use only manual or Plug-in activities.

Add plug-in activity

Proceed as follows to include a new plug-in activity in your alert reaction procedure:

1. Select the relevant step.
2. Select Automatic Activity --> New
3. Choose "Select a Plug-in" and select the required plug-in from the value help.
4. Rename the activity
5. Maintain the plug-in attributes per your requirements
6. Select OK
7. While the new activity is selected press "Help text" --> New
8. Enter a name for the documentation
9. Enter a suitable documentation text. Press save.
10. Use the buttons "Move up or Move down " to change the order of activities in the step

Add Plug-in activities for alert reactions

The following plug-in activities are especially useful for alert reaction procedures

• Alert Handling: Check Current Rating of Alert

This Plug-in activity displays the status of the alert and the rating of its related metrics in a table.

It provides a context sensitive navigation link into the alert details for the alert.

It can be included at the beginning of the guided procedure to show the metrics at the time when the alert was raised and additionally at the end in the verification step to show whether the alert is resolved and related metrics have changed to green.

• Alert Handling: Check Related Alerts for Current System

This Plug-in activity displays all currently open alerts for the system, it's instances, hosts and database in a table. It provides a navigation link into the alert inbox. This plug-in supports alert correlation. For example: If the system is down and the system database is down at the same time you can conclude that database unavailability might be root cause for system unavailability.

• Alert Handling: Check SDA availability

This plug-in activity checks if the simple diagnostics agents related to the system or database in scope of the guided procedure are reachable from SAP Focused Run. It provides a context sensitive navigation link to the agent framework.

• Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of ABAP or JAVA systems. It executes SAP Control functions for the ABAP or JAVA system in the scope of the guided procedure and displays the result in the table. The following sapcontrol functions are supported:

• Alert Handling: Execute Safe SAP Control Operation

This plug-in activity can be used to check the status of databases. It executes SAP host control operations for databases. If the guided procedure is executed for a technical system, it executes the operations for the system database of the technical system. If the guided procedure is executed in the scope of a database it executes the operations for the database. of the system in the scope of the guided procedure and displays the result in the table. The following SAP host control operations are supported:

• Alert Handling: Execute Custom Operation New

This plug-in activity triggers custom operations which are deployed on the SAP host agents of the managed systems host. Results of the custom operation are displayed in the guided procedure HTML result report.

An operation is, in SAPHostAgent, a set of command, executed in the configured order until the first failure happens, or the complete set was successfully executed. The result of every single command is returned in the result of the operation itself. The configuration of the operations is expected in the sub-directory operations.d of the exe directory where SAPHostAgent was installed (generally on UNIX /usr/sap/hostctl/exe, and on windows C:\Program Files\SAP\hostctrl\exe).

It is intended that custom operations are defined by the customer and distributed to the system landscape by customer specific deployment means. SAP Focused Run does not deliver custom operations for Alert reactions or alert reaction and does not provide a deployment and distribution mechanism for custom operations.

  How to define and test custom operations

Example:

You would like to define a custom operation that checks if the database is accessible from an ABAP instance by executing R3trans -d under the sidadm user . Required input parameter is the system identifier. The custom operation shall work for Unix systems

• Log on to your test server where an ABAP instance is installed.
• Navigate to /usr/sap/hostctrl/exe/operations.d
• Create a file with the extension .conf  (e.g. GP_Check_R3trans.conf ) and the following contents:
  • Command: R3trans -d
  • Username:$[SID:#required#tolower]adm
  • Name: GP_Check_R3trans
  • Description: DB connectivity test via R3trans.
  • ResultConverter: flat
  • Platform: Unix
• Test the custom operation as follows:
• Navigate to /usr/sap/hostctrl/exe. Execute ./saphostcontrol -function ExecuteOperation -name GP_Check_R3trans SID=<3 character identifier of the ABAP System>

   How to configure the custom operation plug-in

1. Select Automatic Activities – New
2. Select Plugin “Alert Handling: Execute Custom Operation New”
3. Rename the activity
4. Enter the operation name (without .conf file extension) at the end.
5. Enter the arguments required by the custom operation. If several arguments are required, separate them by “;” without blanks). Some arguments can be replaced at runtime:
   • ${SID} will be replaced by 3 letter short SID of system
   • ${DBID} will be replaced by 3 letter id of DB
   • ${INSNR} will be replaced by instance number of ABAP, JAVA System or HANA DB
   • Example: the custom operation expects an input parameter “SID” with the system short SID.  Enter SID=${SID}
6. Select the execution scope to define on which hosts of the managed system or database the custom operation shall be executed
   • All hosts Custom operation will be executed on all hosts agents   of technical system and on host agent of system database
   • Technical System: Custom operation will be executed on all hosts agents   of technical system but not on host agent of system database (if system database is on separate host)
   • HANA Master Node: Custom operation will be executed on the host agent where the HANA Master node of the database is installed
   • Database hosts: Custom operation will be executed on all hosts agents   of the database but not on host agents of the technical system
   • Central host: Custom operation will be executed on the host agent where the ABAP or JAVA Central Server is installed
7. Enter timeout until how long the activity will wait to retrieve the result in seconds (recommended: 20)
8. Set Wait for result:
   • Yes: Custom operation will wait for result and log a success message whether custom operation has been executed successfully and display the result in the GP HTML result report
   • No: fire and forget
9. Define helper class (optional)
   In some cases it is required to do some evaluation if the result was ok or not and to display the result of the custom operation in a better readable format. This can be done by a helper class. This is a customer specific ABAP class that needs to be implemented using the interface “IF_CUST_OPERATION_HELPER_IMPL”. It must be developed especially for the defined custom operation. If you have developed such a helper class, enter it here. Otherwise leave the field blank.

Result:

When the guided procedure is executed manually or automatically, the custom operation will be triggered on the specified hosts agents. If wait for result=yes, the returning data will be displayed. If the custom operation has a helper class, the result can be semantically evaluated and a corresponding log message will be written.

Examples:

1. Configure an activity to automatically check connectivity from ABAP central instance to database via R3trans-d. Prerequisite is that the custom operation “GP_Check_R3trans” has been defined and deployed to the host agents
   Select Automatic Activities – New
   • Select Plugin “Alert Handling: Execute Custom Operation New”
   • Rename the activity to “Check DB connectivity”
   • Enter the operation name GP_Check_R3trans.
   • Enter the argument SID=${SID}
   • Select the execution scope to Central host:
   • Enter timeout 20
   • Set Wait for result to yes
2. Configure an activity to automatically check connectivity to host agents via operation SAPHostAgent.Ping. The operation SAPHostAgent.Ping is available in host agents by default.
   Select Plugin “Alert Handling: Execute Custom Operation New”
   • Rename the activity to “Check Host Agent Availability”
   • Enter the operation name SAPHostAgent.Ping.
   • Leave arguments empty.
   • Select the execution scope to All Hosts
   • Enter timeout 20
   • Set Wait for result to yes

Add Manual Activity

1. Select Manual activities – new
2. Enter a name for the activity
3. If required, select New to define a navigation link.
4. Select ok
5. Select help next new and define an appropriate instruction text

Manual activities to analyze log and trace files
Very often it is required to analyze critical errors in log and trace files. To support the analysis of relevant log and trace files without manual log on to the managed system or operating system level SAP Focused Run provides the following:

• The file system browser that allows to browse the file system of managed hosts and display or download files.
• Expert navigation links in the guided procedure provide the possibility to open the file system browser context sensitively in the relevant log or trace file directory of the hosts of the managed system or database

The following expert navigation link classes are available:

• CL_GPCR_FSB_ABAP_JAVA_WORKDIR
  provides navigation links to open file system browser in the work directory for all instances of an ABAP or JAVA system
• CL_GPCR_FSB_JAVA_SERVERLOGDIR
  provides navigation links to open File System Browser in the server log  directory for all servers of a JAVA  system
• CL_GPCR_FSB_BOBJLOGDIR
  provides navigation links to open File System Browser in the logging directory for all servers of a BOBJ system
• CL_GPCR_FSB_BOBJSERVERPIDDIR
  provides navigation links to open File System Browser in the serverpids directory for all servers of a BOBJ system
• CL_GPCR_FSB_HANA_TRACEDIR
  provides navigation links to open File System Browser in the trace  directory for all instances of a HANA Database

You can define a manual activity to check relevant log and trace files of the managed system as follows:

1. Select Manual activities – new
2. Enter a name for the activity
3. Select New to define a navigation link.
4. Select Expert
5. Choose the expert link class according your requirements (see above)
6. Select ok

Remove Activity

Select the relevant activity and press button "Remove" to remove the relevant activity if it is obsolete.

Change the contents and appearance of the alert reaction result report

1. Select button "Report Configuration".
2. Set flag "Include help documentation" if the alert reaction report should also contain step and activity documentation. If not keep this flag disabled.
3. Some activities provide context sensitive navigation link into the relevant application in FRUN or the managed system. With the flag "Include Navigation Link" you can decide whether the navigation links should be included in the HTML report or not.
4. Choose "CSS class name"CL_GPCR_CSS_STYLE_MIME_REP" to choose a more suitable style sheet for the HTML report than the standard one. Alternatively, you could define you own style sheet class for the html report to adapt it to your needs.
5. Select include navigation links if it the report should include the activity specific navigation links
6. Select include log messages and select all and all messages
   Select Ok
7. Select Save

Activate the customer specific Alert reaction guided procedure

1. Select button "Check". If the check reports inconsistencies, fix them.
2. Select button. "Activate"

Test the customer specific Alert reaction guided procedure

1. Try to generate the alert which is mapped to the guided procedure for the relevant system , instance or database.
2. Open Alert Management. Search for the relevant alert.
3. If the outbound adapter “GP alert handling” is activated, the guided procedure should have been triggered automatically in background and should be visible in the alert details in the guided procedure section. If not:  Select actions à search guided procedure. Check that you find the alert reaction procedure under “Filtered guided procedures”. Select Execute in background or Execute Manually.
4. Click on the report link to review the report
5. Click on the GP link and process the guided procedure

If the alert reaction guided procedure needs to be transported:

1. Press button "Transport".
2. Include the guided procedure into a new transport request or choose an existing.
3. Release and import the requests per your transport rules.