Extractor Framework Setup and Administration

Extractor Framework Setup

With SAP Solution Manager 7.1 only two extractor setup types (SOLMAN and RCA) existed. All extractors were scheduled during Basic Configuration or the Managed System Configuration.

With SAP Solution Manager 7.2 we have currently 24 different setup types. The extractors are scheduled during the setup of their different use cases. The setup and the scheduling of the extractors is mostly automated. Though there are some important steps during the Basic Configuration and in Managed System Configuration that influences the settings of the scheduled extractors.

The first setup step that influences the extractor setup is in the step “Specify SAP BW System” in the Basic Configuration. The extractors write data to the BW system in SAP Solution Manager, so the decision which BW configuration you want to use in your SAP Solution Manager, influences this setup. If you change the BW system setup you will have to make sure the extractors are adjusted accordingly.


As explained earlier, the Resource Manager is responsible for scheduling extractors when they are due for running. The resource manager is a batch job which is scheduled on the step “Configure Automatically” in the activity “Schedule Sol. Manager Background Jobs”


The following jobs relevant for the Extractor Framework (EFWK) are scheduled:

  • Resource Manager (E2E_EFWK_RESOURCE_MGR) runs every minute
  • EFWK Housekeeper (E2E_EFWK_HOUSEKEEPER) runs daily
  • BI Housekeeper (E2E_HK_CONTROLLER) runs daily

In the activity “Setup Extractor Framework” the extractors of setup type SOLMAN are scheduled. This includes extractors for CCDB and Change Analysis.


During the managed system setup the extractors for root cause analysis (setup type RCA) are scheduled for the managed systems.


The extractors for the other setup types are scheduled in the actual scenario setup. This is not always done in a separate step or activity, but sometimes in the background during the activation of the functionality.

Extractor Setup Types

The following table contains the currently used setup types for the extractors.

Use Case / Setup Type Description Support Component
RCA Root Cause Analysis (Mainextractor   <> E2E*SSR*) SV-SMG-DIA-WA / SV-SMG-DIA-EA
  Statistical Records (Mainextractor =   E2E*SSR*) BW-BCT-CMS

SOLMAN

Root Cause Analysis (SolMan   System only)

SV-SMG-DIA-WA / SV-SMG-DIA-EA

EEM

End User Experience Monitoring   (Technical Scenario)

SV-SMG-MON-EEM

EEM_SM

End User Experience Monitoring (SolMan System only)

SV-SMG-MON-EEM

MAI

Monitoring and Alerting relevant   Extractors

SV-SMG-MON-PRV

SPR

Service Provider Reporting

SV-SMG-SUP-REP

TWB

Test Workbench Reporting

SV-SMG-TWB

PIMF

PI Message Flow

SV-SMG-MON-PI

BPMON

Business Process Monitoring

SV-SMG-MON-BPM

E2EREP

Interactive Reporting

SV-SMG-MON-REP

ESREP

IT Management Reporting – BI Self   Service

SV-SMG-SER-ESR

ULDE

Usage and Landscape Data Exchange

SV-SMG-SER-ESR

SUGEN

SUGEN Data Exchange

SV-SMG-SER-ESR

CCDB_ALL

CCDB Technical System Extractor

SV-SMG-DIA-CA

CONN_MON

Connection Monitoring

SV-SMG-MON-CON

E2E_PA

Problem Analysis

SV-SMG-MON-ALR

DVM

Data Volume Management

SV-SMG-DVM

SMUG

SAP Solution Manager Usage Scenario

SV-SMG-UMP

MASSMAN

Setup for Benchmark Project (Massman   Data)

SV-SMG-DVM

CUP

Customer Usage Provision

SV-SMG-SER-ESR

EXC_MGMT

Exception Management

SV-SMG-DIA-APP-EM

TRP

Transport Request Management

SV-SMG-SUP-REP

DBMS

Database Setup

BC-DB-SDB-CCM

IC_MON

Interface Channel Monitoring

SV-SMG-MON-IC

JSM

Job Scheduling Management

SV-SMG-PSM

If you have issues with an extractor and want to know the setup type to choose the correct CSN component, open the EFWK Admin UI via transaction SM_WORKCENTER -> Solution Manager Administration -> Infrastructure -> Framework -> Extractor Framework.


Change the settings of the POWL table that contains the extractors, by clicking on “Settings” in the upper right corner of the table.


Make the column “Setup Type” visible.


Now you find the information on the extractor's setup type in the last column of the POWL table.

Administration User Interface

The Extractor Framework is administered using the new Administration UI. From here you can display the status of the extractors and start an error analysis or configuration of extractors.

To open the Admin UI call transaction SM_WORKCENTER and switch to the SAP Solution Manager Administration workcenter. Depending if you want to check extractors in general or for a specific technical system you have to choose either the Landscape view (system specific) or the Infrastructure view (general).


The first screen is the status overview screen for the extractor framework. Here you see status information for all registered extractors, except MAI extractors. MAI extractors are administered in the “Alerting Framework”.


Extractors have two status, the status of activation and the status of the last run.


An Extractor can become inconsistent, e.g. if the technical configuration of the system is changed. Banned means the extractor is not executed anymore, due to previous problems it was causing. An extractor gets banned if he causes 3 timeouts in the EFWK. A timeout can occur if the extractor runs too long and is blocking a resource (i.e. an RFC destination) for too long.

Please note that all data transported in the Extractor Framework is always in UTC time zone. The Required Record algorithm and the Logging are also in UTC. The extractor makes sure the data is delivered in UTC time zone. However, before the collected data is written to the BW, that data is converted into the time zone of SAP Solution Manager.

When you select an extractor in a table below you find the status of the last N runs (per default N would be 20). When you select a run you find details on the selected extractor run in the table below. Which phases an extractor has, depends on the extractor.

 In case of an error you will an error message in the column Message.


On the tab “Extractor Detail” you find further details on the extractor:


On the tab “Resource Manager” you find status information for the Resource Manager job.


Exceptions for the extractor framework that have been written to application log, can also be found on the tab “Exception”.

Configuration Options

There are certain settings than can be made to tune the EFWK. DO only change these settings if you know exactly what you are doing or when you are advised to do so by SAP.

Configuration changes are made via the tab "Configuration" of the EFWK Admin UI.


Changing the Resource Cap for RFC Resources

The EFWK is controlled by the Resource Manager and is subject to the so called resource cap. This resource cap makes sure that the Resource Manager doesn't open too many RFC destinations to a managed system and might overload the managed system. The resource cap applies to all extractors for a system, RCA as well as E2E Monitoring and Alerting and other use cases.

Usually a resource is blocked as long as the extractor is running. Additionally an enqueued lock is set per extractor to make sure it cannot be started several time at once in SAP Solution Manager. There is also a timeout in EFWK, after which a resource is released no matter if the extractor finished or not. If an extractor hits a timeout 3 times, it is banned from further execution.

Sometimes it can be necessary to adjust the resource cap. Per default it is set to 1. But if the managed system is very busy or huge and a lot of data has to be collected and if RCA and E2E Monitoring and Alerting is both active, it can be necessary to adjust the resource cap.

You can set resource caps for each READ RFC destination to a managed system and local RFC destinations. Also the number of ABAP dialog work processes that can be used by the EFWK is subject to the resource cap. To change the resource cap switch to the tab “Configuration” in the EFWK Admin UI and change resource cap for the desired RFC destination.


RFC resources are:

  • SOLMAN_DIALOG_WORKPROCESSES: Maximum number of Dialog work processes on SAP Solution Manager, which can be used by the Extractor Framework
  • SM_<Log Sys>_READ: Used for RFC calls from SAP Solution Manager to the managed systems ABAP stack
  • SOLMANDIAG: Used for RFC calls to Java-Stack of SAP Solution Manager via JCO (Introscope)
  • WEBADMIN: Used for RFC calls to Java-Stack of SAP Solution Manager via JCO (EEM)
  • NONE: Used for local RFC calls

Please note: If you adjust the number of SOLMANDIAG or WEBADMIN resources, be aware that you might also have to adjust the number of JCO listeners on the Java Stack of SAP Solution Manager. This can be done in the basic configuration of SAP Solution Manager.


Changing the Logon-Group for EFWK

You can also specify logon groups for the SAP Solution Manager system. The Resource Manager then only runs extractors in the instances in the logon group. On the tab “Configuration” below the RFC Management you find the Logon Group maintenance for the EFWK. Here you can create and set logon and server groups.


Changing the Number of WLI per EFWK RM Run

In the last section you can change how many work list items (basically the number of extractors) are processed within one EFWK Resource Manager run. You can change this number in case there is a huge backlog of extractors that have to be processed. Changing this value might prolongs the runtime of the EFWK Resource Manager job.


Since each Resource Manager enqueued is protected this will not lead to problems, but the new instance of the RM will not be started. You will find a message like this in the spool output of the RM job instance.

EFWK Housekeeping

The housekeeping in EFWK is performed by the job “E2E EFWK HOUSEKEEPING” on a daily basis. The job usually runs at 10pm.

The job performs the following activities:

  1. Landscape consistency check (Technical Systems, PPMS)
  2. EFWK internal consistency checks
    • BI Destination
    • SOURCE RFC Destination
    • Check against E2E_RESOURCES
  3. Release banned extractors
  4. Check for Duplicates and LUW inconsistencies
  5. Cleanup the tables E2E_EFWK_STATUS and E2E_EFWK_LOG
  6. Housekeeping for DPC (Data Provider Connector)
  7. Generate Self-Monitoring Alert

The details for the house keeping job can be found in the application log (SLG1) under the Object E2E_ALERTING and the sub object EFWK.

BI Housekeeping

The EFWK writes data in the BW of SAP Solution Manager. To make sure this data doesn't grow too much and too fast a BI housekeeping is implemented for the data from the EFWK.

Housekeeping for Standard BW Cubes

The housekeeping is done by the program E2E_HK_CONTROLLER. The program is scheduled in the daily job E2E BI HOUSEKEEPING.

Data in either aggregated and transferred from cubes with e.g. hourly resolution to cubes with daily resolution, or it is deleted, e.g. data from the higher volume cubes (H-Cubes) is deleted if it is older than the default of 91 days. SAP Note 1480588 describes the housekeeping process in detail.


The status of the housekeeping can be checked in the application log (transaction SLG1). To find the log information for the housekeeping, enter the value E2E_HK_CONTROLLER in the field External ID.


The spool output of the job E2E BI HOUSEKEEPING contains information on the aggregation and deletions that have taken place.

The settings for the housekeeping are maintained in the tables E2E_BI_AGGREGATE for the aggregation and the table E2E_BI_DELETE for the data deletion. It is not recommended to change these settings!

The default aggregation and deletion settings are:

  • Root Cause Analysis - The default values for aggregation are: Hourly data is stored for 90 days, daily data is stored forever. In addition, the daily data is finally aggregated to monthly values after one year for database related metrics.
  • Monitoring and Alerting - The default values for aggregation are: Minutely data is stored for 8 days and then aggregated to hourly data. Hourly data is stored for 31 days, daily data is stored for 100 days, weekly data is stored for 52 weeks and monthly data is stored for 47 month. This can be adjusted in the infrastructure configuration for technical monitoring.


Housekeeping for Twin Cubes

To ease the housekeeping process the twin cube concept was introduced with SAP Solution Manager 7.1.

In the twin cube concept the same cube exists two times. In the initial phase only cube 1 filled by the data loader. Once cube 1 is full, cube 2 is filled instead. After a certain time period the housekeeping kicks in and the data from cube 1 is dropped. The data loader changes its target and cube 1 is filled again until in the next housekeeping phase the data from cube 2 is dropped. This way of dropping the whole table instead of searching out single records by their age, speeds up the housekeeping process tremendously.


The following figure shows the data retention periods of the twin cubes: