How to Enhance CCDB with Content from OS Level Scripts (EventStore)

This document describes how to enhance the data collection of the CSA by creating a custom table-based EventStore using an OS script as datasource. In contrary to snapshot-based Stores an EventStore uses a delta mechanism allowing to upload events in a table-based format. Such an EventStore consists of multiple table columns: One event timestamp with precision of seconds, n key columns and m data columns.

The operation system command interface DAGT_HA_EXE of the host agent must be used like for regular OS scripts. Because of this, please refer to 'Enhance with OS Level Scripts' to see how to create a Store Template and an OS script.


OS Script

Script Configuration File

File Content: /usr/sap/hostctrl/exe/operations.d/<SCRIPT_NAME>.conf 
Command:/usr/sap/hostctrl/exe/operations.d/custom_operations/CCDB/<SCRIPT_NAME>.sh $[FROM_TIMESTAMP:#required] $[TO_TIMESTAMP:#required] $[MAX_RECORDS:#required]

Platform: Unix

ResultConverter: flat

The script configuration file contains three parameters

- FROM_TIMESTAMP (Input format: YYYYMMDDHHMMSS)
- TO_TIMESTAMP (Input format: YYYYMMDDHHMMSS)
- MAX_RECORDS (Input format: Number)

Script Output Format

For EventStores, there is a special output format based on JSON that ends with the exitcode of the script. The JSON format consists of a table array containing the individual columns and time range control information.

The element FIELDLIST defines the column structure of an event delivered by the element DATA. The timestamp of an event must be called EVENT_TS and has the FIELDROLE = 'T'. The UTC timestamp value must follow the format YYYYMMDDHHMMSS. In addition, there can be n key columns and m data columns. All column names in FIELDLIST and DATA must be upper case and consist of the characters A-Z, 0-9 and the underscore. The maximum length of a fieldname is 30.

A column value must not exceed 1024 characters.

{


    "RESULT": {


        "FIELDLIST": [{


                "FIELDNAME": "EVENT_TS",


                "FIELDROLE": "T"


            }, {


                "FIELDNAME": "KEY1",


                "FIELDROLE": "K"


            }{


                "FIELDNAME": "DATA1",


                "FIELDROLE": "D"


            }{


                "FIELDNAME": "DATA2",


                "FIELDROLE": "D"


            }


        ],


        "DATA": [{


                "EVENT_TS": "20230111110039",


                "KEY1": "189e51cd-9682-4e11-af22-965205e9d16c",


                "DATA1": "value of data1 row1",


                "DATA2": "value of data2 row1"


            }, {


                "EVENT_TS": "20230111110039",


                "KEY1": "e671571f-3e9f-41d1-abc6-5af4edb46bc5",


                "DATA1": "value of data1 row2",


                "DATA2": "value of data2 row2"


            }, {


                "EVENT_TS": "20230111110039",


                "KEY1": "37159e74-090f-4e90-89d3-a5d43ae322f0",


                "DATA1": "value of data1 row3",


                "DATA2": "value of data2 row3"


            }, {


                "EVENT_TS": "20230111110045",


                "KEY1": "e0bdd5e7-00ef-438b-b952-acd136e35367",


                "DATA1": "value of data1 row4",


                "DATA2": "value of data2 row4"


            }, {


                "EVENT_TS": "20230111110057",


                "KEY1": "8e42494c-18da-4ae8-bb48-66786233e44a",


                "DATA1": "value of data1 row5",


                "DATA2": "value of data2 row5"


            }, {


                "EVENT_TS": "20230111110057",


                "KEY1": "29996272-05be-4b3a-bd68-eadad0b39f54",


                "DATA1": "value of data1 row6",


                "DATA2": "value of data2 row6"


            }


        ]


    },


    "REQUEST_TIMERANGE_RESULT": {


        "MAX_RECORDS": "1000",


        "INITIAL_MODE": "",


        "FROM_TIMESTAMP": "20230111110000",


        "TO_TIMESTAMP": "20230111110100",


        "EXTRACTED_DATA_FROM": "20230111110000",


        "EXTRACTED_DATA_TO": "20230111110057",


        "MORE_DATA_AVAILABLE": ""


    }


}

exitcode=0

Timerange Control

A script has the input parameter FROM_TIMESTAMP, TO_TIMESTAMP and MAX_RECORDS. Via the element REQUEST_TIMERANGE_RESULT it must specify which events have been delivered:

MAX_RECORDS: Must return the value if the input parameter

MAX_RECORDS
INITIAL_MODE: Unused yet. Always initial.
FROM_TIMESTAMP: Input value of FROM_TIMESTAMP
TO_TIMESTAMP: Input value of TO_TIMESTAMP
EXTRACTED_DATA_FROM: Always equal to FROM_TIMESTAMP
EXTRACTED_DATA_TO: It is equal to TO_TIMESTAMP if all events of the requested time range could be delivered by the script. If not all records could be delivered - either because f the restriction MAX_RECORDS or because of technical or semantic reasons - the value of EXTRACTED_DATA_TO defines the end of the delivered time range. This is either the EVENT_TS of the last event or can be even later.
MORE_DATA_AVAILABLE: In case not all events could be delivered you can set this value to 'X'. Up to know this has only informative character on the FRUN side.
All timestamp must follow the format YYYYMMDDHHMMSS.

Error handling

The script must deliver the exitcode as the last line of the output. exitcode=0 represents a successful operation and all others are treated as an error which is displayed in the CSA Administration at the corresponding collector item.

It is recommended deliver an error return code in case a problem is occurring during script execution.

Testing the script

A script can be tested using the SAP HostAgent operation called 'ExecuteOperation':

/usr/sap/hostctrl/exe/saphostctrl -function ExecuteOperation -name <SCRIPT_NAME> FROM_TIMESTAMP=<YYMMDDHHMMSS> TO_TIMESTAMP=<YYMMDDHHMMSS> MAX_RECORDS=1000

Template Definition

Depending on the requirements a script can either be assigned to a host or to a technical system that is most often the case. Here, you must make sure that the script is available on all hosts of the technical system. This is required because it is not possible to specify which host shall be used.

Template Definition (Example) of a script assigned to a ABAP technical system

The following values must be adapted:
SCI_ID, START_COND (HOURLY_MM or DAILY_HH is recommended), DDIC_TABLENAME, GDATA_TYPE, STORE_NAME, Column Parameters, MAX_RECORDS and name (SCRIPT_NAME).

<?xml version="1.0" encoding="utf-8"?>


<StoreCollectItems VERSION="">


  <StoreCollectItem APPL_ID="CCDB" MODIFIED="20240731123330" OBJVERS="A" SCI_DESC="Script Event Demo" SCI_ID="Z00169">


    <Variant VARIANT_ID="01">


      <Condition>


        <SCVS/>


      </Condition>


      <CF>


        <EC COLL_DEST_ID="COLL_DAGT_00" EC_OB_CIM_CLASS="*" EC_OB_CIM_NTYP="TECHNICAL_SYSTEM" EC_OB_DEST_ID="DAGT_HA_EXE" EC_OB_PROP="" EC_RT_CIM_CLASS="*" EC_RT_SYSTEM_TYPE="ABAP" EC_RT_TYPE="TS"/>


        <EX EXTR_NAME="" EXTR_TYPE="TIMERANGE" PRIORITY="5" START_COND="HOURLY_MM" START_COND_EXT="" START_IMMEDIATE="X" TRANSFER_TYPE="IQ"/>


        <EXTR_XDATA_INPUT>


          <Supplier SUPPL_CONV="" SUPPL_INTF="" SUPPL_NAME="" SUPPL_PROTECT="N" SUPPL_TYPE="">


            <Store CARDINALITY="1" DDIC_COL_RENAME="" DDIC_TABLENAME="CCDB_DATA_503" GDATA_TYPE="ZGDATA_503_18" STORE_DEF_ID="01" STORE_NAME="OS_SCRIPT_TEST" STORE_NAME_EXT="" STORE_TYPE="EVENT">


              <Parameter NAME="CCDB_Generic_Column-01" VALUE="K-KEY1-GCOL_11"/>


              <Parameter NAME="CCDB_Generic_Column-02" VALUE="D-DATA1-GCOL_19"/>


              <Parameter NAME="CCDB_Generic_Column-03" VALUE="D-DATA2-GCOL_20"/>


              <Parameter NAME="MAX_RECORDS" VALUE="300"/>


              <Parameter NAME="name" VALUE="SCRIPT_NAME"/>


            </Store>


          </Supplier>


        </EXTR_XDATA_INPUT>


      </CF>


    </Variant>


  </StoreCollectItem>


</StoreCollectItems>

After the creation of the template item definition it can take one hour or longer until the changed template is used. For testing, you can alternatively select a technical system in the CSA Administration and use button 'Setup'.

There are various situation where the delta mechanism requires a synchronization. This is automatically performed and may take one hour. During this time synchronization warnings are displayed for this item that can be ignored.