Т.с. ещё один шанс ускорить загрузку...
Скопировал на всякий случай, а то у SAP они бывают.
Shared Memory in BW Transformation
created by Karthick Pandi on Dec 5, 2013 2:46 PM, last modified by Karthick Pandi on Dec 5, 2013 3:20 PM
Version 1
After I read this blog on Shared Memory , I got curious and gave a try, if this could be used in BW transformation routines.
Shared Memory in BW Transformation
Very often in a BW data load process, we may need to hit the DB tables for lookup in the start/end/expert routine. So here the same set of DB tables is hit as many times as the number of data packets in a data load. Apart from hitting the DB table several times, the same data is stored in internal tables across several instances of the transformation class (Data packet).This results in reduced performance by hitting the persistent layer andconsumes more application layer memory by storing the same data in multiple instances.
We ever wonder, if this could be avoided by reading the persistent table, while first data packet loads and storing the data in internal tables defined in the global part of the transformation (CLASS-DATA),so that the data is visible other data packets as well. But this would work only if our DTP processing mode is ‘Serially in background processes’.
So, here comes the concept ‘Shared Memory’, by which the data can be stored in SAP memory once and could be read in the transformation routines for lookup.
What is Shared Memory?
It is the concept of storing the data in SAP memory, for rapid data retrieval without the need to physically read the persistent database tables. It also allows us to apply certain business logics to the data, before it is stored. As the name implies, the data can be used across users or sessions. This makes this technique more flexible than the session memory concepts.
Illustration
Let’s take a simple example: A transformation, where in the end routine, vendor master table is read repeatedly (for every data packet) from the database. In order to reduce the DB access, we decide to make use of shared memory objects, to store the data in SAP memory, instead of repeatly executing the ‘select statement’.
End routine code:
For our illustration, the below end routines code will be fine tuned to access the shared memory.
*************************************************************************************************************
TYPES: BEGIN OF ty_vendor,
vendor TYPE /bi0/oivendor,
accnt_grpv TYPE /bi0/oiaccnt_grpv,
addr_numbr TYPE /bi0/oiaddr_numbr,
altitude TYPE /bi0/oialtitude,
bpartner TYPE /bi0/oibpartner,
city TYPE /bi0/oicity,
city_2 TYPE /bi0/oicity_2,
country TYPE /bi0/oicountry,
dbduns_num TYPE /bi0/oidbduns_num,
fax_num TYPE /bi0/oifax_num,
id_txnumb3 TYPE /bi0/oiid_txnumb3,
id_xcpd TYPE /bi0/oiid_xcpd,
industry TYPE /bi0/oiindustry,
langu TYPE langu,
latitude TYPE /bi0/oilatitude,
logsys TYPE rsdlogsys,
longitude TYPE /bi0/oilongitude,
END OF ty_vendor.
DATA:wa_vendor TYPE ty_vendor,
it_vendor LIKE TABLE OF wa_vendor.
FIELD-SYMBOLS: TYPE ty_vendor.
SELECT
vendor
accnt_grpv
addr_numbr
altitude
bpartner
city
city_2
country
dbduns_num
fax_num
id_txnumb3
id_xcpd
industry
langu
latitude
logsys
longitude
FROM /bi0/mvendor
INTO TABLE it_vendor
WHERE objvers = 'A' AND
country = 'CA'.
IF sy-subrc EQ '0'.
SORT it_vendor BY vendor.
ENDIF.
LOOP AT RESULT_PACKAGE ASSIGNING .
READ TABLE it_vendor
ASSIGNING
WITH KEY
vendor = -vendor
BINARY SEARCH.
IF sy-subrc EQ '0'.
ENDIF.
UNASSIGN .
ENDLOOP.
UNASSIGN .
*************************************************************************************************************
Steps for Defining Shared Memory Objects
1.Create an ordinary class using transaction SE24.Define attributes (the data going to be held in the shared memory) and method for accessing and work with these attributes, in the end routine. This class is called the ‘root class’, since this will be used in step, while defining the memory area. More importantly, the class should be defined as ‘Shared Memory-Enabled’.
1. 2.Allocate a new memory are for our shared object. This could be done from transaction SHMA.
The next section will give us more information on how to do the above steps.
Step 1: Root Class
As mentioned earlier, the concept of shared memory is based on memory areas, each related to a global area root class. The root class contains attributes and methods needed to be used in the end routine. The root class will need attribute to hold data you want store (in our example internal table storing the vendor data), as well as methods to manipulate the data. In our case, we will have methods to initialize the table with specific data from the persistent layer and to retrieve data from it.
Defining Attributes of Root Class
First, we need to define data to be stored in our memory object. We need vendor data from vendor master data. So we simply create a table attribute as below.
I already defined a suitable table type ‘IT_VENDOR’ as below,
Defining Methods of Root Class
Now we need atleast two methods in our root class, one for initializing the attribute ‘X_IT_VENDOR’ with the data from the underlying DB table(Vendor Master data) and another method for retrieving this data from the memory object in the end routine.
One more method (IF_SHM_BUILD_INSTANCE~BUILD), which is highlighted below is for ‘automatic pre-load’ of memory object, which we can discuss in further sections. But to make this method available in the class, an interface “IF_SHM_BUILD_INSTANCE” needs to be included in the class.
Methods SET_VENDOR
The SET_VENDOR method is used to retrieve vendor data from the DB layer(Vendor master data table).When our shared memory object is created for the first time, it will be empty, meaning we need to invoke this method to populate X_IT_VENDOR attribute.
In order to avoid reading the complete vendor master data into the shared memory, we will limit ourselves to specific country using the method attribute I_COUNTRY.
METHOD set_vendor.
SELECT
vendor
accnt_grpv
addr_numbr
altitude
bpartner
city
city_2
country
dbduns_num
fax_num
id_txnumb3
id_xcpd
industry
langu
latitude
logsys
longitude
FROM /bi0/mvendor
INTO TABLE x_it_vendor
WHERE objvers = 'A' AND
country = i_country.
ENDMETHOD.
Methods GET_VENDOR
This method is used to retrieve data from X_IT_VENDOR attribute table of the object. This is the method we call from our end routine, instead of selecting records from the DB layer.
METHOD get_vendor.
ex_it_vendor = me->x_it_vendor.
ENDMETHOD.
ex_it_vendor is the exporting parameter of this method
Methods IF_SHM_BUILD_INSTANCE~BUILD
This method is for automatic pre-loading of our memory area. This method ensures that the memory area which is not yet initialized automatically started when an application tries to access it. This means that if we try to read vendor data from the shared memory, the memory area will be initialized on the fly, even if it is not initialized before. So there is no need of handling the exception ‘CX_SHM_NO_ACTIVE_VERSION’ in our end routine.
METHOD if_shm_build_instance~build.
DATA:lcl_vendor_root TYPE REF TO zcl_vend_shared_obj,
lcl_vendor TYPE REF TO zcl_tst_vendor.
lcl_vendor = zcl_tst_vendor=>attach_for_write( ).
CREATE OBJECT lcl_vendor_root AREA HANDLE lcl_vendor.
lcl_vendor->set_root( lcl_vendor_root).
lcl_vendor_root->set_vendor(
EXPORTING
i_country = 'CA'
).
lcl_vendor->detach_commit( ).
ENDMETHOD.
The above code initializes the memory object with some vendor data. This is done by invoking the attach_for_write (lock the memory area for writing) of the memory area, followed by the instantiation of the root class. When this is done, the SET_VENDOR method of the memory object is called in order to fill the attribute table X_IT_VENDOR.
Step 2: Creating the Memory Area
We need to create a shared memory area for our memory object; we can do this from transaction SHMA
Here as highlighted above, we assigned our new class ‘ZCL_VEND_SHARED_OBJ’ as the root class and constructor class to the memory area. Also we have checked here ‘Automatic Area Creation’ to make sure that the memory object is automatically created whenever the invoked from end routine-if it does not exist already. We can keep rest of the properties as defaulted here (Please refer to SAP documentation on Shared memory for more details).
Once the memory area is activated, a corresponding class is generated automatically (It has the same name as memory area).This class provides methods for establishing a handle for memory area.
Step 3: Using shared memory objects in end routine of transformation
Using memory object in any program normally follows the following steps,
· Create a reference to the shared memory area class
· Call one of the ATTACH methods of the area class
· Call any of the data manipulation methods of the area’s root class
· Call a DETACH method to “free” the area handle
The ATTACH methods are used for requesting locks on the memory area. This is a prerequisite for accessing the related memory area. We normally use ‘attach_for_read’,’attach_for_update’ methods. ‘attach_for_write’ method is usually called only during the creation of memory object (as seen in the code for ‘IF_SHM_BUILD_INSTANCE~BUILD’ in the previous section).
The end routine code is now modified by replacing the select statement with the highlighted in red below.
TYPES: BEGIN OF ty_vendor,
vendor TYPE /bi0/oivendor,
accnt_grpv TYPE /bi0/oiaccnt_grpv,
addr_numbr TYPE /bi0/oiaddr_numbr,
altitude TYPE /bi0/oialtitude,
bpartner TYPE /bi0/oibpartner,
city TYPE /bi0/oicity,
city_2 TYPE /bi0/oicity_2,
country TYPE /bi0/oicountry,
dbduns_num TYPE /bi0/oidbduns_num,
fax_num TYPE /bi0/oifax_num,
id_txnumb3 TYPE /bi0/oiid_txnumb3,
id_xcpd TYPE /bi0/oiid_xcpd,
industry TYPE /bi0/oiindustry,
langu TYPE langu,
latitude TYPE /bi0/oilatitude,
logsys TYPE rsdlogsys,
longitude TYPE /bi0/oilongitude,
END OF ty_vendor.
DATA:wa_vendor TYPE ty_vendor,
it_vendor LIKE TABLE OF wa_vendor.
FIELD-SYMBOLS: TYPE ty_vendor.
DATA:lcl_vendor_shmo TYPE REF TO zcl_tst_vendor,
lv_exception TYPE REF TO cx_root,
lv_text TYPE string.
TRY.
lcl_vendor_shmo = zcl_tst_vendor=>attach_for_read ( ).
CATCH cx_shm_no_active_version
cx_shm_read_lock_active
cx_shm_change_lock_active
cx_shm_exclusive_lock_active
cx_shm_inconsistent
INTO lv_exception.
ENDTRY.
IF NOT lv_exception IS INITIAL.
CALL METHOD lv_exception->if_message~get_text
RECEIVING
result = lv_text.
WRITE lv_text.
ELSE.
IF lcl_vendor_shmo IS NOT INITIAL.
CALL METHOD lcl_vendor_shmo->root->get_vendor(
IMPORTING
ex_it_vendor = it_vendor
).
ENDIF.
SORT it_vendor BY vendor.
ENDIF.
LOOP AT RESULT_PACKAGE ASSIGNING .
READ TABLE it_vendor
ASSIGNING
WITH KEY
vendor = -vendor
BINARY SEARCH.
IF sy-subrc EQ '0'.
ENDIF.
UNASSIGN .
ENDLOOP.
UNASSIGN .
lcl_vendor_shmo->detach ( ).
The ‘detach’ method at the end, release the lock at the end of the routine on the memory object, imposed by the ‘attach_for_read’ method.
Step 4: Examining the Shared Memory Object
The shared memory objected created can be checked from transaction SHHM.
The shared memory object is now in the SAP memory and it will be there until the SAP instance goes down. It could be deleted by selecting it and using the delete button.
In our case we can delete the shared memory object at the end of the DTP load using the method ‘FREE_INSTANCE’of the class ‘ZCL_TST_VENDOR’.It will be created again, when the DTP run for the next time.
Vendor table data stored in shared memory:
Conclusion:
I just did a test runs of the DTP with the end routine code included with ‘SELECT STATEMENT’ and ‘SHARED MEMORY’ separately. For 202 records, the first one took 19s and where as the later one took 12s.
SELECT from DB table:
READ from Shared Memory:
The shared memory concept in BI transformation, apart from reducing the DB load and ‘application memory’ required to run a DTP, also lead to performance improvement of the DTP load. This could be more helpful in transformations, where a number DB tables are used for look up and in DTP loads involving more number of data packets.
I will keep explore this option of using shared memory and keep you guys posted.
2 комментария:
Все бы хорошо, но как себя поведет Shared Memory когда идет запуск на разных серверах приложений, не стоит забывать о синхронизации данных между двумя, тремя и более серверах приложений.
В трансформациях зачастую достаточно использовать кеширование на уровне статических данных в классе трансформации. В каких случаях использование кеширование в shared memory предпочтительно?
Проблема в том, что если грузить данные паралельно, то статические данные в классе живут в рамках потока. Попробуйте, попишите в логи, состояние таблицы в паралельной загрузке.
Отправить комментарий