Loading Knowledge Modules (LKM)

1. Definition of LKM:
• LKM (Loading Knowledge Module) is responsible for moving data from a source server to a target server (which could be a staging area or the final target).
• It's used in mappings when source datastores are not on the same server as the target or staging area.
2. Purpose:
• The LKM helps execute logic that needs to run on the source server.
• It allows transferring data from the source server to the target server, even if they are on separate machines.
3. Types of LKM:
• PERSISTENT:
• Retrieves a result set from the source server and loads it into a “C$” staging table.
• The data is physically stored in the target server for further processing.
• TRANSPARENT_SOURCE:
• Sets up a transparent access mechanism that allows the target server to access data from the source server directly, without physically transferring it.
• TRANSPARENT_TARGET:
• Sets up a transparent access mechanism to allow the source server to directly access and load data into the target datastores.
4. Execution Flow:
• Depending on the LKM type, the process may involve either physical data transfer (PERSISTENT) or transparent access (TRANSPARENT_SOURCE and TRANSPARENT_TARGET).
5. Usage:
• LKMs are typically used in mappings for data extraction, transformation, and loading, where source and target servers are different or when efficient data movement strategies are required.

 

Notes on LKM Process in Steps

1. Creation of C$ Temporary Table:
• The LKM creates a C$ temporary table in the staging area.
• This table will serve as a holding area for the records that are loaded from the source server.
2. Obtaining Pre-Transformed Records:
• The LKM retrieves a set of pre-transformed records from the source server.
• For SQL-based LKMs (e.g., when the source is an RDBMS), this is done by executing a SQL SELECT query to fetch the data.
• For sources that do not support SQL (such as flat files or non-database applications), the LKM uses the appropriate method to read the data, such as file reading or API execution.
3. Loading Records into the C$ Table:
• Once the pre-transformed records are retrieved, the LKM loads these records into the C$ staging table in the target server (the staging area).

 

Notes on LKM Process for TRANSPARENT_SOURCE in Steps

1. Create Transparent Access Mechanism:
• The LKM of type TRANSPARENT_SOURCE sets up a transparent access mechanism that allows the target server to access the data on the source server directly.
• This eliminates the need to physically move the data from the source to the target server, providing a more efficient way to access the data.
2. Generate Code Generation Metadata Object:
• The LKM creates a code generation metadata object. This object contains important information about the source data and the mapping logic used to access and transform that data.
3. Pass Metadata Object to Target IKM:
• The code generation metadata object is passed to the target IKM (Integration Knowledge Module), which uses this information to further process the data and load it into the final target datastores.

 

Notes on LKM Process for TRANSPARENT_TARGET in Steps

1. Create Transparent Access Mechanism:
• The TRANSPARENT_TARGET LKM creates a transparent access mechanism that allows the source server to directly access the target datastores. This mechanism enables data to be loaded into the target datastore without moving it physically between servers.
2. Obtain Pre-Transformed Records:
• The LKM retrieves a set of pre-transformed records from the source server. It executes the necessary transformations on the source data before accessing it, ensuring the data is ready for loading into the target.
3. Load Data into Target Datastore:
• Using the transparent access mechanism created in Step 1, the LKM loads the data directly into the target datastore. The data is transferred directly from the source to the target, without the need for staging.
4. Multiple LKMs for Different Sources:
• When a mapping involves data from multiple source datastores, the mapping may require several LKMs to handle the different sources, especially if they are on different data servers.
5. No LKM Required for Same Data Server:
• If all the source datastores are on the same data server as the staging area, no LKM is necessary, as the target server can directly access the source data without needing a separate loading mechanism.