Loading Using the Agent

1. Run-time Agent Setup:
• The run-time agent is responsible for reading data from the source server and writing it into the staging area. This is done using JDBC (Java Database Connectivity) to interact with both the source and the staging area.
2. JDBC SELECT Command on Source:
• The knowledge module (KM) must include a SELECT command on the source to retrieve the required data. This SELECT query defines what data should be fetched from the source.
3. JDBC INSERT Command on Target:
• Corresponding to the SELECT command, there must be an INSERT command to write the fetched data into the loading table in the staging area.
4. Row-by-Row Data Processing:
• The agent reads data row-by-row from the source using the array fetch feature, which retrieves the data in batches but processes each row individually.
• Similarly, the data is written row-by-row to the loading table in the staging area using the batch update feature, which allows the insertion of multiple rows at once but processes each row separately.
5. Limitations for Large Volumes of Data:
• This method is not suited for large volumes of data because it processes the data row-by-row and may not be efficient for large datasets.
• The array fetch and batch update features allow for some optimization, but handling large datasets can still be slow and resource-intensive.