As we all know, BI Apps is a package of tools with out of the box configuration catered to specific business area. It consists of pre-build Informatica ETL mappings, Dac Tasks and OBIEE data models & dashboards.
More often than none, we have to customized these pre-build objects in order to have it fit in to our specific project environments. These customization includes adding new fields across the ETL, changing the data models, addingincremental logic to non-incremental mapping and so on. Today, let’s talk about how to configure Incremental Loading logic in BI Apps environment.
There are many different ways to implement incremental loading in BI Apps environment, the way I am going to do is by passing parameters from DAC into Informatica for incremental loading. The logic of the mapping is more or less similar to the standard Informatica practice.
For beginners, let’s break down the process as follow”
1. Determine how many columns of the target table are unique columns that should be primary keys.
2. Filtering the source based on certain columns to determine what data are new data and what are old data. Usually, for each type II or type III Slowly Changed Dimension tables, “last_update_date” is almost the best column to filter on in source qualifier transformation.
3. Use lookup transformation to compare the unique columns from source to target and provide 1 output.
4. The value of the lookup output is either null or not-null as the result of comparing the source and target field. If everything in the source already exist in the target, then the lookup outcome is (depending on the lookup condition) null or not.
5. In the expression transformation, create conditions for null or non-null scenarios by assigning values such as “I” or “U”.
6. In the update strategy transformation, using DD_Insert, DD_update commends in update strategy expression for each of those values from expression transformation.
So let’s start with an easy example that there is 1 source and 1 target, which is PA_EVENT table. This table has 3 unique columns “Task_ID”, “PROJECT_ID” and “Event_Num”.
Before we start with the mapping, let’s make these 2 columns the primary key in target designer:
In Dac, make sure we have 2 global parameters, or source system parameters:
$$LAST_EXTRACT_DATE: This is a variable that the value is determined at run time. DAC_SOURCE_REFRESH_TIMESTAMP is the default run time variable that stores the history of timestamps of all the updates made in the source table in the database.
$$LAST_UPDATE_FILTER: This is a parameter that I have created for informatica to use in it’s source filter. This parameter’s value is static with the logic: LAST_UPDATE_DATE > TO_DATE(‘@DAC_$$LAST_EXTRACT_DATE’,’MM/DD/YYYY HH24:MI:SS’)
In other words, when I enter this parameter in the source qualifier, at run time the above column will be added as a where clause in the sql statement, filtering out all of the records that the last_update_date is less than last_extract_date from the source level.
So now let’s start implementing the incremental logic in Informatica.
As you can see, the beginning of the mapping is very simple and straight-forward. We are going from source to applying the filter in SQ transformation, now here we are.
So now, everything that comes before lookup transformation should be the records that it’s corresponding last update date is greater than the previous recorded timestamp, not all of the records stored in the source table.
Therefore, from these filtered records, we need to see whether we are going to insert them in the target or update them if it already exist in the target.
How do we know what to do? Well, we will look at the three unique columns “TASK_ID”, “PROJECT_ID”and “EVENT_NUM” and see if they match from both source and target. This is where lookup transformation comes into the picture:
From there as we can see, there are 6 columns in the lookup, 3 are from the source, the primary key columns. The other 3 are created to be the lookup columns, there I am calling them LOOKUP_INPUT10, 11 and 12. These columns in reality, will be referring to the target columns of the same. So I am really just going from source.Task_ID to target.Task_ID here. Although it may look fancy.
Now, we need to set the lookup conditions as how we want to compare these fields. We have the option of setting source.Task_ID equal target.Task_ID or great or less or whatever. This time, I am setting the condition to be equal:
Now at run time, informatica doesn’t know what lookup_input10 is TASK_ID. The value is going to come from DAC which we will get to later. Therefore, we need to specifically tell Lookup transformation about these things, therefore, we define lookup override in the property:
What I entered there is:
SELECT
$$LOOKUP_INPUT10 AS LOOKUP_INPUT10,
$$LOOKUP_INPUT11 AS LOOKUP_INPUT11,
$$LOOKUP_INPUT12 AS LOOKUP_INPUT12
FROM
$$TARGET_TABLE —
Each of the “$$” represents a parameter name that we will later define in DAC. So this SQL will be executed at run time and the result of this statement will be used to compared to the 3 columns from the source based on the lookup conditions.
In our particular case, think of it like this:
Source: The filtered 3 unique columns from the source has a list of records that are recently updated after the last loading.
Target: Select 3 columns from the target table
Lookup condition: (if ) Source = Target
Then generates 1 lookup output that holds the result of each comparison and bring it into expression transformation for assigning meanings to the lookup output:
So now we know that we have 1 output from lookup transformation and it is brought into the expression transformation.
![]()
![]()
![]()
![]()
![]()
So from there, as we know that the lookup output are either going to have a list of records that satisfies the lookup condition or it is NULL when it doesn’t satisfy the condition. In expression, will define meanings for the NULLs and No-Nulls by creating an ‘update Flag’ port:
The input port here is LOOKUP_INPUT10 which comes from lookup transformation. The output port “update Flag” gets its value from a condition that is based on the input port:
IIF(ISNULL(LOOKUP_INPUT10),’I’,’U’)
So here I am assigning “I” and “U” for either LOOKUP_INPUT is Null or No_Null. You can choose other symbols as you like. It doesn’t have to “I” or “U”.
In other words, if Lookup_Input is Null, that means the lookup condition: Source.Task_ID = Target.Task_ID, Source.Project_ID = Target.Project_ID, Source.EVENT_NUM = Target.EVENT_NUM are not satisfied. Meaning that this particular record exist in the source that are recently populated (based on last update date filter ), but not in the target. Therefore, this record is flagged with “I”. If the lookup_Input is not Null, then flag it with “U”.
Depending on the data and the environment that we are working, sometimes just looking at whether 1 column is Null or not may not be enough. If those cases occurs where you need to make sure all of the lookup inputs satisfy the conditions or not, you can concatenate them in the expression. Converting non-text datatype before concatenation is also possible. An example will be:
In the above situation, I bring all of the needed lookup input columns into expression, then in the update_flag expression, I have:
IIF(ISNULL(to_char(LOOKUP_INPUT10)||’~’||to_char(LOOKUP_INPUT11)||’~’||
LOOKUP_INPUT12||’~’||LOOKUP_INPUT13||’~’||TO_CHAR(LOOKUP_INPUT14)||’~’
||TO_CHAR(LOOKUP_INPUT15)),’I’,’U’)
You can explore your own environment and find out what works for you.
Now bring update Flag along with all of the columns from the source into Update Strategy Transformation where we will actually tell the transformation what to do when loading to the target:
In the update strategy expression, enter:
IIF(UPDATE_FLAG = ‘I’,DD_INSERT,IIF(UPDATE_FLAG = ‘U’,DD_UPDATE,DD_REJECT))
Again here, we are associating “I” with DD_Insert, “U” with DD_Update, and everything else will be DD_Reject. These are the actual commands that Update strategy uses to interact with the target table.
Now that the mapping is built. Yet, because we are using parameters that the values are passing from DAC, therefore, in order for Informatica to receive these values, it needs to know what they are. For that reason, we have to add the mapping parameters in this specific mapping:
Here we are adding 6 parameters to the list, because they are what’s used in this mapping:
$$LAST_EXTRACT_DATE — Referred by the expression in $$LAST_UPDATE_FILTER
$$LAST_UPDATE_FILTER — Used in Source Qualifier Transformation
$$LOOKUP_INPUT10 — Lookup Override
$$LOOKUP_INPUT11 — Lookup Override
$$LOOKUP_INPUT12 — Lookup Override
$$TARGET_TABLE — Lookup Override
So from this point, we are done with the mapping.
In the next entry, we will continue on with the configuring the tasks and workflow in Workflow Manager.
we just went through the whole process of building the InformaticaMapping with incremental logic. This time, let’s create the session and workflow.
![]()
![]()
![]()
The process of doing it is pretty standard and straightforward, I will just high line the few areas that needs attention.
First of all, we will have 2 sessions and 2 workflows, one for full load and the other for incremental load. Both session will use the same mapping that we just created.
So I have 2 sessions: SDE_MCK1213_PA_EVENTS and SDE_MCK1213_PA_EVENTS_FULL
The SDE_MCK1213_PA_EVENTS session is for incremental loading, therefore there are a few difference from the full load versions:
1. Treat Source Row As Data Driven Instead of “Insert” for Full Load
2. Target Load Type is “Normal” instead of “Bulk” for Full Load
3. The Source Filter for the incremental session should have the parameter “$$LAST_UPDATE_FILTER”. For the Full Load Session, it should be removed and left blank.
After that, I created 2 workflows and 1 for each session. For simplicity, I am giving the same name as session names. That’s all there is for setting up 2 workflows and 2 sessions.
We will now go into DAC with the 2 workflows:
SDE_MCK1213_PA_EVENTS and SDE_MCK1213_PA_EVENTS_FULL
we have been configuring Informatica to implement the incremental mapping logic. The last piece is to configure DAC to run these workflows and to ask DAC to tell informatica what these parameter values should be. It is a fairly simple configuration that I will take you through now.
![]()
![]()
Here in DAC, we already have 2 source system parameters, so now we can go ahead and create a task for PA_EVENT:
Notice that in the Task set up, there are 2 fields:
Commend for Incremental Load and Commend for Full Load
In there, we will enter the specific Informatica workflow names for each field. Simple as that.
Now, we need to define the values for the parameters used in lookup override. In this case, the value for $$Lookup_input10 to 12 will be the name of each of the 3 primary key columns. Make sure the name of the columns should match the order of the parameters in the lookup transformation. Then the $$Target_Table will be the name of the target table in the mapping, or “PA_EVENT” in this case:
Notice that these are task parameters, not the source system parameters. Therefore, these parameters are only applicable to this specific task while $$last_extract_date being the source system parameter can be applied to all tasks.
This is really the end of the configuration in DAC. Moving forward, just add the task in certain subject area and run it as part of an execution plan. The corresponding workflow will be triggered depending on the situation.
As I mentioned before in part 1, there are many different ways to implement incremental loading in BI Apps, none is better nor worse. They all have advantages in certain situation.
This approach basically lets DAC to be in control in the event of any changes. We can simply edit the value of the parameters in DAC while leaving Informatica as it is.
Also, if we see that there are a list of tables that all have similar patterns of update, such as having the same number of primary key columns with the same data types, we can simply create a mapplet doing the lookup part, then apply it to all of those mappings.
The bottom-line is, depending on the real life situation, we get to choose how we want to set things up. Nonetheless, the logic of incremental loading is more or less the same, I hope this series of articles will give beginners a detailed and comprehensive understanding of how to implement such kind of mappings in BI Apps.
Thanks and until next time.
The post DAC Configuring incremental load Oracle BI Apps appeared first on Oracle for All.