There is a huge amount of data in your systems and databases. As you would like to optimize processes that won’t be inefficient, one of the most common use case is loading only the latest data. As latest data we mean data that has been created i.e. Yesterday.
No matter if your data source is Facebook, Twitter, Salesforce or Google Analytics you probably face the problem of loading incremental data. Let’s go through several use cases that will show you how you can develop and automate en ETL process in CloudConnect to solve this. Read following blogpost and learn how to use the “inception” graph pattern and how to create a solution on the top applications like Google Analytics or Facebook.
Inception graph pattern
What you need to do is to set component to download data from specific time frame. Sometimes it is straightforward, sometimes you need to use the pattern that we call “inception pattern”. The process is quite easy and consists of two graphs. See it in detail:
- Set up graph that generates “YESTERDAY” parameter based on actual date and save it to some file. Secondly, run the second graph that do the main business case.
- Read previously created parameter from the file use it in the given component.
Google Analytics component doesn’t allow you to insert “YESTERDAY” parameter directly into the configuration parameters. This means you need to use the “inception” graph to generate it. See the screenshot above that is based on real Google Analytics example.
Salesforce is transactional system that stores data about your sales and marketing. Since sales cycle is the process that is changing over time and your sales opportunity may have different status over time, the implementation of complete end to end analytics little bit more complex. You need to deal with historization and slowly changing dimensions. Anyway, from high level point of view, everything you need to do is to compose corresponding SOQL - Salesforce Query Language
Facebook, similar to Salesforce, has its own query language (FQL) so you are able to specify your request and receive selected metrics as a response. Facebook has input value so you can also use incoming edge with data generator component to generate “yesterday” parameter.
Due to the changes in the Facebook Platform API, we dropped the support of the FQL option in CloudConnect (the Facebook Reader). Only the Graph API option is available. Сheck your graphs in CloudConnect, and make sure that they are using Graph API.
We will show you more in more detail how to implement incremental loading principles based on real examples from all data sources that are shown above.