Incremental Reading

[Important]Important

Incremental reading does not work on the GoodData server. We recommend that you do not use it.

Some Readers allow to use so called incremental reading. If the graph reads the same input file or a collection of files several times, it may be interested only in those records or files, that have been added since the last graph run.

In the following four Readers, you can set the Incremental file and Incremental key attributes. The Incremental key is a string that holds the information about read records/files. This key is stored in the Incremental file. This way, the component reads only those records or files that have not been marked in the Incremental file.

The Readers allowing incremental reading are as follows:

The component which reads data from databases performs this incremental reading in a different way.

Unlike the other incremental readers, in this database component, more database columns can be evaluated and used as key fields. Incremental key is a sequence of the following individual expression separated by semicolon: keyname=FUNCTIONNAME(db_field)[!InitialValue]. For example, you can have the following Incremental key: key01=MAX(EmployeeID);key02=FIRST(CustomerID)!20. The functions that can be selected are the following four: FIRST, LAST, MIN, MAX. At the same time, when you define an Incremental key, you also need to add these key parts to the Query. In the query, a part of the "where" sentence will appear, for example, something like this: where db_field1 > #key01 and db_field2 < #key02 . This way, you can limit which records will be read next time. It depends on the values of their db_field1 and db_field2 fields. Only the records that satisfy the condition specified by the query will be read. These key fields values are stored in the Incremental file. To define Incremental key, click this attribute row and, by clicking the Plus or Minus buttons in the Define incremental key dialog, add or remove key names, and select db field names and function names. Each one of the last two is to be selected from combo list of possible values.

[Note]Note

Since the version 2.8.1 of CloudConnect Designer, you can also define Initial value for each key. This way, non existing Incremental file can be created with the specified values.