Global Dynamic Variables in the CloudConnect

A CloudConnect graph can use static parameters which are loaded from the parameter (.prm) files, as global constants in the graph. These constants are set during the graph initialization to the values contained in the parameter files, and the values remain intact without any change during the whole graph computation.

The value of a parameter can be evaluated during the graph execution at the moment it is called (so it looks like dynamic). However, such parameter is only a shortcut for stand-alone CTL2 function without input parameters, nothing more.

But there are also cases where we need to have some way to create and use global variables, which can be updated and read throughout a graph’s execution. CloudConnect provides a handful of ways to accomplish this task.

Prerequisites

• CloudConnect - Using Parameters
• Dictionary object
• Lookup Table and the CloudConnect - Using Lookups

The Dictionary Element

If the list of the parameter names is known in advance, the Dictionary element can be used. The names of the Dictionary entries, which serve as graph parameters, must be set in the graph before deploying it. There are many ways to assign and use values stored in the Dictionary entries. They can be assigned, accessed and changed by a simple assignment statement in the CTL2 language, e.g.:

$out.0.parameterValue = dictionary.parameterName;
dictionary.parameterName = $in.0.parameterValue;

The Dictionary entries can be used as source or target of data in the CSV Reader and the CSV Writer component as well.

The Lookup Table (internal, shared) and the GD Lookup Table

The internal Lookup Table is effective only within a single graph run. A shared Lookup Table can be shared between graphs, but its content is forgotten when ETL run is finished The external GD Lookup Table on the other hand has its content saved on the CC server. The parameter names and their values are kept as persistent information which can be used by more than one ETL process and can even be used by the ETL processes of other projects.

For the external GD Lookup Table, the most efficient way to access it is through-direct reading and writing to the (server located) table. This method preserves written values in the event that graph fails after writing a parameter value to the table. Below is an example of how to read and write from this table using CTL2 code.

Reading from the GD Lookup Table:

$out.0.LAST_SUCCESSFUL_START = lookup(paramsLookup).get("LAST_SUCCESSFUL_START").value;

Writing to the GD Lookup Table (even with the introduction of new parameter names):

gdCustomProperties keyValue;
keyValue.key = "LAST_SUCCESSFUL_START";
keyValue.value = '${CURRENT_START}';
lookup(paramsLookup).put(keyValue);

Summary

The .prm (static parameter) files are appropriate in the case where the parameters are known in advance, are remaining unchanged, and are used by subsequent graphs within the same ETL run. These parameter values are typically prepared by a separate graph in advance of their use later in the ETL process.

The Dictionary element allows a user to make simple dynamic changes to the parameter values during the execution of a single graph, but the list of parameters must be fully known in advance and the updated variable values are forgotten after finish of the graph run.

The Lookup Table allows the dynamic creation of parameters and the use of the parameter values expressions within the same graph (or within the same ETL in the case of a shared lookup table).

The GD Lookup Table allows “remote access” to (and change of) the parameter values from independent ETLs, so it can be used to hand-over values of global variables between different ETLs.