Edge Memory Allocation
Manipulating large volumes of data in a single record is always an issue. In CloudConnect Designer, sending big data along graph edges means this:
Whenever there is a need to carry many MBs of data between two components in a single record, the edge connecting them expands its capacity. This is referred to as dynamic memory allocation.
If you have a complicated ETL scenario with some sections transferring huge data then only the edges in these sections will use dynamic memory allocation. The other edges retain low memory requirements.
An edge which has carried a big record before and allocated more memory for itself will not 'shrink' back again. It consumes bigger amount of memory till your graph execution is finished.
By default, the maximum size of a record sent along an edge is 32 MB. This value
can be increased, theoretically, up to tens of MBs by setting the Record.RECORD_LIMIT_SIZE
property
(Changing Default CloudConnect Settings). Record.FIELD_LIMIT_SIZE
can also be 32 MB by default. Naturally, all fields in total cannot use more memory than
Record.RECORD_LIMIT_SIZE.
There is no harm in increasing Record.RECORD_LIMIT_SIZE to whatever size
you want. The only reason for
keeping it smaller is an early error detection. For instance, if you start
appending to a string field and forget to reset record (after each record), the
field size can break the limits.
![[Note]](figures/note.png) | Note |
|---|---|
Let us look a little deeper into what happens in the memory.
Initially, a record starts with 64k of memory allocated to it.
If there is a need to transfer huge data, its size can dynamically
grow up to the value of |