Data Loading Modes in CloudConnect
CloudConnect is a legacy tool and will be discontinued. We recommend that to prepare your data you use the GoodData data pipeline as described in Data Preparation and Distribution. For data modeling, see Data Modeling in GoodData to learn how to work with Logical Data Modeler.
CloudConnect can use two modes for loading data to the GoodData platform using the GD Dataset Writer component:
- Sequential mode loads data for each dataset separately.
- Batch mode loads data for multiple datasets in one process. Batch mode allows for faster data upload and is the recommended option for CloudConnect.
In the latest versions of CloudConnect (see the table below for details), batch mode is the default loading mode for both local Cloud Connect runs and runs on the GoodData platform.
For backward compatibility, older CloudConnect versions and also projects belonging to older tokens (see the table below for details) have sequential mode as the default loading mode, with an option to switch to batch mode, if needed.
We strongly recommend that you use batch mode as the default loading mode, and use sequential mode only for certain graphs, when absolutely necessary.
The following table describes options and solutions on how you can switch between the modes depending on what version of CloudConnect you are using and when your token was created.
| Token date / CloudConnect version | What is the default loading mode for the GD Dataset writer component? | How can I temporarily switch to another mode for a specific CloudConnect graph execution? | How can I make batch mode the default? (recommended) | |
|---|---|---|---|---|
| Execution on the GoodData platform in a project | Token created on Jan 16, 2016, or later | batch | To switch to sequential mode, use the run parameter GDC_USE_BATCH_SLI_UPLOAD=FALSE | n/a |
Token created before Jan 16, 2016 | sequential | To switch to batch mode, use the run parameter GDC_USE_BATCH_SLI_UPLOAD=TRUE |
| |
| Local run in CloudConnect | 109.0.0 or later | batch | To switch to sequential mode, use the run parameter GDC_USE_BATCH_SLI_UPLOAD=FALSE | n/a |
| 102.0.1 | We strongly recommend that you upgrade CloudConnect to version 109.0.0 or later. Some partial builds within version 102.0.1 may have sequential mode set as the default, and the other builds have batch mode by default. This may result in unexpected behavior. This situation has been fixed in CloudConnect version 109.0.0 where the default loading mode is batch. | |||
| before 102.0.1 | sequential | To switch to batch mode, use run parameter GDC_USE_BATCH_SLI_UPLOAD=TRUE | Upgrade CloudConnect to version 109.0.0 or later. | |