Deploy a Process with the Data Integration Console

You can deploy a process to the currently selected project using the Data Integration Console.

To deploy a process, you can use:

  • Data pipeline brick
  • Life Cycle Management (LCM) brick
  • Automated Data Distribution (ADD) v2
  • Ruby script from the GoodData Git repositories
  • CloudConnect graph

If you are using Automated Data Distribution (ADD) v1 (see Automated Data Distribution Reference), you do not have to deploy the process. The process is automatically created after you configure the output stage parameters for the workspace where you want to load data from Data Warehouse (see Use Automated Data Distribution). The process is created under the name of 'Automated Data Distribution'. You can create schedules for it in the Data Integration Console (see Schedule a Process on the Data Integration Console).

Contents:

Deploy a process for a data pipeline brick

Data pipeline bricks help you perform specific tasks within the scope of the data preparation and distribution pipeline. Depending on the type, a data pipeline brick can download data from a data source, transform the data, or upload it to Data Warehouse for further distributing to workspaces.

For more information, see Data Preparation and Distribution Pipeline and Brick Reference.

Steps:

  1. From the Data Integration Console (see Accessing Data Integration Console), click Projects to open the Projects page.
  2. Click the name of the project where you want to deploy a data pipeline brick.
  3. Click Deploy Process.
    The deploy dialog opens.
  4. From the Component dropdown, select the brick (downloader, executor, or integrator) that you want to deploy.
  5. Fill in the fields to provide required information, such as path the configuration file and S3 properties.
  6. Enter a descriptive name for the process. The process name must be unique within the project.
  7. Click Deploy.
    The process is deployed.

You can now schedule the ETL process (see Scheduling a Process).

Deploy a process for a Life Cycle Management (LCM) brick

LCM bricks help you complete specific tasks for the projects that are managed via Life Cycle Management (LCM).

For more information, see Managing Projects via Life Cycle Management and Bricks.

Steps:

  1. From the Data Integration Console (see Accessing Data Integration Console), click Projects to open the Projects page.
  2. Click the name of the project where you want to deploy an LCM brick.
    By default, you deploy the brick to the service workspace.
  3. Click Deploy Process.
    The deploy dialog opens.
  4. From the Component dropdown, select the LCM brick that you want to deploy.

  5. Enter a descriptive name for the process. The process name must be unique within the project.
  6. Click Deploy.
    The LCM brick is deployed.

You can now schedule the brick (see Scheduling a Process).

Deploy a process for Automated Data Distribution (ADD) v2

You deploy a process for ADD as part of setting up direct data distribution from data warehouses (Snowflake or Redshift). For more information, see Direct Data Distribution from Data Warehouses and Integrate Data Warehouses Directly to GoodData based on Your DW Schema.

Steps:

  1. From the Data Integration Console (see Accessing Data Integration Console), click Projects to open the Projects page.
  2. Click the name of the project where you want to deploy an ADD process.
  3. Click Deploy Process.
    The deploy dialog opens.
  4. From the Component dropdown, select Automated Data Distribution.
  5. From the Data Source dropdown, select the Data Source that you want to use within the ADD process.

    From this dialog, you can create a new Data Source or edit an existing one. For the descriptions of the parameters, see 'Set up a connection between Snowflake and the GoodData platform' in Integrate Data Warehouses Directly to GoodData based on Your DW Schema.

  6. Enter a descriptive name for the process. The process name must be unique within the project.
  7. Click Deploy.
    The process is deployed.

You can now schedule the ETL process (see Scheduling a Process).

Deploy a process from a Ruby script

You can use one of the following options:

  • A GoodData-provided Ruby script
    To get the path to the script, contact GoodData Support.
  • Your own Ruby script
    That script must be placed in the GoodData repositories. To have your script uploaded to the repositories, do the following:
    1. Prepare a Ruby script together with Gemfile, Gemfile.lock, and info.json. If you need help with creating these files, contact GoodData Support.
    2. Contact GoodData Support with a request to commit your files to the GoodData repositories.
      GoodData Support will have your files verified and will send you the path to them in the GoodData repositories.

      The verification service may be subject to a charge. With questions, contact GoodData Support.

You can also deploy Ruby scripts via API for deploying a Ruby process with bundled dependencies.

Steps:

  1. From the Data Integration Console (see Accessing Data Integration Console), click Projects to open the Projects page.

  2. Click the name of the project where you want to deploy a Ruby script.
  3. Click Deploy Process.
    The deploy dialog opens.

  4. From the Component dropdown, select Generic Ruby.
  5. Make sure that the git option button is selected.

    Uploading a Ruby script using the zip file option is currently not available.

  6. Enter the path to the script that you obtained from GoodData Support.
  7. Enter a descriptive name for the process. The process name must be unique within the project.
  8. Click Deploy.
    The process is deployed.

You can now schedule the ETL process (see Scheduling a Process).

Deploy a process from a CloudConnect graph

Extract the graph from CloudConnect, and save it as a ZIP file. The file must contain all resources required for the process to execute and must be less than 5MB.

Steps:

  1. From the Data Integration Console (see Accessing Data Integration Console), click Projects to open the Projects page.

  2. Click the name of the project where you want to deploy a CloudConnect graph.
  3. Click Deploy Process.
    The deploy dialog opens.

  4. From the Component dropdown, select  CloudConnect Graph .
  5. Click Browse, and navigate your local environment to select the ZIP file to upload.
  6. Enter a descriptive name for the process. The process name must be unique within the project.
  7. Click Deploy.
    The process is deployed.

You can now schedule the ETL process (see Scheduling a Process).