Provisioning Brick

The provisioning brick creates clients' workspaces under the appropriate segments, and deploys the reports, dashboards, filters, logical data model (LDM), ETL, and metadata from the master workspace to the clients' workspaces within a segment. Optionally, the provisioning brick deletes obsolete clients and their workspaces.

For information about how to use the brick, see How to Use a Brick.

Contents:

Prerequisites

Before using the provisioning brick, make sure that the following is true:

  • The release brick (see Release Brick) has been executed.
  • You have an input source with a list of the client workspaces to be provisioned.
    The preferred source is GoodData Data Warehouse (ADS) or one of the supported cloud data warehouses (see Types of Input Data Sources).

How the Brick Works

The provisioning brick creates the new workspaces specified in the input source and associates them with the appropriate segment.

Depending on how the delete_mode parameter is set, the brick may also delete the existing client workspaces that are no longer in the input source. If the titles of some workspaces have changed in the input source, the brick renames these workspaces.

Other than deleting and renaming, the brick does not touch the existing workspaces.

Input

The provisioning brick expects to receive the data about which client should be provisioned under which segment.

The input data format is the following:

segment_idclient_idproject_titleproject_token
basic_segmentclient_1workspace_onevertica_token
premium_segmentclient_2workspace_two
  • Values in the "segment_id" and "client_id" columns must match following regex:
    [a-zA-Z0-9_\-]+

    If you store "client_id" values in a database, store them with the VARCHAR data type.

  • If the "project_title" column is missing from the input data, the workspace name is populated from the data in the "client_id" column.
  • If the "project_token" column is missing from the input data or is empty, the PostgreSQL authorization token of the master workspace (see the tokens parameter in Release Brick) is used for provisioning client workspaces.
    If you have a Vertica authorization token, you can provision client workspaces on Vertica. To do so, specify your Vertica token in the "project_token" column for those workspaces.

In addition, you have to add the parameters when scheduling the brick process.

Parameters

When scheduling the deployed brick (see How to Use a Brick and Schedule a Data Loading Process), add parameters to the schedule.

NameTypeMandatory?DefaultDescription

organization

string

yes

n/a

The name of the domain where the brick is executed

segments_filterarrayyesn/a

The segments that you want to provision

You must encode this parameter using the gd_encoded_params parameter (see Specifying Complex Parameters).

Example:

"segments_filter": ["BASIC", "PREMIUM"]
input_sourceJSONyesn/a

The input records that identify active client workspaces

You must encode this parameter using the gd_encoded_params parameters (see Specifying Complex Parameters).

The preferred source is ADS or one of the supported cloud data warehouses. For more information about setting up the input source on the Data Warehouse instance as well as the other types of the input sources, see Types of Input Data Sources.

Example:

"ads_client": {
  "jdbc_url": "jdbc:gdc:datawarehouse://mycompany.com/gdc/datawarehouse/instances/kluuu4h3sogai9x2ztn4wc0g8lta7sn8"
},
"input_source": {
  "type": "ads",
  "query": "SELECT DISTINCT client_id, segment_id, project_title FROM lcm_workspace;"
}
data_productstringnodefaultThe data product that contains the segments that you want to provision
delete_modestringnonone

Defines how the brick should process the clients and their workspaces when detecting that one or more clients existing in the segment are not present in the input source.

Possible values:

  • none leaves the clients who are not in the input source intact (the clients remain in the segment and their workspaces are accessible). This is the default.
  • delete_extra removes from the segment the clients who are not in the input data source and disassociates their workspaces from the segment (the workspaces will still exist but you will not be able to refer to them by client_id).
  • delete_projects deletes the clients who are not in the input source and physically deletes their workspaces (the workspaces will not be accessible).
technical_usersarraynon/a

The users that are going to be added as admins to each client workspace

The user logins are case-sensitive and must be written in lowercase.

You must encode this parameter using the gd_encoded_params parameter (see Specifying Complex Parameters).

Example:

"technical_users": ["dev_admin@gooddata.com", "admin@gooddata.com"]

Example - Brick Configuration

The following is an example of configuring the brick parameters in the JSON format:

{
  "organization": "myCustomDomain",
  "delete_mode": "delete_projects",
  "gd_encoded_params": {
    "segments_filter": ["BASIC"],
    "ads_client": {
      "jdbc_url": "jdbc:gdc:datawarehouse://analytics.myCustomDomain.com/gdc/datawarehouse/instances/kluuu4h3sogai9x2ztn4wc0g8lta7sn8"
    },
    "input_source": {
      "type": "ads",
      "query": "SELECT DISTINCT client_id, segment_id, project_title FROM lcm_workspace;"
    },
    "technical_users": ["dev_admin@myCustomDomain.com", "admin@myCustomDomain.com"]
  }
}

Troubleshooting

Provisioning brick fails

The provisioning brick fails with the following error message:

Master release not found. You must call client synchronization to create a master release before provisioning new projects.

Why has this happened?

When executing, the provisioning brick relies on the platform metadata objects that the rollout brick generates (see "Rollout and Provisioning Metadata" in Rollout Brick). These objects are automatically cleaned up three years after the rollout process was last executed.

How can you fix this?

  1. Synchronize the client workspaces to create the platform metadata objects that the provisioning brick uses. To do so, either run the rollout brick or use the synchronization API.
  2. Run the provisioning brick again.

Advanced Settings

This section describes advanced settings of the provisioning brick.

Change these settings only if you are confident in executing the task or have no other options. Adjusting the advanced options in a wrong way may generate unexpected side effects.

Proceed with caution.

NameTypeMandatory?DefaultDescription
dynamic_paramsJSONnon/aSee dynamic_params.
include_deprecatedBooleannofalse

Specifies how to handle deprecated objects in the logical data model (LDM) while one is being generated in a client's workspace based on the LDM of the master workspace

  • If not set or set to false, the objects that are marked as deprecated in the master workspace LDM are not included in the generated MAQL diff that will be used for generating the LDM of the clients' workspaces. The deprecated objects will not be propagated to the LDM of the clients' workspaces.
  • If set to true, the deprecated objects are included in the generated MAQL diff and will be propagated to the LDM of the clients' workspaces where they be marked as deprecated.
skip_actionsarraynon/a

The actions or steps that you want the brick to skip while executing (for example, deleting clients or collecting dynamically changing parameters)

The specified actions and steps will be excluded from the processing and will not be performed.

NOTE: Using this parameter in a wrong way may generate unexpected side effects. If you want to use it, contact the GoodData specialist who was involved in implementing LCM at your site.

dynamic_params

The dynamic_params parameter lets you add client-specific, dynamically changing parameters to all or some schedules in all or some client workspaces. If the parameters pass sensitive data (for example, passwords), you can send them in encrypted form and add them as secure parameters (see Configure Schedule Parameters).

The dynamic_params  parameter should be used sparsely and in very specific use cases (see the examples further in this section). The most use cases are automatically covered by Life Cycle Management (LCM; see Managing Workspaces via Life Cycle Management). Specifically, the CLIENT_ID parameter is automatically propagated to schedules.

An example of the special use case when you may want to use the dynamic_params parameter is when you use CloudConnect for managing data load processes (see Loading Data Using CloudConnect), and each client receives data from a different source. In this case, you need to propagate data source credentials to the schedules in the appropriate client's workspaces (for example, if Client A's workspaces are populated with data from Google Analytics, the schedules in those workspaces should get the Google Analytics credentials). Any sensitive data (such as passwords, secret keys, client secrets, and so on) must be encrypted first so that it does not appear in clear-text form in the input source of the brick (see input_source in Parameters).


Example 1: No parameters encrypted
You want to add the MODE parameter with the value set to specific_mode to the schedules named "main.rb" for the client with the client ID client_A.

Steps:

  1. Make sure that your input data (see input_source in Parameters) returns correct information when queried for the client ID, schedule name, parameter name, and parameter value.
    That is, the input data must contain the following:

    client_idschedule_titleparam_nameparam_value
    client_Amain.rbMODEspecific_mode
    • client_id is the client ID of the client.

    • schedule_title is the name of the schedules where you want to add the parameter.

    • param_name is the name of the parameter that you want to add.

    • param_value is the value of the specified parameter.

  2. Create the JSON structure for the dynamic_params parameter, and add it to the brick schedule. Because it is a complex parameter, include it in your gd_encoded_params parameter (see Example - Brick Configuration).

    "gd_encoded_params": {
      ...
      "dynamic_params": {
        "input_source": {
    	  "type": "ads",
    	  "query": "SELECT p.client_id, 'main.rb' AS schedule_title, 'MODE' AS param_name, p.param_value FROM lcm_dynamic_params p;"
    	}
      }
    }


Example 2: Some parameters encrypted
You want to add the following parameters to the schedules named "provisioning" for the client with the client ID client_B:

  • The login parameter with the value set to admin@myCustomDomain.com (as a regular parameter)
  • The password parameter with the value set to my_password (as a secure parameter)

The value of the password parameter should not appear in clear-text form in the input source of the brick (see input_source in Parameters) and therefore must first be encrypted. When the password parameter is passed to the client's schedules, its value will be decrypted, and the parameter will be added as secure and with its original value.

Steps:

  1. Encrypt the value of the password parameter (which is my_password).
    To encrypt the parameter, use the OpenSSL enc command (see https://wiki.openssl.org/index.php/Enc) or any compatible encryption tool that you trust. Use a cipher algorithm with the key size of 256 bits or higher (for example, AES-256-CBC).
    When encrypting, use the encryption password my_secret_encryption_password.
    The resulting encrypted value is wCmxeJhyzdM4O9S9+LPJ6w==.

  2. Make sure that your input data (see input_source in Parameters) returns correct information when queried for the client ID, schedule name, parameter name, parameter value, and the indicator whether the specified parameter should be added to the schedules as a regular or a secure parameter.
    That is, the input data must contain the following:

    client_idschedule_titleparam_nameparam_secureparam_value
    client_BprovisioningloginFALSEadmin@myCustomDomain.com
    client_BprovisioningpasswordTRUEwCmxeJhyzdM4O9S9+LPJ6w==
    • client_id is the client ID of the client.

    • schedule_title is the name of the schedules where you want to add the parameter.

    • param_name is the name of the parameter that you want to add.

    • param_secure is a Boolean flag that specifies whether the parameter should be added to the schedules as a secure parameter (see Configure Schedule Parameters).
      param_secure is optional. If it is not found in the input data or does not contain any value, it defaults to FALSE (the parameter is added as a regular, not secure parameter).
      Note the following:
      • The login parameter does not pass sensitive data and therefore can be added as a regular, not secure parameter. That is why its param_secure is set to FALSE. If you leave param_secure empty, it will default to FALSE and also result in adding the login parameter as a regular parameter.
      • The password parameter passes sensitive data and must be added as a secure parameter. That is why its param_secure is set to TRUE.

    • param_value is the value of the specified parameter.
      Note that the value of the password parameter is encrypted (see Step 1 of this procedure).

  3. Create the dynamic_param parameter, and add it to the brick schedule. Because it is a complex parameter, include it in your gd_encoded_params parameter (see Example - Brick Configuration).

    "gd_encoded_params": {
      ...
      "dynamic_params": {
        "input_source": {
          "type": "ads",
    	  "query": "SELECT p.client_id, p.schedule_title, p.param_name, p.param_secure, p.param_value FROM lcm_dynamic_params p;"
        }
      }
    }
  4. Add the dynamic_params_encryption_key parameter to the brick schedule as a secure parameter and set it to my_secret_encryption_password (this is the value of the encryption password that you used to encrypt the password; see Step 1 of this procedure).
    The dynamic_params_encryption_key parameter will be used to decrypt the encrypted value of the password parameter when it is passed to the client's schedules. That is, the password parameter will be added to the client's schedules with the value of my_password and not wCmxeJhyzdM4O9S9+LPJ6w==.

    If you do not add the dynamic_params_encryption_key parameter to the brick schedule, the password parameter will still be added to the specified client's schedules as a secure parameter but its value will not be decrypted and will be added to the schedules in its encrypted format (which is wCmxeJhyzdM4O9S9+LPJ6w==). A corresponding warning message will be recorded in the execution logs (see Review a Log of Data Loading Process Execution).


Additional options of dynamic_params

  • To add dynamic parameters to all schedules for "client_A", do not explicitly specify schedule_title in the query:

    "gd_encoded_params": {
      "dynamic_params": {
        "input_source": {
    	  "type": "ads",
    	  "query": "SELECT 'client_A' AS client_id, schedule_title, 'MODE' AS param_name, 'specific mode' AS param_value"
    	}
      }
    }
  • To add dynamic parameters to the schedules named "main.rb" in all clients, do not explicitly specify client_id in the query:

    "gd_encoded_params": {
      "dynamic_params": {
        "input_source": {
    	  "type": "ads",
    	  "query": "SELECT client_id, 'main.rb' AS schedule_title, 'MODE' AS param_name, 'specific mode' AS param_value"
    	}
      }
    }
Powered by Atlassian Confluence and Scroll Viewport.