Provisioning Brick

The provisioning brick creates clients' workspaces under the appropriate segments, and deploys the reports, dashboards, filters, logical data model (LDM), data loading processes, and metadata from the master workspace to the clients' workspaces within a segment. Optionally, the provisioning brick deletes obsolete clients and their workspaces.

For information about how to use the brick, see How to Use a Brick.

Prerequisites

Before using the provisioning brick, make sure that the following is true:

  • The release brick (see Release Brick) has been executed.
  • You have an input source with a list of the client workspaces to be provisioned. The preferred source is GoodData Data Warehouse (ADS) or one of the supported cloud data warehouses/object storage services (see Types of Input Data Sources).

How the Brick Works

The provisioning brick creates the new workspaces specified in the input source and associates them with the appropriate segment.

Depending on how the delete_mode parameter is set, the brick may also delete the existing client workspaces that are no longer in the input source. If the titles of some workspaces have changed in the input source, the brick renames these workspaces.

Other than deleting and renaming, the brick does not touch the existing workspaces.

Provisioning Brick and Custom Fields in the LDM

If you have custom fields set up in the LDM, consider the following:

  • If you use Object Renaming Utility to set up custom fields in the LDM of a segment’s master workspace (see Object Renaming Utility), you may need to run Object Renaming Utility after the provisioning brick to make sure that newly created client workspaces have the custom fields properly set.
  • If you have custom fields in the LDMs of the client workspaces and you have created new client workspaces, run Custom Field Creator after the provisioning brick to make sure that the custom fields are added to those newly created workspaces. For more information, see Add Custom Fields to the LDMs in Client Workspaces within the Same Segment.

Input

The provisioning brick expects to receive the data about which client should be provisioned under which segment.

The input data format is the following:

segment_idclient_idproject_titleproject_token
basic_segmentclient_1workspace_onevertica_token
premium_segmentclient_2workspace_two 
  • Values in the “segment_id” and “client_id” columns must match the following regex: [a-zA-Z0-9_\-]+

  • If the “project_title” column is missing from the input data, the workspace name is populated from the data in the “client_id” column.

  • If the “project_token” column is missing from the input data or is empty, the PostgreSQL authorization token of the master workspace (see the tokens parameter in Release Brick) is used for provisioning client workspaces. If you have a Vertica authorization token, you can provision client workspaces on Vertica. To do so, specify your Vertica token in the “project_token” column for those workspaces.

In addition, you have to add the parameters when scheduling the brick process.

Parameters

When scheduling the deployed brick (see How to Use a Brick and Schedule a Data Load), add parameters to the schedule.

NameTypeMandatory?DefaultDescription
organizationstringyesn/aThe name of the domain where the brick is executed
segments_filterarrayyesn/a

The segments that you want to provision

You must encode this parameter using the gd_encoded_params parameter (see Specifying Complex Parameters).

Example:

"segments_filter": ["BASIC", "PREMIUM"]
input_sourceJSONyesn/a

The input data that identifies the client workspaces and the location of that input data

You must encode this parameter using the gd_encoded_params parameters (see Specifying Complex Parameters).

The preferred source is ADS or one of the supported cloud data warehouses/object storage services. For more information about setting up the input source on an ADS instance as well as the other types of the input sources, see Types of Input Data Sources.

Example:

"ads_client": {
  "jdbc_url": "jdbc:gdc:datawarehouse://mycompany.com/gdc/datawarehouse/instances/kluuu4h3sogai9x2ztn4wc0g8lta7sn8",
  "username": "john.doe@gooddata.com",
  "password": "${ads_client_password}"
},
"input_source": {
  "type": "ads",
  "query": "SELECT DISTINCT client_id, segment_id, project_title FROM lcm_workspace;"
}
data_productstringnodefaultThe data product that contains the segments that you want to provision
delete_modestringnonone

Defines how the brick should process the clients and their workspaces when detecting that one or more clients existing in the segment are not present in the input source.

Possible values:

  • none leaves the clients who are not in the input source intact (the clients remain in the segment and their workspaces are accessible). This is the default.
  • delete_extra removes from the segment the clients who are not in the input data source and disassociates their workspaces from the segment (the workspaces will still exist but you will not be able to refer to them by client_id).
  • delete_projects deletes the clients who are not in the input source and physically deletes their workspaces (the workspaces will not be accessible).
disable_kd_dashboard_permissionBooleannofalseShared dashboard permissions for user groups are enabled by default. Even when the parameter is not explicitly specified in the syntax, it is assumed. Set to true to disable synchronizing user group permissions for shared dashboards. For more information, see LCM and Shared Dashboards.
technical_usersarraynon/a

The users that are going to be added as admins to each client workspace

The user logins are case-sensitive and must be written in lowercase.

You must encode this parameter using the gd_encoded_params parameter (see Specifying Complex Parameters).

Example:

"technical_users": ["dev_admin@gooddata.com", "admin@gooddata.com"]

LCM and Shared Dashboards

In a segment’s master workspace, dashboards can be private or shared with all or some users/user groups.

The sharing permissions are propagated from the master workspace to the client workspaces in the following way:

  • Dashboards set as private in the master workspace remain private in the client workspaces.
  • Dashboards shared with all users in the master workspace become dashboards shared with all users in the client workspaces.
  • Dashboards shared with some users/user groups in the master workspace become dashboards shared with some user groups in the client workspaces

In addition to the sharing permissions, the dashboards can be configured:

  • To allow only administrators to update the dashboards (that is, editors cannot update such dashboards)
  • To be displayed to users when they drill to these dashboards from facts, metrics, and attributes, even if the dashboards are not explicitly shared with those users

These settings are propagated to the client workspaces exactly as they are set in the master workspace. For more information about these settings, see Share Dashboards.

Example - Brick Configuration

The following is an example of configuring the brick parameters in the JSON format:

{
  "organization": "myCustomDomain",
  "delete_mode": "delete_projects",
  "gd_encoded_params": {
    "segments_filter": ["BASIC"],
    "ads_client": {
      "jdbc_url": "jdbc:gdc:datawarehouse://mycompany.com/gdc/datawarehouse/instances/kluuu4h3sogai9x2ztn4wc0g8lta7sn8",
      "username": "john.doe@gooddata.com",
      "password": "${ads_client_password}"
    },
    "input_source": {
      "type": "ads",
      "query": "SELECT DISTINCT client_id, segment_id, project_title FROM lcm_workspace;"
    },
    "technical_users": ["dev_admin@myCustomDomain.com", "admin@myCustomDomain.com"]
  },
  "ads_client_password": "enter_as_a_secure_parameter"
}

Troubleshooting

Provisioning brick fails

The provisioning brick fails with the following error message:

Master release not found. You must call client synchronization to create a master release before provisioning new projects.

Why has this happened?

When executing, the provisioning brick relies on the platform metadata objects that the rollout brick generates (see “Rollout and Provisioning Metadata” in Rollout Brick). These objects are automatically cleaned up three years after the rollout process was last executed.

How can you fix this?

  1. Synchronize the client workspaces to create the platform metadata objects that the provisioning brick uses. To do so, either run the rollout brick or use the synchronization API.
  2. Run the provisioning brick again.

Advanced Settings

This section describes advanced settings of the provisioning brick.

NameTypeMandatory?DefaultDescription
dynamic_paramsJSONnon/aSee dynamic_params.
metric_formatJSONnon/aSee metric_format.
include_deprecatedBooleannofalse

Specifies how to handle deprecated objects in the logical data model (LDM) while one is being generated in a client's workspace based on the LDM of the master workspace

  • If not set or set to false, the objects that are marked as deprecated in the master workspace LDM are not included in the generated MAQL diff that will be used for generating the LDM of the clients' workspaces. The deprecated objects will not be propagated to the LDM of the clients' workspaces.
  • If set to true, the deprecated objects are included in the generated MAQL diff and will be propagated to the LDM of the clients' workspaces where they be marked as deprecated.
skip_actionsarraynon/a

The actions or steps that you want the brick to skip while executing (for example, deleting clients or collecting dynamically changing parameters)

The specified actions and steps will be excluded from the processing and will not be performed.

NOTE: Using this parameter in a wrong way may generate unexpected side effects. If you want to use it, contact the GoodData specialist who was involved in implementing LCM at your site.

abort_on_errorBooleannotrue

Specifies whether the abort_on_error option when running the brick.

  • If not set or set to true, the abort_on_error option is enabled. If an error occurs, the brick fails.

  • If set to false, the  abort_on_error option is disabled. If an error occur, the brick keeps running and the error will be exported to the execution log.

The brick results in one of the following:

  • SUCCESS - when the brick finishes without any client workspace errors.

  • WARNING - when the brick finishes with at least one client workspace success and one client workspace error.

  • ERROR - when the brick finishes without any client workspace successes.

dynamic_params

The dynamic_params parameter lets you add client-specific, dynamically changing parameters to all or some schedules in all or some client workspaces. If the parameters pass sensitive data (for example, passwords), you can send them in encrypted form and add them as secure parameters (see Configure Schedule Parameters).

 

Example 1: No parameters encrypted  You want to add the MODE parameter with the value set to specific_mode to the schedules named “main.rb” for the client with the client ID client_A.

Steps:

  1. Make sure that your input data (see input_source in Parameters) returns correct information when queried for the client ID, schedule name, parameter name, and parameter value. That is, the input data must contain the following:

    client_idschedule_titleparam_nameparam_value
    client_Amain.rbMODEspecific_mode
    • client_id is the client ID of the client.
    • schedule_title is the name of the schedules where you want to add the parameter.
    • param_name is the name of the parameter that you want to add.
    • param_value is the value of the specified parameter.
  2. Create the JSON structure for the dynamic_params parameter, and add it to the brick schedule. Because it is a complex parameter, include it in your gd_encoded_params parameter (see Example - Brick Configuration).

    "gd_encoded_params": {
      ...
      "dynamic_params": {
        "input_source": {
          "type": "ads",
          "query": "SELECT p.client_id, 'main.rb' AS schedule_title, 'MODE' AS param_name, p.param_value FROM lcm_dynamic_params p;"
        }
      }
    }
    

 

Example 2: Some parameters encrypted  You want to add the following parameters to the schedules named “provisioning” for the client with the client ID client_B:

  • The login parameter with the value set to admin@myCustomDomain.com (as a regular parameter)
  • The password parameter with the value set to my_password (as a secure parameter)

The value of the password parameter should not appear in clear-text form in the input source of the brick (see input_source in Parameters) and therefore must first be encrypted. When the password parameter is passed to the client’s schedules, its value will be decrypted, and the parameter will be added as secure and with its original value.

Steps:

  1. Encrypt the value of the password parameter (which is my_password). To encrypt the parameter, use the OpenSSL enc command (see https://wiki.openssl.org/index.php/Enc) or any compatible encryption tool that you trust. Use a cipher algorithm with the key size of 256 bits or higher (for example, AES-256-CBC). When encrypting, use the encryption password my_secret_encryption_password. The resulting encrypted value is wCmxeJhyzdM4O9S9+LPJ6w==.

  2. Make sure that your input data (see input_source in Parameters) returns correct information when queried for the client ID, schedule name, parameter name, parameter value, and the indicator whether the specified parameter should be added to the schedules as a regular or a secure parameter. That is, the input data must contain the following:

    client_idschedule_titleparam_nameparam_secureparam_value
    client_BprovisioningloginFALSEadmin@myCustomDomain.com
    client_BprovisioningpasswordTRUEwCmxeJhyzdM4O9S9+LPJ6w==
    • client_id is the client ID of the client.
    • schedule_title is the name of the schedules where you want to add the parameter.
    • param_name is the name of the parameter that you want to add.
    • param_secure is a Boolean flag that specifies whether the parameter should be added to the schedules as a secure parameter (see Configure Schedule Parameters). param_secure is optional. If it is not found in the input data or does not contain any value, it defaults to FALSE (the parameter is added as a regular, not secure parameter). Note the following:
      • The login parameter does not pass sensitive data and therefore can be added as a regular, not secure parameter. That is why its param_secure is set to FALSE. If you leave param_secure empty, it will default to FALSE and also result in adding the login parameter as a regular parameter.
      • The password parameter passes sensitive data and must be added as a secure parameter. That is why its param_secure is set to TRUE.
    • param_value is the value of the specified parameter. Note that the value of the password parameter is encrypted (see Step 1 of this procedure).
  3. Create the dynamic_param parameter, and add it to the brick schedule. Because it is a complex parameter, include it in your gd_encoded_params parameter (see Example - Brick Configuration).

    "gd_encoded_params": {
      ...
      "dynamic_params": {
        "input_source": {
          "type": "ads",
          "query": "SELECT p.client_id, p.schedule_title, p.param_name, p.param_secure, p.param_value FROM lcm_dynamic_params p;"
        }
      }
    }
    
  4. Add the dynamic_params_encryption_key parameter to the brick schedule as a secure parameter and set it to my_secret_encryption_password (this is the value of the encryption password that you used to encrypt the password; see Step 1 of this procedure). The dynamic_params_encryption_key parameter will be used to decrypt the encrypted value of the password parameter when it is passed to the client’s schedules. That is, the password parameter will be added to the client’s schedules with the value of my_password and not wCmxeJhyzdM4O9S9+LPJ6w==.

 

Additional options of dynamic_params

  • To add dynamic parameters to all schedules for “client_A”, do not explicitly specify schedule_title in the query:

    "gd_encoded_params": {
      "dynamic_params": {
        "input_source": {
          "type": "ads",
          "query": "SELECT 'client_A' AS client_id, schedule_title, 'MODE' AS param_name, 'specific mode' AS param_value"
        }
      }
    }
    
  • To add dynamic parameters to the schedules named “main.rb” in all clients, do not explicitly specify client_id in the query:

    "gd_encoded_params": {
      "dynamic_params": {
        "input_source": {
          "type": "ads",
          "query": "SELECT client_id, 'main.rb' AS schedule_title, 'MODE' AS param_name, 'specific mode' AS param_value"
        }
      }
    }
    

metric_format

The metric_format parameter lets you specify a custom number format for metrics in each client workspace. For example, you can set up different currency codes for client workspaces with data from different countries (USD for the clients operating in the USA, EUR for the clients operating in Germany, and so on).

For more information about the number formatting, see Formatting Numbers in Insights.

A custom format is applied to the metrics in a specific client workspace based on tags that you add to the metrics in advance. The custom format is applied everywhere where a number format is used in the GoodData Portal (see GoodData Portal). The custom format does not rewrite the format that is defined for a metric in a specific report/insight (for more information about setting a number format in a report/insight, see Formatting Table Values Using the Configuration Pane and Format Numbers).

Steps:

  1. Add tags to the metrics that you want to apply a custom format to (see Add a Tag to a Metric). For example, you can use format_# to tag metrics using the COUNT function, format_$ to tag currency metrics, format_% to tag metrics with percentages.

  2. Create a table that maps the tags to number formats and the client IDs of the client workspaces where the number formats should be applied. Name the table columns tag, format, and client_id, respectively.

    tagformatclient_id
    format_#

    [>=1000000000]#,,,.0 B;

    [>=1000000]#,,.0 M;

    [>=1000]#,.0 K;

    [>=0]#,##0;

    [<0]-#,##0

    client_id_best_foods
    format_%#,##0%client_id_zen_table
    format_%#,#0%client_id_best_foods
  3. Save the table in the location where your input data is stored (see input_source in Parameters):

    • If you use a data warehouse (for example, ADS, Snowflake or Redshift), save the table as a database table named metric_formats.
    • If you use a file storage (for example, S3, Azure Blob Storage or a web location), save the table as a CSV file named metric_formats.csv.
  4. Create the JSON structure for the metric_format parameter, and add it as a parameter under the input_source parameter (see Parameters). The metric_format parameter must contain a query for the metric_formats database table in the data warehouse or point to the metric_formats.csv file in the file storage.

    • The metric_formats table is located in a data warehouse. For example, in ADS:

      "input_source": {
        "type": "ads",
        "query": "SELECT DISTINCT client_id, segment_id, project_title FROM lcm_workspace;",
        "metric_format": {
          "query": "SELECT client_id, tag, format FROM metric_formats;"
        }
      }
      

      Or, in Redshift:

      "input_source": {
        "type": "redshift",
        "query": "SELECT client_id, segment_id, project_title FROM lcm_workspace;",
        "metric_format": {
          "query": "SELECT client_id, tag, format FROM metric_formats;"
        }
      }
      
    • The metric_formats.csv file is located in a file storage.

    For example, in an S3 bucket:

    ``` text
    "input_source": {
      "type": "s3",
      "file": "/data/upload/users.csv",
      "metric_format": {
        "file": "/data/upload/metric_formats.csv"
      }
    }
    ```

    Or, in a web location:

    ``` text
    "input_source": {
      "type": "web",
      "url": "https://files.acme.com/data/upload/users.csv",
      "metric_format": {
        "url": "https://files.acme.com/data/upload/metric_formats.csv"
      }
    }
    ```