Custom Field Creator

Custom Field Creator is a utility that supports the data preparation and distribution pipeline (see Data Preparation and Distribution Pipeline). Custom Field Creator creates additional fields (facts, attributes, or dates) in the logical data model (LDM) of one or more workspaces based on the criteria that you have defined (what fields to create and in what workspaces to do so).

How Custom Field Creator Works

When Custom Field Creator runs, it creates custom fields in the workspaces according to the criteria that you have defined in queries:

• The types of the custom fields to create (fact, attribute, or date)
• The properties of the custom fields (name, datatype, and so on)
• The workspace where to create the fields in
• The dataset in the LDM where each field should be created

Custom Field Creator does not modify or delete any fields from the LDM.

Configuration File

Custom Field Creator does not require any parameters in the configuration file.

Schedule Parameters

When scheduling Custom Field Creator (see Phases of Building the Data Pipeline -> Production Implementation), provide the parameters from this section in the schedule.

General Parameters

Some parameters must be entered as secure parameters (see Configure Schedule Parameters).

Queries

The queries define the criteria for processing the custom fields: what fields to create and in what workspaces to do so.

Query Structure

You can specify the queries in a GoodData Data Warehouse (ADS) table with the following columns:

Here is an example of the query table where workspace IDs are used to identify the workspaces:

Here is an example of the query file where client IDs are used to identify the workspaces and the data product is specified for each workspace:

Query Source

Custom Field Creator reads the queries from ADS. Specify the query as an SQL query to ADS.

{
  "ads_client": {
    "username": "john.doe@example.com",
    "password": "secret",
    "ads_id": "rtmmgjsqc4zmf64egtu6l6xv2xhxempi"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT source_identifier, dataset, cf_identifier, cf_type, cf_datatype, cf_title, cf_folder_title, x__timestamp FROM custom_fields_input"
  }
}

Queries as Schedule Parameters

When providing the queries as schedule parameters, you have to code them using the gd_encoded_params parameter (see General Parameters).

To code the queries, follow the instructions in Specifying Complex Parameters. Once done, you should have the gd_encoded_params parameter coding the queries and reference parameters encoding sensitive information that you will be adding as secure parameters.

Example: The queries obtained from ADS (sensitive information present: ADS password) to be coded by the gd_encoded_params and a reference parameter encoding the ADS password

The following will become the value of the gd_encoded_params parameter:

{
  "ads_client": {
    "username": "john.doe@example.com",
    "password": "${ads_client_password}",
    "ads_id": "rtmmgjsqc4zmf64egtu6l6xv2xhxempi"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT source_identifier, dataset, cf_identifier, cf_type, cf_datatype, cf_title, cf_folder_title, x__timestamp FROM custom_fields_input"
  }
}

In addition, you have to add the reference parameter ads_client_password as a secure parameter.

Schedule Example

The following is an example of how you can specify schedule parameters:

Logs and Error Handling

Custom Field Creator generates a log with all the changes made and errors occurred.

• If processing for one workspace fails, Custom Field Creator logs an error and continues the execution as usual.
• An execution with some workspaces processed and some failed finishes with a state of “Warning”.
• An execution with all workspaces failed finishes with a state of “Failed”.

By default, Custom Field Creator fails when an error occurs. You can change this behavior by setting the dont_fail_on_error parameter to true (see General Parameters).

Best Practices

• If you have a lot of client workspaces, consider using the x__timestamp column (see Query Structure), and run Custom Field Creator in incremental processing mode to reduce its run time during repeated executions.
• Run Custom Field Creator in the following cases: 
  • After you have defined new custom fields to make sure that they will be added to the existing client workspaces.
  • After you have created new client workspaces (see Provisioning Brick) to make sure that the custom fields are added to those newly created workspaces.