Data Modeling API (Cloud Connect)

CloudConnect is a legacy tool and will be discontinued. We recommend that to prepare your data you use the GoodData data pipeline as described in Data Preparation and Distribution. For data modeling, see Data Modeling in GoodData to learn how to work with Logical Data Modeler.

The Logical Data Model API allows you to create logical data models without using CloudConnect LDM Modeler or composing low-level MAQL DDL scripts manually. It also allows you to store datasets in the database and create a custom LDM programmatically. You can build a modular solution on top of the GoodData platform. If, for example, you have 50 data sources, one data source per dataset, and you want to select just two of them for Client A and other three for Client B, using the LDM API helps you automate the process.

How It Works

You can use the LDM API to create a new LDM in an empty project or to update an existing LDM.

This is how it works:

Create a JSON definition that describes your LDM.
Use the API to generate a diff and a MAQL DDL script.
Use the API to execute the script.

Create or Update an LDM Using the API

Imagine that you want to create a new LDM for the project that is used in HR Data Analysis, or update an existing LDM to correspond with the following LDM:

The JSON definition of this LDM would look as follows:

{
  "projectModel": {
    "datasets": [
      {
        "dataset": {
          "identifier": "dataset.department",
          "title": "Department",
          "anchor": {
            "attribute": {
              "identifier": "attr.department.department",
              "title": "Department",
              "labels": [
                {
                  "label": {
                    "identifier": "label.department.department",
                    "title": "Department"
                  }
                }
              ]
            }
          }
        }
      },
      {
        "dataset": {
          "identifier": "dataset.employee",
          "title": "Employee",
          "anchor": {
            "attribute": {
              "identifier": "attr.employee.employee",
              "title": "Employee",
              "folder": "Employee",
              "labels": [
                {
                  "label": {
                    "identifier": "label.employee.employee",
                    "title": "Employee",
                    "dataType": "INT",
                    "type": "GDC.text"
                  }
                },
                {
                  "label": {
                    "identifier": "label.employee.employee.name",
                    "title": "Employee Name",
                    "dataType": "VARCHAR(128)",
                    "type": "GDC.text"
                  }
                }
              ]
            }
          },
          "facts": [
            {
              "fact": {
                "identifier": "fact.employee.age",
                "title": "Employee Age",
                "folder": "Employee"
              }
            }
          ],
          "references": [
            "dataset.department"
          ]
        }
      },
      {
        "dataset": {
          "identifier": "dataset.salary",
          "title": "Salary",
          "anchor": {
            "attribute": {
              "identifier": "attr.salary.factsof",
              "title": "Records of Salary",
              "folder": "Salary"
            }
          },
          "facts": [
            {
              "fact": {
                "identifier": "fact.salary.payment",
                "title": "Payment",
                "folder": "Salary",
                "dataType": "DECIMAL(12,2)"
              }
            }
          ],
          "references": [
            "dataset.employee",
            "payment"
          ]
        }
      }
    ],
    "dateDimensions": [
      {
        "dateDimension": {
          "name": "payment",
          "title": "Payment"
        }
      }
    ]
  }
}

This structure is compatible for integration using many programming languages such as Ruby, Python and so on. You can store this definition in a database, add datasets, and provision those models.

The following procedure assumes that you access your projects at https://secure.gooddata.com/.

If you are a white-labeled customer, replace secure.gooddata.com with your white-labeled domain in the procedure steps.

Steps:

Use the API for generating a diff and a MAQL DDL script:
- API resource: https://secure.gooddata.com/gdc/projects/{workspace_id}/model/diff You can append one or more optional parameters (by default, all of them are set to false and not applied):
```
https://secure.gooddata.com/gdc/projects/{workspace_id}/model/diff?includeDeprecated=true&includeNonProduction=true&includeGrain=true&includeCA=true&excludeFactRule=true
```
  - includeDeprecated=true includes deprecated attributes and facts in the LDM.
  - includeNonProduction=true includes non-production datasets (data uploaded from a separate file) in the LDM.
  - includeGrain=true includes the Fact Table Grain (see Set the Grain of a Fact Table to Avoid Duplicate Records) in the LDM.
  - includeCA=true includes computed attributes (see Use Computed Attributes) in the project LDM.
  - excludeFactRule=true skips number format validation (up to 15 digits, including max. 6 digits after decimal point).
- Method: POST
- Request body:
```
{
  "diffRequest": {
    "targetModel": {above_JSON_definition}
  }
}
```
The API returns the link for polling for status of the diff and MAQL DDL generation.
Poll for the status until the diff and MAQL DDLs are returned.

In the returned MAQL DDLs, locate the updateScripts -> updateScript section.

"updateScripts": [
  {
    "updateScript": {
      "maqlDdl": "{MAQL_statement}",
      "preserveData": true,
      "cascadeDrops": false
    }
  }
]

Copy the MAQL statement from the maqlDdl key.
Use any free online JSON escape tool to make the following changes in the MAQL statement:
1. Remove all instances of \n.
2. Replace \" with ".
Use the API for executing a MAQL DDL script:
- API resource: https://secure.gooddata.com/gdc/md/{workspace_id}/ldm/manage2
- Method: POST
- Request body:
```
{
  "manage": {
    "maql": "{updated_MAQL_statement}"
  }
}
```
The link for polling for status of the LDM update is returned.
Poll for the status until the OK task status is returned.
(Optional) Log in to the GoodData Portal, go to your project, and check the LDM (see View a Workspace Model).
If you were updating the LDM only to add some objects, you do not have to worry about losing any object in the LDM.
If you were updating the LDM to delete some objects, check the results. All reports and dashboards will be dropped with corresponding objects.

Work with Fact Table Grain Using CloudConnect Modeler