Create a Logical Data Model from Your Cloud Object Storage Service

This tutorial guides you through the process of creating a logical data model (LDM) in your workspace using files in your object storage service (for example, S3 or Azure Blob Storage). A newly created workspace does not have an LDM therefore you are going to create an LDM from scratch.

You create an LDM in the LDM Modeler. To do so, perform the following steps:

  1. Create a Data Source.
  2. Download the files from the object storage service.
  3. Add datasets.
  4. Publish the LDM.

Create a Data Source

Create a Data Source for the object storage service that holds the files with data that you want to upload to your workspaces. For more information, see Create a Data Source.

During data load, the GoodData platform connects to the object storage service using the information from the Data Source, downloads the data from the files stored there, and uploads it to the workspaces according to how the datasets in the LDM are mapped to the files.

Download the Files from the Object Storage Service

Download the files from the object storage service to your computer.

While you can upload data from the files of various formats, you need CSV files to create the LDM. Therefore, if your files are not in the CSV format, convert them to CSV files to be able to create the LDM. For data loading, use the files in their original format that are stored in your object storage service. For the data loading to be successful, make sure that the structure of the files in your object storage service matches the structure of the CSV files that you used to create to the LDM.

For the supported file formats for your object storage service, see the following articles:

Add Datasets

You are now going to create datasets by importing the CSV files to the LDM Modeler.

When a CSV file is being imported, the LDM Modeler tries to auto-detect the types of the data in the file. The data can be detected as one of the following:

  • Fact, a numerical piece of data, which in a business environment is used to measure a business process (see Facts in Logical Data Models)
  • Attribute, data that is to be used in grouping or segmenting the values resulting from the computed functions (see Attributes in Logical Data Models)
  • Primary key, an attribute that serves as a unique identifier for a row of data in a file and as a connection point that allows you to connect this dataset to another dataset (see Connection Points in Logical Data Models)
  • Reference, a connection point (foreign key) from another dataset (see Connection Points in Logical Data Models)
  • Date, data representing dates Dates are managed through a separate object, the Date dataset (see Dates in Logical Data Models). If you are importing a CSV file that contains dates, not one but two datasets will be added to your LDM: one Date dataset for the dates, and the other one with the rest of information from the CSV file. These two datasets will be automatically connected with a relationship, and the Date dataset will become a reference in that other dataset.

Steps:

  1. On the top navigation bar, select Manage.

  2. Click Data Integration Console on the left. The Data Integration Console for your workspace opens.

  3. Click Model data on the top. The LDM Modeler opens. You see a blank canvas area in view mode.

  4. Click Edit. The LDM Modeler is switched to edit mode.

  5. To add a dataset, drag a CSV file and drop it in the blank canvas area. The data preview opens. The preview shows the data from the file and looks similar to the following: 

     Based on the column names and the contents of those columns in the file, the column names and the types of the data are suggested for the dataset that will be created from this file.

  6. Review the suggested column names and the data types. Update them if needed. For more information about the data types and how to set them correctly, see Create a Logical Data Model from CSV Files.

  7. Once done, click Import. The file is imported, and the dataset is added to the LDM Modeler. Every column in this dataset is mapped to the appropriate column in the file. During data load, the data from a column in the file will be loaded to the corresponding fact or attribute in the dataset. For more information about the mapping, see Mapping between a Logical Data Model and the Data Source.

  8. Repeat Steps 5-7 to add more datasets. For more information about how to add different types of datasets to the LDM, see Create a Logical Data Model from CSV Files.

  9. Update the LDM as needed: add relationships between datasets, modify attributes or facts in the datasets, and so on. For more information, see Update a Logical Data Model.

Your LDM is ready. You can now publish it.

Publish the LDM

To publish the LDM, follow the instructions from Publish a Logical Data Model.