Types of Input Data Sources

The data that the bricks use as input for further processing can be stored in different locations.

The 'input_source' parameter of a brick points to the location from which the input data should be obtained. This parameter accepts the location described by a JSON structure. For information about how to enter parameters represented by JSON, see Specifying Complex Parameters.

Contents:

Staging Area

The expected data format is CSV.

{
  "input_source": {
    "type": "staging",
    "path": "file_name.csv"
  }
}


Example:

{
  "input_source": {
    "type": "staging",
    "path": "users.csv"
  }
}

For more information, see Project Specific Data Storage.

Amazon S3 Bucket

The expected data format is CSV.

{
  "aws_client": {
    "access_key_id" : "your_access_key_ID",
    "secret_access_key" : "your_secret_access_key"
	"region" : "region_of_your_bucket"
  },
  "input_source": {
    "type": "s3",
    "key": "path/to/file.csv",
    "bucket": "bucket_name"
  }
}


Example:

{
  "aws_client": {
    "access_key_id" : "123456789",
    "secret_access_key" : "1a2b3c4d5e"
	"region": "us-east-1"
  },
  "input_source": {
    "type": "s3",
    "key": "/folder/users.csv",
    "bucket": "my_bucket"
  }
}

Data Warehouse

The expected data format is a query to Data Warehouse (DW).

{
  "ads_client": {
    "username": "your_username@domain.com",
    "password": "your_password",
    "ads_id": "DW_instance_id"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT some_data FROM some_table"
  }
}


If your domain is white-labeled, replace 'ads_id' with 'jdbc_url':

{
  "ads_client": {
    "username": "your_username@domain.com",
    "password": "your_password",
    "jdbc_url": "jdbc:gdc:datawarehouse://white_label_hostname/gdc/datawarehouse/instances/DW_instance_id"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT some_data FROM some_table"
  }
}


Example (for not white-labeled domains):

{
  "ads_client": {
    "username": "john.dow@gooddata.com",
    "password": "secret",
    "ads_id": "123456abcdef7890"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT * FROM users_table"
  }
}


Example (for white-labeled domains):

{
  "ads_client": {
    "username": "john.dow@gooddata.com",
    "password": "secret",
    "jdbc_url": "jdbc:gdc:datawarehouse://my.company.com/gdc/datawarehouse/instances/123456abcdef7890"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT * FROM users_table"
  }
}

For more information, see Data Warehouse.

Web Location

The expected data format is CSV.

{
  "input_source": {
    "type": "web",
    "url": "URL/to/file_name.csv"
  }
} 


Example:

{
  "input_source": {
    "type": "web",
    "url": "https://myfilestorage.com/123/456/users.csv"
  }
}