Types of Input Data Sources

The data that the bricks use as input for further processing can be stored in different locations.

The input_source parameter of a brick points to the location from which the input data should be obtained. This parameter accepts the location described by a JSON structure. For information about how to enter parameters represented by JSON, see Specifying Complex Parameters.

Contents:

Project Specific Data Storage

The expected data format is a CSV file.

For more information about the data storage, see Project Specific Data Storage.

{
  "input_source": {
    "type": "staging",
    "path": "{file_name}.csv"
  }
}


Example:

{
  "input_source": {
    "type": "staging",
    "path": "users.csv"
  }
}

GoodData Data Warehouse (ADS)

The expected data format is a query to GoodData Data Warehouse (ADS).

For more information, see Data Warehouse.

{
  "ads_client": {
    "username": "{your_username@domain.com}",
    "password": "{your_password}",
    "ads_id": "{ads_instance_id}"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT {some_data} FROM {some_table}"
  }
}


If your domain is white-labeled, replace ads_id with jdbc_url:

{
  "ads_client": {
    "username": "{your_username@domain.com}",
    "password": "{your_password}",
    "jdbc_url": "jdbc:gdc:datawarehouse://{your.domain.com}/gdc/datawarehouse/instances/{ads_instance_id}"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT {some_data} FROM {some_table}"
  }
}


Example (for not white-labeled domains):

{
  "ads_client": {
    "username": "john.doe@gooddata.com",
    "password": "secret",
    "ads_id": "123456abcdef7890"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT * FROM users"
  }
}


Example (for white-labeled domains):

{
  "ads_client": {
    "username": "john.doe@gooddata.com",
    "password": "secret",
    "jdbc_url": "jdbc:gdc:datawarehouse://my.company.com/gdc/datawarehouse/instances/123456abcdef7890"
  },
  "input_source": {
    "type": "ads",
    "query": "SELECT * FROM users"
  }
}

Snowflake Warehouse

The expected data format is a query to your Snowflake data warehouse.

{
  "snowflake_client": {
    "connection": {
      "url": "jdbc:snowflake://{your_account}.snowflakecomputing.com",
      "authentication": {
        "basic": {
          "userName": "{your_username}",
          "password": "{your_password}"
        }
      },
      "database": "{database_name}",
      "schema": "{schema_name}",
      "warehouse": "{warehouse_name}"
    }
  },
  "input_source": {
    "type": "snowflake",
    "query": "SELECT {some_data} FROM {some_table}"
  }
}


Example:

{
  "snowflake_client": {
    "connection": {
      "url": "jdbc:snowflake://acme.snowflakecomputing.com",
      "authentication": {
        "basic": {
          "userName": "GOODDATA_INTEGRATION",
          "password": "secret"
        }
      },
      "database": "ACME",
      "schema": "ANALYTICS",
      "warehouse": "GOODDATA_INTEGRATION"
    }
  },
  "input_source": {
    "type": "snowflake",
    "query": "SELECT * FROM users"
  }
}

Google BigQuery Warehouse

The expected data format is a query to your Google BigQuery data warehouse.

{  
  "bigquery_client": {
    "connection": {
      "authentication": {
        "serviceAccount": {
          "clientEmail": "{your_username@domain.com}",
          "privateKey": "{your_private_key}"
        }
      },
      "schema": "{schema_name}",
      "project": "{project_name}"
    }    
  },
  "input_source": {
    "type": "bigquery",
    "query": "SELECT {some_data} FROM {some_table}"
  }
}


Example:

{  
  "bigquery_client": {
    "connection": {
      "authentication": {
        "serviceAccount": {
          "clientEmail": "john.doe@gooddata.com",
          "privateKey": "secret"
        }
      },
      "schema": "ANALYTICS",
      "project": "ACME"
    }    
  },
  "input_source": {
    "type": "bigquery",
    "query": "SELECT * FROM users"
  }
}

Amazon Redshift Warehouse

The expected data format is a query to your Amazon Redshift data warehouse.

Basic authentication:

{
  "redshift_client": {
    "connection": { 
      "url": "jdbc:redshift://{your_account}.amazonaws.com",
      "authentication": {
        "basic": {
          "userName": "{your_username}",
          "password": "{your_password}"
        }
      },
      "database": "{database_name}",
      "schema": "{schema_name}"
    }
  },
  "input_source": {
    "type": "redshift",
    "query": "SELECT {some_data} FROM {some_table}"
  }
}


IAM authentication:

{
  "redshift_client": {
    "connection": { 
      "url": "jdbc:redshift://{your_account}.amazonaws.com",
      "authentication": {
        "iam": {
          "dbUser": "{your_username}",
          "accessKeyId": "{your_access_key}",
          "secretAccessKey": "{your_secret_access_key}"
        }
      },
      "database": "{database_name}",
      "schema": "{schema_name}"
    }
  },
  "input_source": {
    "type": "redshift",
    "query": "SELECT {some_data} FROM {some_table}"
  }
}


Example (for basic authentication):

{
  "redshift_client": {
    "connection": { 
      "url": "jdbc:redshift://examplecluster.us-west-2.redshift.amazonaws.com",
      "authentication": {
        "basic": {
          "userName": "GOODDATA_INTEGRATION",
          "password": "secret"
        }
      },
      "database": "ACME",
      "schema": "ANALYTICS"
    }
  },
  "input_source": {
    "type": "redshift",
    "query": "SELECT * FROM users"
  }
}

Amazon S3 Bucket

The expected data format is a CSV file.

{
  "aws_client": {
    "access_key_id": "{your_access_key_ID}",
    "secret_access_key": "{your_secret_access_key}",
	"region": "{region_of_your_bucket}"
  },
  "input_source": {
    "type": "s3",
    "key": "/path/to/file.csv",
    "bucket": "{bucket_name}"
  }
}


Example:

{
  "aws_client": {
    "access_key_id" : "123456789",
    "secret_access_key" : "secret",
	"region": "us-east-1"
  },
  "input_source": {
    "type": "s3",
    "key": "/folder/users.csv",
    "bucket": "users_data"
  }
}

Web Location

The expected data format is a CSV file.

{
  "input_source": {
    "type": "web",
    "url": "URL/to/file_name.csv"
  }
} 


Example:

{
  "input_source": {
    "type": "web",
    "url": "https://myfilestorage.com/data/upload/users.csv"
  }
}