Supported File URL Formats for Writers

Home \| Table of Contents	Supported File URL Formats for Writers
Prev	Common Properties of Writers	Next

The File URL attribute may be defined using the URL File Dialog.

The URL shown below can also contain placeholders - dollar sign or hash sign.

Important

You need to differentiate between dollar sign and hash sign usage.

Dollar sign should be used when each of multiple output files contains only a specified number of records based on the Records per file attribute.
Hash sign should be used when each of multiple output files only contains records correspoding to the value of specified Partition key.

	Note
	Hash signs in URL examples in this section serve to separate a compressed file (`zip`, `gz`) from its contents. These are not placeholders!

	Important
	To ensure graph portability, forward slashes must be used when defining the path in URLs (even on Microsoft Windows).

Here we present some examples of possible URL for Writers:

Writing to Local Files

/path/filename.out
Writes specified file on disk.
/path1/filename1.out;/path2/filename2.out
Writes two specified files on disk.
/path/filename$.out
Writes some number of files on disk. The dollar sign represents one digit. Thus, the output files can have the names from filename0.out to filename9.out. The dollar sign is used when Records per file is set.
/path/filename$$.out
Writes some number of files on disk. Two dollar signs represent two digits. Thus, the output files can have the names from filename00.out to filename99.out. The dollar sign is used when Records per file is set.
zip:(/path/file$.zip)
Writes some number of compressed files on disk. The dollar sign represents one digit. Thus, the compressed output files can have the names from file0.zip to file9.zip. The dollar sign is used when Records per file is set.
zip:(/path/file$.zip)#innerfolder/filename.out
Writes specified file inside the compressed files on disk. The dollar sign represents one digit. Thus, the compressed output files containing the specified filename.out file can have the names from file0.zip to file9.zip. The dollar sign is used when Records per file is set.
gzip:(/path/file$.gz)
Writes some number of compressed files on disk. The dollar sign represents one digit. Thus, the compressed output files can have the names from file0.gz to file9.gz. The dollar sign is used when Records per file is set.

	Note
	Although CloudConnect can read data from a `.tar` file, writing to a `.tar` file is not supported.

Writing to Remote Files

ftp://user:password@server/path/filename.out
Writes specified filename.out file on remote server connected via ftp protocol using username and password.
sftp://user:password@server/path/filename.out
Writes specified filename.out file on remote server connected via sftp protocol using username and password.
zip:(ftp://username:password@server/path/file.zip)#innerfolder/filename.txt
Writes specified filename.txt file compressed in the file.zip file on remote server connected via ftp protocol using username and password.
zip:(ftp://username:password@server/path/file.zip)#innerfolder/filename.txt
Writes specified filename.txt file compressed in the file.zip file on remote server connected via ftp protocol.
zip:(zip:(ftp://username:password@server/path/name.zip)#innerfolder/file.zip)#innermostfolder/filename.txt
Writes specified filename.txt file compressed in the file.zip file that is also compressed in the name.zip file on remote server connected via ftp protocol using username and password.
gzip:(ftp://username:password@server/path/file.gz)
Writes the first file compressed in the file.gz file on remote server connected via ftp protocol.
http://username:password@server/filename.out
Writes specified filename.out file on remote server connected via WebDAV protocol using username and password.
http://access_key_id:secret_access_key@bucketname.s3.amazonaws.com/filename.out
Writes specified filename.out file on Amazon S3 web storage service to the bucket bucketname using the access_key_id as the of ID of access key and secret_access_key as the personal access key.

Writing to Output Port

port:$0.FieldName:discrete
If this URL is used, output port of the Writer must be connected to another component. Output metadata must contain a FieldName of one of the following data types: string, byte or cbyte. Each data record that is received by the Writer through the input port is processed according to the input metadata, sent out through the optional output port, and written as the value of the specified field of the metadata of the output edge. Next records are parsed in the same way as described here.

Writing to Console

-
Writes data to stdout.

Using Proxy in Writers

http:(direct:)//seznam.cz
Without proxy.
http:(proxy://user:password@212.93.193.82:443)//seznam.cz
Proxy setting for http protocol.
ftp:(proxy://user:password@proxyserver:1234)//seznam.cz
Proxy setting for ftp protocol.
ftp:(proxy://proxyserver:443)//server/path/file.dat
Proxy setting for ftp protocol.
sftp:(proxy://66.11.122.193:443)//user:password@server/path/file.dat
Proxy setting for sftp protocol.

Writing to Dictionary

dict:keyName:discrete¹⁾
Writes data to dictionary. Creates ArrayList<byte[]>
dict:keyName:stream²⁾
Writes data to dictionary. Creates WritableByteChannel

Legend:

1): The discrete processing type uses byte array for storing data.

2): The stream processing type uses an output stream that must be created before running a graph (from Java code).

Sandbox Resource as Data Source

A sandbox resource, whether it is a shared, local or partitioned sandbox, is specified in the graph under the fileURL attributes as a so called sandbox URL like this:

sandbox://data/path/to/file/file.dat

where "data" is code for sandbox and "path/to/file/file.dat" is the path to the resource from the sandbox root. URL is evaluated by CloudConnect Server during graph execution and a component (reader or writer) obtains the opened stream from the server. This may be a stream to a local file or to some other remote resource. Thus, a graph does not have to run on the node which has local access to the resource. There may be more sandbox resources used in the graph and each of them may be on a different node. In such cases, CloudConnect Server would choose the node with the most local resources to minimalize remote streams.

The sandbox URL has a specific use for parallel data processing. When the sandbox URL with the resource in a partitioned sandbox is used, that part of the graph/phase runs in parallel, according to the node allocation specified by the list of partitioned sandbox locations. Thus, each worker has its own local sandbox resource. CloudConnect Server evaluates the sandbox URL on each worker and provides an open stream to a local resource to the component.