The File URL attribute may be defined using the URL File Dialog.
The URL shown below can also contain placeholders - dollar sign or hash sign.
Important | |||
---|---|---|---|
You need to differentiate between dollar sign and hash sign usage.
|
Important | |
---|---|
To ensure graph portability, forward slashes must be used when defining the path in URLs (even on Microsoft Windows). |
Here we present some examples of possible URL for Writers:
/path/filename.out
Writes specified file on disk.
/path1/filename1.out;/path2/filename2.out
Writes two specified files on disk.
/path/filename$.out
Writes some number of files on disk. The dollar sign
represents one digit. Thus, the output files can have the names
from filename0.out
to
filename9.out
.
The dollar sign is used when Records per file is set.
/path/filename$$.out
Writes some number of files on disk. Two dollar signs
represent two digits. Thus, the output files can have the names
from filename00.out
to
filename99.out
.
The dollar sign is used when Records per file is set.
zip:(/path/file$.zip)
Writes some number of compressed files on disk. The dollar
sign represents one digit. Thus, the compressed output files can
have the names from file0.zip
to
file9.zip
.
The dollar sign is used when Records per file is set.
zip:(/path/file$.zip)#innerfolder/filename.out
Writes specified file inside the compressed files on disk.
The dollar sign represents one digit. Thus, the compressed output
files containing the specified filename.out file can have the
names from file0.zip
to
file9.zip
.
The dollar sign is used when Records per file is set.
gzip:(/path/file$.gz)
Writes some number of compressed files on disk. The dollar
sign represents one digit. Thus, the compressed output files can
have the names from file0.gz
to
file9.gz
.
The dollar sign is used when Records per file is set.
Note | |
---|---|
Although CloudConnect can read data from a
|
ftp://user:password@server/path/filename.out
Writes specified filename.out
file on
remote server connected via ftp protocol using username and
password.
sftp://user:password@server/path/filename.out
Writes specified filename.out
file on
remote server connected via sftp protocol using username and
password.
zip:(ftp://username:password@server/path/file.zip)#innerfolder/filename.txt
Writes specified filename.txt
file
compressed in the file.zip
file on remote
server connected via ftp protocol using username and
password.
zip:(ftp://username:password@server/path/file.zip)#innerfolder/filename.txt
Writes specified filename.txt
file
compressed in the file.zip
file on remote
server connected via ftp protocol.
zip:(zip:(ftp://username:password@server/path/name.zip)#innerfolder/file.zip)#innermostfolder/filename.txt
Writes specified filename.txt
file
compressed in the file.zip
file that is also
compressed in the name.zip
file on remote
server connected via ftp protocol using username and
password.
gzip:(ftp://username:password@server/path/file.gz)
Writes the first file compressed in the
file.gz
file on remote server connected via
ftp protocol.
http://username:password@server/filename.out
Writes specified filename.out
file on
remote server connected via WebDAV protocol using username and
password.
http://access_key_id:secret_access_key@bucketname.s3.amazonaws.com/filename.out
Writes specified filename.out
file on
Amazon S3 web storage service to the bucket bucketname
using the access_key_id
as the of ID of access key and
secret_access_key
as the personal access key.
port:$0.FieldName:discrete
If this URL is used, output port of the
Writer must be connected to another
component. Output metadata must contain a
FieldName
of one of the following data types:
string
, byte
or
cbyte
. Each data record that is received by the
Writer through the input port is processed
according to the input metadata, sent out through the optional
output port, and written as the value of the specified field of
the metadata of the output edge. Next records are parsed in the
same way as described here.
-
Writes data to stdout
.
http:(direct:)//seznam.cz
Without proxy.
http:(proxy://user:password@212.93.193.82:443)//seznam.cz
Proxy setting for http protocol.
ftp:(proxy://user:password@proxyserver:1234)//seznam.cz
Proxy setting for ftp protocol.
ftp:(proxy://proxyserver:443)//server/path/file.dat
Proxy setting for ftp protocol.
sftp:(proxy://66.11.122.193:443)//user:password@server/path/file.dat
Proxy setting for sftp protocol.
dict:keyName:discrete
1)
Writes data to dictionary. Creates
ArrayList<byte[]>
dict:keyName:stream
2)
Writes data to dictionary. Creates
WritableByteChannel
Legend:
1): The discrete
processing type uses byte
array for storing data.
2): The stream
processing type uses an output
stream that must be created before running a graph (from Java
code).
A sandbox resource, whether it is a shared, local or partitioned sandbox, is specified in the graph under the fileURL attributes as a so called sandbox URL like this:
sandbox://data/path/to/file/file.dat
where "data" is code for sandbox and "path/to/file/file.dat" is the path to the resource from the sandbox root. URL is evaluated by CloudConnect Server during graph execution and a component (reader or writer) obtains the opened stream from the server. This may be a stream to a local file or to some other remote resource. Thus, a graph does not have to run on the node which has local access to the resource. There may be more sandbox resources used in the graph and each of them may be on a different node. In such cases, CloudConnect Server would choose the node with the most local resources to minimalize remote streams.
The sandbox URL has a specific use for parallel data processing. When the sandbox URL with the resource in a partitioned sandbox is used, that part of the graph/phase runs in parallel, according to the node allocation specified by the list of partitioned sandbox locations. Thus, each worker has its own local sandbox resource. CloudConnect Server evaluates the sandbox URL on each worker and provides an open stream to a local resource to the component.