StructuredDataWriter
We assume that you have already learned what is described in:
If you want to find the right Writer for your purposes, see Writers Comparison.
Short Summary
StructuredDataWriter writes data to files with user-defined structure.
| Component | Data output | Input ports | Output ports | Transformation | Transf. required | Java | CTL |
|---|---|---|---|---|---|---|---|
| StructuredDataWriter | structured flat file | 1-3 | 0-1 | no | no | no | no |
Abstract
StructuredDataWriter writes data to files (local or remote, delimited, fixed-length, or mixed) with user-defined structure. It can also compress output files, write data to console, output port, or dictionary.
Icon
Ports
| Port type | Number | Required | Description | Metadata |
|---|---|---|---|---|
| Input | 0 | yes | Records for body | Any |
| 1 | no | Records for header | Any | |
| 2 | no | Records for footer | Any | |
| Output | 0 | no | For port writing. See Writing to Output Port. | One field (byte,
cbyte, string). |
StructuredDataWriter Attributes
| Attribute | Req | Description | Possible values |
|---|---|---|---|
| Basic | |||
| File URL | yes | Attribute specifying where received data will be written (flat file, console, output port, dictionary). See Supported File URL Formats for Writers. | |
| Charset | Encoding of records written to the output. | ISO-8859-1 (default) | <other encodings> | |
| Append | By default, new records overwrite the older ones. If
set to true, new records are appended to
the older records stored in the output file(s). | false (default) | true | |
| Body mask | Mask used to write the body of the output file(s). It can be based on the records received through the first input port. See Masks and Output File Structure for more information about definition of Body mask and resulting output structure. | Default Body Structure (default) | user-defined | |
| Header mask | 1) | Mask used to write the header of the output file(s). It can be based on the records received through the second input port. See Masks and Output File Structure for more information about definition of Header mask and resulting output structure. | empty (default) | user-defined |
| Footer mask | 2) | Mask used to write the footer of the output file(s). It can be based on the records received through the third input port. See Masks and Output File Structure for more information about definition of Footer mask and resulting output structure. | empty (default) | user-defined |
| Advanced | |||
| Create directories | By default, non-existing directories are not created.
If set to true, they are created. | false (default) | true | |
| Records per file | Maximum number of records to be written to one output file. | 1-N | |
| Bytes per file | Maximum size of one output file in bytes. | 1-N | |
| Number of skipped records | Number of records to be skipped. See Selecting Output Records. | 0-N | |
| Max number of records | Maximum number of records to be written to all output files. See Selecting Output Records. | 0-N | |
| Partition key | Key whose values define the distribution of records among multiple output files. See Partitioning Output into Different Output Files for more information. | ||
| Partition lookup table | 1) | ID of lookup table serving for selecting records that should be written to output file(s). See Partitioning Output into Different Output Files for more information. | |
| Partition file tag | By default, output files are numbered. If it is set to
Key file tag, output files are named
according to the values of Partition key
or Partition output fields. See Partitioning Output into Different Output Files for more
information. | Number file tag (default) | Key file tag | |
| Partition output fields | 1) | Fields of Partition lookup table whose values serve to name output file(s). See Partitioning Output into Different Output Files for more information. | |
| Partition unassigned file name | Name of the file into which the unassigned records should be written if there are any. If not specified, data records whose key values are not contained in Partition lookup table are discarded. See Partitioning Output into Different Output Files for more information. | ||
Legend:
1) Must be specified if second input port is connected. However, does not need to be based on input data records.
2) Must be specified if third input port is connected. However, does not need to be based on input data records.
Advanced Description
Masks and Output File Structure
Output File Structure
Output file consists of header, body, and footer, in this order.
Each of them is defined by specifying corrresponding mask.
After defining the mask, the mask content is written repeatedly, one mask is written for each incoming record.
However, if the Records per file attribute is defined, the output structure is distributed among various output files, but this attribute applies for Body mask only. Header and footer are the same for all output files.
Defining a Mask
Body mask, Header mask, and Footer mask can be defined in the Mask dialog. This dialog opens after clicking corresponding attribute row. In its window, you can see the Metadata and Mask panes. At the bottom, there is a button.
You can define the mask either without field values or with field values.
Field values are expressed using field names preceded by dollar sign.
If you click the button, a simple XML structure appears in the Mask pane.
You only need to remove the fields you do not want to save to the output file and you can also rename the suggested left side of the matchings. These have the form of matchings like this:
<sometag=$metadatafield/>. By default after clicking the button, you will obtain the XML structure containing expressions like this:<metadatafield=$metadatafield/>. Left side of these matchings can be replaced by any other, but the right side must remain the same. You must not change the field names preceded by a dollar sign on the right side of the matchings. They represent the values of fields.Remember that you do not need to use any XML file as a mask. The mask can be of any other structure.
Default Masks
Default Header mask is empty. But it must be defined if second input port is connected.
Default Footer mask is empty. But it must be defined if third input port is connected.
Default Body mask is empty. However, the resulting default body structure looks like the following:
< recordName field1name=field1value field2name=field2value ... fieldNname=fieldNvalue />This structure is written to the output file(s) for all records.
If Records per file is set, only the specified number of records are used for body in each output file at most.
