Data Policy
Data policy can be set in some Readers. Here we provide their list:
CSVReader
ParallelReader (Commercial Component)
XLSDataReader
DBFDataReader
DBInputTable
XMLXPathReader
MultiLevelReader (Commercial Component)
When you want to configure these components, you must first decide what should be done when incorrect or incomplete records are parsed. This can be specified with the help of this Data Policy attribute. You have three options according to what data policy you want to select:
Strict. This data policy is set by default. It means that data parsing stops if a record field with an incorrect value or format is read. Next processing is aborted.
Controlled. This data policy means that every error is logged, but incorrect records are skipped and data parsing continues. Generally, incorrect records with error information are logged into
stdout. Only CSVReader enables to sent them out through the optional second port.![[Important]](figures/important.png)
Important If you set the Data policy attribute to
controlledin CSVReader, you need to select the components that should process the information or maybe you only want to write it. You must select an edge and connect the error port of the CSVReader (in which the data policy attribute is set to controlled) with the input port of the selected writer if you only want to write it or with the input port other processing component. And you must assign metadata to this edge. The metadata must be created by hand. They consist of 4 fields:number of incorrect record,number of incorrect field,incorrect record,error message. The first two fields are ofintegerdata type, the other two arestrings. See Creating Metadata by User for detailed information about how metadata should be created by user.
Lenient. This data policy means that incorrect records are only skipped and data parsing continues.