FastSort
Commercial Component
We assume that you have already learned what is described in:
If you want to find the right Transformer for your purposes, see Transformers Comparison.
Short Summary
FastSort sorts input records using a sort key. FastSort is faster than ExtSort but requires more system resources.
| Component | Same input metadata | Sorted inputs | Inputs | Outputs | Java | CTL |
|---|---|---|---|---|---|---|
| FastSort | - |  | 1 | 1-N | - | - |
Abstract
FastSort is a high performance sort component reaching the optimal efficiency when enough system resources are available. FastSort can be up to 2.5 times faster than ExtSort but consumes significantly more memory and temporary disk space.
The component takes input records and sorts them using a sorting key - a single field or a set of fields. You can specify sorting order for each field in the key separately. The sorted output is sent to all connected ports.
Pretty good results can be obtained with the default settings (just the sorting key needs to be specified). However, to achieve the best performance, a number of parameters is available for tweaking.
Icon
Ports
| Port type | Number | Required | Description | Metadata |
|---|---|---|---|---|
| Input | 0 |  | for input data records | the same input and output metadata [ 1)] |
| Output | 0 | | for sorted data records | |
| 1-N | | for sorted data records | ||
[ 1)] As all output metadata must be same as the input metadata, they can be propagated through this component. | ||||
FastSort Attributes
| Attribute | Req | Description | Possible values |
|---|---|---|---|
| Basic | |||
| Sort key | | List of fields (separated by semicolon) the data records are to be sorted by, including the sorting order for each data field separately, see Sort Key | |
| Estimated record count | [ 1)] | Estimated number of input records to be sorted. A rough guess of the order of magnitude is sufficient, see Estimated Record Count. | auto (default) | 1-N |
| In memory only | If true, internal
sorting is forced and all attributes except Sort
key and Run size are ignored. | false (default) | true | |
| Temp directories | List of paths to temporary directories for storing sorted runs, separated by semicolons | system TEMP dir (default) | other dir | |
| Advanced | |||
| Run size (records) | [ 1)] [ 2)] | Number of records sorted at once in memory. Largely affects speed and memory requirements, see Run Size | auto from (if set) Estimated record count | 20,000 default | 1000 - N |
| Max open files | Limits the number of temp files that can be created during the sorting. Too low number (500 or less) significantly reduces the performance, see Max Open Files. | unlimited (default) | 1-N | |
| Concurrency (threads) | Number of worker threads to do the job. The default value ensures the optimal results while overriding the default may even slow the graph run down, see Concurrency. | auto (default) | 1-N | |
| Number of read buffers | [ 2)] | How many chunks of data will be held in memory at a time, see Number of Read Buffers. | auto (default) | 1-N |
| Average record size (bytes) | [ 2)] | Guess on average byte size of records, see Average Record Size. | auto (default) | 1-N |
| Maximum memory (MB, GB) | [ 2)] | Rough estimate of maximum memory that can be used, see Maximum Memory. | auto (default) | 1-N |
| Tape buffer (bytes) | Buffer used by a worker for filling the output. Affects the performance slightly, see Tape Buffer. | 8192 (default) | 1-N | |
| Compress temporary files | If true, temporary files are
compressed. For more information see Compress Temporary Files. | false (default) | true | |
| Deprecated | |||
| Sorting locale | Locale used for correct sorting order | none (default) | any locale | |
| Case sensitive | By default (Sorting locale is none), upper-case
characters are sorted separately and precede lower-case characters
that are sorted separately too. If Sorting locale is set,
upper- and lower-case characters are sorted together - if Case sensitive
is true, a lower-case precedes corresponding
upper-case while false preservers the order, data strings appears in the input in. | false (default) | true | |
[ 1)] Estimated record count is a helper attribute which is used for calculating (rather unnatural) Run size automatically as approximately Estimated record count to the power 0.66. If Run size set explicitly, Estimated record count is ignored. Reasonable Run sizes vary from 5,000 to 200,000 based on the record size and the total number of records.[ 2)] These attributes affect automatic guess of Run size. Generally, the following formula must be true: Number of read buffers * Average record size < Maximum memory | |||
Advanced Description
Sorting Null Values
Remember that FastSort processes the
records in which the same fields of the Sort key
attribute have null values as if these
nulls were equal.
FastSort Tweaking
Basically, you do not need to set any of these attributes, however, sometimes you can increase performance by setting them. You may have a limited memory or you need to sort agreat number of records, or these records are too big. In similar cases, you can fit FastSort to your needs.
Basic attribute which lets FastSort know a rough number of records it will have to deal with. The attribute is complementary to Run size; you don't need to set it if Run size is specified. On the other hand, if you don't want to play with attributes setting much, giving the rough number of records spares memory to be allocated during the graph run. Based on this count, Maximum memory, records size, etc., Run size is determined.
The core attribute for FastSort; determines how many records form a "run" (i.e., a bunch of sorted records in temp files). The less Run size, the more temp files get created, less memory is used and greater speed is achieved. On the other hand, higher values might cause memory issues. There is no rule of thumb as to whether Run size should be high or low to get the best performance. Generally, the more records you are about to sort the bigger Run size you might want. The rough formula for Run size is Estimated record count^0.66. Note that memory consumption multiplies with Number of read buffers and Concurrency. So, higher Run sizes result in much higher memory footprints.
FastSort uses relatively large numbers of temporary files during its operation. In case you hit quota or OS-specific limits, you can limit the maximum number of files to be created. The following table should give you a better idea:
Dataset size Number of temp. files Default Run size Note 1,000,000 ~100 ~10,000 10,000,000 ~250 ~45,000 1,000,000,000 20,000 to 2,000 50,000 to 500,000 Depends on available memory Note that numbers in the table above are not exact and might be different on your system. However, sometimes such large numbers of files might cause problems hitting user quotas or other runtime limitations, see Performance Bottlenecks for a help how to solve such issues.
Tells FastSort how many runs (chunks) should be sorted at a time in parallel. By default, it is automatically set to 1 or 2 based on the number of CPU cores in your system. Overriding this value makes sense if your system has lots of CPU cores and you think your disk performance can handle working with so many parallel data streams.
You can set the maximum amount of memory dedicated to a single component. This is a guide for FastSort when computing Run size, i.e., if Run size is set explicitly, this setting is ignored. A unit must be specified, e.g., '200MB', '1gb', etc.
You can set Average record size in bytes. If omitted, it will be computed as an average record size from the first 1000 parsed records.
This setting corresponds tightly to the number of threads (Concurrency) - must be equal to or greater than Concurrency. The more read buffers the less change the workers will block each other. Defaults to Concurrency + 2
Along with Temporary files charset this option lets you reduce the space required for temporary files. Compression can save a lot of space but affects performance by up to 30% down so be careful with this setting.
Size (in bytes) of a file output buffer. The default value is 8kB. Decreasing this value might avoid memory exhaustion for large numbers of runs (e.g. when Run size is very small compared to the total number of records). However, the impact of this setting is quite small.
Tips & Tricks
Be sure you have dedicated enough memory to your Java Virtual Machine (JVM). Having plenty of memory available, FastSort is capable of doing astonishing job. Remember that the default JVM heap space 64MB can cause FastSort to crash. Don't hesitate to increase the memory value up to 2 GB (but still leaving some memory for the operating system). It is well worth it. How to set the JVM is described in Program and VM Arguments section.
Performance Bottlenecks
Sorting big records (long string fields, tens or hundreds of fields, etc.): FastSort is greedy for both memory and CPU cores. If the system does not have enough of either, FastSort can easily crash with out-of-memory. In this case, use the ExtSort component instead.
Utilizing more than 2 CPU cores: Unless you you are able to use really fast disk drives, overriding the default value of Concurrency to more than 2 threads does not neccessarily help. It can even slow the process back down a bit as extra memory is loaded for each additional thread.
Coping with quotas and other runtime limitations: In complex graphs with several parallel sorts, even with other graph components also having huge number of open files,
Too many open fileserror and graph execution failure may occur. There are two possible solutions to this issue:increase the limit (quota)
This option is recommended for production systems since there is no speed sacrifice. Typically, setting limit to higher number on Unix systems.
force FastSort to keep the number of temporary files below some limit
For regular users on large servers increasing the quota is not an option. Thus, Max open files must be set to a reasonable value. FastSort then performs intermediate merges of temporary files to keep their number below the limit. However, setting Max open files to values, for which such merges are inevitable, often produces significant performance drop. So keep it at the highest possible value. If you are forced to limit FastSort to less than a hundred temporary files, even for large datasets, consider using ExtSort instead which is designed for performance with limited number of tapes.