CloverETL product family can easily fit different usage scenarios:Open source CloverETL Engine for very small or hobby projectsStandalone CloverETL Designer for small projectsCloverETL Server (includes Designer) for medium to large projectsCloverETL Enterprise Server for large projects, optionally can support clustering for even better performance
Each metadata defines a record structure which is used when parsing the data in reader components or writing the output in writer components. To make the work easier, each record has its own name. Note that the name does not have to be unique within a graph – Clover uses internal identifiers (metadata id) to distinguish between different records with the same name. It is however strongly recommended that the record names are unique to prevent confusion during the development of larger graphs.Each record name has to be a valid identifier and therefore can only contain letters, numbers and underscores. Record names are case-sensitive when used in code.Each record can contain any number of fields of various types. Fields only have simple types and it is not possible to nest records into each other like in Java or other popular languages. Each field has to have a name which is unique within the record. Field names have to be identifiers as well and therefore they have to conform to the same rules as record names.
Record types:Delimited: whole record defines a delimiter between records and each field can have its own delimiter which separates it from the next field. Clover supports delimiters with multiple characters and each delimiter can be different.Fixed-length: each field has predefined width and no delimiters are used.Mixed: some of the fields are delimited and some of the fields have fixed length.Field types:boolean: simple true/false valueinteger: signed integer number, 32-bit (minimum value is -2 147 483 648; maximum is 2 147 483 647)long: signed integer number, 64-bit (minimum is -9 223 372 036 854 775 808; maximum is -9 223 372 036 854 775 807)number: a floating-point number (64-bit IEEE 754 double precision, same as Java double data type).string: a character string. All strings are unicode and are represented in UTF-16. The maximum length of a string is 2^32-1 (the maximum value of integer).date: represents a date with millisecond precision. Note that it is possible to specify the date formatting via custom format string. Clover supports two libraries for date formatting – built-in Java standard library and Joda time library. More details about the formatting options provided by these libraries can be found online:Built-in Java library: formatting performed by java.util.SimpleDateFormat class, online documentation at http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.htmlJoda time: formatting performed by org.joda.time.format.DateTimeFormatter class with configuration specified as DateTimeFormat class, online documentation for formatting strings can be found at http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.htmldecimal: a fixed-precision number with configurable precision. Two parameters – length and scale – define the total number of digits and the number of digits after decimal point.byte, cbyte: array of bytes. cbyte is compressed in memory so it can be used for larger data.It is also possible to use containers for field values. Clover supports lists and maps of primitive data types. For maps, the map is always map[string, X] where X is the data type specified in the metadata. Note that not all components support these types in full (for example it is not possible to use them as keys for sorting etc.).
Programmable components in CloverETL all use set of functions which are called in a specific order when processing the data. In general, the components contain initialization functions (init and preExecute) which are called before the record processing starts. Then usually one “main” function (like transform or generate) which is called once for each incoming record. Finally, after all data is processed, postExecute is called.Only the main function is mandatory – other functions (initialization and post-processing) are optional and you do not need to provide their implementations. All onError functions (like transformOnError) are optional as well – if they are not specified, the processing fails whenever an error is encountered.Some of the components (e.g. RollUp) contain multiple functions which are called for each record. See later modules for more details on advanced programming in Clover.The function prototypes are automatically created for you when you create new transformation (i.e. when you first try to open transformation code after adding a component to the graph). Optional functions are commented out in the source and you can uncomment them if needed.
Onlytransform and append functions are mandatory. All other functions mentioned on the slide can be left unimplemented.Note: it is not possible to access fields from the input record in transform function. The transformation will crash if you attempt to use anything from the input port. It is therefore necessary to store all the data you need in a variable. For example, it is possible to create instance of a record and copy the data via wildcard mappings:InputMetadata temp;temp.* = $in.0.*;InputMetadata is of course name of the metadata coming through the input port.Note: it is not possible to access output ports in append function.