http://www.infiniflux.com/download Append Protocol (High-Speed Input Protocol)

1. Append Protocol
(High-Speed Input Protocol)
www.infiniflux.com

2. Table of Contents
Background
CLI API
JDBC API
Multi Table Insert
1
2
3
4

3. Background

4. 4
Background– Limitations of Conventional Protocol
Client
Server
Idle time
……
• Limitations of Conventional Synchronous Protocol (Request - Reply):
Synchronous protocol is to check the results of executions each time a record was entered. Thus;
• Rapidly increase the number of I/O processing in accordance with the number of records
• Must check the result of a record each time
• Difficult to insert more than 1,000 data per second
Client
Client
Client
Server
Server
Idle time
Idle time

5. 5
Background – Improved Protocol
Flush Append
Flush Append
Append Protocol Diagram
• Introduced Optimistic Asynchronous Protocol (Append Protocol)
• Pile up input records on the buffer consecutively and flush them when the buffer is full.
• Not returning results if append succeeds.
• Return error code and message to the client when append was failed.
• Able to insert more than millions of records per second
• Most of cases are successfully inserted, and return errors rarely. Thus, the improved protocol is the best alternative.
Client
Client
Client
No Idle time
No Idle time
Server
Server
Server

6. 6
Background –API Structure
Title Contents
API for establishing communication
channels
• Open a channel for appending
• Specify a table where data will be inserted
• Insert data at a high-speed through this channel
API for data insert
• Binding column data that conform to the given table schema
• Return errors when the length and format are not matched
• Guarantee of fast data transfer through buffering
API for data flush
• Force to send the data through network even when the buffer was not
filled.
API for setting an error callback
• Setting up asynchronous user-callback in order to process error code
and message received from the server
• Asynchronously operate with data input
API for closing communication channel
• Close the protocol that is currently appending
• Send all the records left on the buffer through network

7. CLI API

8. 8
Confirming CLI Installation
• Makefile
• Header file
• Library file
Check the files, shown below, in the “install”, “include”, and “lib” directories where the InfiniFlux has been
installed.
$IFLUX_HOME/install/iflux_env.mk
$IFLUX_HOME/include/iflux_sqlcli.h
$IFLUX_HOME/lib/libifluxcli.a
$IFLUX_HOME/lib/libifluxcli_dll.so

9. 9
Makefile & Compile
• Move to “sample/cli” directory.
• Check the contents of Makefile.
include $(IFLUX_HOME)/install/iflux_env.mk
INCLUDES += $(LIBDIR_OPT)/$(IFLUX_HOME)/include
all : sample4_append2
sample4_append2 : sample4_append2.o
$(LD_CC) $(LD_FLAGS) $(LD_OUT_OPT)$@ $< $(LIB_OPT)ifluxcli$(LIB_AFT) $(LIBDIR_OPT)$(IFLUX_HOME)/lib $(LD_LIBS)
sample4_append2.o : sample4_append2.c
$(COMPILE.cc) $(CC_FLAGS) $(INCLUDES) $(CC_OUT_OPT)$@ $<
clean :
rm -f *.o sample4_append2
• Compile it.
Check and compile “$IFLUX_HOME/sample/cli/makefile”.
$ cd $IFLUX_HOME/sample/cli
$ make

10. 10
Structure of Append Program
SQLAllocEnv
SQLAllocConnect
SQLDriverConnect
SQLAllocStmt
SQLAppendOpen
SQLAppendDataV2
SQLAppendClose
SQLAppendSetErrorCallback
SQLFreeStmt
SQLDisconnect
SQLFreeConnect
SQLFreeEnv
Extended Functions of InfiniFlux
CLI Standard Connect Functions
CLI Standard Disconnect Functions
Description of “$IFLUX_HOME/sample/cli/sample4_append1.c“

11. 11
SQLAppendOpen
• Open a channel for inserting data into a specified table.
• The channel remains open if the channel is not closed.
• Able to set up to 1024 statements for each connection.
• Use SQLAppendOpen for each statement.
SQLRETURN SQLAppendOpen( SQLHSTMT aStatementHandle,
SQLCHAR *aTableName,
SQLINTEGER aErrorCheckCount );
• aStatementHandle : statement handle to append
• aTableName : name of target table to append
• aErrorCheckCount : set the number of evaluations to detect server errors based on the number of inputs.
If the number is 0, evaluation is not required.

12. 12
SQLAppendDataV2
• Function to insert data for the channel.
• Supported from the InfiniFlux v2.0.
• Able to insert data types of both text and binary.
SQLRETURN SQLAppendDataV2( SQLHSTMT aStatementHandle,
SQL_APPEND_PARAM *aData );
• aStatementHandle: statement handle to append.
• aData : a pointer to parameter array, “SQL_APPEND_PARAM”
Defined “SQL_APPEND_PARAM” in the “iflux_sqlcli.h”

13. 13
SQLAppendClose
• Close the current channel.
• Able to check the number of append success or failure.
SQLRETURN SQLAppendClose( SQLHSTMT aStatementHandle,
int *aSuccessCount,
int *aFailureCount );
• aStatementHandle: statement handle to append
• aSuccessCount : the number of successful append
• aFailureCount : the number of failed append

14. 14
SQLAppendSetErrorCallback
• Setting up the callback function to be called when an error was occurred while appending.
• If this function was not set, the client will ignore even if an error was occurred in the server.
SQLRETURN SQLAppendSetErrorCallback( SQLHSTMT aStatementHandle,
SQLAppendErrorCallback *aFunc );
• aStatementHandle: statement handle to append
• aFunc : a pointer to callback function when an error occurs.
typedef void (*SQLAppendErrorCallback)( SQLHSTMT aStatementHandle,
SQLINTEGER aErrorCode,
SQLPOINTER aErrorMessage,
SQLLEN aErrorBufLen,
SQLPOINTER aRowBuf,
SQLLEN aRowBufLen );
• aStatementHandle: statement handle to append
• aErrorCode : 32 bit error code which causes the error
• aErrorMessage : the string for the error code
• aErrorBufLen : the length of “aErrorMessage”
• aRowBuf : the string that has the content of the record that caused the error
• aRowBufLen : the length of “aRowBuf”

15. 15
Other Functions
• Additionally, provide useful functions.
• Refer to website at http://www.infiniflux.com/document for more information.
Name of Functions Function Declaration Description
SQLAppendDataByTimeV2
SQLRETURN SQL_API
SQLAppendDataByTimeV2(
SQLHSTMT aStatmentHandle,
SQLBIGINT aTime,
SQL_APPEND_PARAM *aData)
Like “SQLAppendDataV2()”, it inserts data into a table. However,
the value of hidden column, “_arrival_time”, will be used when
it was set to a value of a certain time rather than the present
time. In other words, send the time of log files as the value of
“aTime”.
SQLAppendFlush
SQLRETURN SQL_API
SQLAppendFlush(
SQLHSTMT aStatementHandle)
Transfer all the data left in the buffer of the current channel to
the InfiniFlux server.
SQLSetConnectAppendFlush
SQLRETURN SQL_API
SQLSetConnectAppendFlush(
SQLHDBC aHdbc,
SQLINTEGER aTimeOption)
Set whether to transfer data to InfiniFlux at a specified interval.
0: auto flush off
1: auto flush on
SQLSetStmtAppendInterval
SQLRETURN SQL_API
SQLSetStmtAppendInterval(
SQLHSTMT aStatementHandle,
SQLINTEGER aValue)
When set “auto flush” on by using
“SQLSetConnectAppendFlush()”, adjust the interval of flush or
set to “auto flush” off.
Don’t flush it if “aValue” is 0.
The default value is 1000 and unit is “ms”.
The value should be multiples of 100.

16. JDBC API

17. 17
Confirm the Installation of Library
• Library file
• Log4j file
 Support Java 1.6 version or more
• Sample
• Makefile
Check the file, shown below, in the “lib” directory where the InfiniFlux has been installed.
$IFLUX_HOME/lib/iflux.jar
$IFLUX_HOME/lib/ifluxLog.jar
$IFLUX_HOME/lib/log4j.jar
$IFLUX_HOME/sample/jdbc
$IFLUX_HOME/sample/jdbc/Makefile

18. 18
Append Program Structure
DriverManager.getConnection
Connection.createStatement
executeAppendOpen
executeAppendData
executeAppendClose
executeSetAppendErrorCallback
ResultSet.close
Statement.close
Connection.close
Extended Functions of InfiniFlux
Standard Connect Functions
Standard Disconnect Functions
Description of “$IFLUX_HOME/sample/jdbc/sample4_append.java”

19. 19
executeAppendOpen
• Open a channel for inserting data into a specified table.
• The channel remains open if the channel is not closed.
• Able to set up to 1024 statements for each connection.
ResultSet IfluxStatement.executeAppendOpen(
String aTableName,
int aErrorCheckCount );
• aTableName : name of target table to append
• aErrorCheckCount: set the number of evaluations to detect server errors based on the number of inputs. If the number is 0,
evaluation is not required.

20. 20
executeAppendData
• Function to insert data for the channel.
int IfluxStatement.executeAppendData( ResultSetMetaData aRsmd,
ArrayList aData );
• aRsmd: information of column metadata about the table
• aData: array list of data to append

21. 21
executeAppendClose
• Close the current channel.
Run “int IfluxStatement.executeAppendClose();”,
and then call “getAppendSuccessCount()” and “getAppendFailureCount()” to check the number of append
success or failure.

22. 22
executeSetAppendErrorCallback
• Setting up the callback function to be called when an error was occurred while appending.
• If this function was not configured, the client will ignore even if an error was occurred in the server.
int IfluxStatement.executeSetAppendErrorCallback( IfluxAppendCallback aCallback);
• aCallback : to call “callback” function when an error while appending data
IfluxAppendCallback cb = new IfluxAppendCallback() {
@Override
public void onAppendError(long aErrNo, String aErrMsg, String aRowMsg) {
System.out.format("Append Error : [%05d - %s]n%sn", aErrNo, aErrMsg, aRowMsg);
}
};
stmt.executeSetAppendErrorCallback(cb);

23. 23
Other Functions
• Additionally provide useful functions.
• Refer to the website at http://www.infiniflux.com/document for more information.
Name of Functions Function Declaration Description
executeAppendDataByTime
int
executeAppendDataByTime(Res
ultSetMetaData aRsmd,
long aTime,
ArrayList aData)
Like “executeAppendData()”, it inserts a data into the channel.
However, the value of hidden column, “_arrival_time”, will be
used when it was set to a value of a certain time rather than
the present time. In other words, send the time of log files as
the value of “aTime”.
executeAppendFlush int executeAppendFlush()
Transfer all the data left in the buffer of the current channel to
the InfiniFlux server.
executeAppendSuccessCount long getAppendSuccessCount() Return the number of records that were appended successfully.
executeAppendFailureCount
long
executeAppendFailureCount()
Return the number of records that were failed to append.

24. Multi Table Insert

25. 25
Multi Table Insert Structure
Description of “$IFLUX_HOME/sample/cli/sample8_multi_session_multi_table.c”
• Create thread in the main function, and each thread will process the tasks written below.
• Each thread will process Alloc and Free functions respectively against SQLHENV, SQLHDBC, and SQLHSTMT.
• Only one statement is allowed to be inserted per table.
• The reason for tasks above is that an error occurs while a thread is sharing statement or connection with
another thread, it might kill the whole process. If possible, it is safe not to share a resource of thread with
others.
• When input cycle is irregular and has long interval, conduct “auto timed flush” by using
“SQLSetConnectionAppendFlush()” and “SQLSetStmtAppendInterval()”.

26. The World's Fastest
Time Series DBMS
for IoT and Big Data
www.infiniflux.com
info@infiniflux.com
InfiniFlux