Using popular software development tools like maven, git, jenkins and docker with kdb+/q.

1. Pragmatic Software
Development in q/kdb+
Ajay Rathore

2. My preferred tools for kdb+/q
development
● Sublime Text3
● Maven
● Git
● Jenkins
● Docker

3. Sublime Text3
● Light on the CPU, faster startup
● Symbol indexing across projects, which allows you to
use Goto Definition and Goto Symbol to jump to related
code in other files. Pretty huge
● Lots of open source plugins to enhance and customize
● Kdb+ syntax highlighting sublime-q
Screenshot of my IDE below

5. Maven
Maven’s primary goal is to allow a developer to comprehend the
complete state of a development effort in the shortest period of
time. In order to attain this goal there are several areas of concern
that Maven attempts to deal with:
● Making the build process easy
● Providing a uniform build system
● Providing quality project information
● Providing guidelines for best practices development
● Allowing transparent migration to new features

6. Apart from the above benefits, one of the biggest
advantage of using maven is that it allows certain level of
modularity in big kdb+ projects. Using maven we can break
kdb+ libraries and services into smaller, manageable
artifacts and using nexus repositories we can publish and
share those artifacts across different teams.
Let me give a simple example.
Suppose we want to create a logging library which can be
shared across different kdb services without duplicating the
code.

7. Maven project layout will look something like below

8. Assembly.xml (This will create a zip archive maintaining the
dir structure relative to src/main/kdb)

9. pom.xml

12. When we run “mvn install” on above project, maven will create a zip
artifact “logging-library-1.0-SNAPSHOT-dist.zip” and push that into
nexus repository.
In order for other projects to use this logging library, they will need to
declare a maven dependency in their project pom.
<dependency>
<groupId>com.kdb.ajay</groupId>
<artifactId>logging-library</artifactId>
<version>1.0-SNAPSHOT</version>
<classifier>dist</classifier>
<type>zip</type>
</dependency>
Provided those other projects are also assembled as zip
distribution, maven dependency plugin will pull any dependencies

13. and package everything together in a single artifact while
maintaining the individual project’s directory structure. Now for
using the functions in the logging library, other q files will need to
simply load the library using
l code/logging.q
For deployment purpose, we can use maven-antrun-plugin to scp
the final zip archive to any target server and unzip it and initialize
the service.

14. Git
Git is probably the most popular distributed version control
system out there, it can be used as a server out of the box.
Dedicated Git server software helps, amongst other features,
to add access control, display the contents of a Git repository
via the web, and help managing multiple repositories.
For hosting the code, there are couple of options for free
web-based Git repository hosting service like github and
bitbucket. I prefer using bitbucket which allows private
repositories.
Git can be integrated with maven using <scm> tag, this allows
maven to automatically push any tagged artifacts when user
runs mvn release:prepare and release:perform.

15. Jenkins
Jenkins is an open source continuous integration server.
Basically Continuous Integration is the practice of running
your tests on a non-developer machine automatically every
time someone pushes new code into the git repository. It
gets tightly integrated with maven and git.
There are multiple ways of setting up jenkins git repository
monitoring, whenever new code is pushed to git either
master or feature branch, jenkins can download any
changes and build the branch and can deploy the artifact to
the dev or uat server. All of this can be automated and
speeds up develop-test-release cycles.

16. Docker
Docker is an open source technology and provides a
convenient way to ship any infrastructure code and
supporting libraries and tools along with the application
code. You can look into references section for more
resources on docker.
We can make use of couple of maven plugins for creating
docker images of our kdb services and push those into
docker registries via jenkins job.

17. Project structure will look something like
For building docker images and integrating into maven
lifecycle we can create a maven docker profile

19. To invoke the above docker profile we can simply call
mvn clean install -P docker
This will result in following sequence of events, first of all maven will
build project zip artifact and put it inside project build directory which
by default is project-root/target/
After that maven-resource plugin will run in the prepare-package
phase and will copy all files inside src/main/docker dir to target/docker
and project build artifact from target to target/docker dir, it will also
replace any property place holders via resource filtering. Now
everything we need to build docker image is inside target/docker dir
Next we invoke maven-antrun-plugin in package phase. It will perform
two tasks, first it sets up appropriate file permission on
target/docker/build.sh and then run that script to produce docker
image.

20. build.sh
The above script simply invokes docker build command inside
target/docker/ dir which has dockerfile with all the instructions for
building the kdb service image.

21. Dockerfile

22. Once the docker image is produced, we can simply launch the
container using container port mapping and docker volumes for
loading data from SAN or NAS.
docker run -v
/Users/ajay/deploy/kdb-tick/data/marketdata:/home/ajay/services/kdb-t
ick/data/marketdata --name marketdata -td -p 5000:5000 -p
5001:5001 -p 5002:5002 ajayrathore/kdb-tick-marketdata
The above container will launch a standard kdb tick triplet with
tickerplant on port 5000, realtime on port 5001 and hdb on port 5002.
Historic data is mounted from localhost using docker -v flag.

23. References
Maven
Git
Jenkins
Docker