FME and git do not work so well together at the moment - but a git workflow is an attractive option when creating FME-based products. Indeed, in some IT companies using git during development is part of the company policy. It is not only using git and the attractive git workflow; Gitlab (and, of course, GitHub) is also a very attractive tool for developers: Gitlab is a server-based solution which provides an ecosystem to support the development process, and it can facilitate tracking issues, and bugs, managing versions, permissions and can even help with auto-building and deploying software.
This presentation will cover the challenges and benefits involved when applying a git-based workflow to FME development and discuss how Gitlab can be used to help run a pipeline to auto-deploy an FME-based project.
Handwritten Text Recognition for manuscripts and early printed texts
Gitflow with FME and Autobuilding a Project with the Gitlab Build Pipeline
1. Gitflow with FME and
autobuilding a project with
the Gitlab build pipeline
2. FME
User
Conference
20
22
Matt Smillie
FME Guy at Virtual City Systems (VCS)
● **Disclaimer: Not from a software development
background - I only know a fraction of what I’m talking
about**
● Two years in public sector (Geology data)
● Two years as Safe Software (Various)
● Four Years at VCS (XML/GML, 3D, BIM, Raster, Web)
● 6 Years (almost) exclusive FME user
linkedin.com/in/mattsmillie
6. 20
22
FME
User
Conference
Virtual City Systems
• Developed an ecosystem of tools to support the
creation, maintenance, web visualisation and usage
of city models (VC Suite).
• Founded in 2005
• Based in Berlin, Germany
• A strong focus on CityGML
• CesiumJS
• website: vc.systems
8. 20
22
FME
User
Conference
Gitflow and parallel
development
• Branches and feature based development
• master, release-*, feature-*, hotfix-*
• an .fmw file is text based and commits can be
merged using git but you can forget about a text
based review.
• all positions
• changing uuids
• 1 movement of a transformer = 100’s lines
changed
• FME 2022 to the rescue?
• Parallel development would require the splitting of
big workspaces and/or using more linked custom
transformers
• Packages to the rescue?
9. 20
22
FME
User
Conference
Gitflow and parallel
development
• Easiest to work in a linear method (i.e in the
master branch)
• Splitting up features and fixes is nice, however, can
come with significant overhead
• Reviewing can now be done better with FME 2022
but it is still external to git
• Fixes may need to be done twice (in master and
release)
11. 20
22
FME
User
Conference
Versioning
How do you handle versioning in an FME
workspace - not just tracking changes?
● As a “developer” I want to know the version
number of the workspace a customer is
using from the logfile
○ Workspace Metadata/Description?
○ Private/Scripted Parameter?
○ FME hub?
● FME Server Project is defined
12. 20
22
FME
User
Conference
Versioning
● Follow semver versioning
● We use a private parameter called VERSION
● Simple startup script to add it to the logfile
● Can see version in log and in the workspace
import fme
import fmeobjects
logger = fmeobjects.FMELogFile()
v = fme.macroValues.get('VERSION')
logger.logMessageString("version: "+v)
14. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
Goal: Automate building and packaging of FME Server
project - align with the companies DevOps strategy
• Automate fetching dependencies (e.g., python, linked
custom transformers, formats)
• Automate Testing?
• Automate Versioning (npm)
• Automate FME Server project creation
• Run a pipeline for each push to remote git repo
Lots of new technology to learn and play with
15. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
• Gitlab can use docker containers to run jobs
• Each job runs in it’s own container
• Results from previous jobs can be passed
to future jobs
• Can access defined services (e.g., FME
Server*, Postgres)
# .gitlab-ci.yml
#example only (does not create the below image)
stages:
- build
- test
build-dependencies-job:
stage: build
script:
- ./build/deps/get_dependices.sh
test-check-defaults:
stage: test
script:
- ./build/test/check_params.sh
16. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
• Created FME Desktop docker image
• included nodejs and git (to help with versioning)
• can write workspaces to do things (versioning,
tests)
• workspaces triggered by shell scripts in the
.gitlab-ci.yml file
• A combination of shell scripts and fme
workspaces
#!/bin/bash
#example script within ci/cd job
#validates that the workspace has the defaults set
correctly and that services are registered correctly
ERROR_CODE=0
echo "validating default parameter configuration"
fme ./build/tests/default-parameter-validation.fmw &>
/dev/null
status=$?
if [ $status -eq 0 ]
then
echo "default-parameter-validation.fmw passed"
else
ERROR_CODE=$status
echo "default-parameter-validation.fmw failed" >&2
fi
exit $ERROR_CODE
18. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
Automate versioning
• Use npm (node package manager) to manage
project version
• Use FME workspace to read version from JSON and
update workspace private parameter (a hack), FME
Project version and documentation
• git commands used to commit changes to project
after updated files and to set version tags
20. 20
22
FME
User
Conference
CI/CD Build Pipeline (Gitlab)
FME Server project
• Use FME Server and the REST api to build the
project and then export it
• needs FME Server in pipeline (no easy feat)
• OR - Compile *.fsproject file
• Does not need FME Server in pipeline
• Difficult to validate successful compilation
21. 20
22
FME
User
Conference
Room for improvement
• Work needs to be done to make merging and
rebasing in git work more seamless
• FME 2022 change detector?
• Splitting up of a workspace into components?
• Like a shapefile?
• Ability to embed linked Custom Transformers
from command line - vote on the idea
• A defined method for workspace versioning
• some command line to update a version string
- vote on the idea
23. 20
22
FME
User
Conference
Please Share Your Session Feedback
Log in & navigate to the
agenda. Select the
session you attended
Scroll down to “Session
Feedback”
Download EventMobi app
Event code: fmeuc
Click “Launch App”
Fill out the survey
and submit