3. Workflow Management
What are we talking about?
Using the power of FME Workbench to design workflows
Moving and Copying Files
Data Transformation
Quality Control
Database Updates
Notification (Emails, Mobile Alerts)
Etc.
4. Workflow Management
Why do we care?
Because many of you are doing this already!
Users have noticed that FME Workbench is a
powerful process control tool
Many of you running complex workflows with
multiple workspace already
FME Server Survey 2012 #1 Enhancement
Request was “Automated Workflows”
5. Workflow Management: 2 Case Studies
1. “Call Before you Dig” Client (FME Desktop based)
2. Norwegian Environment Agency (thanks to VAR Norkart)
Please note: Demos are very loosely based on these real
projects
7. Case Study #1 “Call Before You Dig”
Customers (ie Utilities) submit DGN Files – 1000s of
them!
Need to:
Find out which territory each file belongs to
Update database with files names and territory
Copy file into directory by territory
8. Case Study #1 “Call Before You Dig”
Reading a directory – Directory Reader
Processing files with WorkspaceRunner
Copying Files with File Copy Writer
9. Case Study #2 Norwegian EA
Goal: automatically add valid data from the SOSI-format to
the ArcSDE operational database
• geometry must be validated and inside the database before
running any other database updates
• split between
geometryhandling, attributehandling, databasehandling
• easily add other procedures. Currently working new one to take
attributes from the features and stores these in another database.
View the Application here:
http://innsyn.naturbase.no/
10. Case Study #2 Norwegian EA
SOS
I-file
ArcSDE
Operational
DB
MSSQL
Logging etc
SOSI-
file, PDF-
report, text-
report
1 2 3 4
Verfiy Data
and Database
updates
Generate
paths/published
parameters based
on attributes in
SOSI-file
To Do all checks:
- Geometry
- SOSI
- Attributename/values
- Exisiting Ids
Create logfiles (in SOSI, txt, 2D / 3D
PDF) and add geometry to ArcSDE
All data verified and all logs updated
To Do:
- Wrap-up files from previous FMW-
files into one package and send by
e-mail to supplier
11. Case Study #2 Norwegian EA
Controller Workspace Running two consecutive
tasks
1. Quality Control Workspace
2. Database Update Workspace
Notification at all stages (Success and Failure)
- Obviously data transformation is huge and most of what FME does but there are so many other things FME can do
There was a packed presentation on this in FME World Tour 2012 Stockholm, tons of people trying to do this stuffNot just power usersPerhaps admit we’ve been behind what are user are doing here
Demos are dumbed down versions of real workflowCustomer workflows are often much more complex that what we do but or goal is to show the potential
Grouped by Readers/WritersVery powerful to read all of the file names in a directoryFile copy writer lets you move things around based on attributes or anything reallyTransformersFMEServerJobSubmitter has had a very important enhancement, if you set it to WAIT it does not need another engineThis means you can for example , check data…then load data after the checking is completeNotification ServiceTell people that a process had completed, that data passed, that new data arrived, that workspace failed etc
Processing lots of files can be painful and FME can automate thisDavid Eagle once used FME to copy 100 000’s of files because it was faster that the Windows Explorer or DOS
WorkflowDirectory reader to read all file namesWorkspace runner to actually read each file and figure out where it isCopy the files into the right place for each territorDatabase record of all of this was key – need to track all files
Thank SigbjørnTillerliHerstad of Norkart for this exampleThese points came from himHe is still not happy because would prefer to use the FMEServerWorkspaceRunner in sync mode (wait) without using an engine
The goal is a sequential workflow where the second task cannot begin until the first task is doneThis is now possible with FMEServerJobSubmitterAlso in the output from the with FMEServerJobSubmitter in Controller to the output dataset from a child via the output_datasets{}.path list attributeWe can use FMEServerNotifier if a child failsIf we want to know if the controller failed – when we publish it there is Topics to Notify on Failure for any service