I like to give a technical presentation about how to synchronize and release ArcGIS Portal Items between different instances of ArcGIS Portal like migrating certain items between a Staging and a Production environment. Those Items can be service types, maps, apps, or groups. In doing so I am using FME as a central spot of interaction between those environments. The aim is to maintain and synchronize several ArcGIS Enterprise instances that belong to one bigger deployment including TEST and PROD environments without a lot of manual maintenance. The developed workspaces make us of ArcGIS functionality based on the Python and REST API for ArcGIS. In this project, I am using FME Desktop and FME Server (Automations, FME Server Apps, Dynamic Engines).
6. 20
22
FME
User
Conference
System Architecture
DEV
PROD
LIVE STAND BY / DISASTER RECOVERY
DEV LIVE
PROD LIVE
DEV
STANDBY
PROD
STANDBY
Source of graphic source:
https://www.esri.com/en-us/arcgis/products/index
∙ Live cluster and stand by
cluster
∙ Each cluster with DEV and
PROV environment
🡪 four ArcGIS Portals
7. 20
22
FME
User
Conference
Migration of ArcGIS Portal Items
Complex Architectures / Scenarios demand
migration mechanisms
● Release new item versions from DEV to
PROD within release cycles
● Manage to support Backup and Disaster
Recovery Scenarios by synchronizing items
between LIVE and STAND BY within
acceptable time ranges
8. 20
22
FME
User
Conference
Main Objective
∙ Migration of Web Mapping Applications and
all related items (Maps and Services)
∙ No manuell creation of ArcGIS Portal Items in
each environment
∙ Create the Web App once and migrate all to
other environments / Cluster
∙ Develop a mechanism that supports
automatic migration
9. 20
22
FME
User
Conference
Solution Workflow
∙ Create Services in ArcGIS Pro and export
them as Service Definition (*.sd) File
∙ Publishing of Services Definition File to ArcGIS
Portal
∙ In ArcGIS Portal:
∙ Create Web Map
∙ Create Web Mapping Application
∙ Migration of all components that are part of
the Web Mapping App from source to
(several) target ArcGIS Portal
∙ Adjust parameters depending on local environment
∙ Replace item Ids
🡪 FME Server is the central component
regarding publishing services and migration of
ArcGIS Portal Items!
.sd
Source of graphic source:
https://www.esri.com/en-us/arcgis/products/index
https://www.safe.com/
10. 20
22
FME
User
Conference
Technical Solution
∙ FME Server Apps and Automations to trigger Migration steps
∙ FME Server Apps: migration from DEV to PROD
∙ FME Server Autoamtion: synchronization between LIVE and STAND BY
∙ Interaction with ArcGIS Portal / ArcGIS Server via ArcGIS REST API and Python
API for ArcGIS
∙ Create token
∙ Get item meta data and configs
∙ Create and copy items
∙ Start Uploading and caching Jobs
∙ Share items
∙ Manage Item IDs in SQLite Database
12. 20
22
FME
User
Conference
Publishing of Service Definitions
Generate token for
target (and source)
portal
Get infos
from SD File
Publish Web Map
and Image Services
Publish Feature
Service and
Vector Tile Layer
Package
Update item ID
mapping table
Share published
Services with
Groups and
Organization
Create result
report in HTML
Cache service data
13. 20
22
FME
User
Conference
Publishing of Service Definitions
∙ Selection of used REST API Endpoints (<HOST>/<WEBADAPTOR>…)
∙ Generate Token
∙ POST /sharing/rest/generateToken
∙ Publish Services (Map / Image Service)
∙ POST admin/uploads/upload
∙ GET admin/uploads/upload/<ITEM_ID>/serviceconfiguration.json
∙ POST sharing/rest/services/System/PublishingTools/GPServer/Publish%20Service%20Definition/submitJob
∙ Publish Feature Service
∙ POST /sharing/rest/content/users/<USER>/addItem
∙ POST /sharing/rest/content/users/<USER>/publish
∙ Manage Services
∙ POST /admin/services/@Value(sd_serviceName).@Value(sd_type)/start
∙ POST /rest/services/System/CachingTools/GPServer/Manage%20Map%20Cache%20Tiles/submitJob
∙ GET /rest/services/System/CachingTools/GPServer/Manage%20Map%20Cache%20Tiles/jobs/<JOB_ID>
∙ GET /rest/services/System/PublishingTools/GPServer/Publish%20Service%20Definition/jobs/<JOB_ID>)
∙ Manage items and groups
∙ POST /sharing/rest/search
∙ POST /sharing/rest/content/users/<USER>/items/<ITEM_ID>/delete
∙ POST admin/services/<SERVICE>/delete
∙ POST /sharing/rest/content/users/<USER>/shareItems
∙ POST /sharing/rest/community/groups
∙ ….
14. 20
22
FME
User
Conference
Publishing of Service Definitions
∙ Selection of used Python API Endpoints
∙ portal = GIS(url, token=, referer, …) # connect to source portal via token….
∙ added_item = portal.content.add(item_properties, data) # add item to portal, e.g. SD / VTPK file
∙ published_added_item = added_item .publish(file_type, overwrite, item_id) # publish added item
15. 20
22
FME
User
Conference
Migration of Web Maps and Web Mapping Apps
Generate token for
target (and source)
portal
Share migrated
items with Groups
and Organization
Check for existence
of target item and
referenced Services
and Web Maps
Update item ID
mapping table
Invoke migration
of Web Map
Create new Item in
target Portal and
modify configuration
16. 20
22
FME
User
Conference
Migration of Web Maps and Web Mapping Apps
∙ Selection of used REST API Endpoints (<HOST>/<WEBADAPTOR>…)
∙ Manage items
∙ GET /sharing/rest/content/items/<ITEM_ID>
∙ GET /sharing/rest/content/items/<ITEM_ID>data
∙ POST /sharing/rest/content/users/<USER>/addItem
∙ GET /sharing/rest/content/users/<USER>/items/<ITEM_ID>
∙ POST /sharing/rest/content/users/<USER>)/items/<ITEM_ID>/update
∙ POST /sharing/rest/content/users/<USER>/shareItems
∙ POST /sharing/rest/community/groups/<GROUP_ID>
∙ ….
18. 20
22
FME
User
Conference
Synchronization Workflow between LIVE and
STAND BY cluster
∙ FME Server Automation triggers scheduled synchronization cycle
∙ Migration of Web Mapping Application and Web Maps that are shared with a
certain ArcGIS Portal group
∙ Get Services that are embedded within Web Maps
∙ Compare last updated timestamp off all selected Items in mapping table
∙ Publish and migrate all updated items (overwrite) / remove all removed items
19. 20
22
FME
User
Conference
Key takeaways
∙ FME complements ArcGIS Portal regarding migration and release of certain
Portal Items between separate Portals
∙ No full backup of ArcGIS Portal is needed to restore only certain items
∙ FME Server Apps and Automations are useful functionalities to trigger manual
or automated migration / release workflows
20. 20
22
FME
User
Conference
Challenges
∙ Design and creation of Workspaces independent of environment
∙ Interaction with ArcGIS REST API
∙ Figure out the right workflow order and API endpoints
∙ Documentation
∙ Manage item IDs in ArcGIS Portal (keep the same item id)
∙ User, Role and Ownership Management in ArcGIS Portal
21. 20
22
FME
User
Conference
Useful “by the way” developments
∙ Maintaining FME Server Synchronization by schedule and event based
∙ Backup and Restore
∙ Custom Transformer
∙ Error handling + Reporting