Presented at Activate 2018

1. Migration Station
Todd Lack
Sr IT Software Programmer, SAS Institute, Inc.
@toddlack
#Activate18 #ActivateSearch

2. Agenda
• Making changes in your Fusion instance
• Source Control for Fusion configurations
• Dev/Test/Prod deployment cycle
• Java-based command line tool for copy and
deployment

3. Migration Station
Making changes
• Fusion webapp
• Interactive, easy to use
• Instant verification
• Specific to server
• No versioning or backup

4. Migration Station
Making changes
• REST API
• Well documented
• Comprehensive
• Admin webapp uses rest API
• Use third party tool like Postman
• Use any language for programming.

5. Migration Station
Source Control
• Essential:
no source control = no control
• Once tests are done, create a release
• Allows for recovery in case of system failure
• Makes rollback possible

6. Migration Station
Source Control - file structure
• Use a structure where each entity type
has its own directory.
• Collections are a separate thing. Solr
config files should be bundled with the
collection definition
• Modeled after Object API json

7. Migration Station
Source Control - file structure
Json response from Objects/export API.
/api/apollo/objects/export?collection.ids=intranet&deep=
true

8. Migration Station
Source Control - file structure

9. Migration Station
Source Control - file structure

10. Migration Station
Source Control
• After verifying changes, download to your
local machine
• Check –in to source control.
• Push changes to a test server for others to
test
• Tag releases before updating production
servers

11. Migration Station

12. Migration Station
Command Line tools to deploy changes
• Easy to incorporate into a CI tool (Bamboo, Jenkins)
• Bash script running curl commands
• Quick to get up and running
• Using REST api
• Tedious to make changes.

13. Migration Station
Command Line tools to deploy changes
• Java CLI
• Using rest API
• Easy to debug
• Easy to modify

14. Migration Station
Continuous Delivery/Integration
• Once tagged, you can use your CI too to check
out the release from source control
• Have CI tool run update commands to remote
server.
• All updates are automated, increasing traceability
of changes.

15. Migration Station
Continuous Delivery/Integration
Using Bamboo
• Build: creates a deployment artifact from source
control
• For Fusion, this is just a packaging of all the
source files, along with the command script to
deploy.

16. Migration Station

17. Migration Station

18. Migration Station

19. Migration Station
• The artifact is the directory of json files and the scripts to deploy
them via REST

20. Migration Station
Create release – this is what gets deployed

21. Migration Station
Bamboo Deployment plan
• Executes program that runs installs the artifact
to a remote destination.
• The same artifact can deployed to many
different places – use environment
variables to differentiate.

22. Migration Station
Bamboo Deployment plan
• In our example, the variables we have are the
fusion server, and credentials:

23. Migration Station
Bamboo Deployment plan
• Download the release we made earlier

24. Migration Station
Bamboo Deployment plan

25. Migration Station
Bamboo Deployment plan
Do the deployment:

26. Migration Station
Bamboo Deployment plan

27. Migration Station
Bamboo Deployment plan
Roll back a problematic deployment:

28. Migration Station
Java command line application
https://github.com/toddlack/fusionConfigManager
• Properties file
• Any properties to set for the server
• Settings can be overridden by specifiying on the command line
• Command file
• List of commands to send through the CLI
• Can be piped in
• Can use regular expressions

29. Example 1
Copy matching collections, query pipelines, and datasources from another server.
Properties file contents:
Command list:
config.directory=temp/stage313
authentication.realm=ldap.demo.com
authentication.clear=demo:demo123
#server.name=lucidstage1.unx.sas.com
server.port=8443
server.protocol=https
collection.name=intranet
#Test commands for CLI
var stageServer=https://lucidintstage1.unx.sas.com:8443
var devServer=https://lucidintdev1.unx.sas.com:8443
var c360Regex=c360_de.*
set server.url ${devServer}
get collection intra.*
get collection ${c360Regex}
get query_pipeline,datasource intra.*

30. CLI Example
java -jar target/fusion-mgr.jar -pf src/test/resources/cli/test1.properties
-cmf src/test/resources/cli/act-demo-cmd.txt
Running fusion config manager:
-pf src/test/resources/cli/test1.properties -cmf src/test/resources/cli/act-demo-cmd.txt
var stageServer https://lucidintstage1.unx.sas.com:8443 --var stageServer assigned the value of
https://lucidintstage1.unx.sas.com:8443
var devServer https://lucidintdev1.unx.sas.com:8443 --var devServer assigned the value of https://lucidintdev1.unx.sas.com:8443
var c360Regex c360_de.* --var c360Regex assigned the value of c360_de.*
set server.url ${stageServer}
get collection intra.*
Oct 04, 2018 12:47:26 PM com.sas.itq.search.configManager.ConfigManager writeEntityToFile
get collection ${c360Regex}
Oct 04, 2018 12:47:26 PM com.sas.itq.search.configManager.ConfigManagerCLI runGet
INFO: Command: collection intra.* ---run with 11 results
Oct 04, 2018 12:47:37 PM com.sas.itq.search.configManager.ConfigManager writeEntityToFile
Oct 04, 2018 12:47:37 PM com.sas.itq.search.configManager.ConfigManager writeEntityToFile
get query_pipeline,datasource intra.*
Oct 04, 2018 12:47:37 PM com.sas.itq.search.configManager.ConfigManagerCLI runGet
INFO: Command: collection c360_de.* ---run with 20 results
Oct 04, 2018 12:47:39 PM com.sas.itq.search.configManager.ConfigManagerCLI runGet
INFO: Command: query_pipeline,datasource intra.* ---run with 37 results
Process finished with exit code 0

31. Command line tool example
Before command is run
After command is run

32. Command line tool example
Collection and associated files

33. Command line tool example
Fusion Before uploading collection
Fusion AFTER uploading
collection

34. Migration Station
Fusion Config manager
What it does NOT do
• Migrate between versions (upgrade) – this is
done by Lucidworks supplied tools.
• Migrate content from Solr – the config files are
updated, but you will have to index in the regular
manner.

35. Migration Station
Fusion Config manager
What is does not do … yet
• Integrate password file for datasources
• We have used the authentication files from Fusion
2.4
• Version 4 – need to add the APPS API
• Apps seem to be the main focus now.
Perhaps I will have it ready by the time of this
presentation.

36. Thank you!
Todd Lack
Sr IT Software Programmer, SAS Institute, Inc.
https://github.com/toddlack/fusionConfigManager
@toddlack
#Activate18 #ActivateSearch