This document discusses data migration in an agile open source world. It provides an overview of data migration, including definitions and common patterns like extract-transform-load (ETL). It also discusses agile principles and practices like Scrum and extreme programming (XP). Open source tools are presented for continuous integration, testing, ETL/ESB and more. The document aims to help organizations perform data migration in an agile and open source manner.

1. Data Migration
in an Agile Open Source World
Craig Smith
Suncorp

2. Welcome…
Image http://www.neopoleon.com/blog/images/excel/7.jpg

3. According to Wikipedia:
Data migration is the
process of transferring
data between ...
computer systems ... is
usually performed
programmatically to
achieve an automated
migration ... required
when organizations ...
change ... or upgrade
to new systems, or
when systems merge...
Image: http://i.ehow.com/images/GlobalPhoto/Articles/2065416/dictionary_Full.jpg

4. “In” End
Point
Extract Extract
Transform
Load Transform
(ETL) Load
Pattern
“Out” End
Point

5. File transfer
(Enterprise) Shared
database
Application
Integration Remote
Patterns
Procedure
Invocation
Messaging
Image http://www.eaipatterns.com/eaipatterns.html

6. Data
Migration
+
Application
Integration
=
Similar
Approach
Image http://kansolutions.net/yahoo_site_admin/assets/images/integration.302161703_std.jpg

7. Image http://blog.theworkinggroup.ca/wp-content/uploads/2009/04/agile-manifesto1.gif
Agile Manifesto

8. Scrum Practices
Image http://blogs.conchango.com/Admin/ImageGallery/blogs.conchango.com/Colin.Bird/Scrum%20Overview%20Diagram.png

9. Image Beck, Kent – Extreme Programming Explained: Embrace Change
XP Practices

10. Development Practices
Fixing a Fast (<3 Use It
100% Nulls
broken build mins) Or Lose
Code Are Evil
is always the Builds It
highest Coverage
priority Test
Changing The Code Is
Following Things Prod
If In Doubt, Code Is A Use
Is A Team Code
Spike It Liability Outlandish
Out (On A Decision... Class
Branch) Statics Names
Are Evil
Molecular
No Code
Over
Ownership
Atomic
All Prod Create Use
Tests
Code Small Delegation
Paired (Or Commit Classes And Over
Reviewed) Often Methods Inheritance
Source http://www.assembla.com/wiki/show/burro/Development_Principles

11. Image http://iqizone.com/images/paper%20pile.jpg
Break It Down... Iteratively

12. Agile Builds Confidence
Image http://static.soxfirst.com/soxfirst.com/imgname--derivatives_time_bomb_for_big_banks---50226711--bomb.jpg

13. Continuous (Inte)(Mi)gration

14. Throttle Data
Image http://i.ehow.com/images/GlobalPhoto/Articles/2289906/P1120995RPM_Full.jpg

15. Image http://www.mikkeman.nl/blog/wp-content/uploads/2009/10/priority-300x137.jpg http://ralphlosey.files.wordpress.com/2009/10/playing-cards.jpg
Prioritisation & Categorisation

16. Volatility Categorisation
Image http://silverspaceship.com/static/shot_1.png

17. Image http://www.domainlanguage.com/_library/images_people_working/designing_at_whiteboard.jpg
Analysis = BDUF NDUF SDUF

18. Implement “Big Bang”
Image http://silverspaceship.com/static/shot_1.png

19. Image http://sapplanning.files.wordpress.com/2008/07/integrate-technology.jpg
Implement “Master + Deltas”

20. Data in non-binary format > diff
Image http://www.spycomponents.com/images/xml_at_work.gif

21. Image http://images.yogee.com.au/cat-dump-truck-609-2.jpg
Master Database Dumps

22. Activity Migration
Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg

23. Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg
Reconciliation

24. Open Source Continuous
Integration / Build Tools
Continuous Integration Servers
 Hudson: https://hudson.dev.java.net
 Cruise Control:
http://cruisecontrol.sourceforge.net
Build Tools
 Ant: http://ant.apache.org
 Maven: http://maven.apache.org
 Gradle: http://www.gradle.org
 SCons: http://www.scons.org
 Phing: http://phing.info
 Rake: http://rake.rubyforge.org/

25. Open Source Developer Tools
IDE’s (multiple language support)
 Eclipse: http://www.eclipse.org
 NetBeans: http://www.netbeans.org
 RadRails: http://www.aptana.com/rails
Database Clients
 DBVisualizer: http://www.dbvis.com/
 SQuirreL: http://squirrel-sql.sourceforge.net
Web Service / SOAP Tools
 soapUI: http://www.soapui.org
Version Control Systems
 Subversion: http://subversion.tigris.org
 Git: http://git-scm.com
Database Versioning
 Liquibase: http://www.liquibase.org/

26. Open Source Testing Tools
Test Specification – xUnit Tools
 JUnit: http://www.junit.org
 PHPUnit: http://www.phpunit.de/
Test Specification Tools – Expressive/BDD
 FitNesse: http://fitnesse.org
 Concordion: http://www.concordion.org
 Cucumber: http://cukes.info
 easyb: http://www.easyb.org
Test Execution / Record / Log / UI
 Selenium: http://seleniumhq.org
 Watir: http://wtr.rubyforge.org
Performance Test
 JMeter: http://jakarta.apache.org/jmeter/
 Grinder: http://grinder.sourceforge.net/

27. Open Source ETL/ESB Tools
ETL
 Donkey:
http://www.assembla.com/wiki/show/burro
 Pentaho (Kettle): http://kettle.pentaho.org/
 Talend: http://www.talend.com
 Clover: http://www.cloveretl.com/
 Scriptella: http://scriptella.javaforge.com/
ESB
 Mule: http://www.mulesoft.org
 Apache ServiceMix:
http://servicemix.apache.org
 Apache Camel: http://camel.apache.org/

28. Questions ??
Craig Smith
http://www.suncorp.com.au
http://www.agileacademy.com.au
craig.smith@suncorp.com.au
@smithcdau
Suncorp is one of Australia and New Zealand's largest
diversified financial services providers, supplying banking,
insurance and wealth management products to around 7 million
customers through well-established and recognised brands such
as AAMI, Australian Pensioners Insurance Agency, Shannons,
Vero, Asteron and Tyndall, as well as Suncorp and GIO. Today,
Suncorp is Australia's fifth largest bank and second largest
domestic general insurance group, with over 16,000 staff.
Suncorp has representation in 450 offices, branches and
agencies throughout Australia and New Zealand.