1. Contributing to OSS in
a commercial non-OSS
environment
Mike Taczak
mtaczak@mailtrust.com
25/11/08
Mailtrust, a division of Rackspace
www.mailtrust.com | www.rackspace.com

2. Overview
□
What is Mailtrust
□
What OSS does Mailtrust use
□
How has Mailtrust contributed to Funambol
□
Design challenges
□
Review of integration strategies
2

3. □
Formerly Webmail.us
□
Now a division of Rackspace
◊ Recently went public
□
Business-class email hosting
□
Noteworthy Webmail
◊ Full-featured AJAX-y webmail application
◊ Competitive collaboration suite
3

4. Mailtrust and Open Source
□
Heavily Used □
Main Contributions
◊ PHP ◊ Funambol
◊ mySQL ◊ Dovecot
◊ Hadoop
◊ Postfix
◊ amavisd
◊ policyd
◊ Many others
4

5. Contributions to Funambol
□
DS-Server
◊ Webmail Connector
□
Outlook Client
◊ Support for 'custom fields'
□
Blackberry PIM Client
◊ Initial development in-house
◊ Now part of Funambol's core clients
□
iPhone client
◊ 2 weeks in Italy
5

6. Goals for Sync Service
□
Synchronize shared data
◊ Read-only
□
Give back to open source community
◊ But keep proprietary systems private
□
Focus on a few highly refined clients
◊ Outlook
◊ Blackberry
◊ Windows Mobile
6

7. System Architecture Evolution, Part 1
Initial design:
●
Very Simple
●
Connector hit DBs directly
Webmail
Webmail
Database
Data
Webmail
DS-Server Authen
Module tication
User
Database
7

8. System Architecture Evolution, Part 1
Lessons Learned:
●
2 code bases to maintain!
●
Data validation duplicated
●
Proprietary DB schemas in open
source code! Webmail
●
We could not launch with this
architecture
Webmail
Database
Data
Webmail
DS-Server Authen
Module tication
User
Database
8

9. System Architecture Evolution, Part 2
Introducing the Webmail-Sync API
●
Implementation of SyncSource
interface in API form
●
Data parsing now in php
●
Implemented a large Vobj library Webmail
●
Hope to open-source it! Database
●
Uses existing structures and validation
●
HTTP + jsON
●
Connector generic enough for other uses
●
Clear separation between Funambol
and Webmail
Webmail
DS-Server API Webmail
Module
9

10. System Architecture Evolution, Part 2
Lessons Learned
●
PHP has an execution time limit
●
Don't do too much at once
●
Know your technologies intimately
●
Distance matters Webmail
●
Reliability, speed deteriorates Database
●
Retry failed requests
●
getSyncItemByKey is slow
●
Page calls and cache data
DS-Server
Webmail 2000 API Webmail
Module Miles!
10

11. System Architecture Evolution, Part 3
Batching calls to API
●
Can't batch calls w/ standard code
●
addItem requires returning a GUID
●
Went one layer higher
●
New BatchedSyncStrategy Webmail
●
New BatchedSyncSource Database
●
Completely compatible with original
●
Reduced calls by 50-90%
Strategy
Webmail
DS-Server API Webmail
Module
11

12. System Architecture Evolution, Part 3
Lessons Learned:
●
The Strategy is complicated
●
Test, Test, Test
●
Funambol is very flexible to these
kind of changes Webmail
Database
Strategy
Webmail
DS-Server API Webmail
Module
12

13. System Architecture Evolution, Part 4
New Problems
●
Cannot upload to webmail and
sync at the same time!
●
Do not want load from API
processing affecting webmail Webmail
users Database
Strategy
Webmail
DS-Server API Webmail
Module
13

14. System Architecture Evolution, Part 4
Lessons Learned:
●
Dedicate systems whenever Webmail
possible
●
Scale services independently
Webmail
Database
Dedicated
Strategy
Webmail Webmail
DS-Server API
Module For
Sync
14

15. System Architecture Evolution, Part 5
New Problems, Again!
●
Network connectivity issues Webmail
caused duplicates
●
Requests to addItems failed, and
were retried
●
Sync was hitting webmail
databases too hard
●
50 calendars can have a lot of data Webmail
Database
Dedicated
Strategy
Webmail Webmail
DS-Server API
Module For
Sync
15

16. System Architecture Evolution, Part 5
Lessons Learned:
●
Don't retry 'dangerous' commands Webmail
●
Database connections are limited
resources
●
Cache everything reasonable!
●
Database requests
●
API requests! Webmail
●
Open source has a solution Database
memcached
memcached
Dedicated
Strategy
Webmail Webmail
DS-Server API
Module For
Sync
16

17. System Architecture – Misc details
□
SyncSourceRedirectSynclet
◊ Replaces remote URI in requests
◊ Uses content types sent by client, has defaults
◊ 'contacts' => 'contacts-vcard21' or 'contacts-sif'
◊ Easier for users to configure devices
◊ Guarantees clients get the preferred type, if
available
17

18. System Architecture – Misc details
□
SessionConnectionLimiterSynclet
◊ Disconnects a session that has lasted 'too long'
◊ Protects service from runaway clients
18

19. System Architecture – Misc details
□
Authentication
◊ Moved queries to mysql stored procedures
◊ Keeps DB schema out of code and configuration
◊ Allows installations to redefine these procedures
19

20. Summary
□
Use abstractions to hide proprietary
details
◊ Write open logic, and share it!
□
Physically and logically separate open and
closed source programs
◊ API separated synchronization logic from
our proprietary systems and data
◊ Stored procedures hide our database
behind an interface
20

21. Summary, cont.
□
Rely on configurations for details
◊ Funambol does this already – model after
them
◊ Requires flexible code, just the kind open
source needs
□
Work at a great company
◊ The kind that values open source
◊ The kind that lets you give back!
21

22. Summary, cont.
□
Coordinate with your community
◊ Communication is key
◊ Work out a way to share code and plans
often
◊ Still working on this one ;)
□
Go to Italy!
22

23. The End
Questions? Comments?
mtaczak@mailtrust.com 23