Slides for my presentation at the Seattle Hadoop/NoSQL Meetup (http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/events/40509972/). These slides are based on this earlier presentation: http://www.slideshare.net/MichaelRys/scaling-with-sql-server-and-sql-azure-federations.

1. mrys@microsoft.com

2. 





3. MySpace: the Business Problem
 223M users
 900 Terabytes of data
 450 SQL Servers
 Required (eventual) data
consistency across databases

4. MySpace’s Data Consistency Problem
1-1000 1001-2000
2001-3000 3001-4000
4001-5000 5001-6000

5. MySpace’s Solution
 Propagate data changes from one DB to
other DBs using reliable, async Message
Service (Service Broker)
 And also used for

6. MySpace’s Service Dispatcher
 Coordination point between all SQL Servers
 Load-balanced across 30 SQL Servers
 Enables multicast/broadcast functionality
 18,000 ~2k msgs/sec per dispatcher SQL
Server

7. MySpace Architecture
1-1000 3001-4000
My DB I change
gets updated my status
Service
Service
TX3 Broker
TX2 TX1
Dispatcher Service userId=1024
Broker
2001-3000 Service
1001-2000
Broker
TX4 TX5
4001-5000 5001-6000 Web Tier
Data Tier

8. Many other customers using similar
patterns
 Online electronic stores (cannot give names
)
 Travel reservation systems (e.g. Choice
International)

10. 



11. Social
Social
Networks
Network
Management s
Ops Services
Data Backend
Data Backend
Services
Data Backend
Services
Services
MSN
MSN Games
Games l
Web Portal
Front Door
GameBar Router
WLM Services
WLM Games
Host
Games Web Portal
Bing
Bing Auth
Games STS
STS
Games l
Web Portal
Azure Data Centers

12. 



14. •
•
•

15. Partitioned over 100 SQL Azure DBs
Find Friends’ Profiles
Social Get my Profile
Social
Services
User … DB Service Publish feed, read feed
Find Friends’ Profiles
Get Friends highscores
Gamer Last Played
Gamer
Services Favorites
STS Services Game Preferences
STS
Leaderb … Social Leaderboards
DB
oard
Partitioned over 298 SQL Azure DBs Game
Game Disable/Enable
Ingestion Games from
Front Door Write user specific game infos Ingestion
accessing services
Router
Services
Game
Game binaries
User … DB Catalog
Game metadata
250 instances
Partitioned over 100 SQL Azure DBs 250 instances

16. • Fanout: Parallel calls to
multiple database
partitions
• Quorum: Able to
tolerate a percentage of
request failures during
Fanout
• Retry: Retry on database
requests error

17. Statistics per game

18. 




19. 






20. 



21. 




22. 






23. Readable
Replica
Primary Shard
Readable
Replica Dynamic OLAP
Workloads
(Scale-out queries,
OLTP Workloads often using
(mostly touching 1 to eventual consistent
Readable scale-out
low number of shards) Replica
frameworks like
Hadoop)
Primary Shard
Readable
Replica
Readable
Replica
Primary Shard
Readable
Replica

24. 





25.  Federation
Azure DB with Federation Root
Represents the data being sharded
 Federation Root
Federation Directories, Federation Users,
Database that logically houses federations,
contains federation meta data Federation Distributions, …
 Federation Key
Value that determines the routing of a piece
of data (defines a Federation Distribution)
 Federation Member (aka Shard)
Federation “Orders_Fed”
A physical container for a set of federated (Federation Key: CustomerID)
tables for a specific key range and reference
tables
Member: PK [min, 100)
 Federated Table
Table that contains only atomic units for the AU AU AU
member’s key range PK=5 PK=25 PK=35
 Reference Table
Non-sharded table
 Atomic Unit Member: PK [100, 488)
All rows with the same federation
key value: always together! AU AU AU
Connection
PK=105 PK=235 PK=365
Gateway
Member: PK [488, max)
AU AU AU
Sharded PK=555 PK=2545 PK=3565
Application 25

26. •
•
•

27. 





28. 
http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseSt
udyID=4000008310

http://cacm.acm.org/magazines/2011/6/108663-
scalable-sql
http://download.microsoft.com/download/9/E/9/9E9F240D-0EB6-472E-
B4DE-
6D9FCBB505DD/Windows%20Azure%20No%20SQL%20White%20Paper.p
df
http://blogs.msdn.com/b/cbiyikoglu/archive/2011/03/03/nosql-genes-in-
sql-azure-federations.aspx

mrys@microsoft.com
http://sqlblog.com/blogs/michael_rys/default.aspx

30. Connection:
Server=az1cl321.db.windows.net;
Database=MyDB;
User=AppUser;
Passwd=****;
Gateway
CREATE FEDERATION
Existing Database
CREATE FEDERATION sales
(customer_id bigint RANGE)
30

31. Federation with a Single Shard
Existing Database
CREATE FEDERATION sales
(customer_id bigint RANGE)
Database root contains:
sales
• Federation root = DB level object
containing federation scheme
Gateway
• Federation users
• Federation metadata incl. federation
Connection:
map
Server=az1cl321.db.windows.net;
Range: Min...Max
Database=MyDB;
User=AppUser;
Passwd=****;
Federation Member
31

32. Introducing Two Connection Modes
• Filtered Connection
– Guarantees that any queries or DML will produce the
same results independent of changes to the physical
layout of the federation members
– Scoped to an “Atomic Unit”
• Unfiltered Connection
– Scoped to a Federation Member
– Management Connection
32

33. Create Schema on Member: Management Connection
Existing Database
sales
Gateway
USE FEDEDERATION sales (customer_id=0)
Connection:
WITH FILTERING=OFF, RESET;
Server=az1cl321.db.windows.net;
CREATE TABLE …
Range: Min...Max
Customer Order Product
Database=MyDB;
User=AppUser;
Passwd=****;
Federation Member
© 2011 Microsoft Corporation. Microsoft Materials
- Confidential. All rights reserved. CITA # 33
MSFT101120_A

34. DDL
CREATE TABLE customer (
c_id bigint PRIMARY KEY,
…
)
FEDERATED ON (customer_id=c_id);
CREATE TABLE order (
item_num int,
customer_id bigint,
date_sold datetime2,
…,
CONSTRAINT PK_Order PRIMARY KEY (item_num, customer_id, date_sold),
CONSTRAINT FK_Cust FOREIGN KEY customer_id REFERENCES
customer (customer_id)
)
FEDERATED ON (customer_id=customer_id);
CREATE TABLE product (
product_name varchar(100) NOT NULL,
unit_price money,
item_num int PRIMARY KEY,
…
);
34

35. More Detail
• Supported data types for federation key : bigint, int, GUID, and varbinary
(900)
– Only range partitioning
• Federation key must be part of unique index
• Foreign key constraints only allowed between federated tables and from
federated table to reference table
• Not all Azure programmability features supported
– Sequence, timestamp
• Additional surface area restrictions
– Indexed views, drop database (members)
• Schemas are allowed to diverge over time
– Furthermore, in v1, schema updates to existing members must be done in each
member (where the change is needed)
• USE FEDERATION “rewires a connection”
– Connection is reestablished
– All existing settings and context of the connection is lost (sp_reset_connection)
– Must be in a batch by itself
35

36. Connect to Atomic Unit: Filtered
Existing Database
When using into a specific key
value, SELECT will only return
sales
records from federated tables that
match that value. It will still return
Gateway
all records from non-federated
tables.
USE FEDEDERATION sales (customer_id=3)
Connection:
Inserts and UPDATES operating
WITH FILTERING=ON, RESET;
outside of the value will fail.
Server=az1cl321.db.windows.net;
Range: Min...Max
customer order product
3
Database=MyDB;
SELECT * from customer 3
User=AppUser;
Passwd=****;
SELECT * from product
SELECT * from order
Federation Member
36

37. More on Connection Filtering
• Most operations behave differently in filtered vs
unfiltered connections
• Connection filtering is a property of the session
– Filter injected dynamically at runtime
– Cannot inspect source code to determine how it behaves
• E.g., running stored proc written for filtered mode on unfiltered
connection could lead to unintended results
• There are several operations that will not work in
filtered connection in v1
– DDL, DML on reference tables, …
• Fan-out, bulk operations not efficient in filtered mode
– For now, filter=off is our best offer
37

38. Support Matrix
Connection Type Filtered Unfiltered Named
Operation (unfiltered)
Dynamic SELECT P P P
DML* (federated tables) P P P
DML* (reference tables) X P P
DDL X P P
Views (not indexed) P P P
UDF - activate P P P
Stored Proc - activate P P P
Trigger (all modes) - activate P P P
CREATE/UPDATE Stats X P P
Bulk Ops
openrowset bulk, bcp, bulk
insert X P P
* not including SELECT & modules
^ autostats will work on all connections
System stored procs, intrinsics will be unaffected (run unfiltered)
38

39. Splitting a Member
Existing Database
ALTER FEDERATION sales
sales SPLIT AT (customer_id=50)
Gateway
USE FEDERATION ROOT
Connection:
WITH RESET
Using to the
Server=az1cl321.db.windows.net;
federation ROOT
will pop you out of
a member back Range: Min...Max
into the database
that hosts the customer order product
federation 3 3
Database=MyDB;
User=AppUser;
Passwd=****;
40 58
58 58
Federation Member
39

40. Two New Members
Existing Database
ALTER FEDERATION sales
sales SPLIT AT (customer_id=50)
Gateway
USE FEDEDERATION ROOT
WITH RESET
Connection:
Server=az1cl321.db.windows.net;
Range: Min...50 Range: 51...Max
customer order product customer order product
3
Database=MyDB;
3
User=AppUser;
58
Passwd=****;
40 58 58
Federation Member Federation Member
40

41. Two New Members
Existing Database
sales
Gateway
USE FEDEDERATION sales (customer_id=40)
Connection:
WITH FILTERING=ON, RESET;
Server=az1cl321.db.windows.net;
Range: Min...50 Range: 51...Max
customer order product customer order product
Database=MyDB;
SELECT *
User=AppUser;
58
Passwd=****;
from customer 40
40 58 58
SELECT * from order
41