Building highly-available and highly-scalable applications are one of the main reasons for using NoSQL database systems and processing frameworks over traditional relational database systems. Relational database systems have taken notice and are increasingly moving forward to provide solutions for these class of applications. In this presentation we will showcase how the Windows Gaming Experience is using SQL Server Azure to build a highly-available and highly-scalable application that is used to create new experiences for millions of casual gamers in the next version of the Bing search engine and integrate Microsoft games with social-networking sites. They employ several of the NoSQL architectural patterns such as sharding. We will be presenting the architecture, lessons learned and also provide an insight into how the SQL Server Azure service is evolving to support NoSQL application development patterns such as sharding and open schema support to make SQL Server Azure a Not Only SQL database engine.

1. mrys@microsoft.com

4. Social
Social
Networks
Network
Management s
Ops Services
Data Backend
Data Backend
Services
Data iBackend
Services
S
Services
MSN
MSN Games
Games l
Web Portal
Front Door
GameBar Router
WLM Services
WLM Games
Host
Games Web P t l
W b Portal
Bing
Bing
g Auth
Games
G STS
STS
Games l
Web Portal
Azure Data Centers

7. •
•
•

8. Partitioned over 100 SQL Azure DBs
Find Friends’ Profiles
Social Get my Profile
Social
Services
User … DB Service Publish feed, read feed
Find Friends’ Profiles
Fi d F i d ’ P fil
Get Friends highscores
Gamer Last Played
Gamer
Services Favorites
STS Services Game Preferences
STS
Leaderb Social Leaderboards
… DB
oard
Partitioned over 298 SQL Azure DBs Game
Game Disable/Enable
Ingestion Games from
Front Door Write user specific game infos Ingestion
accessing services
g
Router
R t
Services
Game
Game binaries
User … DB Catalog
Game metadata
250 i
instances
Partitioned over 100 SQL Azure DBs 250 instances

9. • Fanout: Parallel calls to
multiple database
partitions
• Q
Quorum: Abl t
Able to
tolerate a percentage of
request failures during
Fanout
F t
• Retry: Retry on database
requests error

10. Statistics per game

17. Federation
Azure DB with Federation Root
− Represents the data being sharded
Federation Root
Federation Directories, Federation Users,
− Database that logically houses federations,
contains federation meta data Federation Distributions, …
Distributions
Federation Key
− Value that determines the routing of a piece
of data (defines a Federation Distribution)
Federation Member (aka Shard)
Federation “Orders_Fed”
− A physical container f a set of federated
h i l t i for t ff d t d (Federation Key: CustomerID)
tables for a specific key range and reference
tables
Member: PK [min, 100)
Federated Table
− Table that contains only atomic units for the AU AU AU
member s
member’s key range PK=5 PK=25 PK=35
Reference Table
− Non-sharded table
Atomic Unit Member: PK [100, 488)
− All rows with the same federation
key value: always together! AU AU AU
Connectio
PK=105 PK=235 PK=365
Gateway
Member: PK [488, max)
on
y
AU AU AU
Sharded PK=555 PK=2545 PK=3565
Application 17

18. •
•
•

20. http://www.microsoft.com/casestudies/Case_Study Detail.aspx?CaseSt
p // / / y_ p
udyID=4000008310
http://cacm.acm.org/magazines/2011/6/108663-
scalable-sql
l bl l
http://download.microsoft.com/download/9/E/9/9E9F240D-0EB6-472E-
B4DE-
6D9FCBB505DD/Windows%20Azure%20No%20SQL%20White%20Paper.p
df
http://blogs.msdn.com/b/cbiyikoglu/archive/2011/03/03/nosql genes in
http://blogs msdn com/b/cbiyikoglu/archive/2011/03/03/nosql-genes-in-
sql-azure-federations.aspx
mrys@microsoft.com
y
http://sqlblog.com/blogs/michael_rys/default.aspx

22. Connection:
Server=az1cl
l321.db.wind
dows.net;
Database=MyD
DB;
User=AppUser
r;
Passwd=****;
;
Gatew
way
CREATE FEDERATION
g
Existing Database
(customer_id bi i
CREATE FEDERATION sales
i bigint RANGE)
22

23. Federation with a Single Shard
Existing Database
g
CREATE FEDERATION sales
(customer_id bigint RANGE)
Database root contains:
sales
• Federation root = DB level object
containing federation scheme
way
• Federation users
Gatew
• Federation metadata incl. federation
Connection:
map
net;
er=az1cl321.db.windows.n
Range: Min...Max
base=MyDB;
=AppUser;
wd=****;
Serve
Datab
User=
Passw
Federation Member
23

24. Introducing Two Connection Modes
• Filtered Connection
– Guarantees that any queries or DML will produce the
same results independent of changes to the physical
layout of the federation members
–SScoped to an “Atomic Unit”
d “A i U i”
• Unfiltered Connection
– Scoped to a Federation Member
– Management Connection
24

25. Create Schema on Member: Management Connection
Existing Database
g
sales
way
Gatew
Connection:
USE FEDEDERATION sales (customer_id=0)
WITH FILTERING=OFF, RESET;
net;
CREATE TABLE …
er=az1cl321.db.windows.n
Range: Min...Max
Customer Order Product
base=MyDB;
=AppUser;
wd=****;
Serve
Datab
User=
Passw
Federation Member
© 2011 Microsoft Corporation. Microsoft Materials
‐ Confidential. All rights reserved. CITA # 25
MSFT101120_A

26. DDL
CREATE TABLE customer (
c_id bigint PRIMARY KEY,
…
)
FEDERATED ON (customer_id=c_id);
( )
CREATE TABLE order (
item_num int,
customer_id bigint,
date_sold datetime2,
…,
CONSTRAINT PK_Order PRIMARY KEY (item_num, customer_id, date_sold),
CONSTRAINT FK_Cust FOREIGN KEY customer_id REFERENCES
customer (customer id)
(customer_id)
)
FEDERATED ON (customer_id=customer_id);
CREATE TABLE product (
product_name varchar(100) NOT NULL,
unit_price money,
item_num int PRIMARY KEY,
…
);
26

27. More Detail
• Supported data types for federation key : bigint, int, GUID, and varbinary
Supported data types for federation key : bigint int GUID and varbinary
(900)
– Only range partitioning
• Federation key must be part of unique index
• Foreign key constraints only allowed between federated tables and from
federated table to reference table
• Not all Azure programmability features supported
– Sequence timestamp
Sequence, timestamp
• Additional surface area restrictions
– Indexed views, drop database (members)
• Schemas are allowed to diverge over time
g
– Furthermore, in v1, schema updates to existing members must be done in each
member (where the change is needed)
• USE FEDERATION “rewires a connection”
– Connection is reestablished
Connection is reestablished
– All existing settings and context of the connection is lost (sp_reset_connection)
– Must be in a batch by itself
27

28. Connect to Atomic Unit: Filtered
Existing Database
g
When using into a specific key
value, SELECT will only return
sales
records from federated tables that
match that value. It will still return
way
all records from non‐federated
Gatew
tables.
Connection:
USE FEDEDERATION sales (customer_id=3) Inserts and UPDATES operating
WITH FILTERING=ON, RESET;
outside of the value will fail.
net;
er=az1cl321.db.windows.n
Range: Min...Max
customer order product
3
base=MyDB;
3
=AppUser;
SELECT * from customer
wd=****;
SELECT * from product
SELECT * from order
Serve
Datab
User=
Passw
Federation Member
28

29. More on Connection Filtering
• Most operations behave differently in filtered vs
unfiltered connections
• C
Connection filtering is a property of the session
ti filt i i t f th i
– Filter injected dynamically at runtime
– Cannot inspect source code to determine how it behaves
Cannot inspect source code to determine how it behaves
• E.g., running stored proc written for filtered mode on unfiltered
connection could lead to unintended results
• There are several operations that will not work in
There are several operations that will not work in
filtered connection in v1
– DDL, DML on reference tables, …
• Fan‐out, bulk operations not efficient in filtered mode
– For now, filter=off is our best offer
29

30. Support Matrix
Connection Type Filtered Unfiltered Named
Operation (unfiltered)
Dynamic SELECT
DML (federated tables)
DML* (federated tables)
DML* (reference tables) X
DDL X
Views (not indexed)
UDF ‐ activate
Stored Proc ‐ activate
Trigger (all modes) ‐ activate
CREATE/UPDATE Stats
CREATE/UPDATE Stats X
Bulk Ops
openrowset bulk, bcp, bulk
insert X
* not including SELECT & modules
^ autostats will work on all connections
System stored procs, intrinsics will be unaffected (run unfiltered)
30

31. Splitting a Member
Existing Database
g
ALTER FEDERATION sales
l
sales SPLIT AT (customer_id=50)
way
Gatew
USE FEDERATION ROOT
Connection:
WITH RESET
Using to the
net;
federation ROOT
er=az1cl321.db.windows.n
will pop you out of
a member back Range: Min...Max
into the database
that hosts the customer order product
federation 3 3
base=MyDB;
=AppUser;
wd=****;
40 58
58 58
Serve
Datab
User=
Passw
Federation Member
31

32. Two New Members
Existing Database
g
ALTER FEDERATION sales
l
sales SPLIT AT (customer_id=50)
way
Gatew
USE FEDEDERATION ROOT
WITH RESET
Connection:
net;
er=az1cl321.db.windows.n
Range: Min...50 Range: 51...Max
customer order product customer order product
3
base=MyDB;
3
=AppUser;
wd=****;
58
40 58 58
Serve
Datab
User=
Passw
Federation Member Federation Member
32

33. Two New Members
Existing Database
g
sales
way
Gatew
USE FEDEDERATION sales (customer_id=40)
Connection:
WITH FILTERING=ON, RESET;
net;
er=az1cl321.db.windows.n
Range: Min...50 Range: 51...Max
customer order product customer order product
base=MyDB;
SELECT *
=AppUser;
wd=****;
from customer 58
40
40 58 58
SELECT * from order
Serve
Datab
User=
Passw
33