1. Samba as a gateway to OpenAFS
Fabrizio Manfredi Furuholmen
SambaXP April 2007

2. Agenda
Goal
Solution
Gateway Architecture
Gateway Configuration
Integration Tools
Tuning
Performance
Result
Page 2

3. Goals
Project Goal
Primary goal of the project was to design and build
an inexpensive storage system
Requirements:
Handle terabytes of data
Transparent to final user
Working in WAN environment
Good level of scalability
Page 3

4. Solution
Considerations
Centralize Storage (hardware solution)
SAN
Blockdevice interface
Performance
NAS
Filesystem interface
Shared filesystem
Distributed Filesystem (software solution)
Filesystem interface
Single file system across multiple computer nodes
Page 4

5. Solution
Considerations
Big Server vs Small Server (Google Techs)
Small number of inexpensive fileservers
provides similar performance to client side
Increase in capacity are inexpensive
Better manageability and redundancy.
Page 5

6. Solution
Storage Price
Terabyte Cost (SAS/FB) Components NAS SAN DFS
14k euro NAS/SAN
4k euro DFS Storage1.5 Tb with 52.000 52.000
10 disks (110/150)
Disks Size Storage 14TB 200.000 200.000
143 vs 300 SAS/FB reduce 30% 100 disks (110/150)
Disks Type 3 Server Storage 9.000
250/500 SATA Disk reduce >50% 500Gb (SAS)
14 Server Storage 56.000
1Tb (SAS)
Installation
4 FB interface 1.600
Software 2 Switch FB 6.000
2 Server Gw 2.000 2.000
Discount 2 Switch Gb 1.200 1.200 1.200
TOTAL for 1.5 Tb 53.200 62.800 12.200
Administration
TOTAL for 14 Tb 201.200 210.800 59.200
Page 6

7. Solution
Solution
Distributed Filesystem
AFS
Free available and stable
Support of large installations (>200TB with 40 milion Files)
More then 20 platforms are supported
Aggressive Roadmap ($350,000 per year from CSG)
Samba (Gateway)
AFS windows client uses internal file server emulation (slow)
Clientless
Fast and stable
User Identity
Heimdal Kerberos Autentichation (SSO)
KA emulation
LDAP backend
2b protocol (large kerberos ticket)
Openldap
Centralize storage
User administration scripts (custom provisioning)
Page 7

8. Solution
AFS Features
Transparent Access and Uniform Namespace
Cell
Partitions and volumes
Mount Points
In-use volume moves
Scalability
Client Caching
Replication
Load balance among servers while data is in use
Security
Authentication and secure communication
Authorization and flexible access control
System Management
Single system interface
Administration tasks without system outage
Delegation
Backup
Page 8

9. Gateway Architeture
Architeture
Scalability
Storage scalability
(Filesystem layer)
User scalability
(Samba Gateway layer)
Performance
Load balancing
Roaming user/branch office
Clientless
Centralized Identity
Kerberos
Ldap
Page 9

10. Gateway Configuration
Enable AFS in Samba
Compile Options
Enable KA server emulation
--with-fake-kaserver
Enable AFS ACL mapping
--with-vfs-afsacl
Don’t use AFS clear text password switch (old not supported)
--with-afs
Setting Samba Trusting (undocumented)
Obtain KeyFile from AFS fileserver (/usr/afs/etc/)
Import an OpenAFS KeyFile into secrets.tdb:
net afs key AFSKeyFile
Custom script for AFSKeyFile sync (Key rotation)
Useful command (undocumented)
Impersonate user, create a token for user@cell:
net afs impersonate <user> <cell>
Page 10

11. Gateway Configuration
smb.conf
Mapping Domain User<-> Pts
Single domain/unique identification:
afs username map = %u@zero.it
Multiple domain/duplicated identification
Store DOMAIN+user:
afs username map = %D+%u@zero.it
Store the SID in pt server:
afs username map = %s@zero.it
Enable AFS share
afs share = yes
Page 11

12. Gateway Configuration
smb.conf locking
Access only from samba server
Samba default
Access only from samba and local gw
Disable oplocks , level2 oplocks ..
Only with Byte-range locking on AFS client (AFS>1.5.X)
Access from all system
Enable strict locking option (mandatory lock)
Page 12

13. Gateway Configuration
Samba scalability and HA
Primary server HA (DFS Root)
Heartbeat
VIP associated to primary Samba Server
Transparent Access (MSDFS)
No compile option required
Enable DFS on Primary Samba server
host msdfs = yes
Samba Scalability
DFS Proxy,
Share redirection
Name resolved with DNS (link is FQDN)
(ex. msdfs proxy = gw1.intranet.zeropiu.itshare)
DFS root ,
Directory link
Fault tolerance
(ex. ln –s msdfs:server1share1,server2share1 share1)
Page 13

14. Gateway Configuration
Identity Storage
Heimdal integration
Compile
Enable ldap backend (--with-openldap)
Configuration
Enable ldap backend
Enable 2b token for Kerberos V integration
Only if have old client: enable-kaserver / afs3-salt
LDAP
Openldap 2.3 (SASL EXTERNAL)
Extending Schema (Samba,hdb ..)
nss_switch with ldap support
PAM
PAM Kerberos V integration
Page 14

15. Integration Tools
Identity Administration
Custom user administration script (iauser.pl)
Unix user (ldap)
Samba user (ldap)
Kerberos user (ldap)
Pt server user
Volume and mount point
Groups administration script (iagroup.pl)
Create unix group (ldap)
Create samba group (ldap)
Create pt server group
Syncronization administration script (ptsSync.pl)
Synchronization user from ldap to pt server
Page 15

16. Performance
Test Enviroment
Hardware
3 FileServer Linux
2 GB of RAM, 3GHz Xeon processor
2x36Gb SAS RAID 1 for operating system partition
4x 143GB SAS RAID5 storage
2 Server Gateway Linux
2 GB of RAM, 3GHz Xeon dual processor
2x36Gb SCSI RAID 1 for operating system partition
Software
Samba 3.22
OpenAFS 1.4.2
IOzone 3.8
Page 16

17. Performance
Samba Client
write
45000
40000
Client: 35000
30000
40000-45000
35000-40000
Windows XP sp2 25000 30000-35000
25000-30000
20000
20000-25000
Server:
15000
15000-20000
10000 10000-15000
16384
5000 5000-10000
Linux 2.6.9
1024
0 block 0-5000
64
64
256
1024
4096
Samba 3.22
4
16384
65536
262144
kb
Write: 30-43MB/sec read
Read: 40-50MB/sec 70000
60000
50000
60000-70000
40000 50000-60000
40000-50000
30000
30000-40000
20000 20000-30000
16384 10000-20000
10000
1024 0-10000
0 block
64
64
256
1024
4096
4
16384
65536
262144
kb
Page 17

18. Performance
write
AFS Client 40000
35000
Client: 30000
35000-40000
25000
Linux 2.6.9
30000-35000
20000 25000-30000
20000-25000
openafs 1.4.2
15000
15000-20000
10000 10000-15000
16384 5000-10000
Server:
5000
1024 0-5000
0 block
64
64
256
Linux 2.6.9
1024
4096
4
16384
65536
262144
kb
openafs 1.4.2
read
Write: 20-37MB/sec 90000
80000
70000
80000-90000
60000 70000-80000
Read
50000 60000-70000
50000-60000
40000
40000-50000
Cold Cache: 22-28MB/sec
30000
30000-40000
20000 20000-30000
10000 10000-20000
Warm Cache: >45MB/sec 0
131072
16384 43
0-10000
4
16
64
256
2048
1024
4096
16384
a
Page 18

19. Performance
write
Samba GW 25000
Server: 20000
Linux 2.6.9 15000
20000-25000
15000-20000
openafs 1.4.2 10000 10000-15000
5000-10000
Gateway: 5000 16384
1024
0-5000
Linux 2.6.9
0 block
64
64
256
1024
4096
4
16384
65536
262144
OpenAFS 1.4.2 kb
Samba 3.22 read
Client:
Windows XP sp2 45000
40000
35000
40000-45000
30000 35000-40000
Write: 18-25MB/sec 25000
20000
30000-35000
25000-30000
20000-25000
15000
15000-20000
10000
Read
16384 10000-15000
5000 5000-10000
1024
0 0-5000
Warm Cache: 30-40MB/sec
64 block
64
256
1024
4096
4
16384
65536
262144
kb
Page 19

20. Performance
Throughoutput Coparison
write
45000
40000
35000
30000
AFS
25000
Samba
20000
Samba-GW
15000
10000 read
5000 90000
0
80000
2
4
8
6
2
4
8
6
72
44
88
2
4
9
9
38
76
53
51
70000
10
21
42
10
20
40
81
16
32
65
13
26
52
filesize 60000
50000 AFS
Samba
40000 Samba-GW
30000
20000
10000
0
2
24
48
96
92
4
8
6
72
44
88
38
76
53
51
10
21
42
10
20
40
81
16
32
65
13
26
52
Page 20

21. Performance
Tuning
Samba Configuration (increase 30%)
Enable socket options = TCP_NODELAY (Default)
Increase SO_RCVBUF (16384)
Increase SO_SNDBUF (32768)
AFS Cache Manager (increase 20%)
Increase block size (chunksize 19)
Increase cache elements (dcache 5000)
Increase server daemon (daemons 6)
Increase rx packet (rxpck 2000)
Increase data cache file (files 50000)
Increase Cache size (cache size 4gb)
Use separated disk for cache
Page 21

22. Result
Benefit
Reduced cost
Reduced storage cost 40.000 Euro (1.5TB Storage)
Reduced down time
Increase performance
Client side
Simplify System Administration task
Data accessible from everywhere
High security level (kerberos base)
Single sign-on
Disaster recovery (Volume replication)
Page 22

23. Next
Under Testing
OpenAFS
Lock subsystem, support AFS 1.5.X (Byte range)
Windows client, support AFS 1.5.X
Inode interface
Socket communication vlserver/fileserver
Memory cache
Disable fsync on write (AFS 1.5.X + patch)
WebDav
Samba
Cluster
External project (www.beolink.org)
Ptserver with ldap backend (ptsldap)
Web Administration interface (AFS Manager)
Page 23

24. The End
Manfred at zeropiu.it
www.beolink.org
Page 24