A brief presentation on the ATA-over-Ethernet protocol and how it can be used to create an inexpensive enterprise-ready SAN.

1. ATA Over Ethernet
7 March 2008
KULUA Meeting
Lawrence Public Library, Lawrence, KS

2. What is it?
• Encapsulation of ATA commands inside
Ethernet frames
• Non-routeable: transmitted directly via
Ethernet, rather than (e.g.) TCP/IP
• Hosts can use AoE-attached disks as if they
were directly (e.g. via SATA) attached.
• Cheaper lower-level alternative to iSCSI

3. How to get it?
• Purchase EtherDrive appliance from Coraid.
This is a commodity x86(64) server, generally
SuperMicro, with a number of hot-swap SATA
trays and Coraid OS.
• Purchase a device based on LayerWalker
MiniSAN chip
• Roll your own

4. Roll your own?
• Build commodity Linux server.
• Put physical disks for export (or RAID arrays)
into an LVM VG.
• Create as many separate LVM LV’s as you want
to export.
• Export LV’s using vblade(d), kvblade, or
ggaoed
• Physical volumes need not be (S)ATA!

5. Wasn’t this ATA over Ethernet?
• LVM appears to take care of all of it.
• I have successfully created and exported LV’s
via AoE on top of SATA, SCSI, and Adaptec SCSI
RAID PV’s.
• This means you can throw a bunch of old SCSI
disks into a Linux box, export them via AoE,
and use them for stuff.

6. Example configuration
Gigabit Ethernet switch
OS on SSD LAN
AOE shelf AOE shelf
AOE shelf AOE shelf

7. Example configuration explained
• Linux server has OS mounted on SSD
• Connected to AOE “shelves” via dedicated
gigabit network
• AOE shelves configured with appropriate RAID
• Each shelf exports a single logical device
• Logical devices configured on server as PV’s in
single VG
• LV’s created on server in VG

8. Advantages
• In the example configuration, it is trivially easy
to add storage to the VG on the server:
• Attach another AOE shelf (e.g. #5)
• Extend VG with device exported by shelf (e.g.
0)
• vgextend vg00 /dev/etherd/e5.0
• Extend appropriate LV(s), filesystems

9. Caveats
• AoE is non-routeable. SAN must be on same logical
Ethernet LAN as server
• Set up AoE devices on server as PV’s and put them into
an LVM VG. Otherwise not so easy to extend later.
• Set up syslog on related AoE shelves to point to single
host for ease of management.
• 1500 byte MTU only allows for two 512b sectors.
Increase MTU to 9000 for better performance.
• AoE volumes potentially accessible via multiple
machines. If mounting the same volume from two
machines, use an FS that supports this, like ClusterFS.

10. Required Software for Linux
• AoE modules have been in the kernel since
2.6.11
• aoetools – tools to detect and display
available AoE volumes, as well as tools to
export LV’s/partitions via AoE –
http://aoetools.sourceforge.net/

11. More information
• http://en.wikipedia.org/wiki/ATA_over_Ethernet
• http://www.coraid.com/
• http://kitstechblog.blogspot.com/search/label/aoe
(my personal experiments with AoE)
• http://www.lbserver.org/aoe/ (AoE bootable CD, can
both mount and export volumes)

12. I can help!
• Available to consult on AoE
installations/deployments
• popefelix@gmail.com
• (843) 819-9065