Rangarajan discusses the evolution of virtualization at Gilt

1. O’REILLY
Virtualization at Gilt
Rangarajan Radhakrishnan, ranga@gilt.com
Infrastructure Engineering
Jun 6, 2013

2. O’REILLY
What is Gilt Groupe?
> Founded in 2007, Gilt provides a curated,
personalized experience featuring products by today’
s top designer labels, at up to 60% off retail
> Products across four stores: Men, Women, Home,
and Baby & Kids

3. O’REILLY
Gilt Tech:
> The engine behind the Gilt experience
Infrastructure Eng:
> Tasked with improving efficiency and reliability

4. O’REILLY
Tech at Gilt
Offices: NYC, Dublin and Portland
Infra Eng: NYC and Dublin
Architecture: NYC, Portland and Dublin
Engineering: NYC, Portland and Dublin

5. O’REILLY
Gilt Tech History
> 2007: Small Debian footprint with Ruby on Rails
> 2008: Migrated to OpenSolaris (from RoR to
Java services)
> 2010: Migrated to CentOS (from Java services to
Scala services)
=> =>

6. O’REILLY
Current Technology at Gilt

7. O’REILLY
Technology
> Databases: PostgreSQL, MongoDB
> Search: Solr, Splunk
> Frameworks: Play
> Languages: Scala, Java, Ruby, Node.js
> Config mgmt: Puppet
> Continuous integration: Jenkins
> Messaging: RabbitMQ

8. O’REILLY
Virtualization Goals
> Build on what is available
> Avoid a one-size-fits-all approach
> Offer seamless integration with existing
application stack
> Use lightweight containers that offer great
performance

9. O’REILLY
Additional Virtualization Goals
> Automate provisioning as much as possible
> Feature immutable containers -- avoid
uncertainties as a result of changes over time
> Avoid repeated enforcement of config
management policies

10. O’REILLY
Virtualization History at Gilt:
The Past
> Bare metal: Debian (2007 - 2008)
> OpenSolaris zones (2008 - 2010)
> Migration to CentOS (201

11. O’REILLY
Virtualization History at Gilt:
The Present
> Xen VMs (since 2010)
> LXC (since 2012): strong performance, low
overhead
> EC2 (off and on since 2008), with varying
degree of prod/non-prod use (mostly spillover
traffic)
> Apache CloudStack (since 2013): to increase
utilization and maximize efficiency

12. O’REILLY
Why CloudStack for
Virtualization?
> open source
> support for multiple
hypervisors (KVM, Xenserver and, recently, LXC)
> great web UI and Rest API
> increasingly active community
> committer in-house

13. O’REILLY
Virtualization at Gilt with
Cloudstack and Galactica
> CloudStack primarily for use with LXC containers
> CloudStack integration with Galactica (in-house)
> Galactica implements a meta-data model
> Galactica integrates with Puppet/LDAP etc.

14. O’REILLY
Galactica ...
> is a family of services
> allocates containers (via CloudStack)
> allows for provisioning of other services
> implements a meta-data model for:
>> HW/SW provisioning/management
>> User access provisioning/management
>> IP address/DNS provisioning/management

15. O’REILLY
> maintains desired state of services
> provides DNS updates for services, especially
when provisioning or decommissioning
> updates load balancer config
> integrates with monitoring system/audit
> integrates with ioncannon (automated and
continuous deployment)
Galactica also:

16. O’REILLY
Why LXC?
> lightweight: minimal performance overhead
> tested in staging with good results
> proof of concept with live traffic showed good
performance
> significant work to support LXC in cloudstack
"LXC will be implemented as a hypervisor in CloudStack and will be a
first class citizen similar to other hypervisors ..."
- Phong Nguyen, Gilt (CloudStack committer)

17. O’REILLY
Other Key Pieces
> config management system: Puppet 3.1
(for first-time run on container, then turned off)
> Cobbler 2.2.3 for kickstarting bare metal servers
(with both Ubuntu and CentOS profiles)
> Jenkins: used for continuous integration and
testing

18. O’REILLY
Recent and Ongoing Work
> Testing provisioning LXC containers with
different libvirt versions
>> Up to 1.0.5 on CentOS 6.3 and 0.9.8 on
Ubuntu
> Testing entire application stack on CentOS 6.4
LXC container on Ubuntu host
>> Trying to avoid libvirt issues that came up
with CentOS host

19. O’REILLY
Recent and Ongoing Work
(cont.)
> Testing entire application stack on Ubuntu 12.04
LTS LXC container on Ubuntu host
>> Challenge: requires different packaging
mechanism
> Provisioning LXC with and without libvirt to
identify issues with libvirt

20. O’REILLY
For More Info:
> Gilt Tech blog: http://tech.gilt.com
> Gilt Twitter: @gilttech
> Gilt Facebook: facebook.com/gilttech
> Work for us: gilt.com/careers/tech
Thanks to:
> CloudStack community
> Gilt Tech team
> NYCDevOps