Танки в облаках

Copyright © 2016 Mirantis, Inc. All rights reserved
www.mirantis.com
Tanks in The Clouds
Ivan Krovyakov, Mirantis

What we are going to discuss
Contents
● Wargaming.net: overview
● Cloud use case: what and why?
● How we did it
● Learnings and achievements
● What’s next
Speakers
Ivan Krovyakov
Cloud Solutions Architect, Mirantis

Wargaming and G-Core Labs in a nutshell
Wargaming.net
● International game developer
● 18 years on the market
● 2 Guinness World Records
● 15 offices worldwide (incl. Austin, TX)
● 4000+ employees
● 150M+ users / 1,114,000 peak online
G-Core Labs
● Luxembourg-based IT provider
● Europe/CIS/Asia/US DC coverage
● Direct peering to major ISPs
● HTTP/P2P CDN
● 500+ GbpS bandwidth

Wargaming: initial state
People, processes, technologies
Release Managers
Dev/QA DevOps
Administrators
G-Core team L1 support

Project Business Goals and Use Cases
● Business Goals
● Reduce costs by embracing open source platform and increasing utilization
● Reduce time-to-production by using API and automation for manual operations
● Use Cases: four-step workload lifecycle
● Staging Trunk: personal virtual sandboxes to debug/troubleshoot the code
● Staging Stable: pre-release testing of the new software versions
● Prod Test: pre-production small scale public A/B testing
● Production: publically available production workloads

Mirantis approach: Assess – Build – Sustain
SustainBuildAssess
Assess ● Analyze the business and technical goals
● Map the goals to OpenStack capabilities
● Lock down the MVP
Build ● Develop all the plugins needed
● Build staging environment
● Roll out the production environment(s)
Sustain ● Product support
● Plugins and customizations maintenance
● Roadmap including scaling and upgrades

Mirantis approach: Assess
Architecture Design Assessment (ADA)
● Identify cloud users and their use cases (technical and business)
● Collect and prioritize all the requirements (any level)
● Map MOS capabilities to the customer’s needs, lock MVP and get inputs for SoW
Goals
● Executive Summary and Overview
● A review of business objectives, critical success factors, core use cases, users and stakeholders.
● Technical Response (MVP)
● Focus on the technologies and products required for delivering the expected business value
● Functional & Operational Response
● Recommendations on successful operation of proposed technical solution for MVP
Format ● 3-5 days onsite workshop with Mirantis architects and customer stakeholders
● Offsite work to design and document the future cloud solution
● 10-15 business days project in total including customer review and readout
Artifacts
SustainBuildAssess

SustainBuildAssess
Mirantis approach: Assess
Wargaming ADA deliverables

Mirantis approach: Build
Phase 1: Development
SustainBuildAssess
Low-level design
● Component-level decision making
● Project scope refinement
Mirantis OpenStack plugin development
● Dedicated lab environment
● Separate QA process
Quality Assurance preparation activities
● Test planning
● Test automation using common tools: Rally, Tempest, Mirantis Health Check (OSTF)

Wargaming development phase deliverables
● Fuel plugins
● Active Directory as authentication backend for Keystone
● NFS as a storage backend (ephemeral, image, persistent)
● Disabling anti-spoofing rules for VMs (multi-IP containerized apps)
● Integration with external Puppet master
● Test plans and test report templates (Rally, Tempest)
● Acceptance process
● Demo scenarios
SustainBuildAssess

Phase 2: Deployment
SustainBuildAssess
Hardware preparation
Network infrastructure configuration
Handover to Mirantis support
Pilot cloud deployment
QA acceptance according to test plan
Wargaming Ops start using cloud
Feedback provided
Pilot cloud
deployment
(MVP 0.1)
Remaining
clouds
rollout
10 days each
CRs, scope refinement
Plugins / configuration
changed according to
feedback
Hardware preparation
Network infrastructure configuration
Handover to Mirantis support
Documentation
Pilot cloud deployment
QA acceptance according to test plan
Handover to Wargaming Ops
Extra changes based on
customer feedback
End of project
Retrospective

Wargaming deployment phase deliverables
● 4x clouds built on Mirantis OpenStack 7 (Kilo/Ubuntu 14.04/KVM)
● 1x development MVP (WG internal sandbox)
● 1x staging MVP (12x computes, 288x CPUs, 1.5TB RAM)
● 2x production MVPs (19x computes, 372x CPUs, 2.6TB RAM)
● Basic set of OpenStack services (Glance, Keystone, Nova, Cinder, Ceilometer,
Heat, Murano) + set of developed plugins
● Control plane HA: Mirantis reference architecture
● Neutron: OpenVSwitch/VLANs, provider networks
● Basic CMDB integration via Murano application (made by Wargaming)
SustainBuildAssess

Mirantis approach: Train
Extra step to make it work
Custom-tailored training based on the variety of existing offerings has been delivered
● OS100 – vanilla OpenStack 3-day course which covers various aspects of running
OpenStack cloud core components
● Architecture
● Best practices
● Provisioning workflow
● Component interaction
● Real-world challenges faced by OpenStack experts
● FUEL100 - 2-day course focused on Mirantis OpenStack extra components
● Fuel
● Murano
● Sahara
● Ceph
SustainBuildAssess
Train

Mirantis approach: Train
Wargaming training deliverables
SustainBuildAssess
Train

Mirantis approach: Sustain
● Commercial enterprise-grade support subscription
● Mirantis OpenStack as-is
● MOS packages
● Reference architecture
● Deployment and orchestration engine
● Publically available plugins and extensions
● Customer case-centric maintenance
● Custom developed content maintenance
● Assistance to roll out maintenance updates
● Upgrades
● One-time upgrade
● Upgrade subscription
SustainBuildAssess
L1 Support
L2 Support
Technical Escalations Team

Wargaming: business goals
Initial results
● Reduce time-to-production
● DevOps and Admin teams were OpenStack enabled
● Self-service IaaS is already in use
● DevOps team provisions virtual environments on their own
● Reduce costs
● Workloads onboarding to OpenStack started
● > 100 VMs are already onboarded
● Data collection for capacity management started
● Cost
● Utilization
Source: http://cloudtweaks.com

Wargaming: lessons learned so far
● Don’t treat OpenStack as vCenter
● Paradigm shift, not just a technology change
● No feature parity now and a few years ahead
● Be prepared to revise your business processes
● Think SLAs, cross-team interaction, working closely with product teams
● Pluggable cloud architecture: the key to long-term success
● Modules: extra Dev $ spent -> Ops $$$ saved
● Localized technical debt -> easier to deploy, sustain and upgrade
● Plan operational readiness beforehand
● Again, people and processes >> technology
● Train and enable Ops on OpenStack before going into production

Wargaming and Mirantis: next steps
● Bare-metal -> OpenStack workloads onboarding
● Kilo -> Mitaka (MOS 7 -> MOS 9) upgrade
● Scale out production clouds
● Advanced IaaS features
● Load Balancing as a Service
● DNS as a Service
● PaaS features
● DB as a Service
● Hadoop as a Service

Tank you for your attention!
http://static-ptl-us.gcdn.
co/dcont/fb/image/military_month_6
84x280_3YmN6Fy.jpg

Wargaming workloads
What is a realm?
YAML which includes:
○ region to deploy to (like “EU”)
○ global services to use (“auth”, “dbms”, ...)
○ games to deploy (“wot”, “wowp”, “wotb”, ...)
■ applications to deploy for each game
● names/IP-addresses of nodes to deploy to
● Extra config info
dbms:
- host: *wowp-aap-host0
version: '5.6'
port: 3306
type: percona
databases:
- &wowp_aap_default
name: aapwowpwgt21
host: *wowp-aap-host0
password: ***
user: ***
mqvhosts:
- &wowp_aap_amqp_default
password: ***
user: ***
vhost: /aapwowpwgt21
wowp:
aap:
version: 3
bonus_check_events: False
pay_events: False
amqp:
default: *wowp_aap_amqp_default
http:
ip: *wowp-aap-ip0
mysql:
db: aapwowpwgt21
password: ***
user: ***
db:
default: *wowp_aap_default
=== CUT ====

Wargaming: core projects
Currently, Wargaming is focused on its team-based MMO war series
dedicated to mid-20th century warfare. This lineup includes the
company’s flagship armored MMO World of Tanks, launched in April
2011; the fast-paced flight combat MMO World of Warplanes; and the
epic naval warfare shooter World of Warships.
150+ million users in Wargaming.net Universe and value each and every player.
■ Best European Publisher at EGA 2012
■ 2 Guinness World Records (project: WoT, years: 2011, 2013);
■ 2 Golden Joystick Awards (project: WoT, title: MMO game of the year, year: 2012 and 2013);

What is Mirantis
Mirantis is the pure play OpenStack company, delivering all the
software, services, training, and support needed for running
OpenStack. More customers rely on Mirantis than on any other
company to get to production deployment of OpenStack at scale.
Mirantis is among the top three companies worldwide in contributing
open source software to OpenStack, and has helped build and deploy
some of the largest OpenStack clouds in the world, at companies such
as Cisco, Comcast, Ericsson, NASA, Samsung and Symantec.

G-Core in a nutshell
G-Core is a Luxembourg-based global IT solutions provider offering a
wide range of services, including hosting, CDN, peering network, and
different levels of support, from smart hands to DDoS protection to
advice on local regulations, and more. Initially developed to support
online gaming, G-Core has three main KPIs: low latency, high
availability and cost optimization. These parameters run through many
online industries, such as banking or streaming services, expanding
client base far beyond gaming.
1 114 000 peak players online

Tank you for your attention!
Any questions?

Wargaming before OpenStack - the Scale
● TBD

Wargaming: initial stack
● 3 level platform stack
○ Hardware
○ Virtualization layer, VMWare (optional)
○ Containerized applications, OpenVZ.
● Several engineering independent teams, responsible for their own
part
○ Release Managers
○ Dev/QA engineers
○ DevOps engineers
○ Administrators
○ G-Core team
○ L1 support team
● JIRA-based provisioning workflow

Wargaming: cloud use cases
4 steps for a workload to get from developer to end-user
● Staging Stable: pre-release testing of the new versions of software
● Production Test: public beta-testing of limited deployments of production-ready versions of
software

What was wrong? The business drivers
Wargaming has asked Mirantis to implement a cloud solution based on
Mirantis OpenStack to ease their current pain points and offer a flexible
solution to their Online Presence team cost effectively and without
compromising on performance and scale.
● Reduce costs of existing and new infrastructure
● Improve infrastructure utilization
● Capability for automation of infrastructure provisioning
● Shorten release cycle and increase business agility

Phase 2: Deployment
Once content is developed and tested OpenStack rollout starts
● Pilot cloud deployed
○ Hardware preparation, network infrastructure configuration
○ QA acceptance according to the plan
○ Handover to Mirantis support
○ Wargaming operations team starts playing with the cloud and provides
feedback.
● Customer feedback addressed by Mirantis
● Rest of the clouds deployed
○ Preparation
○ Deployment
○ QA acceptance
○ Handover to support and operations
SustainBuildAssess

What has been done so far: numbers
4 cloud were built with Mirantis OpenStack 7 (Kilo/Ubuntu 14.04
/KVM)
● 1 development cloud for WG team to play with
● 1 staging cloud, 12x computes, 288x CPUs, 1.5TB RAM
● 2 production clouds, 19x computes, 372x CPUs, 2.6TB RAM

What has been done so far: features
● Control plane High Availability - Mirantis reference architecture
● MOS plugins
○ Active Directory as authentication backend for Keystone
○ NFS as a storage backend (ephemeral, image, persistent)
○ Disabling anti-spoofing rules for VMs (multi-IP containerized apps)
○ Integration with external Puppet master
● Basic set of OpenStack services: Glance, Keystone, Nova,
Cinder, Ceilometer, Heat, Murano
● Neutron - OpenVSwitch/VLANs, provider networks
● Basic integration with external CMDB as a Murano application

Wargaming: business goals
Initial results
● DevOps and Admin teams were trained to operate OpenStack
● Infrastructure as a Service has been provided to the cloud users
○ DevOps team provisions virtual environments on their own
■ “Business agility” driver is being addressed
● Migration of workloads has been started
○ more than 100 VMs have been migrated so far
■ “Cost reduction” driver is being addressed
● Wargaming has started collecting data for long-term number-
driven management of cloud capacity
○ Cost
○ Utilization

Project Business Goals and Use Cases
Wargaming has engaged Mirantis to implement a cloud solution based on
Mirantis OpenStack to ease their current pain points and offer a flexible solution
to their Online Presence team cost effectively and without compromising on
performance and scale.
● Business Goals
● Reduce CAPEX by replacing VMware with MOS and increasing infra utilization (~50% now)
● Reduce time-to-production by using API and automation for manual operations
● Use Cases: four-steps workload lifecycle
● Staging Stable: pre-release testing of the new software versions
● Prod Test: pre-production small scale public A/B testing

Танки в облаках

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (14)

Semelhante a Танки в облаках

Semelhante a Танки в облаках (20)

Mais de OpenStackRussia

Mais de OpenStackRussia (15)

Último

Último (20)

Танки в облаках