Anders Walgren, Chief Technology Officer, Electric Cloud at DevOps Enterprise Summit 2014 Link to video: https://www.youtube.com/watch?v=G_coTcbj1FU Huawei is a $40B company delivering communications technologies for telecom carriers, enterprise and consumers. This talk will provide an overview of Huawei’s Continuous Delivery and DevOps transformation initiatives in a complex embedded software evironment. Ting Zhou Principal Solution Architect for Huawei, alongside Electric Cloud’s CTO Anders Wallgren, will present the business drivers and benefits of their Continuus Delivery solution encompassing build, infrastructure provisioning, deployment and testing plus reporting including release pipeline visualization and progress dashboards. Huawei’s CD and DevOps implementation is a centralized, shared cloud service currently used by 2,000 developers supporting 20 applications and is in process of being extended to 40,000 developers servicing over 1000 applications. Some additional statistics on our current implementation: – More than 2000 releases per year – More than 50,000 compile & builds per day – More than 1million test cases run per day – More than 30million LoC, product is complicated – More than 480K code review/analysis per year – More than 170k system integration testing per year The benefits of our solution include: – Reduced cost of delivering software – Increased resource utilization and productivity – Shorter time to market with higher quality

1. © Electric Cloud | electric-cloud.com
Anders Wallgren
CTO, Electric Cloud
Huawei’s journey to
Continuous Delivery

2. © Electric Cloud | electric-cloud.com
Who is Huawei?
An Innovative Industry Contributor
An ICT Industry Leader
A privately-owned Global Company

3. © Electric Cloud | electric-cloud.com
Who is Huawei?
Revenue by business
 Devices
 Devices Chipset
 Devices Cloud
Consumer
Business Group
 Fixed Network
 Wireless Network
 Telecom Software
and Core Network
 Service
Carrier
Business Group
 Enterprise
Networking
 UC&C
 IT
 SecoSpace
 service
Enterprise
Business Group
22%
Consumer
5%
Enterprise
73%
Carrier
(2012)

4. © Electric Cloud | electric-cloud.com
16
R&D Centers
140+
Countries
14
Regional HQs
28
Joint Innovation
Centers
150K
Employees
Worldwide
45
Training
Centers
33.4%
14.5%
17%
35.1%
Revenue by Geo
China
EMEA
Asia Pacific
Americas
70K
R&D Employees
(2012)

5. © Electric Cloud | electric-cloud.com
• 41,948 patents in China
• 14,494 patents in ROW
• 12,453 international PCTs
• USD$300 million/yr royalties
• 150+ standards organizations
• 30,000+ standards proposal
1.5Bn
2Bn
2.7Bn
3.8Bn
4.8Bn
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Accumulated R&D investments:
USD 23 billion

6. © Electric Cloud | electric-cloud.com
Makin’ it real
Huawei Phone - 2012 Huawei Phone - 2014

7. © Electric Cloud | electric-cloud.com
Previous State
• Large scale R&D environment with
complicated tools requires multi-thousands
of CPU cores available on demand.
• Environment/tools provisioning is very time-
consuming; lab asset utilization is low.
PROD BUILD
300min
Per 10M LOC
DEV BUILD
10min
Per 5M LOC
FULL TEST
1440min
REGRESSION TEST
240min
FEATURE DELIVERY TIME
30days

8. © Electric Cloud | electric-cloud.com
What next?

9. © Electric Cloud | electric-cloud.com
The Evalution
Premise 1:

10. © Electric Cloud | electric-cloud.com
The Evaluation
Design
modeling
Code analysis
CI
Redmine
Func Testing
Requirement
Analysis
Project Mgmt
SCM
Collaboration
Design Development Testing
gUnit,cppUnit
Automation
Unit Testing Selenium
Pef Testing
IOMeter
Others
Hadoop
Provision
Chef
Travis-CI
memcached
Github
Reviewboard
Trac
Puppet
RabbitMQ
ActiveMQ
Hive
HBase
CFEngine
Subversion
Nagios
Monitor
CloudStackOpenStackCloud Platform
SourceForge
Jenkins
PcLint valgrind
Jmeter
MobitestgMock,JMock
CppCheckCpplint
Page Speed
JIRA
GIT
Gerrit
Mercurial
OpenGrok
Bugzilla
Orchestration Jenkins RunDeck ThoughtWorks Go Electric Cloud
CloudFoundry
Docker

11. © Electric Cloud | electric-cloud.com
The implementation: CD-DevOps Solution
20
Rockstars
3
Months to V1
>1000
CD Pipelines on
Commander/Day

12. © Electric Cloud | electric-cloud.com
“Asked for volunteers to try out the system.”

13. © Electric Cloud | electric-cloud.com
Where are we on our journey?
• Bridging the gap between Dev and Ops
• Multi-layer / multi-platform infrastructure
• Devs gaining control over production environment
• Concise definition and automation of processes
• Globally distributed R&D teams
PROD BUILD
300min
Per 10M LOC
DEV BUILD
10min
Per 5M LOC
FULL TEST
1440min
REGRESSION TEST
240min
FEATURE DELIVERY TIME
30days
PROD BUILD
10min
Per 10M LOC
(was 300)
DEV BUILD
1min
Per 5M LOC
(was 10)
FULL TEST
360min
(was 1440)
REGRESSION TEST
60min
(was 240)
FEATURE DELIVERY TIME
7days
(was 30 days)

14. © Electric Cloud | electric-cloud.com
What’s “faster?”
Hours to Minutes
Compile and Build
Days to Hours
Hardware Emulation
Hours to Minutes
Code Analysis
DevelopDesign
Days to Hours
Architect Evaluation
Product Validation
Days to Hours
Full Regression (System)
Days to Minutes
Full Regression (Software)
Weeks to Days
Full Functional Testing
Solution Validation
Months to Weeks
Solution Testing

15. © Electric Cloud | electric-cloud.com
>1M
System
Integrations/ye
ar
10K+
Releases/year
30M
Lines of Code
100K
Builds/day
480K
Code
reviews/year
100M
Test cases run/day

16. © Electric Cloud | electric-cloud.com
Now… Who else wants some?

17. © Electric Cloud | electric-cloud.com
Thanks
Questions?