Google Cloud Networking provides a global, flexible, and secure networking foundation for applications and data. Key elements include: - A global fiber network with over 100 points of presence and hundreds of thousands of miles of cable connecting Google's regions and zones. - The Andromeda network virtualization stack, which powers VPC networking and provides scalable isolation, high performance, and distributed firewall capabilities. - Global and regional load balancing options like HTTP(S) and TCP/UDP load balancing for optimizing application delivery worldwide. - Hybrid connectivity options like Cloud Interconnect, VPN, and Direct Peering to build hybrid cloud architectures connecting on-premises to Google Cloud.

1. Google Cloud Networking
Without networking, there is no cloud
Prajakta Joshi, Product Manager, Google
prajaktajoshi@google.com
Srinivas Krishnan, Tech Lead/Manager, Google
krishnan@google.com
April 6th, 2017

2. 7 Cloud products with 1 billion users

3. Google’s technology infrastructure and innovations available to all

4. Google probably has the best networking
technology on the planet.
— Peter Bakkum, Quizlet
“
”

5. FASTER (US, JP, TW) 2016
Unity (US, JP) 2010
SJC (JP, HK, SG) 2013
GCP Infrastructure
6 regions, 18 zones, over 100 points of presence, and a well-provisioned global network comprised of
hundreds of thousands of miles of fiber optic cable.
Edge points of presence (>100 - all are
not shown on this map)
Leased and owned fiber
#
#
Future regions and number of zones
Current regions and number of zones
3
3
2
3
3 3
3
3
2
4
3
3
2
Frankfurt
Singapore
S Carolina
N Virginia
Belgium
London
Taiwan
Mumbai
Sydney
Oregon Iowa
São Paulo
Finland
Tokyo
Montreal
California
Netherlands
3
3
33
https://peering.google.com
https://cloud.google.com/compute/docs/regions-zones/regions-zones

6. Software
Defined
Load
Balancer
Google Innovations in Networking
Software
Defined Network
Virtualization
Software
Defined
DataCenters
Software
Defined
WAN
2006
2008
2010
2012
2014
Google
Global
Cache
Freedome
Watchtower
BwE
Onix
B4
Jupiter Andromeda
Maglev
QUIC
Espresso
Software
Defined
Edge

7. Google Cloud Networking
Global
Scale
Application delivery at scale
globally or regionally
Cloud Load Balancing
Cloud CDN
Cloud DNS
Global private space,
regional segmentation.VPC
SDN network virtualization
Global Networks
Granular Subnetworks
Connect to on-premises
or another cloud
Cloud VPN
Cloud Router
Cloud Interconnect
Hybrid
Cloud
User control Network IAM roles
Firewalls
Stackdriver
Security Policies
Visibility / diagnostics
Control

8. Global, Flexible Virtual Private Cloud (VPC)

9. Traditional VPC
● Regional
US West US East
Traditional
VPC
10.10.0.0/26
Traditional
VPC
Internet
Web Application
Server
Web Application
Server
10.50.0.0/26

10. Google Global VPC
US West
10.10.0.0/26
US East
10.50.0.0/26
Traditional VPC Traditional VPC
Application
Server
Application
Server
Internet
US West US East
Google VPC
10.10.0.0/26 10.50.0.0/26
Application
Server
Application
Server
● Connect workloads across any regions
● Access any region by interconnecting through a single location, through Google's backbone network
Traditional VPC Google VPC

11. Subnet 10.10.0.0/20
Project
Webserver
Project
Recommendation
Project
Analytics
Webserver
devOps
Recommend
devOps
Security
Network Admin
Organization Node
MY-ORG.com
Database
devOps
VM
instances
VM
instances
VM
instances
Customer
DC
Interconnect
(POP)
Shared VPC Network (XPN)
Private IP connectivity
Shared network VPC (XPN)

12. VPC Network Peering
VPC Producer NetworkVPC Consumer Network
Project
service-prod
Project
customer-prod
Producer
devOps
Consumer
devOps
Consumer
Security
Network
Admin
Organization node example.com
Serving Instance
Compute Engine
Cloud Load
Balancing
Backend
Compute Engine
Producer
Security
Network
Admin
Organization node
SaaS.com
12

13. GCE VM distributed firewalls
● Distributed: enforced on host => no choke points
● Ingress and Egress firewall (allow / deny rules)
● Stateful with Connection tracking
● Flexible grouping mechanism for applications: tagsGCP Firewall
Traditional enterprise
model
Ingress
firewall
Egress
firewallIngress
firewall
Internet
Google Cloud model
Compute
Engine
Compute
Engine
Compute
Engine
Compute
Engine
Red-tag Blue-tag

14. Scale, resiliency and optimized app delivery
across the globe

15. Google Global Load balancing
HTTP(S) Load
Balancing
SSL proxy
Network TCP/UDP
Load Balancing
Internal TCP/UDP
Load Balancing
Global
Regional
TCP proxy Delivered using
Google Front End
infrastructure
(GFE)Delivered using
Maglev
Delivered using
Andromeda
● Google Front Ends (GFEs)= Software-defined, distributed systems that sit in Google POPs and perform
global load balancing in conjunction with other systems/control planes
● Andromeda = Google Cloud’s software-defined network virtualization stack
● Maglev = Distributed systems for network load balancing

16. GFE
Your Backend
compute
Google’s global high-quality network
ISP
Network
Global LB
Software-defined and globally distributed load balancing
GFE
VM VM VM
Global LB
Load Balancer
GCP Datacenters
Google
Edge POP
Google
Network
Google
Edge POP
Serving Instances
Compute Engine
Autoscaling
Software-defined, globally distributed load balancing

17. Google Global HTTP(S) Load Balancing
Maya in California
2001:db8::2
Bob in London
2001:db8::3
Shen in Singapore
2001:db8::4
Cloud
DNS
www.myapp.com
2001:db8::10
120.1.1.1
Region: US-West
Serving Instances
Compute Engine
Autoscaling
Region: Europe
Serving Instances
Compute Engine
Autoscaling
Region: Asia
Serving Instances
Compute Engine
Autoscaling
10.240.0.0/16 192.168.0.0/16 10.2.0.0/16
Google
Network
www.myapp.com
www.myapp.com www.myapp.com www.myapp.com
2001:db8:10
120.1.1.1
Google
Edge POP
Google
Edge POP
Google
Edge POP
2001:db8:10
120.1.1.1
2001:db8:10
120.1.1.1
Google Global Load Balancing (IPv4/IPv6)

18. Google HTTP(S) load balancing performance
0 150 300 450 600
1,600,000
Multiple Regions
(europe-west1-a, europe-west1-b, asia-east1-a)
RPS
Time (s)
1,200,000
800,000
400,000
0
1.3 million requests per second
50X
Actual Traffic was
of expected traffic
Handled with:

19. Optimizing with Content Delivery Networks (CDN)
Google Cloud CDN

20. Google Cloud CDN
User in San Francisco
Frontend
Frontend
US-Central Region
Asia Region
US-East region
Google Cloud Storage
Cloud CDN
User in Iowa
User in New York
Stackdriver Monitoring
& Logging
Autoscaling
Autoscaling
Cloud Load Balancing
HTTP(S) LB
Cache hit!
● 80+ locations
● Single IP across multiple regions
● Enable CDN for HTTP(S) with single check box
● Caches content from instances and storage buckets
● Cache invalidation
● Custom cache keys
● No extra charge for SSL (TLS traffic)
● Industry leading performance (Cedexis)

21. Google Cloud CDN: latency performance
View Cedexis data on CDN latency, throughput, availability at https://www.cedexis.com/google-reports/
*Lower is better

22. Hybrid and multi-cloud

23. Google Cloud Interconnect
Carrier Interconnect
Enterprise-grade connection through a
large partner network of service providers,
VPN required for RFC 1918 communication
VPN
Secure multi-Gbps connection
over VPN tunnels
Direct Peering
Enterprise-grade connection
between you and Google for
your hybrid cloud workloads,
VPN required for RFC 1918
communication
Private Interconnect (new)
Connect N X 10G transport circuits for
private cloud traffic to Google Cloud at
Google POPs, VPN not required for RFC
1918 communication

24. Hybrid cloud: The Home Depot
Home Depot Data Centers
VPN
Gateway
Local
Storage
Local
Compute
Google Cloud
Interconnect / VPN
VPN
Gateway
Compute
Engine
Cloud
Storage
Local
Compute
API
Gateway
HTTP
Cloud Load
Balancing
Compute
Engine
Google Cloud
Interconnect
( Mutual TLS )
Learn more in Ravi’s talk (The Home Depot):
RFC 1918
Public IPs

25. Hybrid/Multi-cloud automation
Internal load balancing templates
Google Cloud provider

26. Network Security

27. Cloud networking security blueprint
Secure
VPC
Cloud Interconnect
3rd party virtual
appliances
Google Global Load Balancer
(GFE)
VPC topologies for isolation,
Distributed firewall, IAM,
Private access to
Google services, Bastion hosts
Built-in L3/L4 DDoS
protection, Intelligent
anycast for Global IPv6
and IPv4 LB, Autoscaling,
Cross-region overflow,
Cross-region failover,
SSL (TLS) termination
Google Network
Direct Peering,
Carrier Interconnect,
VPN,
Private Interconnect Next-gen firewall,
Logging, Monitoring,
Compliance
Cloud Interconnect
3rd party virtual
appliances
Google Global Load Balancer
(GFE)
Google Network
High capacity/High
performance global
network, Protection
against UDP-based
attacks
3rd party DDoS Defense
27

28. Leverage 3rd party appliances and services
Network Security
(Next Gen
Firewall/IPS/WAF)
DDoS/WAF
Endpoint/
Container Security
Scanning, Logging
& Compliance
Encryption/
DLP
TO DO:
FIX ME
Evernote users trust us with billions of their
notes, so the security and privacy of those notes
is top of mind as we move to the cloud.
Google Cloud Platform security capabilities let us
match the protections we provide for customers
today, and improve on them.
Rich Tener, Head of Security at Evernote
“
”https://blog.evernote.com/blog/2017/02/08/evernote-reaches-the-cloud/
28

29. Under the Hood:
Andromeda Network
Virtualization
Srinivas Krishnan
Technical Lead, Google

30. Andromeda
● Scalable, Flexible, Secure, High
Performance Infrastructure for Virtual
Networking
● Powers Google Compute Engine Network

31. FASTER (US, JP, TW) 2016
Unity (US, JP) 2010
SJC (JP, HK, SG) 2013
Edge points of presence (>100 - all are
not shown on this map)
Leased and owned fiber
#
#
Future regions and number of zones
Current regions and number of zones
3
3
2
3
3 3
3
3
2
4
3
3
2
Frankfurt
Singapore
S Carolina
N Virginia
Belgium
London
Taiwan
Mumbai
Sydney
Oregon Iowa
São Paulo
Finland
Tokyo
Montreal
California
Netherlands
3
3
33
GCP Network at a Glance
https://peering.google.com
https://cloud.google.com/compute/docs/regions-zones/regions-zones

32. Belgium Finland
Frankfurt
*SLA availability statements are achievement targets and are
subject to change and per Google terms and conditions
https://peering.google.com
https://cloud.google.com/compute/docs/regions-zones/regions-zones
Singapore
S Carolina
London
Taiwan
Mumbai
Oregon
Iowa
Tokyo
São Paulo
vnet-1: 10.1/16
vnet-2: 10.2/16
vnet-3: 10.2.1/24
Setting the Stage (Control Plane)
● Isolation across
virtual networks
● Built for scale
○ 100k VMs in
184 ms1
1
median
● High Availability
Targets*
○ 99.9% Single Zone
○ 99.99% Single Region
○ 99.999% Globally

33. Setting the Stage (Data Plane)
Core network functions
Guest
VM
Dataplane
Host
Mgmt
Control Plane
Guest
VM
Dataplane
Host
Mgmt
Constantly Adding new functions
● Provide core network functions
● Constantly evolving new
features
○ Supported by Live
Migration and Dataplane
Hitless Upgrades

34. Frankfurt
https://peering.google.com
https://cloud.google.com/compute/docs/regions-zones/regions-zones
Singapore
S Carolina
N Virginia
Belgium
London
Taiwan
Mumbai
Sydney
Oregon
Iowa
São Paulo
Finland
Tokyo
Overview
Cluster Manager
Regional Fabric Manager
Google Fabric Manager API Google Fabric Manager API Google Fabric Manager API
Regional Fabric Manager
VM Controller VM Controller VM ControllerVM Controller
OFE OFE OFE OFE
Google Fabric Manager API
VM
Coprocessor
Guest
VM
Open
vswitch
Host
Mgmt
Andromeda Dataplane
Guest
VMGuest
VM
VM
Coprocessor
Guest
VM
Open
vswitch
Host
Mgmt
Andromeda Dataplane
Guest
VMGuest
VM
VM
Coprocessor
Guest
VM
Open
vswitch
Host
Mgmt
Andromeda Dataplane
Guest
VMGuest
VM
VM
Coprocessor
Guest
VM
Open
vswitch
Host
Mgmt
Andromeda Dataplane
Guest
VMGuest
VM

35. Providing High Availability and Scaling
Fabric Manager
VMC
OFE
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
VMCVMC
Replicated
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
Sharded
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
Horizontal
scaling
VMCVMCVMC
OFE
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
VM
Coprocessor
Open
vswitch
Host
Mgmt
Andromeda Dataplane
VMVM
VM
Sharded VMCVMCVMC
OFE

36. Network Programming Models
● Pre-Programmed Model
○ Programs a full mesh of VM-VM connectivity
○ Works well for small virtual networks
VM
Control
Plane
VM
VM
Hoverboard
VM
● On-Demand Model
○ Traditional OpenFlow learning packet model
○ First Packet always misses to controller
● Google Hybrid Model: Hoverboards
○ Uses a software gateway
○ Pre-Program small networks
○ Larger Networks
■ Packet goes through Hoverboards
■ Flows can be offloaded to the VM hosts

37. Guest OS
Hypervisor
Openvswitch
Hardware NIC
Andromeda 1.0
Guest OS
Userspace NIC
Hardware NIC
Hypervisor
Andromeda 2.1
Bypass to
Userspace NIC
Guest OS
Hypervisor
Userspace NIC
Hardware NIC
Andromeda 2.0
Dataplane in
Userspace
Andromeda 1.5
Guest OS
Hypervisor
Openvswitch
Hardware NIC
Offloads + Live
Migration
Guest OS
Userspace
NIC
Hardware NIC
Hypervisor
Hypervisor
Bypass
Future
Andromeda Dataplane Quick History

38. Andromeda Dataplane
Enabling Native Hw Performance in Software
● Fast Path
○ 2 million pps
VM1
Guest VM
vnic TX vnic RX
Fastpath
Egress
Engine
Ingress
Engine
VM2
Guest VM
vnic TX vnic RX
Coprocessor
Coprocessor
NIC
vswitchd
ctrl
vswitchd
● Coprocessors
○ CPU Intensive functions
● On Host control plane

39. Unique Userspace Architecture
● Secure VM1
Guest VM
vnic TX vnic RX
VM2
Guest VM
vnic TX vnic RX
NIC
● Improved Robustness
● Rapid Releases
vswitchd
Andromeda Dataplane
Fastpath
Egress
Engine
Ingress
Engine Coprocessor
Coprocessor
vswitchd
ctrl
Andromeda Dataplane
Fastpath
Egress
Engine
Ingress
Engine Coprocessor
Coprocessor
vswitchd
ctrl
NO ROOTContinuous FuzzingASLR
Andromeda Dataplane
Fastpath
Egress
Engine
Ingress
Engine Coprocessor
Coprocessor
vswitchd
ctrl

40. Fully Programmable Dataplane
Openvswitch
Google
Extensions
Load Balancing
Stats and Billing
Policy Enforcement
Table-0: Decap
Table-0: Decap
Table-50: Encap
Flow Table
Flow Key Action
Coprocessors
Traffic Shaper
DoS and Abuse
New Features
vswitchd ctrl
NIC
In port=5, src ip=1.2.3.4 Encap, out port=3229
In port=470, Encap Decap, out port=5
● Fully programmable from
control plane
VM Controller
● Programmable flow table
● Custom Google extension
framework
● Easy Network function
implementation

41. Case-Study: Internal Load Balancing

42. Typical Customer Deployment
us-central
FE
europe-west
FE
us-central
FE
europe-west
FE
us-central
FE
us-central
BE
europe-west
BE
us-central
BE
europe-west
BE
us-central
BE
HTTP/S LB
Internal LB
TCP/UDP

43. No Middle Box
Load Balancer
Backend Backend Backend
VIP
Traditional Load Balancers
Client VM
Load
Balancer
Backend Backend
Client VM
Load
Balancer
Backend
GCP Internal Load Balancing
Client VMClient VM

44. Load Balancer in Client as Network Function
VM Controllers
OFEs
Health Reporter
Health
CheckerClient VM
Load Balancer
Backend VMBackend VM
Cluster Fabric Manager
Control
Health
VM-VM
OpenFlow

45. Google Cloud Networking recap

46. Your toolkit is much bigger actually...
Container Engine KubernetesCompute Engine App Engine
Cloud Load
Balancing
Cloud CDN Cloud
Interconnect
Cloud
Functions
Global, Secure
Network
Cloud VPC
Stackdriver
And many many more ...

47. Cloud Networking: Key Takeaways
Google’s global, software-defined network: the underpinnings of high performance,
flexibility, control, and security that Google Cloud provides
Google Cloud’s secure VPC: Global VPC, choice of topologies (org, shared VPC, peered
VPC…), distributed firewall, IAM, secure connectivity (Private Interconnect, Direct
Peering, Carrier Interconnect, Cloud VPN)
Google Cloud’s best-of-breed network services: High performance Global Load
Balancing, Private Internal Load Balancing, High performance Cloud CDN, Cloud DNS.
Google’s technical infrastructure is built for the cloud and employs a layered security
model to secure the entire stack to address stringent regulatory and enterprise
security needs.
1
2
3
4

48. https://cloud.google.com/products/networking/
gcp-networking@google.com