A presentation at Networkshop46, by David Stockdale

1. Campus network
refresh
David Stockdale,
Imperial College London

2. >17,000 students
>8,000 staff
>Main campus – South Kensington, London
>Under construction – White City, London
>6 other large campuses (hospitals, Silwood Park)
>10+ other sites (hospitals, halls, sports grounds)
>2 datacentres – Slough & South Ken
>Centralised ICT
Facts and figures – Imperial

3. >Over 65,000 unique hosts on wired network
>Over 60,000 unique hosts on wireless network
>Over 20,000 concurrent wireless clients at peak time
>~400 active comms rooms (CWCs)
>~20 dark fibre links
>~15 Ethernet circuits
>40G to Janet via two 2x10G trunks
Facts and figures – Network

4. >Routers – 23x Juniper MX & Cisco 6500/6880/N7K
>Smaller sites – 12x Juniper SRX 2xx/3xx
>Firewalls – 2x Juniper SRX 3xxx
>Switches – 1,900x Juniper & Extreme
>Wireless – 2,900x Cisco lightweight APs
>VoIP – 10,000x Cisco handsets
Facts and figures – Equipment

5. Physical infrastructure
Core location
• Core router, dark fibre and optical equipment
Distribution location
• Site router, distribution switch
Building ODF (BODF)
• Passive fibre patching
CWC
• One or more stacks of edge switches

6. >One or more per building
>Fewer, bigger CWCs
>Stacks of edge switches
>Until now cat 5e
>In future cat 6a
>Diverse fibre to BODF
CWC

7. >One per building
>No active equipment
>Single-mode throughout
>Fibre to dist locations
BODF

8. >Typically 2-3 per site
>Site router
>Distribution switch
>Fibre to core locations
Distribution location

9. >Two in London
>Core router
>Lots of dark fibre
>Lots of optical equipment
>Also:
>Firewalls
>Border routers
>Route reflectors
>etc.
Core location

10. >DWDM and CWDM
>Physical topology != active topology
>Up to 40x 10G* links over single fibre pair
>Passive on shorter distances
>Amplifiers on longer links
>Coloured optics in our equipment
>Transponders for links to other equipment
WDM

11. Network hierarchy
Core routers
• 2 in London at different sites
Site routers
• Pairs, each attached to both core routers
Distribution switches
• Pairs, each attached to both site routers
Edge switches
• Each stack attached to both distribution switches

12. >2x MPLS P routers
>No VRFs
>Lots of 10G ports
>Lots of IPv4/IPv6 P2Ps
>OSPF v2/v3 as IGP – loopbacks & P2Ps
>iBGP with route reflectors – other routes
>PIM
>LDP
>ECMP
>Links to border, site, datacentre and wireless routers
Core routers

13. >VRF lite doesn’t scale well
>No per-VRF P2Ps or routing protocols
>VRFs don’t need to exist on intermediate routers
>VRF routes in iBGP
>L3VPN – IPv4 routes for VRFs
>6VPE – IPv6 routes for VRFs
>EoMPLS / VPLS / EVPN
>MVPN – Multicast for VRFs (Draft-Rosen, not MPLS)
MPLS

14. >2x central devices for everything
>One-armed off border routers
>VRFs map to zones
>eBGP session per zone, landing in VRFs on routers
>BGP for failover, rather than HA
>Networks Group runs network side
>Security Group maintains policy side
Firewalls

15. >Production
>BYOD
>Guest
>BMS
>Device management
>Registration
>Banned
>Many smaller VRFs
VRFs / firewall zones

16. >MPLS PE routers
>In pairs
>Limited, expensive 10G ports
>Dual-stack IPv4/IPv6 for production and BYOD
>VRRP / HSRP
>PIM, IGMP
>2x10G to each core router (40G ECMP)
Site routers

17. >Layer 2 only
>MLAG pairs of stacks
>Plentiful, affordable 1/10G ports
>2x10G per stack to each router (40G total)
Distribution switches

18. >1G PoE everywhere
>Interested in 2.5/5G – works over cat5e
>2x1/10G LACP to distribution
>Standard set of per-VRF VLANs
>All edge ports alike – VLANs assigned by RADIUS
>No UPS
>Edge SLA – higher SLA available in datacentre
Edge switches

19. >~30% of our Internet traffic IPv6
>Dual stack on production & BYOD (including wireless)
>AAAAs on most load-balanced services
>Other services enabled:
>Home directories (>95% IPv6!), more storage soon
>Mail, DNS, Skype for Business, HEP systems
>SLAAC rather than DHCPv6 (historical reasons)
>Feature parity mandated in tenders
IPv6

20. >2008 – First subnets enabled, separate firewalls
>2010 – Upstream native IPv6, dual-stack firewalls
>2010/11 – Most production and BYOD enabled
>2010/11 – Some servers including mail & DNS
>2011 – World IPv6 Day: College websites enabled
>2013 – Wireless enabled
>2015 – AAAAs added to most load-balanced VIPs
>During a migration to new hardware
>Single protocol backends
IPv6 - Milestones

21. >IPv4 exhaustion – wireless is now using /17!
>NAT will be inevitable – IPv6 minimizes it
>HEP community – dependent on IPv6, run out of IPv4
>Overseas students – better IPv6 connectivity at home
>People have it at home in UK! Sky, BT, Virgin (soon?)
>Cost us very little, deploying opportunistically
>Could cost a lot to deploy in a hurry!
>We’ve seen very few problems
IPv6 - Reasons

22. >Separate DMZ router connected to border routers
>Initially for HEP, soon to be standard service
>Bypasses firewall
>Avoids upgrading whole path
>Cheaper equipment but fewer features
>Essential moving towards 100G
>ECMP outbound, multiple subnets inbound = 40G!
Science DMZ

23. >We’re gonna need a bigger pipe… :-/
Science DMZ – Success!

24. >Private network for participating HEP sites
>Tagged VLANs on Janet links
>BGP peerings into L3VPN
>DMZ router has BGP peerings into Internet/LHCONE
>More specific LHCONE prefixes preferred
>It works…
LHCONE

25. >Router configs built and managed by Ansible
>Firewall groups fed from host database
>Switch configs automatically generated
>Network built from standard blocks
>All edge ports alike
>MPLS simplifies configuration
>WDM and dark fibre surprisingly affordable
>Simple is better!
Automation & scalability

26. David Stockdale
ICT Networks Group
Imperial College London
I have been…
Exhibition Road, London, SW7 2AZ
T 020 7594 6968
david@imperial.ac.uk
www.imperial.ac.uk

27. Any questions? /
Thank you