IX Best Practices by Tay Chee Yong

1. IXP Best Practices
Tay Chee Yong
MyNOG 3
28 November 2013
1

2. IXP Essentials
• Layer 2 Ethernet network consisting of one or more switches
• Members connects to the network with an assigned IP
address
• Only BGP is allowed
– Bi-lateral (BGP between members)
– Multi-lateral (BGP with route servers)
2

3. IXP Essentials
• Announce own origin and customer routes
• Exchange traffic with all other members to improve traffic
gravity and performance
– Members save cost on Internet transit
– Better user experience (reduced latency)
• One port with many peers
– Allows exchange of routes/traffic among all IXP members
3

4. IXP Benefits
• Keep the local traffic local!
– ISP within the country/region peer with each other
– Doesn’t need to take a long route out and return
– Improved latency and efficiency
• Save money!
– Traffic stays local means save transit bandwidth = save money
• Improve network performance
– Better RTT between end points
– Direct traffic forwarding instead of sub-optimal routing
4

5. Be responsible!
• IXP operator responsible to ensure infrastructure is stable
and secure
– Choice of hardware/software
– Stability of route server daemon
– Security measures
– Competent operational staffs
• Usual BGP best practices still apply to all members
• IXP best practices and etiquettes to be adhered
5

6. Leaking of IX prefix to Internet
• Announce IXP prefix outside of AS boundary is not a good
idea
• Providing free transit for IXP prefix
• Vulnerable to DDOS attacks
• Common reason : redistribute connected to bgp
• Prefix list/route maps to deny IXP prefix announcement
6

7. Routing control discipline
• Same set of routes should be announced over both transit links
and IX port
• Consistent routing policy over different IXP
• Members announcing more specific routes, may result in transit
over the IXP
• No Static/Default route!
7

8. Unwanted protocols towards IXP
• Interior routing protocols : OSPF, IS-IS, EIGRP, RIP
- Generates unwanted broadcast/multicast traffic
• Layer 2 protocols :
- STP, VTP, Proxy Arp
• Network discovery :
- CDP, LLDP, EDP
8

9. Proxy ARP
• Members acting as a arp relay, potentially very
dangerous
• Leading to hijacking of packets destined to other
members
• Usual culprits are of Cisco equipment
• IOS : enabled by default
• IOS-XR : disabled by default
• JUNOS : disabled by default
#sh arp
219 202.yyy.yyy.yyy
225 202.yyy.yyy.yyy
242 202.yyy.yyy.yyy
316 202.yyy.yyy.yyy
0012.7fxx.xxxx
0012.7fxx.xxxx
0012.7fxx.xxxx
0012.7fxx.xxxx
Dynamic
Dynamic
Dynamic
Dynamic
0
0
0
0
15/20
15/20
15/20
15/20
9

10. Proxy ARP
• Tools to detect members with proxy arp enabled
• Violation logs to be sent to NMS monitoring
• Enhance internal monitoring & operational process
• Follow up , Follow up
10

11. Looping back an Ethernet Port…
• Loopback towards on an IXP port is never a good idea
• Result : broadcast storm towards all other members
• Cripple the IXP, and disrupting traffic
11

12. Peering with route servers
• Facilitate implementation of peering arrangement
• Allow new members to join the community easily
• Generally have 2 route servers for redundancy
• Single routing daemon
• Dual routing daemon
• Reduced the number of peering sessions
• Just peer with 2 to get all routes from all members
• Ability to manipulate routing policy via bgp communities
12

13. Port Security
• MAC address filtering
• Only permit specific IP ethertypes
• IPv4, ARP, IPv6
• Drop everything else
• Enforce one-mac-address-per-port rule
• No additional devices are permitted
• Prevent noise from any intermediate L2 devices (eg. STP)
• Inform your IXP if you are doing any migration or change of
device
• Mac address change
13

14. Prefix Filtering
• Applied on route servers
• Per neighbor prefix filtering
• Pros
• Prevent unintentional route hijack or route leak by members
• Treat IXP as a normal upstream provider to update prefix list
• Cons
• Accidental of route denial – reduction in traffic
• Solutions : Route update using IRR where possible
• Challenge : Route objects should be updated regularly
14

15. Configuration Automation
• Fat fingers and human nature at times cause issues in IXP
- Applying incorrect switch configuration
- Forgot to apply port security
- Typo error
- etc
• Reduce errors during provisioning of switch or route servers
• Increase IXP productivity and efficiency
• Standardize configuration across IXP platform
15

16. Transparent AS
• AS-PATH Transparency : Route servers do not insert its
own AS number in the AS-PATH updates to members
• In route servers, well-known BGP attributes (AS-Path,
MED, next-hop, communities) are not modified before
redistributing to other members.
• Peering sessions appears to be directly between members,
but the RS is mediating the session.
• Common problem seen with Cisco routers due to default
behavior
• IOS : no bgp enforce-first-as
• IOS XR : bgp enforce-first-as disable
16

17. Transparent AS
• Non route server setup
10.10.0.0/16
20.20.0.0/16
AS10
Prefix
20.20.0.0/16
AS100
AS-PATH
100 20
AS20
Prefix
10.10.0.0/16
AS-PATH
100 10
17

18. Transparent AS
• With route server setup
10.10.0.0/16
20.20.0.0/16
IXP A
AS 100
AS10
Prefix
20.20.0.0/16
AS20
AS-PATH
20
Prefix
10.10.0.0/16
AS-PATH
10
18

19. Storm Control
• Broadcast storm into an IXP a major challenge for the
operator – beyond their control
• IXP hardware to have better storm control capability or
features to counter
• Various hardware vendors has employed certain level of
storm control detection and mitigation feature
Vendor
Mechanism/Capability
Cisco Nexus
• Interface level
(Threshold : Interface bandwidth)
Brocade MLX
• Interface level ACL/rate-limit
• Global Level / VPLS Level
(Threshold : # of packets)
Extreme
• Interface level ACL/rate-limit
• Global/CPU level
(Threshold : # of packets)
19

20. Summary of Best Practices
Members
Operator
• Disable unwanted traffic
towards IXP
• Do not loop towards IXP
• Do not leak IXP prefix to
Internet
• Peering with route servers
• Consistent route
announcement
•
•
•
•
•
Port Security
Prefix Filtering
Configuration Automation
Transparent AS
Storm Control
20

21. Reference
• AMS-IX
• https://www.ams-ix.net/technical/specifications-descriptions/
config-guide
• Euro-IX
• https://www.euro-ix.net/ixp-bcp
21

22. chee-yong.tay@ap.equinix.com
22