Arquitetura e Troubleshooting da Linha de Switches Catalyst Access Series - Cisco Live 2017

1. Cisco Catalyst Access Series
Switching – Architecture and
Troubleshooting
Kallol Bosu
BRKCRS-3146

2. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 3BRKCRS-3146
Evolution in Enterprise Network

3. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Are You Ready to Digitise Your Network for Tomorrow?
BRKCRS-3146 4

4. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Kallol Bosu
Customer Support Engineer – LAN Switching
kbosu@cisco.com
Kallol works as a TAC Escalation engineer in
APAC- LAN Switching team. Kallol joined
Cisco in 2013 and has around 7 years of
experience in the networking industry. He
handles a variety of Catalyst switching
products like- 3750, 3850, 3650, 6500, 7600,
ME and IE switches.
Kallol holds a Master of Science degree in
Electronics and Telecommunication.
BRKCRS-3146 5

5. • Architecture of Legacy Switches
• Architecture & Packet Walk -Next Generation Switches
• Troubleshooting Next Generation Switches
• Summary
Agenda

6. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Product Portfolio – Overview (Before NGWC)
BRKCRS-3146 7
3560CX
3650 Series- Fixed uplinks

7. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 3K Portfolio (Before NGWC)
BRKCRS-3146 8

8. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 2K Portfolio
BRKCRS-3146 9
2960X and 2960-XR 2960-L
§ LAN lite+ feature set
§ No stacking
§ Standard L2 security & QOS
§ No routing support
§ Zero touch installation
§ Reduced Cost
§ SFP uplink

9. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Compact Switches Portfolio
BRKCRS-3146 10

10. Architecture of Legacy Switches

11. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3750-X Components and Architecture
Port ASICs and TCAM/SRAM
Ø Port ASIC is the BRAIN of the Switch
Ø Processing includes Packet Modification and Decision to forward/drop packets
Ø Lookups include - TCAM/Hash Tables/Forwarding
Ø TCAM stores vital information including IPv4, IPv6 and MAC addresses, ACES etc.
Ø 3750-X TCAM/SRAM is incorporated into the Port ASIC
BRKCRS-3146 12

12. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3750-X Components and Architecture
Switch Fabric and CPU
Ø 128Gbps switching Fabric , provides line rate & local switching within a switch and stack connectivity
Ø 48G + 2X10G + 32 Stack-ports (100Gbps FDX)
Ø 64 Gbps Ring Stackwise Plus
Ø 1 Gbps Ring Inter-connect control path to the Port ASICs to the CPU
Ø CPU updates the MAC and Routing caches attached to each port ASIC
Ø The CPU communicates with the Port ASICs via a dedicated management ring (the yellow ring in the diagram)
BRKCRS-3146 13

13. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
2960-X Architectural Overview
BRKCRS-3146 14

14. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15BRKCRS-3146
2960 GE Models Comparison

15. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Compact Switches Architectural Overview
Ø Power : Models that are PoE/PoE+/UPoE powered and AC Powered
Ø Speeds :10/100 or 10/100/1000.MACSec Phys on the 3K
Ø Capability : 3560c runs IPBase Only. No IPServices
Ø ASICs: Two types of ASICs – Choice depends on speed and capability
Ø TCAM – embedded in ASIC
BRKCRS-3146 16

16. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Module Architectural Overview
3KX-SM-10G : 3750X Service Module
Ø FPGA : Contains logic to implement Netflow engine. Supports 32K flows
Ø Phy : Helps perform switch to switch MACSec in H/W.
Ø CPU : Quad Core Processor with 600 Mhz Frequency. Runs a Linux based environment over IOS
Ø Runs an independent software that can be downloaded from cisco.com
BRKCRS-3146 17

17. Architecture & Packet Walk –
Next Generation Switches

18. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
IOS-XE Evolution
BRKCRS-3146 19

19. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Next Generation Switches Portfolio
BRKCRS-3146 20

20. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 3650 Mini
BRKCRS-3146 21

21. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 3650 MultiGigabit
Introducing MultiGigabit , UPoE & 40G on 3650s
BRKCRS-3146 22

22. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
48 Port SFP+ Model
BRKCRS-3146 23

23. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 3850 & 3650
BRKCRS-3146 24

24. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
One Architecture, Various Possibilities
BRKCRS-3146 25

25. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Uplink Module Options on Catalyst 3850
BRKCRS-3146 26

26. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Uplink Module Options on Catalyst 3650
BRKCRS-3146 27

27. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28BRKCRS-3146
Uplink Options on 3850s

28. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29BRKCRS-3146
Fixed Uplink Options on 3650s

29. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Power Supplies
BRKCRS-3146 30

30. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Stack Cables and Components
BRKCRS-3146 31

31. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 3850 –Under Cover
BRKCRS-3146 32

32. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
UADP ASIC
BRKCRS-3146 33

33. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Key UADP ASIC Capabilities
BRKCRS-3146 34

34. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3850/3650 -24 Port Layout
BRKCRS-3146 35

35. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3850/3650 -48 Port Layout
BRKCRS-3146 36

36. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Catalyst 3850 MultiGigabit—48 Port Layout
BRKCRS-3146 37

37. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3850 Block Diagram and Packet Walk
BRKCRS-3146 38

38. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco StackWise Virtual
BRKCRS-3146 39

39. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco StackWise Virtual
Continued..
BRKCRS-3146 40

40. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco StackWise Virtual
Continued..
BRKCRS-3146 41
FAQ
http://www.cisco.com/c/dam/en/us/products/collateral/switches/catalyst-3850-series-switches/q-and-a-c67-738577.pdf

41. Troubleshooting Next Generation
Switches

42. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

43. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Front Panel LEDs
BRKCRS-3146 44
CSCuj17317: Certain snagless cables may press on the mode
button causing reload
Field Notice FN63697
“no setup express”

44. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Front Panel LED Description
BRKCRS-3146 45
show hardware led stack <Switch-ID>
SWITCH: 1
SYSTEM: GREEN
MASTER: GREEN
STATUS: GREEN BLACK GREEN GREEN GREEN BLACK GREEN GREEN GREEN BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK
DUPLEX: BLINKGREEN BLINKGREEN BLACK BLACK BLACK BLACK BLACK BLACK BLINK GREEN BLACK BLACK BLACK BLACK BLACK
SPEED: BLACK GREEN BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK GREEN GREEN GREEN BLACK GREEN
STACK: GREEN GREEN GREEN GREEN BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK
POE: GREEN GREEN GREEN GREEN BLACK BLACK BLACK BLACK BLACK BLACK BLACK BLACK GREEN GREEN GREEN BLACK GREEN
STACK POWER: GREEN
RJ45 CONSOLE: GREEN
USB CONSOLE: BLACK

45. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Interface Up/Down Status- Checking Through CLI
0x4- Up
0x0- Down
0x6- Admin down
BRKCRS-3146 46

46. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

47. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48BRKCRS-3146
Image Naming Convention
cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin
Converged
Access Access
Switch
Universal
License
S - Digitally Signed
P - Production
A- Key Version
IOS-XE Version
IOSd Version

48. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Booting IOX-XE Software
BRKCRS-3146 49

49. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Software Upgrade Using Install Boot
BRKCRS-3146 50

50. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Software Rollback
software clean file flash:
BRKCRS-3146 51

51. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52BRKCRS-3146
Critical known issues with Install mode

52. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3850/3650 Password Recovery
BRKCRS-3146 53
Ø Password Recovery on 3850/3650 does NOT follow 3750 family procedure
Ø Power cycle switch and hold the Mode button (on the front top left) for a few seconds
(officially 12) until the status LED gets amber, that will get you in Boot Loader prompt
(Switch:)
Ø Set the following variables
Ø Boot the 3850/3650

53. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3850/3650 Password Recovery
BRKCRS-3146 54
Ø Skip the initial configuration dialog and go to enable (no password
required):
Ø Copy startup-config back to running-config

54. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
3850/3650 Password Recovery
BRKCRS-3146 55
Ø Go to global configuration and remove or change the password:
Ø Re-enable reading of startup-config and disable password recovery if needed
Ø Write the configuration.
Switch(config)#no system ignore startupconfig switch all
Switch(config)#system disable password recovery switch all
Switch(config)#no system ignore startupconfig switch all
Switch(config)#no enable password
Switch(config)#no enable secret
Switch(config)#enable secret <NEW_PASSWORD>

55. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
License Mismatch
show license right-to-use slot <>
show license right-to-use mismatch
BRKCRS-3146 56

56. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

57. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Frequently Asked Questions – High Memory Utilisation
BRKCRS-3146 58

58. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Memory Show Commands
Total Memory
IOS-XE process
BRKCRS-3146 59

59. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Memory Show Commands
Which Process?Is it ?
BRKCRS-3146 60

60. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Memory Show Commands
BRKCRS-3146 61

61. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Common Causes for High Memory Utilisation
BRKCRS-3146 62

62. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Commands Summary – Memory Troubleshooting
BRKCRS-3146 63

63. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Frequently Asked Questions - High CPU Utilisation
BRKCRS-3146 64

64. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting High CPU
Identify process
BRKCRS-3146 65

65. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting High CPU
Drill Down Deeper
Interrupt
switched traffic
BRKCRS-3146 66

66. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
CPU Punt Path Architecture
BRKCRS-3146 67

67. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting High CPU
Drill Down Deeper – (continued)
-
Check which Queue is
receiving more traffic
(incrementing)?
BRKCRS-3146 68

68. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting High CPU
Drill Down Deeper – (continued)
BRKCRS-3146 69

69. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting High CPU
Drill Down Deeper – (continued)
-
BRKCRS-3146 70

70. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71BRKCRS-3146
Troubleshooting High CPU
Another way to dump packet from the CPU queue in question (Example)-

71. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Troubleshooting High CPU
Embedded Capture- Control-Plane Traffic
BRKCRS-3146 72

72. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Commands Summary – CPU Troubleshooting
BRKCRS-3146 73
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

73. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Common Causes for Punting Traffic to CPU
BRKCRS-3146 74

74. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75BRKCRS-3146
Known High CPU Issues

75. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76BRKCRS-3146

76. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

77. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78BRKCRS-3146
Stack Ring - Understanding Spatial Reuse

78. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79BRKCRS-3146
What is the status of my stack?

79. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80BRKCRS-3146
What is the status of my stacking ports?
Cable with corrupted EEPROM

80. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81BRKCRS-3146
HA Redundancy on 3850/3650

81. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82BRKCRS-3146
HA State Machine
2
1
3
4
5
6
7

82. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83BRKCRS-3146
Show switch with SSO

83. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84BRKCRS-3146
Did I reach full SSO state?

84. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

85. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
TCAM Utilisation
BRKCRS-3146 86

86. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASIC to Port Mapping and Port-Info
BRKCRS-3146 87

87. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Unicast L2/L3 Forwarding Show Commands
BRKCRS-3146 88

88. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Unicast L2/L3 Forwarding Show Commands
Continued..
BRKCRS-3146 89

89. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Unicast L2/L3 Forwarding Show Commands
Continued..
Destination MAC
Address
Multiple Features can
use same RI
BRKCRS-3146 90

90. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Unicast L2/L3 Forwarding Show Commands
Continued..
BRKCRS-3146 91

91. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Unicast L2/L3 Forwarding Show Commands
• Garb these CLIs’ (as needed) and share with TAC for in depth analysis.
Continued..
Show platform forward <source interface> <Vlan_ID for trunk> <SMAC> <DMAC> IPV4 <SIP> <DIP> L4#
Look at different kind of exceptions, drops, fatal errors. Note that all drop/exception counts do not necessarily mean there is a potential issue.
Please verify the same with help of TAC engineer. Use appropriate switch# and ASIC# for the port in trouble/question.
BRKCRS-3146 92

92. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 93BRKCRS-3146
ASIC level drops and exceptions

93. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multicast –Useful Commands For Verification
BRKCRS-3146 94

94. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multicast –Useful Commands For Verification
Continued..
Receiver ports
for Group
If entry is successfully
installed, failure flags
will be Zero
Resource Handler for entry in
ASIC. If it is non-zero , Resource
is allocated successfully
BRKCRS-3146 95

95. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

96. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Key Differences in QoS- 3850/3650 Vs 3750-X/E
BRKCRS-3146 97

97. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Egress QoS Drops- Deep Dive
Default Buffer allocation for a 1GB port is 300 buffers and for a 10GB port, it is 1800 buffers (1 buffer = 256 bytes).
Port can use up to 400% of the default allocated from common pool with default settings.
BRKCRS-3146 98

98. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Egress QoS Drops- Deep Dive
Continued..
Ø Here all drops are seen in Queue 2 TH2 which is the queue for all regular traffic
when port has no egress policy-map attached.
Ø By default Queue 1 is used for control traffic.
Ø Default Softmax-Multiplier is 100 on the box. Maximum value is 1200.
BRKCRS-3146 99

99. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Egress QoS Drops- Deep Dive
Continued..
BRKCRS-3146 100

100. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Egress QoS Drops- Deep Dive
Continued..
Ø Make sure that the policy-map attached to interface is installed in HW.
BRKCRS-3146 101
Useful command to check the
drops for any specific DSCP
value

101. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 102BRKCRS-3146
Few other known caveats–resolved in 3.6.6 (Recommended)
• CSCvb65304 – Output errors increment incorrectly for egress queuing drops.
• CSCuw50024 3650/3850: Total output drops are counted in bytes, not packets (Documentation)
• CSCux71386 – Reports Gigantic value in Xmit-error after clearing interface counters. (Cosmetic)
• Quite a few major fixes on POE component have been integrated in 3.6.6.

102. Interface status and LED
Software Upgrade, Password Recovery & Licensing
Memory and CPU resources
Stacking and HA
Forwarding path
Egress QoS Drops
Tricks
Troubleshooting Topics

103. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Additional Troubleshooting Commands
BRKCRS-3146 104

104. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Core Dumps and System Reports
BRKCRS-3146 105
• System generates a fullcore, crashinfo and System Report when a process terminates
abnormally
• A System Report is generated each time the switch is rebooted
• System Report contains a dump of all the trace buffers in the system
• When filing a TAC case, please attach the fullcore, crashinfo and System Report files
(whatever is applicable) from the crashinfo: filesystem

105. Summary

106. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Summary
Ø Architectural overview of legacy switches and NGWC gears.
Ø 3850/3650 -Packet Walk.
Ø Software upgrade, Password recovery and Licensing on NGWC switches.
Ø Troubleshooting common issues (including Stacking and HA) on 3850/3650
Ø Benefits of running the recommended release.
Do you have a better understanding of followings :
ü Architectural difference between Legacy switches and NGWC platforms.
ü Troubleshooting common issues on Next Generation Switches.
ü Benefits of running recommended release.
BRKCRS-3146 107