1.
www.wildpackets.com
© WildPackets, Inc.

2.
www.wildpackets.com
WildPackets Seminar Series © WildPackets, Inc.

3.
Corporate Background
• Experts in network monitoring, analysis, and troubleshooting
Founded: 1990 / Headquarters: Walnut Creek, CA
Offices throughout the US, EMEA, and APAC
• Our customers are leading edge organizations
Mid-market, and enterprise lines of business
Financial, manufacturing, ISPs, major federal agencies,
state and local governments, and universities
Over 7,000 customers / 60+ countries / 80% of Fortune 1,000
• Award-winning solutions that improve network performance
Internet Telephony, Network Magazine, Network Computing Awards
United States Patent 5,787,253 issued July 28, 1998
• Different approach to maintaining availability of network services
WildPackets Seminar Series © WildPackets, Inc. 3

4.
What We Do
• Provide network visibility and intelligence …
WatchPoint, OmniPeek, OmniEngines
• Expert systems – we find the problems for you
• Superior drill-down capability – trouble-shoot from anywhere
• Flexible, customizable, extensible – leverage your investment
Professional services, training, best practices
• For all network segments …
Data center to desktop to remote office
LAN, WAN, Wireless …
HTTP, Email, Database, VoIP, Video …
• To …
Network engineers; IT Management; Developers
WildPackets Seminar Series © WildPackets, Inc. 4

5.
With accurate visibility
into the network…
IT staff can improve:
• End-user Productivity
• Network Performance
• Application Performance
• Security
• Compliance
WildPackets Seminar Series © WildPackets, Inc. 5

6.
Select WildPackets Customers
Mid-Market / Enterprise Government & Education
WildPackets Seminar Series © WildPackets, Inc. 6

7.
WildPackets Delivers Network Visibility
WildPackets Seminar Series © WildPackets, Inc. 7

8.
www.wildpackets.com
© WildPackets, Inc.

9.
Feel the Rush…
• Your network is
running great!
• Packets enjoy a
speed-limit ride
on the wire!
• Performance is
awesome!
Are You Dreaming?
• You have few
complaints from
users!
© WildPackets, Inc.

10.
Or Feel the Jam!
• Does your network really look more like this…?
Pinpointing the Problem
CAMP IT © WildPackets, Inc. 10

11.
The BIG Questions
• DID YOU (OR WILL YOU) DETERMINE THE HEALTH
OF YOU NETWORK BEFORE DEPLOYING VOIP?
• DID YOU (OR WILL YOU) RID YOUR NETWORK OF
VoIP-KILLING TROUBLES BEFORE INSTALLATION?
• CAN YOU SUCCESSFULLY ASSESS YOUR
NETWORK WHEN VOIP-KILLING TROUBLES ARISE?
© WildPackets, Inc.

12.
Troubleshooting?
• The formal definition of troubleshooting is…
“the act of shooting or killing troubles”
When troubles are small, they can
seem so innocent and harmless,
but…
Pinpointing the Problem
CAMP IT © WildPackets, Inc. 12

13.
Troubleshooting?
• The formal definition of troubleshooting is…
“the act of shooting or killing troubles”
You’ve got to kill them when they’re
young, or…
Pinpointing the Problem
CAMP IT © WildPackets, Inc. 13

14.
Troubleshooting?
• The formal definition of troubleshooting is…
“the act of shooting or killing troubles”
They will come
back to get you!
Pinpointing the Problem
CAMP IT © WildPackets, Inc. 14

15.
Troubleshooting: Not Just Reactive!
• When we use the word troubleshooting, most folks
immediately think about reacting to a problem
• But proactive troubleshooting identifies troubles
when they are small and are having minimal impact!
• The concept is simple…
Proactive Reactive
=
Troubleshooting Troubleshooting
© WildPackets, Inc.

16.
What Troubles Are We Shooting for VoIP?
Packet
• Before VoIP, network
troubleshooting focused on Loss
factors like application
response time and latency
• With VoIP, we’ve
learned that latency
is just one part of a
three-headed
monster…
Latency
Jitter
The monster attacks RTP with one or more of its weapons!
Pinpointing the Problem
CAMP IT © WildPackets, Inc. 16

17.
Identifying Troubles: The First Step
• Before you begin worry about statistics or packets,
take time to listen to representative calls
• Hearing VoIP troubles is the most natural way to
recognize them
Use analysis application that can playback call audio
• Playback of individual RTP streams
• Playback of complete call
Listen for the telltale of signs of latency, jitter, and packet loss
© WildPackets, Inc.

18.
Understanding the Monster: Latency
• The time it takes for packets to travel across the
network is based on several factors:
Distance latency – unavoidable – a fact of physics
Queue latency
Decision latency
Encryption/decryption
Codec operations
© WildPackets, Inc.

19.
Understanding the Monster: Latency
Queue Latency &
Decision Latency
Network
Propagation
Delay
Encoding / Decoding
Compression / Decompression
Jitter Buffer Latency
© WildPackets, Inc.

20.
Latency Tolerance
Latency is much
800 ms
The ITU more critical for
700 ms
Fax Relay
recommends a
VoIP systems
Broadcast 600 ms maximum one-way
Quality
than for
delay of 150 ms
500 ms
for VoIP
traditional data
400 ms
Satellite
applications
Quality 300 ms
200 ms
High Quality
100 ms
Required for
VoIP 0 ms
Roundtrip latency > 250 ms will be
noticeable for call participants!
© WildPackets, Inc.

21.
Understanding the Monster: Latency
• Excessive latency is a major enemy of VoIP
Often caused by network congestion in the absence of adequate
QoS provisions
For some network segments, especially WAN circuits, elevated
latency may be a way of life
Excessive latency may be one-way or roundtrip, depending on
how traffic is routed through the network
© WildPackets, Inc.

22.
Latency's Effects: Talkover
• “Talkover” occurs when excessive latency delays
audio
– Caller A speaks, but his words are slow to reach Caller B
– As a result, Caller B is slow to respond
– Caller A believes that his words were unheard, so he begins to
speak again, often just as Caller B begins his response
– Caller A is speaking as he begins to hear Caller B
– Caller B may still be speaking when Caller A’s second set of
words begin to arrive
• Conversation cadence is not natural or comfortable
• Callers feel as if they must “push to talk” or say
“over” to control the conversation
© WildPackets, Inc.

23.
Latency's Effects: Echo
• In some cases, excessive latency may produce an
echo effect
The speaker’s voice feeds back into the listener’s microphone
The speaker then hears his own voice returning from the
listener’s end, but delayed due to latency
• Most callers find it difficult to maintain normal
speech when echo delay is prolonged
• Some VoIP systems attempt to cancel echo, but are
not always successful
High latency may also cause additional troubles such as loss of
synchronization between audio and video for multimedia sessions.
© WildPackets, Inc.

24.
Understanding the Monster: Jitter
• Closely related to latency
Jitter is really nothing more than variable latency preventing on-
time delivery of RTP packets
Example
• G.711 needs an RTP packet to be delivered every 20 ms to provide
accurate audio reconstruction
• If the delta time between one RTP packet and the next is 24 ms,
then the jitter is 4 ms
• VoIP devices employ jitter buffers to smooth packet
delivery
Jitter up to about 100 ms may be managed by the buffer
Packets with jitter greater than the jitter buffer are dropped
An large jitter buffer increases latency
© WildPackets, Inc.

25.
Jitter's Effects
• Jitter causes weird “sound effects” that vary with
jitter severity and environmental factors
• Examples include:
Static
Stuttering or uneven audio – abnormal speech rhythm
For multimedia systems, video may be “jerky” or irregular
• If jitter levels are high, packet loss can result
In some cases, severe jitter may sound similar to packet loss,
even if no packets are actually dropped
© WildPackets, Inc.

26.
Measuring Jitter
• Jitter can be measured as instantaneous jitter
The difference in actual packet arrival time vs. expected arrival time
• Each packet is the jitter reference for the next packet
Gives a rather “jerky” view of jitter that may overstate its effects
• However, it does correctly depict the “jerkiness” of a call
• Smoothed jitter is an improved metric defined in RFC 3550
Applies a “filter” to smooth out the instantaneous jitter trend, which
provides a much more useful and accurate view of jitter over time
For smoothed and instantaneous jitter, minimum, maximum, average,
and standard deviation values are very meaningful.
Graphs of these metrics provide good insight into call quality.
© WildPackets, Inc.

27.
Measuring Jitter
• Absolute jitter uses the first packet in the stream as a
constant reference, which catches “clock skew”
Clock skew is the difference in the clocks on the involved VoIP
stations and the measuring device
Absolute jitter graphs can reveal when packets are dropped
due to clock skew between VoIP stations
Calculating maximum, minimum, average, and standard deviation for absolute
jitter is not very helpful because of the clock skew factor.
© WildPackets, Inc.

28.
Understanding the Monster: Packet Loss
• Most commonly caused by:
Packet dropped due to physical layer corruption
Congestion without adequate QoS provisions
Jitter buffer discards due to excessive latency
• Causes missing sounds, syllables, words, or phrases
DSP algorithms may compensate for up to 30 ms of missing data
More than 30 ms of missing audio (e.g. 2 RTP packets for G.
711) is noticeable by listeners
© WildPackets, Inc.

29.
Packet Loss Effects
• An average person speaks at a rate of about 200
words per minute
Do the math – that’s 3.33 words/sec = 300 ms per word
For G.711, we would need to lose 15 consecutive RTP packets
to lose a whole word
Dropping 15 packets/sec for G.711 would be a loss rate of 30%
• But losing only a few packets can still be very
noticeable
Loss of more than 2 consecutive packets will be heard
Loss rates 2% will have a strong impact on quality
Losses of 5 – 10% make calls all but intolerable
Bursty periods of packet loss are worse than more dispersed
loss
© WildPackets, Inc.

30.
Packet Loss Bursts
• A packet loss “burst” is a period of time that begins
and ends with loss in which the number of
consecutive received packets is less than the
minimum number needed (Gmin) to maintain adequate
quality
Gmin for VoIP = 16
Gmin for video services 64 - 128
• The more “bursty” the packet loss, the worse the
quality of the call
• VoIP quality scoring standards consider
Burst length (ms) of the bursts for a given RTP stream
Burst density (% of missing packets) for the RTP stream
© WildPackets, Inc.

31.
Assessing the Monster's Impact
• While traditional network applications are very tolerant of
jitter, latency, and even some degree of packet loss, VoIP is
very sensitive to these troubles
• Levels of jitter, latency, and packet loss that would be easily
tolerated on a data network can be devastating on a
converged VoIP network
• Pre- and post-deployment network assessment are critical
– You must understand your network’s ability to accommodate VoIP
– Current latency, jitter, and packet loss
– QoS capabilities
– Current bandwidth utilization (is there any room for VoIP)
– You must maintain a constant vigil after deployment to watch for
imminent troubles
© WildPackets, Inc.

32.
Network Traffic: Quantitative Analysis
• Most network folks are concerned about the amount
of traffic on their networks
Utilization (percentage of bandwidth)
Throughput (bits or bytes per second)
• You also need to be concerned about individual
utilization components
How much bandwidth and throughput can be attributed to each
application or process?
• Clarifies which application traffic may need to be tuned or controlled
How well or poorly will the baseline (trended) behavior of each
application interact with VoIP
• Don’t forget to also consider the reverse case – VoIP’s impact on
existing applications
© WildPackets, Inc.

33.
The Impact of quot;Just One More Callquot;
• Although a network link may be able to support a
number of concurrent calls, one additional call is
often enough to cause quality problems…
Example: The WAN can support 2 simultaneous calls.
What happens when a third call is attempted???
1st Call
2nd Call x2111
x1111
x2112
x1112 3rd Call
x2113
x1113
© WildPackets, Inc.

34.
Network Traffic: Qualitative Analysis
• The quality of your network traffic is potentially more
important than its quantity when it comes to VoIP
• Many traffic streams are “bursty” in nature
Burstiness my occur over long period of time, or may consist of
rapid, recurring traffic spikes
Prolonged rises in utilization may decrease the number of calls
that can occur simultaneously
Sharp spikes may cause very noticeable quality issues with
ongoing calls
• Your baseline monitoring should consider not only
averages and long-term trends, but also the short-
term peaks and dips that characterize your traffic
flow
© WildPackets, Inc.

35.
Troubleshooting for Latency,
Jitter, and Packet Loss
• Proactive and reactive assessment of VoIP troubles
can be easily accomplished using common utilities
and/or analysis tools:
Performance and protocol analysis applications
Network management systems
PING, Traceroute, etc.
• Use packet sizes that reflect your actual RTP packets
• For example, 218 byte RTP packets are typical for G.711
For reactive assessment (post-deployment) include VoIP stats,
quality scores, and audio evaluation
© WildPackets, Inc.

36.
ping –l 218 –t 206.169.32.70
Pinging 206.169.32.70 with 218 bytes of data:
Reply from 206.169.32.70: bytes=218 time=88ms TTL=57
Reply from 206.169.32.70: bytes=218 time=83ms TTL=57
Reply from 206.169.32.70: bytes=218 time=92ms TTL=57
Reply from 206.169.32.70: bytes=218 time=88ms TTL=57
Reply from 206.169.32.70: bytes=218 time=88ms TTL=57
Reply from 206.169.32.70: bytes=218 time=102ms TTL=57
Reply from 206.169.32.70: bytes=218 time=87ms TTL=57
Ping statistics for 206.169.32.70:
Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 83ms, Maximum = 102ms, Average = 90ms
C:>
Watch for Watch for Watch for Watch for
excessive unsteady variable packet
latency. latency (jitter). routing. loss.
© WildPackets, Inc. 36

37.
C:>tracert 206.169.32.70
Tracing route to 206.169.32.70 over a maximum of 30 hops:
1 1 ms <1 ms <1 ms 10.71.0.1
2 3 ms 4 ms 3 ms 69.46.168.254
3 3 ms 2 ms 2 ms 172.33.37.78
4 4 ms 3 ms 4 ms 66.163.28.157
5 6 ms 2 ms 2 ms 209.167.101.217
6 8 ms 3 ms 2 ms 152.63.133.74
7 15 ms 13 ms 14 ms 152.63.128.117
8 14 ms 14 ms 14 ms 152.63.66.65
9 17 ms 102 ms 68 ms 64.215.195.137
10 17 ms 14 ms 14 ms 67.17.109.118
11 14 ms 15 ms 14 ms 207.200.10.11
12 73 ms 73 ms 72 ms 66.192.254.213
13 87 ms 83 ms 107 ms 206.169.32.70
Trace complete.
Traceroute can show the hop-by-
C:>
hop components of latency to help
locate the source of troubles.
© WildPackets, Inc.

38.
A Picture is Worth a 1000 Words
• A graph of latency,
jitter, or packet loss
can speak volumes
about network
health, either for
proactive or reactive
troubleshooting
• Overlaying this
graph with a graph
of utilization or total
throughput can
reveal even more
about the causes of
VoIP troubles
© WildPackets, Inc.

39.
Quality Scoring for VoIP
• One of the best initial troubleshooting tools for VoIP
traffic
• Mean Opinion Score (MOS) – several flavors
Algorithmic simulation of subjective audio assessment
Most commonly used varieties are MOS-LQ (listening quality) and
MOS-CQ (conversational quality)
Possible range of 1 (poor) to 5 (excellent)
Maximum possible MOS = 4.4 with G.711
Typical range in most networks is 3.5 – 4.2
• R-Factor – several flavors
Based on latency, jitter, packet loss, bit rate, and signal-to-noise
ratio, codec effects (for low bit-rate codecs), recency
• The ITU algorithms consider about 20 quality inputs
Possible range of 0 (poor) to 100 (excellent)
Provides LQ, CQ, and other score variants
© WildPackets, Inc.

40.
Quality Score Trending
• Isolated scores are useful for validating single call
complaints, but overall VoIP health is best seen by
graphing long-term trends
Overlaying VoIP trends
with network utilization,
errors, or other metrics may
reveal previously unseen
performance relationships!
© WildPackets, Inc.

41.
Got QoS?
• One of the most potent weapons for fighting VoIP troubles
is provision of Quality of Service (QoS) parameters
• QoS enables network devices to prioritize and give
preference to packet streams that are sensitive to delay,
packet loss, jitter, and other performance inhibitors
• Standards-based QoS methods include:
RSVP (nearly antiquated)
IP Differentiated Services (DiffServ)
MAC Layer QoS with IEEE 802.1p
VLANs
• QoS may be obtained or supplemented via proprietary
means, such as traffic shaping via various flow
processing algorithms
© WildPackets, Inc.

42.
QoS the Old-Fashioned Way with RSVP
RSVP is fading away since it manages traffic on a flow-by-flow basis, a method
that is not scalable to enterprise and carrier grade networks.
© WildPackets, Inc.

43.
IP Differentiated Services
• Using DiffServ,
routers and other
devices no longer
have to worry about
individual flows that
are identified by IP
addresses and port
numbers
• Instead, devices
only need to
DiffServ Code
examine 6 bits to
Point (DSCP)
know how to
classify and manage
The DiffServ bits provide 64 DSCPs, of which 8 give
traffic
backward compatibility with the older IP Type of Service
(ToS) field. Only 32 DSCPs are now in common use.
© WildPackets, Inc.

44.
DiffServ at the Data Link Layer
• IEEE’s 802.1p specification enables packet prioritization via
3-bit field in 802.1q VLAN tags
VLAN Tag (802.1q)
• This field provides 8 levels of precedence, and only requires
switches to read 3 bits to classify and manage traffic
• Each 802.1p-aware switch allocates 8 different queues to
separate handling for each priority level
• Network administrators must map priority levels with handling
methods
© WildPackets, Inc.

45.
Ready for QoS?
• QoS provisions are based on the “weakest link”
concept
If any device in a data path does not support QoS, then media
streams will not be afforded the preference they require for good
performance
• Pre-deployment assessment must ensure that ALL
devices can recognize and respond to QoS
parameters in packet headers
Switches, routers, firewalls, proxies, and any other devices that
touch RTP packets must be “VoIP-friendly”
© WildPackets, Inc.

46.
Summary
• The main threats to VoIP are latency, jitter, and packet
loss
The presence of these “monsters” may remain unnoticed in a data
network, but will become very obvious and ugly in VoIP systems
• Troubleshooting VoIP is both reactive and proactive
Prudence dictates that you test your network before installing a VoIP
system to identify and correct performance troubles
• You can see the three-headed monster even before VoIP is installed
After VoIP deployment, constant monitoring will
• Validate QoS operations
• Reveal network traffic pattern changes that adversely affect VoIP
• Provide alerts when VoIP performance declines
• Proactive troubleshooting and monitoring is a way of life –
a job that is never done!
© WildPackets, Inc.

47.
www.wildpackets.com
WildPackets Seminar Series © WildPackets, Inc.

48.
Product Line Overview
OmniPeek
Enterprise Packet Capture, Decode and Analysis
• 10/100/1000 Ethernet, Wireless, WAN, 10Gbe
• Portable Capture and OmniEngine Console
• VoIP Analysis and Call Playback
OmniEngine / Omnipliance
Distributed Enterprise Network Forensics
• Packet Capture and real-time analysis
• Stream-to-Disk with Data Mining
• Integrated OmniAdapter network analysis cards
WildPackets Seminar Series © WildPackets, Inc. 48

49.
OmniPeek Product Family Architecture
• OmniEngines / Omnipliance • OmniPeek Analyzers
OmniEngines collect and process OmniPeek Analyzers perform local
data throughout the network. network analysis on a portable
basis and function as consoles to
• Platform Services multiple remote engines.
Filters, alarms, authentication and
• Intelligent Data Transport
communication.
OmniPeek analyzes network traffic
• Interfaces at the engine and intelligently
Packets captured from Ethernet
coordinate analysis results that are
NICs, 802.11 NICs, OmniAdapter
exchanged with the console for
Gigabit and WAN analyzer cards.
maximum efficiency.
WildPackets Seminar Series © WildPackets, Inc. 49

50.
Omnipliance
• Full Line Rate support, Windows or Linux
• Customer upgradable hardware
• Representative Specs
3U, 2x Intel Dual Xeon 3.3 Ghz,
8x 500 Gigabyte SATA or 8x 150 Gigabyte SAS, etc.
Packet filtering in hardware
1U Omnipliance Edge available
WildPackets Seminar Series © WildPackets, Inc. 50

51.
WatchPoint Architecture
• WatchPoint Server
Web server
Centralized data
collector management
• OmniFlow Collector
• NetFlow Collector
• sFlow Collector
• Scalable architecture
• Collectors can be
separated for
increased performance
• Published API and SDK
for easy extension
WildPackets Seminar Series © WildPackets, Inc. 51

52.
WildPackets Key Differentiators
• Visual Expert Intelligence with Intuitive Drill-down
Let computer do the hard work, and return results, real-time
Packet / Payload Visualizers are faster than packet-per-packet diagnostics
Experts and analytics can be memorized and automated
• Automated Capture Analytics
Filters, triggers, scripting and advanced alarming system combine to provide
automated network problem detection 24x7
• Multiple Issue Network Forensics
Can be tracked by one or more people simultaneously
Real-time or post capture
• User-Extensible Platform
Plug-in architecture and SDK
• Aggregated Network Views and Reporting
NetFlow, sFlow, and OmniFlow
WildPackets Seminar Series © WildPackets, Inc. 52

53.
WildPackets, Inc.
1340 Treat Boulevard, Suite 500
Walnut Creek, CA 94597
(925) 937-3200
www.wildpackets.com
© WildPackets, Inc.