New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Novarum wlan
1. High Density WLAN Comparison
Testing: Aruba, Cisco and Juniper
September 2013
Copyright 2013 Novarum Inc.
2. Executive Summary
2
Key Findings
2
Test Environment
3
The Facility
3
Infrastructure Equipment
5
Client Equipment
5
Test Network
6
Testing Methodology
6
Tools
6
WLAN Infrastructure Configuration
7
Client Configuration
9
Comparison of High Density Testing
10
Comparison of Aggregate Capacity
10
Comparison of Retry Rates
13
Comparison of Test Streams
15
A Partial Analysis of Cisco Results
22
Conclusions
23
Key Findings
23
Appendix A – About Novarum
24
Appendix B – Aruba Configuration
25
Appendix C – Cisco Configuration
39
Appendix D – Juniper Configuration
43
High Density Wireless LANs: A Comparison
www.novarum.com
1
3. Executive Summary
There has been an explosion of mobile devices that are used within enterprises and public spaces – and almost all of
these devices are wireless LAN-enabled. This concentration of wireless LAN devices imposes unique stresses on
infrastructure wireless LANs, requiring them to operate at client and data densities that few networks ever have
before.
We chose to study this unique high-density environment by placing over 300 Wi-Fi client devices in a large room and
driving varying traffic loads across subsets of this collection of devices. We evaluated three different wireless network
infrastructures from leading enterprise network vendors - Aruba, Cisco and Juniper - with this collection of user
devices and traffic loads.
This paper describes the results of this comparative test of the three leading WLAN infrastructure systems in a high
density environment. This is the second in a series of high density WLAN test reports conducted by Novarum. Please
be sure to download first report: Novarum High Density WLAN Testing.
Key Findings
Our testing regimen clearly pushed the abilities of each of these networks and it was clear we had found the
maximum capacities of each network. But the networks differed dramatically in where and how maximum capacity
was handled. Each of these networks was clearly at maximum load, with network performance at peak as measured
by the degradation of aggregate throughput and increased MAC frame retry rates as we increased the number of
client devices.
Juniper maintained consistent performance as load increased from under 100 clients to over 300 clients - delivering
not only the highest absolute throughput, but with the best underlying fundamentals as evidenced by the lowest MAC
frame retry rates and the lowest rate of stream failures as load increased. Its maximum performance was at over 50%
more clients than Aruba’s maximum performance and at client numbers and total traffic loads for which the Cisco
network could not even be tested. For high density user device loads, Juniper clearly exhibited the most robust
performance.
Aruba delivered materially lower performance than Juniper - not only lower throughput but, more importantly, with
fewer clients. We noted that the Aruba system had materially higher MAC frame retry rates (1.9x) and materially
higher data stream failures (about 2x on average) than Juniper. The higher stream failure rates indicate that many
more clients were starved for data throughput capacity under the Aruba system than the Juniper system. The Aruba
system did successfully complete the full range of tests. While the Aruba system did not collapse like the Cisco
system, it delivered lower maximum performance than the Juniper system and delivered that maximum performance
with a lower number of client devices.
The Cisco network was challenged by this test and did not come close to completing the full range of tests. We had
great difficulty running the tests due to the very high number of stream failures that substantially increased the time
necessary to run the test. With limited availability of the test facility, we were unable to complete a full suite of tests for
the Cisco network. However, the one completed test is suggestive. Under a maximum client load of 302 clients, with
the downstream-only traffic load, the Cisco network had over 62% stream failures, more than 62% of the test
throughput streams had zero measurable data throughput. This compares to 17% stream failures for Aruba and 11%
stream failures for Juniper. With almost two thirds of the test streams with zero throughput, the Cisco system can be
considered to have collapsed under the network load.
High Density Wireless LANs: A Comparison
www.novarum.com
2
4. Test Environment
The Facility
We conducted the testing in the Juniper Aspiration Dome, a conference facility in Sunnyvale, California, on the
Juniper Networks campus adjacent to Moffett Federal Airfield.
Figure 1 - The Aspiration Dome.
The seating in the dome can be reconfigured to match the needs of
specific events. For our testing the dome was configured as a large
auditorium. There was a grid of seats, 20 per row, in the center of the
facility. We placed a single client device on each seat. The seats were
numbered 1 through 20, and we used rows a through q. The client name
was set to match its seat number:“a1”, “a2”, “a3”, etc. Every time we set
up the tests, the clients were placed in the same location.
Figure 2 - Client Devices in Rows Figure 3 - Dome Layout
Stage
AP1
AP2
AP3
AP4
AP5
AP6
Clients Clients
Chariot and Monitor
Machines
High Density Wireless LANs: A Comparison
www.novarum.com
3
5. During our testing, the permanent Juniper access points in and around the facility were disabled and we verified with
our tools that there were no other Wi-Fi networks that would interfere with our testing.
Figure 4 - Access Point Mounting
The access points under test were mounted on temporary stands about 10 feet above floor level as shown in Figure
4. At the base of each stand we had a MacBook Pro laptop running a packet capture program that was monitoring
the 5 GHz channel being used by that AP.
High Density Wireless LANs: A Comparison
www.novarum.com
4
6. Infrastructure Equipment
The Wi-Fi infrastructure for this testing is current three stream 802.11n enterprise Wi-Fi systems. The detailed
configurations are contained in an appendix later in this document.
Juniper
• WLC880 Wireless LAN Controller
• WLA532 Access Point
Cisco
• WLAN Controller 5508
• AP 3602
Aruba
• WLAN Controller 3600
• AP 135
Client Equipment
One of the significant factors in this high-density testing is our use of real-world client machines rather than traffic
simulators. We used more than 300 Wi-Fi clients. There are three categories of clients - laptops, smartphones and
tablets. We had a variety of clients running different operating systems: Windows 7 laptops from Dell and HP;
MacBook Pros and MacBook Airs; iOS, Windows 8 and Android tablets such as the iPad, iPad Mini, Kindle Fire,
Galaxy Tab 10.1, Microsoft Surface and Surface Pro; and iOS and Android smartphones including iPhone 3, 4, and 5,
Samsung Galaxy 3s, Nexus and more. These clients included all varieties of Wi-Fi devices that will typically be found
in a modern BYOD environment:
• 2.4 GHz 802.11n 1x1 MIMO smartphone clients
• Dual band 2.4 and 5 GHz 1x1 MIMO 20 MHz tablets
• Dual band 2.4 GHz and 5 GHz 1x1 MIMO 40 MHz tablets
• Dual band 2.4 GHz and 5 GHz 2x2 MIMO laptops
• Dual band 2.4 GHz and 5 GHz 3x3 MIMO laptops
High Density Wireless LANs: A Comparison
www.novarum.com
5
7. Test Network
The test network configuration is shown in Figure 5. This is a high-availability, high-density configuration for the entire
network. There are redundant routers, redundant Ethernet switches and redundant wireless LAN controllers.
Figure 5 - Test Network Topology
Testing Methodology
Tools
We used Ixia Chariot to generate traffic and record throughput. Chariot generates different network traffic flows
between pairs of endpoints. Every client machine had the Ixia Endpoint software installed. We had a dedicated
hardware Ixia Chariot server with licenses for up to 500 endpoints.
We collected packet captures in the 5 GHz band from each AP on each 5 GHz channel. During each test run, we
captured packets for 30 seconds in the middle of the data transfer. We extracted data and analyzed the packet
captures using EyePA and Cascade Pilot.
High Density Wireless LANs: A Comparison
www.novarum.com
6
8. We used Channelyzer Pro and AirMagnet from Fluke to do spectrum analysis. The spectrum screen shots in this
paper are from Channelyzer Pro. We drove the network with traffic generated from three different Chariot scripts:
Max Throughput Script
The "Max Throughput Script" creates a single TCP stream for smartphones and tablets. Each laptop client was
configured to support two TCP streams. All TCP streams are running the High Throughput TCP script from the
Chariot server to the client. The high throughput script attempts to send as much data as possible.
Bi-Directional Script
The "Bi-Directional Script" is a single TCP stream per client for smartphone and tablets, with each one running
High Throughput TCP script from the Chariot server down to the client. For laptops there are two TCP streams
per client, one running the High Throughput TCP script down from server to client and one running the script up
from the client to server. The laptops are the only clients with bidirectional traffic in this test.
Low Throughput Script
The "Low Throughput Script" is a single TCP stream per client for all clients. All TCP streams are running the High
Throughput TCP script from the Chariot server to the client; however, the throughput per client is constrained.
For laptops, the throughput is constrained to 5 Mbps per stream, tablets are limited to 2 Mbps and smartphones
are limited to 1 Mbps.
The scripts are configured to run for two minutes, with each client generating continuous streams of traffic. This is
one area where this high-density testing goes beyond the typical enterprise usage scenarios. Most wireless LANs
experience bursty traffic - some very high throughput peaks, followed by longer periods of low activity. Constant load
from all clients for a two minute period is very unusual, and difficult for 802.11 networks to handle. With bursty traffic,
the pressure on the network is relieved during the gaps in offered load. With continuous offered load, retransmissions
compete with new transmissions, congestion builds and there is no opportunity to relieve this pressure. This extreme
traffic pattern generated by our test scripts is useful for high-density testing because it allows us to simulate what
would happen to these networks with many more clients running more typical network traffic patterns.
Each time we conducted a test, we would run the test at least three times to ensure that we were getting consistent
results. With over 300 clients of different types with different operating systems, perfection was not possible. We
spent a great deal of time configuring the clients to ensure that they were active on the network and running the
proper software. Before running each series of tests we ran a script that would contact each client to make sure that
it was active, on the network and running the Chariot endpoint software. However, there were always a few clients
that would not initialize properly and did not complete the tests. The clients could be unavailable for a variety of
reasons: their battery died; they are off-line downloading new firmware; their Wi-Fi turned off; or their network
protocol stack was not running correctly. We decided that this was a realistic part of the test and since the number of
such off line clients was always small, it did not materially affect the results of the test.
WLAN Infrastructure Configuration
We followed the manufacturer recommendations and guidelines for high-density deployments. We configured the
APs manually with static channel assignment and power levels even though the vendors normally recommend
automatic channel tuning. We wanted to maintain a consistency of configuration that would be repeatable throughout
the testing, and we did not want any channel changes to occur during the testing.
High Density Wireless LANs: A Comparison
www.novarum.com
7
9. Our testing area inside the Dome was a large open space. Every client device was close enough to “hear” every AP,
and the APs were all within range of each other. In order to provide maximum capacity, we wanted to avoid sharing
the same channels between APs. This is not possible in the 2.4 GHz band. With only three independent Wi-Fi
channels available, we had to use each 2.4 GHz channel twice. We separated the APs operating on the same
channel as far as possible, but they were clearly overlapping and any associated clients will also share the same
channel. This 2.4 GHz configuration is not ideal, but it exemplifies a real world constraint that most enterprise
networks have to address.
High-Density Channel Configuration
At 5 GHz, there is more unlicensed spectrum and more independent channels are available. For this high-density
environment we wanted as many independent channels as possible. Since our client devices included many modern
802.11n clients, we also wanted to use 40 MHz wide channels in the 5 GHz band for the highest data rates possible.
We initially configured six independent 40 MHz channels in the 5 GHz band, one for each AP. The high-density
channel layout for the APs is shown in the table at right. This channel configuration at 5 GHz used two Dynamic
Frequency Selection (DFS) channels. In order to use these DFS channels, the system must detect radar operating in
those frequencies. If radar is detected, the wireless LAN must abandon that channel and move to an alternate
channel.
During the network configuration and test, we experienced a few radar events a day. (The Dome is located very close
to an active airport, Moffett Federal Airfield.) These DFS events modified the test results and made the testing less
repeatable. Overall, using the DFS channels proved to be too disruptive and time consuming during our testing.
Therefore we changed the 5 GHz channel plan to avoid the DFS channels and moved to a fixed configuration that
was of lower capacity for most of the testing.
Fixed 5 GHz Channel Configuration
For this comparative multi-vendor testing we adopted a fixed 5 GHz channel
configuration that avoids the DFS channels. This fixed configuration is 240 MHz total.
The 2.4 GHz is the same - three 20 MHz channels shared by six APs. The 5 GHz
band is four independent 40 MHz channels and one 20 MHz channel. Channel 165,
the 20 MHz channel, is shared by two APs. The channel layout is shown in the table
at right.
Overall, the potential capacity for the system under test was reduced by 20% to 25%
compared to the highest capacity configuration – using 180 MHz of 5 GHz spectrum
rather than potentially 240 MHz of spectrum. That is acceptable for this high-density
WLAN testing. This fixed configuration is more complicated, since there is a 20 MHz
channel at 5 GHz shared by two APs. The testing scenarios that we constructed will push the systems to their limits
and we should be able to observe how different systems allocate the limited resources to best serve the wireless LAN
clients.
Our initial testing as we were setting up and calibrating the system confirmed that while this channel configuration did
materially decrease the overall absolute capacity of the network, relative performance across vendors was
maintained.
High Density Wireless LANs: A Comparison
www.novarum.com
8
10. Client Configuration
The clients were arranged in rows of 20. When a client machine was entered into the test bed, it was assigned a
specific location, and named after that location; that never changed. For example, D10 is a Dell laptop running
Windows 7 with a 3 stream 802.11n Wi-Fi network adapter. That machine has a physical label D10 and a software
label D10 and would always be at location row D, seat 10.
Figure 6 — Client Devices in Seats
Figure 7 shows the arrangement of clients on the floor. We ran each set of Chariot tests with three different client
configurations - All, Half, and Third. When configuring the tests for fewer clients, we attempted to maintain an even
spread of clients throughout the space. For the Half load configuration, we removed every other row of clients from
the test. For the Third load configuration, we removed more rows as shown in Figure 7 below.
Figure 7 – Client Configurations
High Density Wireless LANs: A Comparison
www.novarum.com
9
11. Comparison of High Density Testing
We replicated the exact same network configuration with systems from Aruba and Cisco. We used a high availability
configuration with two WLAN controllers and used their latest dual band, three stream indoor enterprise APs.
Everything was mounted in the same locations as the Juniper gear.
We configured the networks according to the manufacturer’s recommendations for high density deployments.
However, we did configure the same static channel plan that we created for the Juniper testing. The APs are
mounted in the same locations and set to the same channels and channel sizes. All user devices were in the same
locations.
We were not able to get the Cisco network to complete the testing successfully for a significant number of tests. We
had a very high number of Chariot initialization errors. Many clients dropped out before the tests began. When we did
complete a Chariot test run, the throughput results were often very low. When large numbers of clients do not
complete their streams in the allotted time for the test, the Chariot error processing at the end of test can be very
long - much longer than a successful test.
We were able to complete only one set of tests for the Cisco configuration. At the time of the testing, without
complete analysis of the data, we did not fully understand the results and since the result of the Cisco test was
impractically long test times, we abandoned further Cisco network testing. As we finished the test program and
conducted the full post-test analysis, we decided that the partial Cisco test results told an important story - but not a
story directly comparable to that which we are able to construct from the full suite of tests conducted on the Aruba
and Juniper equipment. We report the Cisco results separately for this reason.
For both the Aruba and Juniper systems, we were able to configure the system properly and we were able to get
consistent behavior and reasonable throughput. We ran through the complete suite of tests with Aruba and Juniper
including all client configurations.
Out of the mountain of data we collected for Aruba and Juniper we believe that there are three themes are of
particular interest:
• The comparison of aggregate throughput capacity;
• The comparison of MAC level frame retry rates - often the WLAN “canary in the coal mine” – first indicator of
underlying wireless LAN system issues; and
• The comparison of distribution of capacity between all the independent streams to various clients – how
equitably the capacity is shared under high load.
Comparison of Aggregate Capacity
Let's first look at the overall throughput capacity under the three types of traffic. Overall throughput is a measure of
the total capacity of the network under a given load ... without consideration of how that capacity is shared among all
the user devices.We are purely looking at total capacity.
Under the low, throttled throughput model, Juniper performs better at low client levels and maintains overall
throughput as the number of client increases. Aruba, surprisingly, has rather low overall throughput with low numbers
of clients but then matches Juniper as the number of clients increases.
High Density Wireless LANs: A Comparison
www.novarum.com
10
12. Figure 8
Under bidirectional traffic, Juniper and Aruba have very similar capacities, though Juniper has modestly better overall
capacity at higher client levels.
Figure 9
0"
100"
200"
300"
400"
500"
600"
0" 100" 200" 300" 400"
Mbps%
Number%of%Clients%
Low%Throughput:%%Aruba%and%Juniper%
Aruba"
Juniper"
0.0#
100.0#
200.0#
300.0#
400.0#
500.0#
600.0#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Bidirec5onal%Throughput:%Aruba%and%
Juniper%
Aruba#
Juniper#
High Density Wireless LANs: A Comparison
www.novarum.com
11
13. Under a purely downstream load, Juniper materially outperforms Aruba - particularly as the number of clients grows.
Figure 10
It is fair to create a composite measurement of these three traffic models, since in practice, aggregate load will be a
composite of these traffic models. If we average the three measurements, we have a composite comparison in
which at low numbers of clients, Aruba and Juniper have similar capacities. However, with increasing numbers of
clients, the Aruba system slowly degrades in overall capacity while the Juniper system essentially maintains -
modestly increasing and then decreasing in capacity.
Figure 11
As we can see from this composite measurement, Juniper not only has a higher total capacity, but reaches peak
system capacity at about 50% more clients than Aruba.
0.0#
100.0#
200.0#
300.0#
400.0#
500.0#
600.0#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Down%Throughput:%Aruba%and%Juniper%
Aruba#
Juniper#
0.0#
100.0#
200.0#
300.0#
400.0#
500.0#
600.0#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Composite%Throughput:%Aruba%and%
Juniper%
Aruba#
Juniper#
High Density Wireless LANs: A Comparison
www.novarum.com
12
14. Comparison of Retry Rates
One of the key measures of underlying WLAN network issues is the MAC frame retry rate - how often MAC frames
are retransmitted due to error or congestion. We looked at three examples (from the many tests) to get the 5 GHz
MAC frame retry rates from the packet capture data:
• Bidirectional traffic with full client load,
• Bidirectional traffic with half client load and
• Low throughput with low client load.
In every case, we can see dramatically higher frame retry rates in Aruba over Juniper. On average, Aruba
consistently is about 1.9x higher in frame retry rates at 5 GHz. Junipers lower frame retry rates generally imply greater
underlying stability and robustness.
Figure 12
For the bidirectional traffic model with full client load, we see Aruba has substantially higher frame retry rates than
Juniper in 4 out of the 5 channels configured.
0%#
10%#
20%#
30%#
40%#
50%#
60%#
36# 44# 149# 157# 165#
Retry&Rate&
5&GHz&Channel&
Bidirec4onal&Retry&Rate:&&302&
Clients&;&Aruba&and&Juniper&
Aruba#
Juniper#
High Density Wireless LANs: A Comparison
www.novarum.com
13
15. Figure 13
For the bidirectional traffic model with half client load (156 clients), Aruba has higher retry rates than Juniper in all 5
channels configured.
Figure 14
For the last case, with the low throttled load model and low client load (95 clients), Juniper consistently has a lower
retry rate on all configured channels.
0%#
10%#
20%#
30%#
40%#
50%#
60%#
36# 44# 149# 157# 165#
Retry&Rate&
5&GHz&Channel&
Birec3onal&Retry&Rate:&156&Clients&9&
Aruba&and&Juniper&
Aruba#
Juniper#
0%#
10%#
20%#
30%#
40%#
50%#
60%#
36# 44# 149# 157# 165#
Retry&Rate&
5&GHz&Channel&
Low&Retry&Rate:&95&Clients&7&Aruba&
and&Juniper&
Aruba#
Juniper#
High Density Wireless LANs: A Comparison
www.novarum.com
14
16. Comparison of Test Streams
In our tests, each device has at least one TCP communications stream (generally down from the Chariot server to the
device), and some have multiple streams - either an additional stream down to the device or, in the bidirectional test,
an upstream stream from the device to the server. It is illustrative to look at the performance of the each network in
terms of the histogram of results over all the clients, as well as the summary metrics of the average speed of each
stream under each type of test, the standard deviation of the range of results, and particularly the frequency of
occurrence of stream failure.
Stream failure is defined as a stream that fails to report results by the end of the 120 second test. We presume zero
throughput for that stream since no results were reported. The number of such failed streams is a measurement of
how equitable and stable the system provides bandwidth to the set of clients. We will see large differences between
Aruba and Juniper, particularly in stream failure rate. And when we consider our partial Cisco testing results, the issue
of stream failure is even more compelling.
Average Stream Throughput
Let's first look at the average stream throughput rate as we vary the number of clients and the type of traffic between
our low, bidirectional and downstream tests.
0.00#
0.50#
1.00#
1.50#
2.00#
2.50#
3.00#
3.50#
4.00#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Down%Stream%Throughput:%%
Aruba%and%Juniper%
Juniper#
Aruba#
Figure 15
For our downstream only tests, Aruba degrades noticeably more rapidly than Juniper, reflecting the aggregate
capacity analysis, in this case, simply divided by the number of streams.
High Density Wireless LANs: A Comparison
www.novarum.com
15
17. 0.00#
0.50#
1.00#
1.50#
2.00#
2.50#
3.00#
3.50#
4.00#
0# 100# 200# 300# 400#
Mpbs%
Number%of%Clients%
Bidirec5onal%Stream%
Throughput:%Aruba%and%Juniper%
Juniper#
Aruba#
Figure 16
Similarly, the average bidirectional stream throughput mirrors the aggregate capacity for this test, as does the low,
throttled stream throughput.
0.00#
0.50#
1.00#
1.50#
2.00#
2.50#
3.00#
3.50#
4.00#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Low%Stream%Throughput:%Aruba%
and%Juniper%
Juniper#
Aruba#
Figure 17
We can see that in the low and downstream tests, Juniper usually delivers more average throughput per stream than
Aruba, and for the bidirectional test, both systems were about the same for all numbers of clients.
If we create a composite average flow, it shows that Juniper has a 12% better average stream throughput than
Aruba over this range of client load.
High Density Wireless LANs: A Comparison
www.novarum.com
16
19. 0%#
5%#
10%#
15%#
20%#
25%#
30%#
0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# 19# 21# 23#
Mbps%
Downstream%Traffic%2%95%Client%Load%
Juniper#
Aruba#
Figure 25
0%#
5%#
10%#
15%#
20%#
25%#
30%#
0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# 19# 21# 23# 25# 27# 29#
Mbps%
Bidirec,onal%Traffic%3%95%Client%Load%
Juniper#
Aruba#
Figure 26
0%#
5%#
10%#
15%#
20%#
25%#
30%#
35%#
40%#
0# 0.5# 1# 1.5# 2# 2.5# 3# 3.5# 4# 4.5# 5#
Mbps%
Low%Traffic%.%95%Client%Load%
Juniper#
Aruba#
Figure 27
Over all traffic models and over all client loads, we can see the pattern that Aruba has more streams with zero (or very
low) measurable bandwidth while Juniper tends to have a tighter (and higher throughput) clustering of bandwidth
distribution between streams.
The stream histogram data demonstrates that Juniper delivers a more equitable and predictable user experience –
there is less variation of the user experience between user devices in the same room than the Aruba network
demonstrated. And as we shall see, the Cisco network could not be tested in this way since its highly unpredictable
performance prevented full testing.
Standard Deviation of Stream Throughput
We saw common themes in the histograms, now lets see some summary data that illustrates some of the differences
between these systems. One indicator is the standard deviation of the collection of the streams during a test. A lower
standard deviation is an indicator of a more consistent and equitable distribution of throughput amongst the clients -
a higher standard deviation indicates a less equitable distribution.
Figure 28
The higher standard deviation for the downstream throughput test suggests that a Aruba has a much less equitable
distribution of throughput at low client load than Juniper, closing the gap as the number of clients increases.
0.00#
2.00#
4.00#
6.00#
8.00#
10.00#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Down%StreamThroughput%
StDev%
Juniper#
Aruba#
High Density Wireless LANs: A Comparison
www.novarum.com
18
20. Figure 29
This pattern of unequal distribution (higher standard deviation) of bandwidth at low client load, that becomes more
equal (lower standard deviation) at high load is repeated for the bidirectional stream tests. In both bidirectional and
downstream cases, Aruba has substantially higher standard deviation in stream throughput for low numbers of clients
and incrementally better (lower) standard deviation at high numbers of clients. In both systems, the standard deviation
decreases with increasing numbers of clients.
Figure 30
For the low throughput test, both systems deliver very similar standard deviations in stream throughput.
Juniper delivers a more consistent (more equitable) variation in flow throughput with variation between flows
remaining more consistent (and delivering a more consistent user experience) as the number of clients increases.
0.00#
2.00#
4.00#
6.00#
8.00#
10.00#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Birdirec5onal%Stream%
Throughput%StDev%
Juniper#
Aruba#
0.00#
2.00#
4.00#
6.00#
8.00#
10.00#
0# 100# 200# 300# 400#
Mbps%
Number%of%Clients%
Low%Stream%Throughput%
StDev%
Juniper#
Aruba#
High Density Wireless LANs: A Comparison
www.novarum.com
19
21. Stream Failure Rate
Perhaps the most revealing measurement from the stream data comes from the stream failure data. While both
systems showed an increased stream failure with increasing client load, Aruba has a dramatically higher stream failure
that appears to have an inflection point around 100 client stations for all three traffic models.
0%#
5%#
10%#
15%#
20%#
25%#
30%#
0# 100# 200# 300# 400#
%age%Streams%
Number%of%Clients%
Down%Stream%Failure%Rate%
Juniper#
Aruba#
Figure 31
0%#
5%#
10%#
15%#
20%#
25%#
30%#
0# 100# 200# 300# 400#
%age%Streams%
Number%of%Clients%
Bidirec7onal%Stream%Failure%
Rate%
Juniper#
Aruba#
Figure 32
High Density Wireless LANs: A Comparison
www.novarum.com
20
22. 0%#
5%#
10%#
15%#
20%#
25%#
30%#
0# 100# 200# 300# 400#
%age%Streams%
Number%of%Clients%
Low%Stream%Failure%Rate%
Juniper# Aruba#
Figure 33
0%#
5%#
10%#
15%#
20%#
25%#
30%#
0# 100# 200# 300# 400#
%age%Streams%
Number%of%Clients%
Composite%Stream%Failure%
Rate%
Juniper# Aruba#
Figure 34
For all types of traffic, Aruba has a materially higher rate of stream failures - clients that receive no data throughput -
than Juniper. This stream failure rate dramatically increases as the number of clients increases from 95 to 156 and
continues to increase to the full client load of 302. Juniper has a much lower and much more stable stream failure
rate through 156 clients, before beginning an increase in stream failure at full client load.
High Density Wireless LANs: A Comparison
www.novarum.com
21
23. A Partial Analysis of Cisco Results
As previously noted, we were not able, within the time constraints of this test, to fully evaluate the Cisco
configuration. We found high failure rates of Chariot test streams which slowed the testing process unacceptably.
And as we have seen above - failure rates of the Chariot test streams is an important distinguishing factor between
Aruba and Juniper.
However, the data from the one complete test of the Cisco configuration is interesting and we think illustrative. This is
for the downstream only, full throughput traffic model with the full 302 client load.
Figure 35
As we can see in Figure 35, the histogram of the distribution of stream throughputs shows that 62% of the streams
had no measurable data throughput. The aggregate throughput of these test runs was 376 Mbps which is OK, but
there is a highly inequitable distribution of that capacity. About 3% of the streams had throughput higher than 10
Mbps. These few streams constituted the bulk of the data throughput while the majority of the devices under test
were completely starved of data capacity.
For similar tests, Aruba had a stream error rate of 17% and Juniper’s error rate was 11%. The 62% stream error rate
for Cisco on this test illustrates why we were unable to complete the testing for Cisco. For the other tests that we
attempted, we could see the traffic generated over the air but the error rate was so high that Chariot did not report
any useful results. While we do not have the measurements for the other traffic tests and other client load models, we
suspect that the same pattern would ensue based on our difficulties in getting these tests to run reliably for the Cisco
network.
0.0%$
10.0%$
20.0%$
30.0%$
40.0%$
50.0%$
60.0%$
70.0%$
0$ 1$ 2$ 3$ 4$ 5$ 6$ 7$ 8$ 9$ 10$ 11$ 12$ 13$ 14$ 15$ 16$ 17$ 18$ 19$ 20$
%age%Streams%
Mbps%
Cisco%Downstream%302%Clients%Stream%
Distribu:on%
High Density Wireless LANs: A Comparison
www.novarum.com
22
24. Conclusions
Almost all enterprise wireless LANs will have some high density areas in their networks and these areas are often in
visible and highly used portions of the network – auditoriums, classrooms, conference areas, etc..
We found meaningful and compelling differences between the network performance delivered by three important
wireless LAN vendors: Aruba, Cisco and Juniper.
Key Findings
We are measuring the limits of capacity of each of these networks under high user density load. Each network was
clearly at maximum load, as measured by the degradation of aggregate throughput and increased MAC frame retry
rates as more client devices were added.
Juniper maintained consistent performance as load increased from under 100 clients to over 300 clients – delivering
not only the highest aggregate throughput, but the lowest MAC frame retry rates and the most stability under load as
measured by the number of simultaneous streams with real, non-zero throughput. Its maximum performance was at
over 50% more clients than Aruba’s maximum performance and at client and traffic loads at which the Cisco network
could not be successfully tested.
Aruba delivered materially lower performance than Juniper – not only lower throughput but, more importantly, at lower
numbers of clients. We noted that the Aruba system had materially higher MAC frame retry rates (1.9x) and materially
higher data stream failures (about 2x on average) than Juniper. The higher stream failure rates indicate that many
more clients were starved for data throughput capacity under the Aruba system than the Juniper system.The Aruba
system did successfully complete the full range of tests.
The Cisco network was challenged by this test. We had great difficulty running the tests due to the very high number
of stream failures that substantially increased the time required to run the test. With limited availability of the facility,
we were unable to complete a full suite of tests for the Cisco network. However, the one completed test is
suggestive. Under a maximum client load of 302 clients, under the downstream only traffic load, the Cisco network
had over 62% stream failures – that is, almost two-thirds of the test throughput streams had zero measurable data
throughput. This compares to 17% for Aruba and 11% for Juniper.
Both the Juniper and Aruba networks delivered a useful network under these conditions of high user density - unlike
the Cisco network which did not deliver a reliable, robust network. The Juniper network delivered the most capable
network of demonstrably higher capacity, greater equity of throughput between user devices, and of higher stability
as the network load increased.
High Density Wireless LANs: A Comparison
www.novarum.com
23
25. Appendix A – About Novarum
Novarum is an independent consulting firm specializing in wireless broadband technology and business. Novarum
provides consulting, strategic advice, analysis and network design for cities, service providers, enterprises and
vendors in the wireless broadband industry. Our technology focus spans Wi-Fi, WiMAX and 4G cellular data systems.
Novarum offers a unique insider perspective from pioneers in the wireless and networking industry who have practical
experience bringing wireless products to market.
Phil Belanger
Phil has over 25 years of broad leadership in the technology, marketing and standards of data networks. Phil
pioneered local area networking technology with Zilog and Corvus and extended that leadership by co-leading the
multi-company technical and marketing efforts that produced the original IEEE 802.11 wireless LAN standard.
Phil defined the original market position of wireless LANs for mobile computing with Xircom. While at Aironet, he
broadened the market for wireless LANs and laid the foundation for Wi-Fi's success with the acquisition of Aironet by
Cisco. Phil was one of the founders of the the Wi-Fi Alliance and served as the group’s initial Chairman, creating the
Wi-Fi brand and promoting Wi-Fi for the entire industry. He helped create the business model for Wi-Fi service
providers with Wayport and expanded the market for Wi-Fi infrastructure with extended range technology of Vivato
and municipal mesh networks at BelAir Networks.
Ken Biba
Ken is a rocket scientist. He also has many years experience in the network information systems industry bringing a
unique background of general management with a strong product and marketing focus in network systems and
information security. Ken was an early engineer of the Internet in 1975. He has co-founded and managed four
notable networking companies — Sytek, which was focused on cable TV-based local and metropolitan data
networks, Agilis which was focused on wireless handheld computers, Xircom, which developed local area network
client products for mobile computing, and Vivato, which was focused on scaling Wi-Fi infrastructure to cover
campuses and metropolitan areas. Ken's perspective as CEO, board member of public and private companies, and
as a technologist brings unique insight to the business, market and technology of bringing useful wireless solutions to
users. Ken has a Bachelor of Science in Physics (Magna Cum Laude, Tau Beta Pi) and a Master of Science in
Computer Science from Case Western Reserve University.
Wayne Gartin
Wayne is a senior executive with world-wide experience at start-ups and Fortune 500 companies. He has built high
level relationships and delivered business partnerships at all levels for companies in the communication, software,
and semiconductor markets. Wayne has worked with industry leading suppliers in all aspects of network technology,
including long haul transport, metropolitan networks, wired and wireless LANs. He has successfully run multi-million
dollar sales teams for companies in the access (last mile) consumer oriented markets, Passive Optical Networks,
VoIP, and IMS. Wayne has held executive and senior level positions at Centillium, Agility (now JDSU), Bandwidth 9
(now NeoPhotonics), Infineon, Lucent, Adaptec, and Intel. He is also the co-founder of a semi-conductor IP
company. Wayne’s experience with multiple channels and leading successful sales teams to multi-million dollar
revenue levels brings a unique insight to the strategies necessary to successfully launch new products and
technologies into the market. Wayne has a BS in Math and an MBA from the University of Utah. He has been a
certified instructor for sales and marketing courses in strategic planning, negotiations, and sales management.
High Density Wireless LANs: A Comparison
www.novarum.com
24
27. netservice svc-noe-oxo udp 5000 alg noe
netservice svc-natt udp 4500
netservice svc-ftp tcp 21 alg ftp
netservice svc-microsoft-ds tcp 445
netservice svc-svp 119 alg svp
netservice svc-smtp tcp 25
netservice svc-gre 47
netservice web tcp list "80 443"
netservice svc-netbios-ns udp 137
netservice svc-sips tcp 5061 alg sips
netservice svc-smb-udp udp 445
netservice svc-cups tcp 515
netservice svc-esp 50
netservice svc-v6-dhcp udp 546
netservice svc-snmp udp 161
netservice svc-bootp udp 67 69
netservice svc-pcoip2-udp udp 4172
netservice svc-msrpc-udp udp 135 139
netservice svc-ntp udp 123
netservice svc-icmp 1
netservice svc-ssh tcp 22
netservice svc-lpd-udp udp 631
netservice svc-v6-icmp 58
netservice svc-http-proxy1 tcp 3128
netservice svc-vmware-rdp tcp 3389
netexthdr default
!
ip access-list session control
!
ip access-list session allow-diskservices
!
ip access-list session v6-icmp-acl
!
ip access-list session validuser
network 169.254.0.0 255.255.0.0 any any deny
any any any permit
ipv6 any any any permit
!
ip access-list session vocera-acl
!
ip access-list session v6-https-acl
!
ip access-list session vmware-acl
!
ip access-list session v6-control
!
ip access-list session icmp-acl
!
High Density Wireless LANs: A Comparison
www.novarum.com
26
28. ip access-list session testing
!
ip access-list session captiveportal
!
ip access-list session v6-dhcp-acl
!
ip access-list session allowall
!
ip access-list session v6-dns-acl
!
ip access-list session https-acl
!
ip access-list session sip-acl
!
ip access-list session ra-guard
!
ip access-list session dns-acl
!
ip access-list session citrix-acl
!
ip access-list session tftp-acl
!
ip access-list session skinny-acl
!
ip access-list session srcnat
!
ip access-list session vpnlogon
!
ip access-list session logon-control
!
ip access-list session allow-printservices
!
ip access-list session v6-allowall
!
ip access-list session cplogout
!
ip access-list session http-acl
!
ip access-list session dhcp-acl
!
ip access-list session v6-http-acl
!
ip access-list session captiveportal6
!
ip access-list session ap-uplink-acl
!
ip access-list session noe-acl
!
High Density Wireless LANs: A Comparison
www.novarum.com
27
29. ip access-list session svp-acl
!
ip access-list session ap-acl
!
ip access-list session v6-ap-acl
!
ip access-list session h323-acl
!
ip access-list session v6-logon-control
!
aaa derivation-rules user test
!
vpn-dialer default-dialer
ike authentication PRE-SHARE aea06b09f946b8ead663bb1b77b7edc345861acb504d2749
!
user-role ap-role
!
user-role guest-logon
!
user-role guest
!
user-role stateful-dot1x
!
user-role logon
!
!
controller-ip vlan 1
interface mgmt
shutdown
!
dialer group evdo_us
init-string ATQ0V1E0
dial-string ATDT#777
!
dialer group gsm_us
init-string AT+CGDCONT=1,"IP","ISP.CINGULAR"
dial-string ATD*99#
!
dialer group gsm_asia
init-string AT+CGDCONT=1,"IP","internet"
dial-string ATD*99***1#
!
dialer group vivo_br
High Density Wireless LANs: A Comparison
www.novarum.com
28
43. config time timezone location 5
config ap packet-dump truncate 0
config ap packet-dump buffer-size 2048
config ap packet-dump capture-time 10
config mgmtuser add encrypt admin 1 fbb528b8da08963b9ad15ce73edca1a6 e94ab7e8ad70417d3445e92e8fc2a3c2b359172a 16
2b48e5806c0c231af24f36a9b5d66d180000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000 read-write
config rfid timeout 1200
config rfid status enable
config rfid mobility pango disable
transfer upload path /
transfer upload datatype config
transfer upload serverip 10.0.1.16
transfer upload filename csco-090413-cfg.txt
transfer download path /
transfer download serverip 10.0.1.16
transfer download filename csco-090413-cfg.txt
High Density Wireless LANs: A Comparison
www.novarum.com
42
44. Appendix D – Juniper Configuration
# Configuration "nvgen'd" at 2013-8-30 15:27:36
# Image 8.0.3.0.143
# Model WLC880R
# Last change occurred at 2013-8-30 15:27:33
set trace lb level 10
set ip route default 10.0.1.1 1
set ip dns domain complab.trapezenetworks.com
set ip dns enable
set ip dns server 8.8.8.8 PRIMARY
set log console disable severity error
set dot1x quiet-period 0
set system name MXR-2-1
set system ip-address 10.0.1.2
set system countrycode US
set timezone PST -8 0
set qos-profile dometest trust-client-dscp enable
set service-profile ag ssid-name ag
set service-profile ag rsn-ie cipher-ccmp enable
set service-profile ag rsn-ie enable
set service-profile ag transmit-rate 11g mandatory 6.0,11.0,24.0 disabled 1.0,2.0,5.5 beacon-rate 6.0 multicast-rate AUTO
set service-profile ag transmit-rate 11ng mandatory 6.0,11.0,24.0 disabled 1.0,2.0,5.5 beacon-rate 6.0 multicast-rate AUTO
set service-profile ag attr vlan-name Enterprise
set service-profile ag-clear ssid-name ag-clear
set service-profile ag-clear ssid-type clear
set service-profile ag-clear auth-fallthru last-resort
set service-profile ag-clear attr vlan-name Enterprise
set service-profile ag-psk ssid-name ag-psk
set service-profile ag-psk auth-fallthru last-resort
set service-profile ag-psk psk-encrypted
040e5e550e201e18581d5247425c0e067329222b61617a425710565007010d075756501d4e0a09530b530407525f550309565615035c095d5f
59724e1b511a5c16
set service-profile ag-psk wpa-ie auth-dot1x disable
set service-profile ag-psk rsn-ie cipher-ccmp enable
set service-profile ag-psk rsn-ie auth-psk enable
set service-profile ag-psk rsn-ie auth-dot1x disable
set service-profile ag-psk rsn-ie enable
set service-profile ag-psk attr vlan-name Enterprise
set service-profile provisionme ssid-name provisionme
set service-profile provisionme ssid-type clear
set service-profile provisionme auth-fallthru web-portal
set service-profile provisionme web-portal-form http://10.0.1.15/
set service-profile provisionme web-portal-acl portalacl
set service-profile provisionme attr vlan-name Guest
set vlan-profile default vlan Enterprise tag 2
set vlan-profile default vlan Guest tag 3
set vlan-profile default vlan Remediation tag 4
set vlan-profile default vlan VoIP tag 5
High Density Wireless LANs: A Comparison
www.novarum.com
43
45. set vlan-profile default vlan Sslvpn tag 10
set radius server IC address 10.0.1.5 encrypted-key 044f0e151b284249584b56
set radius server SP address 10.0.1.16 encrypted-key 03105e1812062f4b1f5b4a
set radius server SBR address 10.0.1.19 encrypted-key 131112011f050a2d7a767b
set radius server NPS address 10.0.1.15 deadtime 0 encrypted-key 051f031c3545400e485744
set server group IC-group members IC
set server group SP-group members SP
set server group SBR-group members SBR
set server group NPS-group members NPS
set radius dac IC address 10.0.1.5 replay-protect disable encrypted-key 071b245f5a001702464058
set radius dac SP address 10.0.1.16 replay-protect disable encrypted-key 0835495d1d100b1043595f
set radius dac AD address 10.0.1.15 replay-protect disable encrypted-key 14031718180d242c757a60
set enablepass password 81254119e3b01232456e0b6f652be87b9891
set accounting dot1x ssid Juniper_Secure_Access ** start-stop SP-group
set accounting web ssid any ** start-stop SP-group
set accounting web ssid Juniper_Guest_Access ** start-stop SP-group
set authentication dot1x ssid Juniper_Secure_Access ** pass-through IC-group
set authentication dot1x ssid Wireless_CAC_Access ** pass-through SBR-group
set authentication dot1x ssid ag ** pass-through NPS-group
set authorization dynamic ssid any SP
set authorization dynamic ssid Juniper_Secure_Access IC
set authorization dynamic ssid Juniper_Guest_Access AD
set accounting system SP-group
set user admin password encrypted 09584b1a0d0c19155a5e57
set device-fingerprint ios-generic device-group ios
set device-fingerprint ios-generic rule 1 type dhcp option-list NOT-CONTAINS 12
set device-fingerprint ios-generic rule 2 type dhcp option 12 NOT-CONTAINS iPhone
set device-fingerprint ios-generic rule 3 type dhcp option 12 NOT-CONTAINS iPad
set device-fingerprint ios-generic rule 4 type dhcp option 12 NOT-CONTAINS iPod
set device-fingerprint ios-generic rule 5 type dhcp option-list CONTAINS 53,55,57,61,51
set device-fingerprint ios-generic rule 6 type dhcp option-list CONTAINS 53,55,57,61,50,51
set device-fingerprint ios-generic rule 7 type dhcp option-list CONTAINS 53,55,57,61,50,54
set device-fingerprint ios-generic rule 8 type dhcp option 55 NOT-CONTAINS 1,3,6,15,119,95,252,44,46
set device-fingerprint ios-generic rule 9 type dhcp option 55 NOT-CONTAINS 1,3,6,15,119,95,252,44,46,47
set device-fingerprint ios-generic rule-expression "((1 or 2) and (1 or 3) and (1 or 4) and (5 or 6 or 7) and (8 and 9))"
set radio-profile default rf-scanning mode passive
set radio-profile default rf-scanning channel-scope operating
set radio-profile default dfs-channels enable
set radio-profile default service-profile ag
set ap 1 serial-id jb0212248283 model WLA532-US
set ap 1 radio 1 channel 1 tx-power 11 mode enable
set ap 1 radio 1 load-balancing group 24ghz rebalance
set ap 1 radio 2 channel 165 tx-power 11 mode enable
set ap 1 radio 2 load-balancing group 5ghz rebalance
set ap 1 local-switching mode enable vlan-profile default
set ap 2 serial-id jb0212248445 model WLA532-US
set ap 2 radio 1 channel 6 tx-power 11 mode enable
set ap 2 radio 1 load-balancing group 24ghz rebalance
High Density Wireless LANs: A Comparison
www.novarum.com
44
46. set ap 2 radio 2 channel 36 mode enable
set ap 2 radio 2 load-balancing group 5ghz rebalance
set ap 2 local-switching mode enable vlan-profile default
set ap 3 serial-id jb0212248218 model WLA532-US
set ap 3 radio 1 channel 11 tx-power 11 mode enable
set ap 3 radio 1 load-balancing group 24ghz rebalance
set ap 3 radio 2 channel 44 tx-power 11 mode enable
set ap 3 radio 2 load-balancing group 5ghz rebalance
set ap 3 local-switching mode enable vlan-profile default
set ap 4 serial-id jb0212248260 model WLA532-US
set ap 4 radio 1 channel 11 tx-power 11 mode enable
set ap 4 radio 1 load-balancing group 24ghz rebalance
set ap 4 radio 2 channel 149 tx-power 11 mode enable
set ap 4 radio 2 load-balancing group 5ghz rebalance
set ap 4 local-switching mode enable vlan-profile default
set ap 5 serial-id jb0211483963 model WLA532-US
set ap 5 radio 1 channel 1 tx-power 11 mode enable
set ap 5 radio 1 load-balancing group 24ghz rebalance
set ap 5 radio 2 channel 157 tx-power 11 mode enable
set ap 5 radio 2 load-balancing group 5ghz rebalance
set ap 5 local-switching mode enable vlan-profile default
set ap 6 serial-id jb0212248253 model WLA532-US
set ap 6 radio 1 channel 6 tx-power 11 mode enable
set ap 6 radio 1 load-balancing group 24ghz rebalance
set ap 6 radio 2 channel 165 tx-power 11 mode enable
set ap 6 radio 2 load-balancing group 5ghz rebalance
set ap 6 local-switching mode enable vlan-profile default
set ip telnet server enable
set band-preference 5ghz
set vlan 1 name Management
set vlan 1 port 1 tag 1
set vlan 2 name Enterprise
set vlan 2 port 2 tag 2
set vlan 3 name Guest
set vlan 3 port 1 tag 3
set vlan 4 name Remediation
set vlan 4 port 1 tag 4
set vlan 5 name VoIP
set vlan 5 port 1 tag 5
set vlan 10 name Sslvpn
set vlan 10 port 1 tag 10
set interface 1 ip 10.0.1.2 255.255.255.0
set interface 3 ip 10.0.3.2 255.255.255.0
set mobility-domain mode seed domain-name PDM
set mobility-domain member 10.0.1.3
set mobility-domain ap-affinity-group address 10.0.1.0 netmask 255.255.255.0
set security acl name portalacl permit udp 0.0.0.0 255.255.255.255 eq 68 0.0.0.0 255.255.255.255 eq 67
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 17.0.0.0 0.255.255.255
High Density Wireless LANs: A Comparison
www.novarum.com
45
47. set security acl name portalacl permit 17.0.0.0 0.255.255.255
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 173.255.192.195 0.0.0.255
set security acl name portalacl permit 173.255.192.195 0.0.0.255
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 69.31.128.0 0.0.15.255
set security acl name portalacl permit 69.31.128.0 0.0.15.255
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 199.7.48.0 0.0.15.255
set security acl name portalacl permit 199.7.48.0 0.0.15.255
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 205.235.112.0 0.0.15.255
set security acl name portalacl permit 205.235.112.0 0.0.15.255
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 10.0.1.16 0.0.0.0
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 10.0.1.15 0.0.0.0
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 216.255.76.102 0.0.0.0
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 216.255.76.73 0.0.0.0
set security acl name portalacl permit ip 0.0.0.0 255.255.255.255 74.203.64.100 0.0.0.0
set security acl name portalacl deny 0.0.0.0 255.255.255.255 capture
commit security acl portalacl
set cluster mode enable
set auto-tune channel band 11a mode disable
set auto-tune channel band 11a schedule Any1910
set auto-tune channel band 11a indo-threshold 65
set auto-tune channel band 11bg mode disable
set auto-tune channel band 11bg schedule Any1910
set auto-tune channel band 11bg indo-threshold 65
High Density Wireless LANs: A Comparison
www.novarum.com
46