WLAN Stress Test - Ruckus, Aruba, Aerohive, Meraki and Cisco

[Net-Ctrl]

Divergent Dynamics have released a new stress test report putting Ruckus, Aruba, Aerohive, Meraki and Cisco against each other. Using mid-range 802.11ac access points (APs), the testing measured each AP’s ability to simultaneously deliver video and data to 60 client devices.

 

I thought it could be of interest to some of you. A lot of the time these reports are co-sponsored/supported by a vendor, but on quick inspection, i can't see any obvious signs of this being the case.

 

You can view the report here: http://divdyn.com/docs/WiFi-vendor-video-stress-test.pdf

 

Ruckus intro piece:

 

Using mid-range 802.11ac access points (APs), the testing measured each AP’s ability to simultaneously deliver video and data to 60 client devices. The results showed that the Ruckus R610 AP was the only tested AP able to maintain stall-free video delivery to each client when data traffic loading was simultaneously applied to the network.

 

“The tests were conducted in a real-world, high client-density environment and carried out using industry best practices and each manufacturer’s recommendations,” said Devin Akin, founder and CEO of Divergent Dynamics. “In this environment, Ruckus APs delivered consistent stall-free video, outperforming five APs from four manufacturers, including two APs with higher performance specifications. The R610 was the only AP to meet real-world relevant test objectives and was nothing short of impressive.”

 

Test Results

 

In the test with 30 video clients, the Ruckus R610 was the only AP that maintained stall-free video delivery to all 30 clients while under heavy network loading from data-only clients. In the test with 60 video clients, the Ruckus R610 was the only AP that maintained stall-free video delivery to all 60 video clients both with and without network loading.

 

Business Implications

 

IT departments are increasingly tasked with delivering a higher quality of service over a wireless local area network (WLAN) without over-spending on infrastructure. In some segments, such as education, the consistent delivery of stall-free video is imperative to the proper functioning of the organization. IT organizations that can provide their users with a better video experience save time by reducing the number of helpdesk tickets they receive. IT organizations that can use the equivalent of one AP for every two rooms rather than one AP for each room reduce the cost of access point infrastructure as well as associated licenses, cable runs and power consumption.

 

“Ruckus is committed to ensuring the highest quality of experience for each user, regardless of traffic type,” said Greg Beach, vice president of wireless products, Ruckus. “Video is becoming more prevalent in every WLAN environment and, in contrast to most types of data traffic, poor video delivery is immediately apparent to the end user. The Divergent Dynamics report is further testament to Ruckus’ unique ability to ensure a great end-user Wi-Fi experience—without over-procuring—even under the most challenging of circumstances.”

[Wave9_Andy]

Morning

 

Not doubting any of the conclusions and as we all know in real world scenarios there are lots of other contributory factors that can and will impact on network performance. But it does appear, contrary to your initial impression, that is was a Ruckus supported test?

 

"The author commends the Ruckus team for keeping each test vendor-neutral and fair, and, in fact, always gave the benefit-of-the-doubt, in favor of the competitor, as needed. All results herein were pulled directly from the raw data collected during the testing, without rounding or massaging in any way. The test methodology was fair to and the same for all, and the cards fell where they fell for each vendor."

 

Perhaps Gartner gives a broader view of the WLAN and Switching Market? Who knows :-)

 

https://files.jocke.no/b/gartner-wireless-august2016.pdf

 

Andy

[Blue_Cookeh]

It's also interesting they left out one of the biggest players in the Wireless market today... Ubiquiti.

 

Definitely taken with a hu-ugeeeeeeeeeee pinch of salt here.

[atcoates]

We use Meraki and it's a two second job to employ bandwidth limits per client, per application to share the bandwidth equally. Surely the test should equally focus on the software as much as the hardware? Otherwise it's only half a test. Car analogy here. Put foot down on a Ferrari Enzo with traction control off, wheel spin etc. Turn on launch control and it's gonna work a lot better!

[Net-Ctrl]

Apologies, I hadn't spotted that bit. I will amend the initial post to reflect Ruckus' involvement.

 

I thought I'd share this as a resource rather than to say, you must buy Ruckus because of this report

 

The results shown are only valid for the testing environment.

 

Ubiquiti doesn't tend to be included as it's still not classed as an enterprise product. That's my understanding anyway.

 

Toms Hardware do a WiFi report that is more in-depth and wider ranging with their tests.

[AlanD]

I think I am right in saying that rhere are only 2 WiFi chip sets and they come with their own micro code which has to be used to get the 802.11 compliance. So AP manufacturers have to work quite hard to distinguish themselves from competitors. I tried a range of access points and found little real difference in tests with 30 clients ....all in fact quite disappointing with quite a significant drop of overall bandwidth courtesy of a single collision domain. Some had a better dashboard, some worked a bit better through walls, some were a bit better with video streams, some were a bit better at load balancing, but largely they were all much of muchness...and not a good replacement for wired networks, especially in dense deployments...but great for a handful of clients or a larger number of tablets.

[TechMonkey]

Morning

 

Not doubting any of the conclusions and as we all know in real world scenarios there are lots of other contributory factors that can and will impact on network performance. But it does appear, contrary to your initial impression, that is was a Ruckus supported test?

 

"The author commends the Ruckus team for keeping each test vendor-neutral and fair, and, in fact, always gave the benefit-of-the-doubt, in favor of the competitor, as needed. All results herein were pulled directly from the raw data collected during the testing, without rounding or massaging in any way. The test methodology was fair to and the same for all, and the cards fell where they fell for each vendor."

 

Perhaps Gartner gives a broader view of the WLAN and Switching Market? Who knows :-)

 

https://files.jocke.no/b/gartner-wireless-august2016.pdf

 

Andy

Seems very open about it. 5th Paragraph on the first page states a Ruckus team carried out the tests witnessed and validated by the author.

 

Don't see why the author would want to align himself with a dodgy test as his company is all about being vendor neutral.

 

Reading through, the methodology seems sound, testing rigorous. If you are dubious maybe you could recreate? Or a few of the vendors could get together with a neutral overseer and recreate the test?

 

Gartner are generally considered more for investors than buyers. Plus they have their own set of biases such as towards larger vendors. They even hide behind a proviso that their reports are pure opinion so they cannot be pulled up in court. They argue legally the reports have no basis on facts.

Edited September 20, 2017 by TechMonkey

[SuperfluousAdjective]

This test was commissioned by Ruckus and is actually quite laughable in how biased the results were created. Unlike most wireless tests the configuration of each product was not released, despite asked for, and in some instances they purposely used outdated/deferred version of code (months prior to release).

 

As Devin points out in his report, QoS is critical to a test like this where we are basically testing wireless video QoS. That is all this test is, a wireless video QoS stresstest/benchmark. But, we know from the results that Ruckus did not enable wireless QoS on competing products, which is why you see the results you have. Best practice configurations of competing products was not used for a specific type of testing that Ruckus knew they would win if they handicapped their competition with deferred code and no QoS.

 

Additionally, Devin who isn't the most unbiased character in the industry, didn't actually run the tests. He simply observed. Ruckus designed and ran the tests themselves. Whenever doing competitive tests like this it is common practice to allow the respective vendors to set up their own gear or to release the configs to show there was no monkey business (as there was in this test). The truth is most of these APs use the same/similar Qualcomm chipset and the performance results for each would not be very far off with a properly configured system and unbiased testing.

Edited November 5, 2017 by SuperfluousAdjective

[Michael]

Ubiquiti not being classed as enterprise ready is frankly ridiculous...

 

And I agree, with the above - chipsets all used are the same/similar, so I wouldn't expected the results to hugely vary.

[AlanD]

Not just the chipsets are the same...all the microcode is the same...I found it very difficult to measure any significant or repeatable data rates between AP s that we used for testing...