Dell EMC's Pure-crushing benchmarks are flawed, says, er, Pure

We've seen selected benchmarks that show #Dell #EMC #Unity arrays giving #Pure a hard time – and, unsurprisingly, Pure says the tests are "obviously flawed." The Unity arrays are reengineered #VNX arrays that were introduced in May. All-flash versions were used in the two benchmarks below. An obvious caveat is that these are selective benchmark results and we don't know if we have seen the whole set of results. What we have seen indicates that the Unity array can outperform Pure Storage Flash Arrays. The benchmarking software followed, we're told, IDC and SNIA testing recommendations for all-flash arrays. It involved varying percentages of reads and writes, different block sizes, and 100 per cent random workloads. Dell EMC believes this represents a typical real-world use case, whereas comparing millions of 4K 100 per cent read sequential IOPS does not.

Here are the results we have seen:

WorkloadPure //m50Unity 600FUnity Advantage
16K IOPS (80% Read)33,46058,8071.8X
256K BW MBps (@3ms)6742,3963.6X
Steady-state IOPS42,000116,0002.8X
Steady-state latency (ms)13.64.43X
The tested configurations were:

Unity 600F
2 active-active controllers
4 x 16Gbit/s FC ports
RAID5 (8 + 1)
16 LUNS, 32 or 16 threads
19 SSDs
FlashArray //m50
2 controller (active-passive backend)
4 x 16Gbit/s FC ports
RAID-3D
16 LUNs, 32 or 16 threads/LUN
40 SSDs (estimated)
We saw a Unity vs FlashArray //m70 benchmark result as well:

WorkloadPure //m70Unity 600FUnity Advantage
8K IOPS (80% Read)31,448163,7415X
256K BW MBps540 (@c5ms)3,450 (@c5ms)7X
Steady-state IOPS12,00086,0007X
Steady-state latency (ms)645.911X
Here are the tested configurations:

Unity 600F
2 active-active controllers
12 x 16Gbit/s FC ports
RAID-6 (10+2, 10+2, 10+2, 6+2)
64 LUNS, 8 threads/LUN
47 SSDs (44 + 3 spares)
FlashArray //m70
2 controllers (active-passive backend)
8 x 16Gbit/s FC ports
RAID-3D
16 threads/LUN
40 SSDs (estimated)
Pure's Matt Kixmoeller, VP of products, said: "Pure always encourages customers to perform real [proof-of-concept tests] with their application data, as synthetic testing has proven [over] time to be a poor indicator of the comparative performance of any AFA. In this case, the alleged test results are so obviously flawed that they serve to discredit themselves. For example, the tests seem to indicate that Pure's //m50 delivers higher bandwidth than the higher-end //m70 (which is an impossibility)."

Kixmoeller continued, "When doing any head-to-head testing, it is important to understand a myriad of configuration questions, such as whether or not compression and deduplication were enabled, whether encryption was enabled, how full the array was, whether data services like snapshots or replication were enabled, and whether the test data set has similar data reduction and locality characteristics to the actual planned workload.

"Pure's products are designed to deliver the highest levels of consistent performance, even with all data services engaged. We encourage customers to perform their own objective [proof-of-concept] comparisons, ideally with their own application data, and we can further enable customers to implement best practices for an all-flash environment, rather than bring along disk environment baggage."

http://www.theregister.co.uk/2016/11/23/dell_emcs_unity_benchmarks/