« NetApp #7 in Fortune's List of 100 Best Companies To Work For | Main | The Top 10 Storage Vendor Blogs »

February 03, 2010



Just to keep things honest, because you know that I love you Vaughn, ;)

I want to discuss perhaps what has been overlooked - I'll try not to cover what you've already mentioned (whether right or wrong, so please don't try to u-turn the conversation into something irrelevent to the point I am making, okay ? :))

You mention that the NetApp array is a FAS3160, (Your words: 1 NetApp FAS3160 Array with 16Gbs of cache) but in the Spec report (and how the 3100 series is deployed) it is a cluster (later mentioned in the SPEC report that it is an active/active cluster) [I won't go into how the BOM seems to lack mentioning various required licenses such as cluster and so forth, this doesn't even count! ;))

So subsequently and taken from the SPEC report this is effectively a two node cluster with 8gb of cache (per each controller) and the PAM module which is another 256GB slapped into the mix which is isolated to each node of the cluster, so if I'm doing my math correctly (Please do correct me if I am wrong)

8*2+256*2 = 528

And it has the NVRAM which I honestly don't count, but for the sake of argument in the SPEC report it shows another 2gb per node - for the sum total of 532GB of ram allocated to the system.

Please do correct me if I am wrong, that the most amount of memory addressable at a single time on a single node within this configuration is in fact 266 (8+256+2) since the data which is stored on the PAM card is not shared across controllers for accessibility from a connection perhaps going into nodeb in the cluster.

(If I am mistaken and the PAMII memory is indeed shared in the Active/Active cluster, where nodea interactively can access the PAMII card of nodeb and vice versa, let me know)

But whether that is true/valid or otherwise - please correct me if I am mistaken, but is this 'comparison' report basically saying that this NetApp configuration is using almost 2x the amount of memory that the compared EMC configuration is using?

If that being the case, 2x the 'cache' thrown at the problem, but producing 1/2 the speeds/feeds in IOPS (and don't even get me started on how I really don't care about speeds and feeds unless I have a SPECIFICALLY designed app which performs this way in real time and in production (rarely)

But putting that aside, as I am exclusiving discussing memory ramifications - and both sides of the coin seem like they deviate from reasonable practice for volume / aggr layout - I won't say how, because I frankly am not sure how the EMC side conforms to best practice, but the Netapp array in this configuration is in violation of 'best practices' (hell, I helped define some of those practices ;)) and I've seen enough disasters to know to NEVER run a prod environment in the exact way the evironment was laid out.

So, taking a step back, it's quite possible everything I've said is a bunch of noise (and I'm sure many people on both sides will agree and disagree with everything I've said)

My Lessons learned are:

Some systems can get really fast speeds (yay?!) and unrealistic expectations without consideration for application behavior can cause us to do crazy things in our environments.

People like to write blog posts about their SPEC reports, and people like to write counter blog posts about SPEC reports and sometimes (rarely: me) I typically hate commenting on SPEC reports when there's so much controversy around them, but I couldn't stand idly by with the level of intimate netapp knnowledge I have.. and be... misled perhaps is the word, on the merits of it's own configuration.

I guess my follow-up lesson learned on this is, I could rip out everything I've said about the EMC environment, and the points and facts I've made about the NetApp SPEC report and my comments are entirely valid (except for areas I specifically defined as needing confirmation/validation :))

We both know that I know VERY specific ways to make both of these reports fail miserably and one side can shine greater than the other when we take into consideration a 'mixed workdload' as is often the common consideration when it comes to a Unified platform - So we won't open up that can of worms ;)

Thanks for your time on this matter Vauhgn and others - and if you disagree with anything I've concretely said or referenced where I may be incorrect or mistaken I welcome the correction; though just like it may be better for me not to post this - It may equally be better not to question the integrity of the facts ;)

Oh yea, and don't write comment responses at 130am :)

/ second comment - the SPEC report you reference for the single FAS6080 - did not link to a valid report [So, I didn't check that in my response]

Disclaimer: My comments represent me and my own personal thoughts and opinions, comments, especially for where they're HORRIBLY WRONG AND MISTAKEN, and even where they're OMG RIGHT ;) Subsequently the content here does not reflect the opinions or thoughts of EMC, NetApp, or even SPEC as an organization!

- All is fair in love and SPEC reports ~me

Chuck Hollis

Hi Vaughn

I checked, and the SPEC test we submitted was simply the result of some lab work done by the engineering team.

They thought it might be interesting to post, and I agree.

There was no breathless press release or announcement. We didn't pick a competitor's gear and do a trumped-up head-to-head.

Nope, none of that carnival-style marketing-stunt stuff we see so often in the industry.

No claim that it represents an actual customer environment, or that it's better than anything else. No representation that someone might actually want to buy such a thing.

Nope, just an interesting engineering result that some might find useful. And many people did find it interesting, including you!

It's important to keep in mind that if you're familiar with how enterprise flash drive costs are starting to come down, it's not entirely unlikely that we'll start seeing "all flash" configs before too long.

And then it does become interesting where the bottleneck moves to in this world -- the controller, the network, the host adaptor, etc.

Which is one of the reasons the team ran the tests.

Best wishes ...

- Chuck

Vaughn Stewart

@Chris - Thanks fro the comments, I have a soft spot for NetApp Alumni. I'll keep it brief.

Does a FAS3160 have 2 storage processors? Yes

Is a PAM II array cache? Well, kind of. While it is a form of memory, it manages data as an extension of WAFL, so it does behave a bit differently than system cache. With that said, who cares, PAM makes systems extremely fast.

The nice thing about SPEC is the bench mark runs the same for all vendors, the only turntables are the array & server configs. As I highlighted, the NetApp config is a stock, production config.

And this is where the rubber meets the road. Who cares about PAM or EFD drives.

You have published its benchmark, so let's make the benchmark relative. What was the cost of EMC config? 10X that of the NetApp config? I bet I'm close to the mark with this guess.

This is where benchmarks are very valuable, they show the performance relative to the cost of the hardware. These data points help customers in selecting a storage vendor.

also, thank you for citing the broken link - its fixed.


Vaughn Stewart

@Chuck - Thank you for the reply. While we go at each other rather aggressively at times I am always appreciative of your comments (even if more often than not we disagree).

On the fall of enterprise flash costs, I agree. in time EFD will be available at commodity prices. When that day draws nearer, NetApp will offer them.

EFD is a game changer. Crazy IOPs, but its not a perfect medium. Have you looked at the write performance of EFD? maybe it would be better stated as the write penalty of EFD.

So while everyone whoops it up over EFD, why do you think slow 15k FC drives with PAM perform so well when the EFD drives have ~30X the IOP capacity?

So today we can meet EFD with a much more cost effective architecture - and customers really like saving money.

As for tomorrow, we all have to face the EFD write challenge. In your opinion, will RAID-1 become the default RAID type for EMC arrays with EFD? RAID-1 has the lowest RAID performance penalty for a traditional storage array, so it seems plausible.

I've just realized that I may be going at you a bit with my comments. My apologies.

I thank you for sharing your thoughts.


Chuck Hollis

Vaughn -- now you've got me totally confused.

My understanding is that PAM is volatile, and not safe for writes. Random reads with locality of reference -- great. Other read profiles -- it depends.

Of course, no one at NetApp would ever come out and say that directly, but it's pretty clear. If I am wrong, please set the record straight so we can minimize FUD based on wrong perceptions.

My other understanding is that the STEC EFD implementation is significantly faster than any configuration of rotating rust for most write profiles. We've seen this so many times in so many customer environments, so trust me on this.

So, when you're talking "write penalty" -- compared to what? Some theoretical number? Or compared to other alternatives in the marketplace today?

Thanks ...

-- Chuck

marc farley

V-man, you were right - that was WAY TOO MUCH NARRATIVE up front - get to the smackage faster. Sometimes the other guys use a lab queen to get their results. That's benchmarking! Netapp could do something similar, couldn't it?

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] Chuck Hollis submitted a comment to EMC Benchmarking Shenanigans

@Chuck You are correct, PAM does not directly address writes; however, the IOPs savings of PAM frees up IOPs on the drives that does make the freed IOPs available for write operations. As for sharing the secret sauce (in order to end the FUD), I dont think theres a benefit to having that level of discussion. First, Im not sure that we can have this level of discussion without a NDA. Second, what does the discussion solve? The proof is all around you.

If EMC arrays with EFD drives were actually as effective as NetApp arrays with PAM and FC drives, then why does NetApp own ~90% of the VMware View (aka VDI) market share? Your team knows that View is very challenging in regards to storage as the array must be able to significantly reduce the price per GB in order for the customer to deploy; however, the array with its inexpensive architecture must be able to address the numerous IO storm events (boot, login, A/V scan, etc...).

Can EFD address the IO challenges? Absolutely. 6,000 IOPs per drive is perfect! Can EFD address the cost savings? Not today, as they are way too expensive. Yet FC + PAM can. Extremely cost effective with outrageous performance.

Imagine when NetApp ships EFD + PAM! I bet you guys already have plans in place in an effort to slow down our momentum in advance of any such release.

Have you guys thought about rolling out a NetApp take out program? Maybe you could identify 2 or 3 key accounts in each sales district and aggressively go after them with over the top pricing discounts? In your opinion, do you think I have a future as an EMC sales strategist?

To close my thoughts, NetApp and EMC have completely different architectures. We have different strengths and weaknesses. Ill even offer today EMC storage arrays can do things NetApp cannot (connect to a VAX, IBM System/360, UNIVAC, etc...) Today Im pretty sure we own you guys when one looks at grading arrays based on storage utilization, performance, data protection and ease of use. Funny enough, these are exactly the foundation attributes required for the deployment of cost efficient virtual datacenters with technologies like VMware and Hyper-V.

Please dont misunderstand my words. I am not underestimating EMCs ability to innovate, deliver, or execute. You guys are very good at all three. I just believe the market has realized we might be a better fit for where most want to go.

Chuck, at some point in the future Id like to work with you. It may never happen at EMC or NetApp, but I would like it happen. Your a fierce competitor who is thought provoking and evangelizes on behalf of your company and indirectly the storage industry. Again, thank you for the comments.

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] marc farley submitted a comment to EMC Benchmarking Shenanigans


Youve got me on this one... Whats a lab queen?

Vaughn Stewart

Someone sent me an email estimating that the cost of the EMC test bed used in the SPEC tests was $6M.

While I am not in sales, I would estimate that the FAS3160 config is in the $300-400k range.

So here's my math: Two FAS3160s out perform the EMC mountain of hardware at 1/7th the cost, with more capacity, less footprint (36u), and and served on fewer file systems.

Can someone check my math?

Mike  Riley

Also, Chuck posed the same PAM question on my blog. I gave a little more detail on the impact of writes with PAM. I don't think I violated any NDA since all of this stuff has been talked about publicly or at least published in one of our whitepapers, KB articles or manuals.



marc farley

A lab queen is a system that you would only find in a lab - too expensive and tricked out for the real world.

Chuck Hollis


Thank you for your thoughtful comments.

"NetApp owns ~90% of the VMware View (aka VDI) market share" -- that's simply amazing!

That's funny, I subscribe to all the independents -- IDC, Gartner, et. al. and I've never seen that stat from anyone.

If it's actually true, that would be an amazing achievement on your part.

If it's a made-up number, that would be more reinforcement of my premise that -- yes -- NetApp is not beyond making stuff up when it suits their purpose.

So, anywhere we can find the data behind it? Someone showed me some slideshow thingie, but no sources were mentioned, or even the methodology.


Adrian Simays

Hey Vaughn - I always enjoy your posts. This is a fun one. :)

One thing has me confused...you mention early on that "had EMC unleashed some new technology which could revolutionize the storage industry?".

The answer is yes and you skipped over this. While the configuration EMC used is a larger & faster frame than what NetApp tested with (and how did we get to comparing the two anyway, since we didn't build an array to target 60k IOPS we don't know that the two would be that different!?), it is still industry changing technology.

The Vmax with EFDs is truly revolutionary and can deliver some amazing performance results, even in smaller configurations than this one. You imply this configuration is not for the real world but maybe you mean it is only a play for the high-end market - a space that NetApp still isn't taken seriously in.

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] Adrian Simays submitted a comment to EMC Benchmarking Shenanigans

@Chuck youll have to wait for more details. I know thats not what you want to hear, but it is what it is.

@Adrian long time no speak! Kudos to you on your new role @ EMC, setting direction for the EMC Microsoft Hyper-V partnership. Well deserved. Following up on our thoughts...

What was tested is the latest and greatest form EMC which has a speculative cost of around $6M. This config achieved roughly 110,000-118,000 IOPs (NFS CIFS respectively. Greta numbers. Cool tech. No challenges there.

My point is pure economics, why would a customer spend $6M on said config when alternatively they could spend approximately $600k-$800k for two mid tier NetApp arrays, where the config consumes significantly less rack space, provides more usable capacity, and achieves 60,000 IOPs from each unit providing a total output of 120,000 IOPs?

NetApp, also has a submission with posted a high end array config, albeit an older publishing achieving 120,000 IOPs with a FAS6080 array. This config provides 344% the storage capacity of the EMC config and costs much less than the suggested $6M of the EMC config.

So, lets get real here.

Customers look to purchase storage on a number of metrics. One is cost per GB, another is cost per usable GB, the third is cost per IOP. The EMC V-Max + Celerra + EFD combo loses to NetApp configurations on all three metrics. I applaud your ability to defend your architecture as being production worthy and feasible for the market.

Do you pay 10X the market value for food? What about consumer electronics? I doubt it. Then why would you expect your customer base to consider the demonstrated configuration as being realistic? Again, I doubt it.

Polly Pearson

Thanks @marcfarley for teaching me what a lab queen was. Sounded scary at first.

- Polly Pearson

Vaughn Stewart

Another great post on this topic is available from Nick.


Mike Workman

Boy do these SPEC arguments devolve into intense battles, huh?

Chuck says EMC did a "casual post" showing what can be done with SSD.

This strikes many as rediculous as the commercial value of the systems under test is dubuious. I think the $/IOP, $/Useable-GB pretty much say it all: You can build it but nobody should want it because it's value is highly questionable. Also, there are clearly better ways of skinning the cat.

It does seem like an odd place to put a "technology demo" Chuck - SPEC results database? No?

Regarding unleasihing a new technology that is revolutionizing the sotrage industry, I have to say there are ingredients of change here, but we have a way to go don't we? After all - just look at the total IOPS EMC is getting relative to the unbottlenecked capability of the EFD's in the set-up: Can you say bottleneck? Maybe a 6X bottleneck? God forbid the SSD's were any faster, or you added any more of them.

I think Marc adapted the term aviators use for a plane that spends all it's time in the Hanger because it is so unreliable - Hanger Queen....replace Hanger with Lab, unreliable with, well, worthless? I don't know.

I see your point Chuck, but it is odd to pop up after two and a half years with a SPEC posting like this one.




I don't know. Let's give EMC the benefit of the doubt on that one. After all, with all that inventory overhang http://bit.ly/5io7Fa they very well might just have piles of EFD lying about the lab.

What I thought was more interesting was how many backend disk IOPS EMC expended for each CIFS op.

They used 96 of these http://bit.ly/c2ZeJd in a V-max so, with somewhere around 6 million IOPs available, The three node Celerra (two active one standby) produced 120K CIFS ops. That works out to something like 50 Disk IOPS on the backend for every CIFS op out.

In the Apple config, for the CIFS, it works out to something like 1 disk IOP for every two CIFS ops. In the Apple NFS config, it works out to something like 1.5 disk IOPS/NFS op. In the NetApp NFS 6080 config, it works out to about 1 disk IOP for every two NFS ops. In the 3160 config, it's more like 1 disk IOP for every 6 NFS ops.

Seems to me that the EMC configuration was very lossy. It's not exactly what I'd call efficient. It takes EMC 50 Disk IOPS to do what it take Apple or NetApp less than 1 disk IOP to do. No wonder EMC needs EFD.



The backend was removed entirely from this test as it was a test of the front end DataMovers.

From the results you can see exactly where the redlines are for CIFS and NFS and that they're comparable regardless of the protocol.

No weird drop off as shown in some of the other results posted by those who tested both protocols and not just one protocol.

The test used exactly the required amount of storage with all shipping components. No hardware or weird cuts of software in the testbed six months before any customer can get their hands on them.

What I find interesting is that in the FAS 6080 test out of the 324 drives used in the test only 72 of them were required to support the test data set.

What were the other 252 drives for?
That's a box configured with 4.5 times the required storage.


A 15K RPM FC drive is good for about 200-250 IOPS. You have to consider BOTH IOPS and Space. @ 1 disk IOP per 2 NFS OPS, the NetApp 6080 config used 324 15K FC spindles.

The EMC Vmax config used 96 EFDs @ 60K IOPS per spindle per the manufacturer documentation. That works out to 50 disk IOPS per CIFS OP. That's an efficiency 100X worse than either the NetApp 6080 result or the Apple Xserve result.


Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] Storagezilla submitted a comment to EMC Benchmarking Shenanigans

@Mark thanks for the comments. Ill agree with you that the FAS6080 results required a lot of hardware. Im not sure why the decision was made to not include PAMII versus gobs of spinning rusted metal (Im paraphrasing Chuck here).

This same logic applies to the question regarding why only two Celerra datamovers? If the V-Max wasnt sweating, than more datamovers would have provided greater results, so why the absence of hardware? Was a $6M config OK, but $7M view as excessive?

I look forward to the next submissions from both EMC and NetApp, I am confident challenges to the existing results will be addressed.


I have results for that right here. Four Data movers was over 200K IOPS. It scales linerarly as you add data movers as....and say it with me...the backend isn't the bottleneck.

But two is always going to be the base DataMover configuration. And that's what's serving the NFS/CIFS IOPS, the DataMovers.

While I'm sure you'd like to make out this was is a systems benchmark if that were the case you'd never have provisioned four and a half times the storage for the required workload in the 6080 test.

One could call shinanigans right there if one so chose. 252 excess disk drives.

But it's not a systems test and all that tells anyone is that there was 4.5 times the rust in the box and assuming it wasn't the backend that choked up the NFS result tops out when you redlined the 6080 controller pair.

When the controllers/DataMovers/whatever they're called in FASland choke that's your SPECsfs number.

There's nothing beyond that in the benchmark, that's the number.

I look forward to reading your CIFS submission.



For that, look to the 3160 result which did use PAM II. It used 56 spindles of good old spinning rust to reach just over half of what EMC did with 96 EFDs. So, just use 2 of the 3160s. With 112 spindles of rust, you get the performance of 96 EFDs on EMC.

So, you're telling me it's ok to use 50 disk ops per CIFS op, when even Apple gets 0.5 disk ops per CIFS op. There's something wrong with that picture.


Val Bercovici

I suspect the only people still reading by now are just the motivated vendor debaters, but alas ... :)

NetApp have only scratched the surface with PAM II powered benchmark results, both block & file. As always we will stick to our emphasis on practical applicability (i.e. affordability) of these results, since we already know we can easily hit any artificial max performance milestone when money is no object.

Throughout 2010, expect NetApp to rock the industry with (revolutionary PAM II accelerated) SATA-based high performance benchmark results. That’s what *real* customers need – and that’s what NetApp is delivering.

The “usable benchmark” bar has been set by NetApp. This year we will keep raising it. Let’s see who else can keep up – or even stay in the same ballpark! :)

Vaughn Stewart

@Val @John

Well said, customers want data that helps them make a sound decision... Period.

This message was sent by my thumbs... fault them for the brevity and poor spelling

The comments to this entry are closed.