vSphere 4.1 and vStorage APIs for Array Integration (VAAI)

What began in late 2005 with Vanderpool continued today with the release of vSphere 4.1…

Vanderpool was the code name of Intel’s Virtualization Technology of x86 (or VT-x), which introduced extended page tables and enabled multiple operating systems to run simultaneously on an x86 computer in a safe and efficient manner. VT-x was the first means of hardware acceleration for VMware and the other x86 hypervisor vendors.

With today’s release of vSphere 4.1 VMware has delivered the first set of vStorage APIs for Array Integration (VAAI). VMware and key storage partners, led by Dell, EMC, and NetApp, have continued the direction originally established by Intel to extend the scaling of virtual infrastructures by enabling hardware acceleration with storage arrays serving ESX/ESXi clusters.

Today’s post is on VAAI, which is merely one of the three primary vStorage constructs available with vSphere 4.1. The other two are:

• Storage I/O Control (SIOC) - Set storage quality-of-service priorities for each virtual machine to guarantee access to storage resources
• Performance Reporting - Deliver key storage performance statistics regardless of storage protocol

Either I or one from the vSolutions Engineering team will post more on SIOC and Performance Reporting later this week.

vStorage APIs for Array Integration - 101

VAAI provides hardware acceleration that allows storage operations, traditionally provided by the hypervisor, to be completed by the storage array. These capacities allow ESX/ESXi to operate more efficiently and scale to greater capacities with reduced resource consumption (host CPU, storage I/O, etc).

The capabilities of VAAI in this initial release include:

Full Copy - enables the storage arrays to make full copies of data within the array without having to have the ESX/ESXi hosts to read and write the data. This API will reduce the time and network load on cloning and migrating VMs with vMotion.

Did you know that without VAAI the process of cloning a 10GB VM requires 10GBs to be read and 10GBs to be written over the storage network?

Block Zeroing - enables storage arrays to zeroing out a large number of blocks to speed up provisioning of VMs. This API will reduce the time and network load when formatting an Eager-Zeroed thick VMDK.

Did you know that without VAAI the process of formatting a 10GB VM requires 10GBs of zeros to be written (or sent) over the storage network?

Hardware Assisted Locking - provides an alternative means to protect the metadata for VMFS cluster file systems and thereby improving the scalability of large ESX server farms sharing a datastore. This API will replace the traditional SCSI-2 locks, thus enabling larger datastores.

Did you know that without VAAI the maximum ESX/ESXi cluster size supported with View deployments where linked clones are used is 8 nodes?

Quite simply, your virtual infrastructure will scale further with VAAI than without. VAAI is a level of measurement of the intelligence within your storage arrays. Much like Data deduplication or TSCS from NetApp, once enabled these technologies allow hardware to scale to greater limits. Doing more with less, now that’s a win!

Click here to view the VAAI on NetApp demo from VMworld 2009


Does VAAI Support VMFS & NFS?

vSphere 4.1 contains the first release in what is planned to be multiple of VAAI. In this initial release, these APIs will only support VMFS datastores.

Now let’s be realistic… SAN connectivity is rock solid and includes very robust multipathing capabilities; however, since June of 2006 VMware on NetApp NFS has had a significant advantage over traditional SANs in the areas of storage array integration with ESX/ESXi.

Zero cost VM clones, I/O offloaded cloning, direct access to storage based snapshot backups, storage transparency with the use of data deduplication, and scaling uber large / dense datastores are just a few of such benefits which come to mind when using NetApp NFS with ESX/ESXi.

It’s no secret some in the storage industry cannot accept the scaling of NFS, which is truly unfortunate especially when they label it as FUD. Do you recall my post on systems from NetApp & Cisco powered the labs at VMware’s TechSummit 2010? What I didn’t share in that post was the entire lab was ran on NFS specifically to meet the scaling requirements of such a demanding lab, with a high level of concurrency running on small storage footprint.

Allow me to be frank, it’s about time SANs received a little love from our engineers and closed the gap with NFS.

If you are a NFS customer don’t despair; VMware and NetApp (and even EMC) have a number of NFS enhancements, which we are engineering for future releases of VAAI. Trust me, when these are delivered, you’re going to be blown away.

 

A Few Housekeeping Notes Before I Close

VAAI is available with vSphere 4.1 Enterprise and Enterprise Plus editions, and supports VMFS datastores served by block-based storage protocols FC, iSCSI, FCoE on arrays running Data ONTAP 8.0.1 in 7-Mode. 8.0.1 is targeted for release in Q4 2010.

Comments

[Jason]

Great article Vaughn!

[Kent]

Just so I am clear if I am still on the latest version of OnTap 7.x and FC block access, I will not receive any VAAI benefits until I upgrade to ONTAP 8.0.1 in 7-Mode?

One further question... I noticed today v6000 Series was not listed as vSphere 4 or 4.1 acceptable SVD solution on the VMware HCL list. What is the delay in supporting the latest ESX release with this Vseries model? Thanks!

[Duncan]

Vaughn, that 8 host limit in most deployments had nothing to do with locking/scsi reservations. For most it was a limit because of HA maximums.

Especially with vSphere 4 and optimistic locking the problems experience with ESX 2.5 and early 3 were not an issue anymore. Besides that most users/admins/consultants matured as well and figured out what a reasonable amount of VMs per VMFS volume would be VS just loading as much as we can.

Good article though,

[marc farley]

Vaughn, I missed the part where you mentioned when Netapp was going to have VAAI integration. Sometime in the future and we'll all be happy about it, I'm sure - but when?

[CDM]

Vaughn, how does this work with SANscreen VM Insight? I see that SANscreen 6.0 (NetApp version) is now available. What's the scoop?

[Vaughn Stewart]

@Jason - Thank you for the compliment

@Kent - That is correct. Data ONTAP 8.0.1 in 7-mode provides the support for VAAI and is targeted for release in Q4 of this year.

@Duncan - Thank you for the clarification, scaling datastores has been an area where both companies have invested a large amount of engineering resources over the years. I'm glad to see the limits being l eliminated. Thank you for the expert level feedback

@Marc - Great question. The info is at the bottom of the post and is basically...
4.1
VMFS on FC, FCoE, or iSCSI
Data ONTAP 8.0.1 in 7-mode
Thanks for asking for clarification

@CDM - I should have more on SANScreen with VMInisght 6.0 posted next week. Thanks for chiming in.

[Andrew VanSpronsen]

Hey Vaughn

Great read. Disappointing NFS is still a second class citizen at VMware. Thank goodness we have RCU in the interim although there are limitations like concurrent users.

At least we have disk performance statistics now for NFS backed datastores.

[Vaughn Stewart]

@Andrew - Second class citizen?!?! No Way!

As for the first release of VAAI... These features ALREADY EXIST in NFS. SAN is attempting to be more NAS-like. The VAAI roadmap is long, and it's chocked full of NFS goodness. Be patient.

The disk performance stats were probably the most demanded feature, as our largest customers tend to select NFS for it's elegance in managing massive volumes of VMs with exceptional simplicity.

There;s more goodness on the horizon...

Thanks for sharing your thoughts.

[Jason]

Looking forward to seeing this in action. We're about to update to the latest RC of 8.0.1 from 7.3.2 as well as upgrade from ESX4 to ESXi 4.1 in one of my environments. This will be with a v3140 virtualizing some CX gear as well as its own internal storage. It will be a mix of FCoE, NFS, and traditional FC over 10Gb network. Should rock and roll.

[Justin]

It's really disappointing to see this only showing up in 8.01, and not the 7.3.x line. Particularly with 8.0 not being up to feature parity with 7.3.3, and 8.01 slipping more than a greased eel.

[Leif]

I agree with Justin. If the capabilities are there in 7.3.x, please add support for this in a future 7.3.x release. Since the FAS2020/2050 filers can't run 8.x code, customers will be left in the dark. I think 2020/2050 setups could greatly benefit from VAAI since there are fewer links and fewer spindles to handle the increased IO from not having VAAI.

I'd like to add an asterisk to the NetApp implementation of VAAI in NFS. The VAAI copy offload capabilities provided in the RCU for cloning only work with a FlexClone license. VAAI won't need a separate feature license for iSCSI/FC. For those without FlexClone, I suppose it would be possible to script out a copy offload to work in NFS using ndmpcopy and then register the cloned machine into vCenter afterwards. But is it currently possible to utilize copy offload in NFS for storage vmotions?

Looking forward to future VAAI features like snapshot offload!

[Lynn Schwab]

Still unclear if these features are supported with V-Series. Vaughn, can you please confirm if VAAI fully copy, block zero and hw assisted locking will work with V-Series if the backend block array supports it? My guess is no, but would like to confirm.

Thanks.

[Existing Customer]

where is t3h NFS!!!

[Peter Learmonth]

Lynn
VAAI will work between ESX and V-Series regardless of back-end support. The V-Series will not use any VAAI capabilities of the back-end, even if present, although that is an interesting idea.
Peter

[Claes]

Hi, Running a vmware cluster towards an OTAP 7.3.3 with very bad disc I/O performance. Anyone know how to tune it to retrieve better flow ?
Any suggestions is welcome

[Vaughn Stewart]

@Claes - If you are having a performance issue I would strongly suggest that you open a case with NetApp Global Services at 888-4-NetApp.

[Dennis Bray]

In the time since this was posted, has the support or capabilities of V Series to pass through VAAI calls to back end non-NetApp storage arrays changed?

[Vaughn Stewart]

@Dennis - When a vSeries virtualized a traditional array, all of the storage virtualization capabilities are delivered by the vSeries. Thus VAAI can be enabled with any array which the vSeries supports whether the back end array support VAAI or not.