« VI3 on NetApp Best Practices Update | Main | Got an iPhone? Got VMware? Heard of VManage? »

June 18, 2009

Comments

Richarb Barlow

At first when I read the EMC pdf I thought that perhaps it was old info. But now I see that its from April 2009 - this is relatively hot off of the press.
I hope EMC answers the question that you're posing becuase I'm sure their customers will be asking as well, especially in mixed environment shops where there may not be a single storage standard. Do I need to keep different templates for my Cellera NFS versus my NetApp NFS/FC/iSCSI or HDS FC storage? Inquiring minds want to know . . .

BK

MBRAlign isn't officially supported by NetApp. This is disappointing. We would love to align our 500 guests but we aren't going to walk blindly off a cliff with an unsupported tool to do it.

What are the chances this will become supported and make it into best practices?

Nick Triantos

MBRalign and MBRscan are now part of NetApp's ESX Host Utilities Kit v5.1 and such are officially supported.

The newly released Kit also introduces support for vSphere and continues to support previous ESX versions.

Vaughn Stewart

BK - Nick is spot on here. The versions in the EHU 5.1 and the upcoming VSC are officially supported.

If you need additional assistance NetApp global support center can also assist you and your team.

Thanks for the feedback.

Chad Sakac

Vaughn - thanks for investigating. Digging into this with the Celerra team.

I agree - there can be only one standard guest-level VM alignment, as customers need to be able to know that moving from one storage platform to another won't require re-aligning.

Errors (as you well know from the old NFS disable lock episode) can be made - we are all human after all - but it's important to catch and correct them if indeed it is an error.

Will get you and everyone else and update.

Vaughn Stewart

Chad - thanks for jumping in to address this issue from the EMC perspective. Customers should be able to have all features on all arrays over any protocol and on-off configs prevent this from happening. Sometimes change requires a little effort (ala like correcting the old NFS locking issue).

Michael

IMHO, if you are going to make the statement like; "I don’t have all of the details that I’d like to have in order to explain why EMC would promote alignment for VMFS deployments on iSCSI, FC, and FCoE but not promote alignment for deployments of NFS on Celerra", then why not wait for an answer *before* you post??

I have been researching this question (do I, or don't I, align VMs on Celerra NFS) and my studies are leaning towards no. You can, but it does not buy you anything. The whole concept is block alignment. There are no "blocks" in the NFS storage exported to vSphere. It is a share. Furthermore, the MS article that explains the reasoning behind alignment in the OS says that the underlying disks need to be multi-disk arrays before it is necessary.

I, for one, am waiting (patiently?) for someone from EMC to explain (down in the nitty-gritty weeds of the bits and bytes) of why not (or why).

Vaughn Stewart

@Michael - I don't understand why you're not swayed by the recommendations of Microsoft, Citrix, VMware & NetApp on this topic. Are you aware that EMC has revisited their data and as a result have changed their recommendations regarding GOS file system alignment?

The previous EMC recommendation on alignment over NFS was simply a mistake. it happens, and to their credit they fixed it. Check out the updated EMC Techbook for VMware on Celerra (H5536):

http://www.emc.com/collateral/hardware/technical-documentation/h5536-vmware-esx-srvr-using-celerra-stor-sys-wp.pdf

Michael

Maybe I'm thick headed, but this makes no sense to me. The MS article (http://support.microsoft.com/kb/929491) states “This issue may occur if the starting location of the partition is not aligned with a stripe unit boundary in the disk partition that is created on the RAID.” This means that it only matters on block level devices where the logical disk is striped over multiple physical disks. NFS is a share. There are no physical blocks to align with.

I am one of those individuals that needs the underlying reasons for things explained.

The Vmware article on the subject does an excellent job of explaining why this matters at all (http://www.vmware.com/pdf/esx3_partition_align.pdf) and on page 5 (with pretty pictures) you can see the blocks and how they can be missaligned. But NFS takes away the bottom two layers and thus, in my logic, the need for alignment.

Can you point me to a NetApp paper that explains *WHY*? I would very much like to read it.

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] Michael submitted a comment to I/O Efficiency and Alignment the Cloud Demands Standards


Michael,

The difference between NFS VMFS is where the file system exists. With VMFS the file system is local to all nodes connected to the shared LUN. With NFS the file system resides on the NFS server. For a Data ONTAP powered array (FAS, vSeries, IBM N-Series) the file system of NFS datastores is WAFL (4kb blocks) and for EMC Clariion the file system is UFS (with block size selected by storage admin).

In the case of VMFS, WAFL, UFS, etc... these file systems must align to the underlying physical storage boundaries.

In the case of Virtual Machines, their local file systems (NTFS, EXT3, etc...) must align to the file system of the datastore (i.e. WAFL, UFS, VMFS).

Lack of alignment results in very poor disk I/O utilization. Put in other word, you can work a storage array twice as hard to server the workload when misaligned. Such an oversight leads to underutilized hardware assets and more hw required to run the operation than what is needed.

Michael

You telling me that NFS is different form VMFS is somewhat making my point. I fully understand alignment and the need for it in block based storage.

However, I found a NetApp paper on the Citrix site(http://www.citrix.com/site/resources/dynamic/partnerDocs/BestPracticesforFileSystemAlignmentinVirtualEnvironments.pdf) entitled "Best Practices for File System Alignment in Virtual Environments" and it states it this way...

1) (This is from section 3, file system alignment) "For optimal performance, the starting offset of a file system should align with the start of a block in the next lower layer of storage." What the authors are stating is that each layer needs to be aligned with the adjacent layer. But in NFS, because it is a share, there is no lower storage block to align to.\

2) (This is from table1 under 2.1 VMWARE) When using NFS on VMware, there is only one layer, the Guest OS. NFS is the ONLY one to have a single layer. There is nothing to align it to.

In essence, the guest OS cannot be aligned with a share. It is not block storage. There is nothing to align it to. Aligning a GOS on NFS would not hurt, but it does not help.

If you think (and obviously you do) I am mistaken, then please point me to a paper that tells me why? Everything I have read substantiates that NFS does not require GOS alignment.

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] Michael submitted a comment to I/O Efficiency and Alignment the Cloud Demands Standards


@Michael I understand why one would think NFS is a single layer; however, it is not. VMFS has a block size which the ESX/ESXi hosts are aware of. NFS has a block size that the NFS server is aware of, so even though this value is unknown by the ESX/ESXi hosts, it still exists.

LUN hosting file system (VMFS or WAFL) GOS file system

All three layers must be aligned. Failure to do so unnecessarily stresses the storage array requiring more hardware to meet the workload than what is required with aligned VMs.

The hardware abstraction provided by server virtualization allows for the non-disruptive migration of VMs between VMFS and NFS file systems. If one align the GOS file systems they ensure the best I/O performance for all VMs, on all storage arrays, one both VMFS NFS.

Michael

Yes, I absolutely see the benefit from a standards perspective. Being able to migrate a VM between storage systems (easily done now with vSphere and storage vMotion) without altering the performance has huge benefits.

As for the performance, I may have had an epiphany. If alignment is necessary in a physical, single disk system, then it would still be (obviously) necessary from a VM on NFS. I am still looking into that as I was under the impression (from the MS article I mentioned earlier) that alignment was only necessary with some sort of RAID (aka multi-disk) configuration.

As for aligning to NFS, this still seems like it would be dependent on how the NFS alignment was done. If, for example, the OS running the NFS (the SAN) auto-aligned each file with the beginning of a physical underlying block, then any GOS/VM would be automatically aligned. Right?

Thank you for taking the time with me on this.

-M

P.s. The idea that NFS is a single layer was from the NetApp article I mentioned. And it is not I was suggesting that NFS was a single layer, but that NFS removed the blocks from the equation. This is pretty much what the article said. But then it went on to say that one should still align all GOSes.

Vaughn Stewart

@Michael - now your starting to get it!

When using VMFS, you need to select the correct lUN type. With NetApp this is a VMware LUN type. When selecting this type the storage controller ensures that VMFS is aligned.

With NFS with NetApp the fiel system, WAFL, is already aligned.

When we create a VMDK it is stored correctly, in aligned blocks.

The challenge is ensuring that the file system created inside of the VMDK, by the GOS, is aligned. Historically OS installations did not align partitons to array block boundaries. This is why Storage arrays have LUN types. By having LUN types we could force the alignment; however, with VMDKs, storage arrays don't have any means to make such adjustments.

So we must manually align the starting partition offset for all legacy OSes, like Windows NT-2003. Note 2008 is aligned by default. Alignment is also an issue with LINUX distributions, but don't know which distributions and version have addressed this issue.

Michael

So as it turns out, I was right to begin with. There is no performance gain from aligning a GOS running on VMware over NFS. I have had several conversations with different individuals both on-line and off and it comes down to this; NFS separates the blocks from the VM and exposes the vDisk to the GOS as a single drive. There is no advantage to aligning a single drive. So the only reason (and it *is* a good one) to align the GOS drives is compatibility with other, block based, storage arrays.

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] Michael submitted a comment to I/O Efficiency and Alignment the Cloud Demands Standards


Michael,

Im sorry but you are incorrect. It is critical to align the partitions on all virtual disks for any GOS that does not provide proper alignment.

J.ross

hi Vaughn,
have you heard about Paragon’s PAT (Paragon Alignment Tool) which has just been released: https://www.paragon-software.com/business/partition-alignment/?

Vaughn Stewart

Re: [NetApp - The Virtual Storage Guy] J.ross submitted a comment to I/O Efficiency and Alignment the Cloud Demands Standards


@J.Ross Thank, I am aware of Paragons Alignment Tool (PAT); however, Ive yet to use it either as a stand alone tool or as a part of their Hard Disk Manager suite. Id like to know more about its capabilities, especially in regards to job management and reporting. If anyone has more to share, Im all ears...

The comments to this entry are closed.

TRUSTe CLICK TO VERIFY