I occasionally have VMware customers ask about jumbo frames, most often in reference to their iSCSI SANs. I attribute the curiosity mostly to VMware now officially supporting jumbo frames for iSCSI and NFS in vSphere. So with vSphere supporting it, and if your NICs, switches and SAN support it, should you use it?
Maybe. It depends on your goal, your equipment, and your workloads.
At the basic byte-counting level, you get 1460 bytes (1500 byte payload minus 20 bytes of IP and 20 bytes of [minimal] TCP) of iSCSI payload out of a 1538 byte iSCSI frame (1518 byte frame plus 8 bytes of preamble/SOF and 12 bytes of interframe gap), which is 94.93% efficient. With 9000 byte jumbo frames, your iSCSI payload is 8960 bytes out of a 9038 byte frame, or 99.14% efficient, a 4.21% gain. Not huge, but not insignificant.
However, the biggest potential gains of jumbo frames is not on the wire efficiency side, but on the host processing side. Each packet requires checksum calculations and protocol computation overheads. If your main CPU is doing these calculations, the conversion to jumbo frames can provide some CPU relief. TOE (TCP Offload Engine) functionality in your NICs (with O/S support) shifts some or all of those calculations to the NIC hardware, and TSO (TCP Segmentation Offloading) and LRO (Large Receive Offload) support can also bring CPU relief. An important note is that VMware does not currently support most TOE functionality, but does provide support for TSO, and support for LRO for a very limited set of NICs (which are conveniently the Intel VT and Broadcom 5708/5709 cards commonly found in Dell servers). See KB 1006143.
But the single most important question to ask yourself before going down the jumbo frames road is this: Am I intending to push my servers and storage so hard that I need the gains from jumbo frames? I support a fairly large number of setups, and I’m rarely designing anywhere near the edge of equipment performance. I like comfortable performance buffers, and so do my customers. I also like having excess performance capacity for growth. If you’re at the performance edge, perhaps the more appropriate question is whether your core design and core equipment are the right choices. Perhaps you need to step them up a notch. If you’re deeply concerned about the CPU load from software iSCSI, buy hardware iSCSI HBAs (but be sure to verify them against the VMware hardware compatibility list!). Also note that some new NICs are being listed as “NIC w/TOE iSCSI”, but these are not the same as a full iSCSI HBA, and you will not get the hardware offloading of iSCSI that you desire.
The best community posting I’ve found on iSCSI with vSphere:
http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html
TOE, TSO, and LRO support by VMware:
http://kb.vmware.com/kb/1006143
--Blaine Kahle (blaine AT blaine kahle DOOT com)