I recently purchased one of the new Western Digital 3TB USB drives for use on one of my DPM servers, which is a Hyper-V guest.  Unfortunately, when I started to try to use the disk, Hyper-V refused to create new VHD files on the disk, or to boot VHD files that were manually copied to the disk.  All other functions seemed fine.

After trying a multitude of formatting and partitioning options and speaking with a couple of higher-ups at Western Digital, I realized there was a fundamental incompatibility between Hyper-V and 4k physical sectors.  I was planning to return the drive, but got caught up in some other projects and got back to this project yesterday to find a new KB article that seems to have been pulled – but thanks to google cache it’s still readable: http://support.microsoft.com/kb/2515143

It references not only the root problem but a hotfix (that is also included in the recently released Service Pack 1 for Win2008R2): http://support.microsoft.com/kb/982018

I decided to download and try the hotfix on a non-production Hyper-V server.  Here is what you get when using an advanced format disk on Windows 2008 R2 RTM Hyper-V:

And following the patch and reboot I now get…

…the exact same error.  Sad face.  Creating the VHD elsewhere and copying it over still doesn’t work either as Hyper-V still squawks about compatibility; looks like MS has more work to do:

UPDATE 4/18/2011: As you can see from my above edit, there is a new KB article specifically addressing this issue WRT hyper-v – and the bottom line is you cannot put VHDs on native 4k sector disks (and disks with 512b emulation will take a performance hit)…  bummer!

The workloads being virtualized consist of four Exchange 2010 servers (2-node DAG and 2 CAS/HT servers running NLB), a couple of file servers, and a hodgepodge of domain controllers, SQL, and web servers with a few Citrix Xenapp servers thrown in (most of the Citrix servers are physical boxes).  The network consists of three distinct internal networks brokered by Juniper firewalls, named DMZ, Trust, and Vault.

When it comes to clustering on 2008/2008R2, the storage solutions that get the most publicity are Fiber Channel and iSCSI.  Both of these solutions offer the ability to connect multiple nodes and, when configured properly, can perform very well.  Both require quite a bit of hardware and tweaking to do so however – including a redundant set of FC or gigE switches (to avoid a single point of failure) and in the case of iSCSI, further network tweaks for performance.  There is a third shared storage option that just doesn’t get talked about very much:   Shared SAS.

Shared SAS

Serial attached SCSI is the successor to the older parallel SCSI interface we all knew and loved (and the command set for which is still used in SAS, FC, even SATA).  SAS performs well (even the older 3gbit spec) and is relatively inexpensive, and while parallel SCSI is no longer legal for shared storage clusters under Server 2008 and beyond, shared SAS is fully supported!

SAN budgets can vary wildly depending on the target application, storage types, fancy features such as snapshots, thin provisioning, etc. but even the cheapest redundant solutions quickly get you into the thousands of dollars range, if not tens of thousands.  You can set up software-based solutions such as Starwind or Sanmelody on commodity hardware you have laying around (and I have done this with great success for lab work) however to build those into redundant SANs requires expensive licenses that can rival the cost of turnkey solutions.

Ebay to the rescue

Ultimately we decided on purchasing a used Dell MD3000 shared-SAS SAN that still had a year of 4-hour response warranty left on it.  The 3U MD3000 is still sold by Dell despite being upstaged by its 6gbps brother, the MD32xx.  While the newer model can only hold twelve 3.5″ drives, the MD3000 can handle fifteen.  The real sweetener on this find was that the SAN came with eight 450GB 15k RPM SAS drives.  To buy this MD3000 redundant configuration new with the eight disks and warranty would have cost us over $20,000.  All told (including the four SAS5/E adapters and cables) we paid under $6,000.  When next July rolls around we will have to re-up on the warranty coverage, but that is still a hefty savings (now I know some of you are used to working with SAN budgets that exceed our entire annual operating budget, but this was big savings for us!).

Networking

After storage is nailed down, the next big pain point of clustering (especially with Hyper-V) tends to be figuring out the best networking configuration.  Spending a lot of money and effort on a cluster to improve uptime does little good if something as simple as a failed ethernet cable or switch port can bring down a VM.  Strangely enough, this is exactly the case – a nic failure or switch failure that causes connectivity failure in a VM is not something a Hyper-V cluster can detect and work around.  So your options either become scripting elaborate routines to check network connectivity and migrate the affected virtual machines, or to turn to NIC teaming.

The Sorrowful Saga of NIC Teaming and Hyper-V

To the surprise of many, Windows has never had any way to aggregate NICs.  Microsoft has always left teaming solutions (and support of them) to the hardware vendors.  Both Broadcom and Intel have stepped in, with varying levels of success.  Typically teaming is very easy to set up, but virtualization can introduce some quirks that need to be understood and worked around.  At this time, both Broadcom and Intel offer support for teaming solutions under Hyper-V.  As always, however, your mileage may vary.  I have read way too many horror stories about both vendor solutions, but as of this writing Intel seems to have the edge and has worked out the kinks (revision 15 ProSet drivers and later).  Intel also offers a special teaming mode called Virtual Machine Load Balancing which is very handy, providing both load balancing and failover options at the VM level.

Our T710s came with the integrated quad port Broadcom nic, and a PCIe quad port Intel nic.  The cluster teaming plan called for more available ports for full redundancy, so each server was outfitted with an additional quad port Intel nic ($200 each on Ebay), as well as another Intel single-port PCIe nic, for a lucky total of 13 ports per node.

What could you possibly need all those NIC ports for?

The 2008 R2 revision of Hyper-V clustering introduces a number of important network-based features and services that increase the need for dedicated gigE interfaces.  The one most people have heard about is Live Migration – Microsoft’s answer to VMWare’s VMotion.  Live Migration allows you to seamlessly transition a running virtual machine from one cluster node to another.  This is obviously incredibly handy for managing server load, as well as shifting workloads off of physical hosts so that they can be taken offline for hardware or software maintenance.  Production-ready Live Migration should have a dedicated gigE nic (10gigE works great too if you have deep pockets).

Next is Cluster Shared Volumes – a handy feature in 2008 R2 that allows you to store multiple virtual machines on a single LUN (trying to provision per-VM LUNs is a headache I don’t even want to think about).  In our case, the use of CSVs allowed us to take our two physical disk arrays on the SAN (8-drive RAID10 and 5-drive RAID5) and format each as a single LUN.  I won’t go into too much detail here on CSV as there are million blogs out there with great information.  CSV should also have a dedicated NIC available to it, as it can be used to relay storage commands in case a node loses its connection to the shared storage (there must be a performance penalty for this, but I suppose it is good enough to limp along until you can live migrate the workloads to another fully functional node – which is why LM and CSV should have separate nics to work with).

The nodes themselves need a NIC for basic network connectivity and administration, so now we’re up to 3 NIC ports per node and we haven’t even talked about the virtual machines yet!  As previously mentioned, the cluster is pointless if a failed NIC or switch port can bring down a VM, so we decided to implement Intel nic teaming using the VMLB mode.  We carved up our two intel quad port nics into three teams (one for each of our aforementioned networks): TeamV for Vault (4 ports), TeamT for Trust (2ports), and TeamD for DMZ (2 ports).  This setup provides at least two ports per network to guard against both nic or switch failure, and avoids the need for configuring VLANs (which is a whole other mess for another blog) – we do have VLANs configured on our cisco switches, but only for trunking purposes - I like to avoid complicating the Hyper-V configuration as it is far easier to train administration staff without having to have them learn the ins and out of networking at that level.  If you prefer you could certainly cable all ports into a giant team and use VLANs to separate out the traffic.  At any rate, we’re now up to 11 NIC ports used (LM, CSV, host administration, and 8 teamed VM ports).

Not all VMs need be highly available

While most of our VMs are fairly critical and will made highly available across the cluster, certain machines are not – either because they just aren’t as important, or because the services they provide offer high availability in other ways.  For example, each of our nodes has a domain controller Hyper-V guest configured to run from local storage (not part of the cluster).  As long as domain clients are configured with both DCs in their DNS settings, they can deal with one or the other going down for a while.  Similarly, each of our nodes has an Exchange 2010 mailbox server (running in a DAG) and an Exchange 2010 CAS/HT server.  This way, the failure of an entire physical node does not derail DC or Exchange services.  It also allows us to play around with fancy storage options for the Exchange servers, but I will save that for another post…

I mention Exchange because we use Windows Network Load Balancing to provide high availability across our two CAS servers – and NLB creates port flooding on switches (by design).  By dedicating one of our NIC ports to each node’s CAS server, we are able to limit the port flooding on the cisco switches to only the ports those dedicated NICs plug into.  There are likely more sophisticated solutions to this problem, but this is what we came up with (had to use those spare broadcom ports for SOMETHING!).

Our count is now up to 12 NICs, but I happened to have a couple of spare Intel PCIe NICs laying around, so I figured it wouldn’t hurt to get an extra in there.  This would allow me to set up a broadcom team to provide switch redundancy on the nodes’ admin IPs, which could always be reconfigured and set up to take over in the event of one of the primary cluster links going down – so 13 it is!  Below is a picture of the color coded NIC cabling (the two short orange cables are crossovers for the LM and CSV links) – I have not attacked it with zip ties yet!  You can also see each node’s two SAS adapters/cables in the photo.

Putting it all together

Now that the hardware part of things was all set, it was time to adjust the software side.  The procedure basically broke down like this:

1. Join each node to the domain (they were previously not domain members to avoid dependence on the virtual domain controllers), configuring DNS to point first to the other node’s virtual DC and second to an offsite domain controller with a reliable WAN link.
2. Install latest NIC drivers and configure the teams.  Configure private networks as desired (for example I have my crossover networks set to 10.10.10.x and 10.10.11.x).
3. Remove all the VHDs from VMs to be made highly available, then export their configurations and copy the VHDs to cluster storage (this step can be done many ways but I prefer manually copying the VHDs instead of having them tied to the Hyper-V export process; saves time if you end up having to re-do the import for some reason).
4. Delete existing virtual networks and create new ones using the nic team interfaces, ensuring that all virtual networks to be used for highly available VMs are named identically across the nodes.
5. Install the Failover Clustering feature and use its administration tool to run the Cluster Validation wizard – hopefully this will pass!
6. Go ahead and create the cluster, and the wizard should be smart enough to set up the proper cluster networks (three of them in my case, the two privately addressed interfaces and the admin interface) – the virtual networks have no protocols bound to them other than the Hyper-V protocol so they are ignored by the cluster wizard.
7. Add any storage disks that were not automatically added by the wizard (mine added all three of mine).
8. Using the Failover Cluster management tool, enable CSVs and turn the appropriate storage disks into CSVs.
9. Using the Hyper-V management tool, import the VM configurations you exported onto the CSVs as desired, and copy/move the VHDs into place and re-add them into the configurations, as well as specifying the appropriate virtual network(s).
10. In Failover Clustering management, configure a service and select Virtual Machine, and select all the VMs you want to make highly available – they should pass validation!
11. Fire up the VMs, and then attempt to Live Migrate one or two over to the other node.  This worked like a charm for me…  The only quirk I ran into was a Live Migration failure after editing a VM config from the Hyper-V management tool – it would appear that any settings changes need to be made from within Failover Clustering and not the Hyper-V tool to avoid this problem!

The entire procedure took roughly 13 hours, however a decent chunk of that time was spent simply waiting for large VHDs to copy, and/or dealing with unexpected surprises such as the fact that the MD3000 had the wrong rail kit with it (despite having the proper part # on the box) – nothing a pair of pliers wasn’t able to fix…  The only single point of failure left is the ethernet drop the colo provides – and in the near future we may add a second drop to remove that last potential failure point:

Thanks for reading!

A couple of things to note about this guide:

• I do not use Vista — I have never really used Vista and now with Windows 7 out there really is no reason to; however the steps here should work just fine with Vista installation sources
• I have only tested this using a USB stick and will only cover that method here – it’s much more flexible (and faster) for installation than using DVD — but DVDs should work just fine: you can find plenty of tutorials on the web that will tell you how to use oscdimg.exe to take the files we create here and turn them into a burnable ISO

WHAT YOU WILL NEED:

• USB stick, at least 4 GB (perhaps larger depending on how many OSes you plan to integrate, I use a 16 GB) that has been properly formatted using Steps 1 and 2 here
• ISOs or DVDs of the operating systems that you plan to integrate
• imagex.exe (can be found for download on the web, or you can download the full 1.7 GB WAIK here)
• computer running Vista or newer operating system (XP should work but I have not tested it)

STEP 1: Extract the operating system files

Take your various ISO files or DVDs and copy their contents into subfolders on your hard drive – in my case I have Windows 7 x86, Windows 7 x64, and Windows 2008 R2, so I created folders called e:\7×86, e:\7×64, and e:\2008r2    (E is the letter of my hard drive, not the USB drive).

STEP 2: Browse the WIM files and extract the desired editions

Open a WAIK command prompt, or browse to whatever folder you downloaded imagex.exe to within an administrative level command prompt.  Start by running the following command:

imagex.exe /info e:\7×86\sources\install.wim

This will display a big verbose mess that, once you parse through it, lists out all the editions embedded in the WIM file (and their associated index #).  Most default Microsoft WIMs will have multiple editions – in this case the ones in the Windows 7 x86 WIM boil down to:

• 1 Starter
• 2 Home Basic
• 3 Home Premium
• 4 Professional
• 5 Ultimate

Since Starter edition is basically worthless, I only want editions 2-5 in my custom WIM file, so I run these commands one after another:

imagex.exe /export e:\7×86\sources\install.wim 2 e:\install.wim “Windows 7 HOMEBASIC (x86)” /compress maximum

imagex.exe /export e:\7×86\sources\install.wim 3 e:\install.wim “Windows 7 HOMEPREMIUM (x86)” /compress maximum

imagex.exe /export e:\7×86\sources\install.wim 4 e:\install.wim “Windows 7 PROFESSIONAL (x86)” /compress maximum

imagex.exe /export e:\7×86\sources\install.wim 5 e:\install.wim “Windows 7 ULTIMATE (x86)” /compress maximum

I now have an install.wim file in the root of my E drive that contains only the editions I specified in these commands.  Next,  I want to integrate Windows 7 x64 so I repeat the above steps using the 7×64 path instead of 7×86.  You will notice that the x64 version of Windows 7 has no Starter edition, so the index numbers are not the same as the x86 source!  Also, since just about any machine new enough to run x64 is likely new enough to handle Aero graphics, I don’t bother with integrating the Home Basic version of x64 into my WIM – so I only add Home Premium, Professional, and Ultimate.

I then repeat these steps again using the Windows 2008 R2 source (here again I discriminate – I only pull Standard, Enterprise, Standard Core, and Enterprise Core because I have no use for the DataCenter or Web versions).

You can of course integrate any OSes and editions you like!  When you are done, you will have an install.wim file of varying size (for those of you planning on burning to DVD, you may have issues with WIM files that are larger than 4 GB…  To get around this see this link)

STEP 3: Finalize the USB stick

You must now choose which OS you want to use as the boot environment – this needs to be an x86 operating system if you plan to install any x86 OSes (because x86 OSes cannot be installed from x64 boot environments).  You also want to use the newest operating system you can, because you may encounter issues if you try to deploy an OS that is older than the boot environment - in my case I use Windows 7 x86 to ensure I can deploy any operating system I like:

• take the custom install.wim file (from E:\) and replace the install.wim file in e:\7×86\sources
• copy all contents from e:\7×86 to your USB stick
• ***EDIT*** please see JAG’s comment below for a link to the extra step required to get the server OS to work properly!

Now boot from the USB stick and you should see all your OSes, like below!

$219 plus $12 next day air got me a fedex delivery the next morning.  Two screws later the back panel is off and the easily upgradeable components are exposed — single ram slot which will be replaced with a $20 2GB module arriving tomorrow — and the stock 4 GB SSD (dell offers larger models but I’m going aftermarket):

Until the replacement RunCore 32 GB SSD I ordered ($110) comes, I’m stuck with the stock 4 GB one, which doesn’t give me much room to load an operating system — so for now I’ll be running Windows Fundamentals for PCs which is a poorly named super-slim version of XP that microsoft offers its software assurance customers to run on older computers they can’t afford to retire.  If you look closely you can see that even with almost all the options checked, the FLP operating system will take up less than 900 MB of space when installed.  Once I got FLP up and running, I got to work installing Mac OS X.  This is pretty straightforward and can be done in a number of different ways, and only requires a USB stick and a legit, retail copy of Mac OS X. For now I’m running OS X off an external USB drive, since my little 4 GB SSD doesn’t have the space to install it. Once I get my 32 GB SSD installed I’ll be dual-booting Windows 7 and OS X:

UPDATE 4/28/2009:

My RunCore 32 GB SSD arrived yesterday, so I replaced the stock 4 GB drive and got Windows 7 up and running – the new drive is significantly faster and doesn’t exhibit the random hiccups/pauses the stock drive did.

First, the RunCore SSD package – comes with a mini-USB cable so that you can use the drive via USB if necessary (helps for imaging etc.). Next you can see the SSD compared to a stick of laptop memory – this thing is small! The 3rd shot is installing the SSD into the mini (2 screws). Finally, the mini now running Windows 7…