Hyper-V 3.0, Server 2012 deduplication (yay), and VHDX files

As you may have heard, Windows Server 2012 has hit RTM and should be available to you technetters, MSDNers and Volume License holders in about a week or so.  This post will be a work in progress documenting the curious issues I’ve encountered while working with the exciting new deduplication feature of Server 2012 for the past 6 months or so, moving from Beta to RC and now (sadly confirmed that the issue is still here) RTM.

I don’t know about you, but pretty much all of our servers these days are virtualized (we are a Hyper-V shop and I’m curious to see how this breaks down over in the VMDK camp), and the idea of “free” deduplication sounded like a fantastic boon for our file servers.  Less storage space being eaten up in the guests means smaller VHD (now VHDX) files, and less storage space eaten up across the board.  If only it were so!

First let me say that I have tested and confirmed this issue on a variety of hardware and configurations (Dell, HP, SAS, SATA, SAN, local storage, you name it) – so this definitely appears to be something inherent to Microsoft’s implementation.  What I have NOT done is test this on fixed VHDX files as they pose two problems:

1) they can’t be shrunk!

2) if you start at your smaller target size, they aren’t large enough to hold your initial data set prior to the dedup pass!

Like many of you, we use DFS Replication (and use it heavily!).  So a new file server build consists of getting the OS and associated DFS (and now dedup) roles and features installed, then letting DFSR seed the data.  With a dynamic VHDX, even if we try to stay on top of it and run dedup passes multiple times during the initial seed, the VHDX files ultimately end up quite a bit larger than the deduped data set.  This is certainly to be expected to a certain degree, but what I was not expecting was my subsequent results when trying to reclaim all that space.

An example of one of our datasets is a 250 GB share that, after being fully deduped, shrunk down to 150 GB of used space being reported by the guest OS (w00t!).  This is a fantastic savings.  Unfortunately, our VHDX file after all this was sitting right around 275 GB (some overhead for DFSR staging and other odds and ends is to be expected).  So I shut down the guest and used the Hyper-V management console to attempt to shrink the VHD.  It worked for about 7 seconds or so and completed.  As you can imagine, the file was no smaller.  After digging up a couple of suggestions on TechNet forums I mounted the VHDX on the host (making sure that dedup was installed on the host first so it could properly work with the deduped volume) and defragged it.  I then used the optmize-vhd commandlet in “full” mode rather than the gui to shrink the file – and it did shrink quite a bit down to 222 GB.  A big improvement over 275, however not so great considering the actual data is only 150 GB and the pre-dedup dataset was only 250!  A second defrag and optimize pass squeezed another 7 GB of blood from the stone.  Obviously the VHDX file’s lack of “shrinkability” makes the gains from deduplication moot, especially considering all the manual steps (which of course can be scheduled and scripted eventually).

Now having confirmed the same behavior under RTM with a couple of different server scenarios and datasets, my next plan is to attempt to take advantage of the fact that Windows Server Backup under 2012 is dedup-aware.  My hope is that WSB may be able to properly create a backup of the 150 GB dataset and restore it without bloating the VHDX file.  I will also attempt restoring to a fixed VHDX (although I’m not sure if WSB will accept a volume smaller than the backup volume was).  Stay tuned!

UPDATE 8/12: Well Windows Server Backup doesn’t appear to be able to do an optimized (dedup-aware) backup of my disks…  Tried on a couple of different servers and am getting the same error when I select the deduped volume for backup:

UPDATE 8/19: At this point I need to move the dedup project forward so my workaround has been this: Take one of the DFS replication partners (we have two at each site) and configure it with a fixed size VHDX, then replicate and dedup, enlarging the fixed VHDX manually as needed.  Once one is finished and we know the proper target size, the other peers are rebuilt with a fixed VHDX of that size, once again seeding and deduping as we go to prevent the virtual disk from filling up.  The process is rather manual and requires a bit of attention, but it isn’t that bad until a better solution can be found (the dedupes can easily be scripted/scheduled).

961112 ADM and forcing Outlook 2010’s Outlook Anywhere settings via Group Policy

Just a brief note about something that tripped me up on a recent installation.  Outlook 2010 is running in an environment where workstations are domain-joined, but have no direct tcp access to the Exchange server.  Default behavior of course is for Outlook to attempt tcp connections, and then after roughly 25-30 seconds it will give up and switch to http and successfully connect.  Of course, this is annoying to users who may or may not have had their coffee in the morning but expect Outlook to connect more or less immediately.

There is a KB article (http://support.microsoft.com/kb/961112) which provides the ability to force the various Outlook Anywhere check box values, but it is hard coded to work with Outlook 2007 – our 2010 clients were not seeing any change.

After much head-scratching I finally remembered having dealt with the exact same issue roughly 10 months ago at another installation (my memory isn’t what it used to be).  The trick is to open up the ADM file from the KB article in a text editor such as notepad and adjust all the 12.0 values (which is Office 2007’s version) to 14.0 (Office 2010’s version – they skipped ver. 13 due to superstition).  Making this change and reimporting the modified ADM into a fresh GPO did the trick.

Hope that helps anyone looking to adjust Outlook Anywhere settings via GPO!

Hyper-V and 4k “advanced sector” hard drives

As you may know, all newer hard drives are being shipped with 4k sector sizes.  If you would like more detail on this, please see http://www.bit-tech.net/hardware/storage/2010/04/01/the-facts-4k-advanced-format-hard-disks/1

I recently purchased one of the new Western Digital 3TB USB drives for use on one of my DPM servers, which is a Hyper-V guest.  Unfortunately, when I started to try to use the disk, Hyper-V refused to create new VHD files on the disk, or to boot VHD files that were manually copied to the disk.  All other functions seemed fine.

After trying a multitude of formatting and partitioning options and speaking with a couple of higher-ups at Western Digital, I realized there was a fundamental incompatibility between Hyper-V and 4k physical sectors.  I was planning to return the drive, but got caught up in some other projects and got back to this project yesterday to find a new KB article that seems to have been pulled – but thanks to google cache it’s still readable: http://support.microsoft.com/kb/2515143

It references not only the root problem but a hotfix (that is also included in the recently released Service Pack 1 for Win2008R2): http://support.microsoft.com/kb/982018

I decided to download and try the hotfix on a non-production Hyper-V server.  Here is what you get when using an advanced format disk on Windows 2008 R2 RTM Hyper-V:

And following the patch and reboot I now get…

…the exact same error.  Sad face.  Creating the VHD elsewhere and copying it over still doesn’t work either as Hyper-V still squawks about compatibility; looks like MS has more work to do:

UPDATE 4/18/2011: As you can see from my above edit, there is a new KB article specifically addressing this issue WRT hyper-v – and the bottom line is you cannot put VHDs on native 4k sector disks (and disks with 512b emulation will take a performance hit)…  bummer!

Building a production-ready Hyper-V Cluster on the cheap

My most recent project was to improve the resiliency and uptime for a server farm consisting of ~18 virtual servers spread across two standalone Hyper-V hosts.  The objective was to convert the two hosts (Dell Powerdge T710 servers) into a 2-node Hyper-V cluster as inexpensively as possible while still maintaining solid performance and leaving headroom for moderate growth.

The workloads being virtualized consist of four Exchange 2010 servers (2-node DAG and 2 CAS/HT servers running NLB), a couple of file servers, and a hodgepodge of domain controllers, SQL, and web servers with a few Citrix Xenapp servers thrown in (most of the Citrix servers are physical boxes).  The network consists of three distinct internal networks brokered by Juniper firewalls, named DMZ, Trust, and Vault.

When it comes to clustering on 2008/2008R2, the storage solutions that get the most publicity are Fiber Channel and iSCSI.  Both of these solutions offer the ability to connect multiple nodes and, when configured properly, can perform very well.  Both require quite a bit of hardware and tweaking to do so however – including a redundant set of FC or gigE switches (to avoid a single point of failure) and in the case of iSCSI, further network tweaks for performance.  There is a third shared storage option that just doesn’t get talked about very much:   Shared SAS.

Shared SAS

Serial attached SCSI is the successor to the older parallel SCSI interface we all knew and loved (and the command set for which is still used in SAS, FC, even SATA).  SAS performs well (even the older 3gbit spec) and is relatively inexpensive, and while parallel SCSI is no longer legal for shared storage clusters under Server 2008 and beyond, shared SAS is fully supported!

SAN budgets can vary wildly depending on the target application, storage types, fancy features such as snapshots, thin provisioning, etc. but even the cheapest redundant solutions quickly get you into the thousands of dollars range, if not tens of thousands.  You can set up software-based solutions such as Starwind or Sanmelody on commodity hardware you have laying around (and I have done this with great success for lab work) however to build those into redundant SANs requires expensive licenses that can rival the cost of turnkey solutions.

Ebay to the rescue

Ultimately we decided on purchasing a used Dell MD3000 shared-SAS SAN that still had a year of 4-hour response warranty left on it.  The 3U MD3000 is still sold by Dell despite being upstaged by its 6gbps brother, the MD32xx.  While the newer model can only hold twelve 3.5″ drives, the MD3000 can handle fifteen.  The real sweetener on this find was that the SAN came with eight 450GB 15k RPM SAS drives.  To buy this MD3000 redundant configuration new with the eight disks and warranty would have cost us over $20,000.  All told (including the four SAS5/E adapters and cables) we paid under $6,000.  When next July rolls around we will have to re-up on the warranty coverage, but that is still a hefty savings (now I know some of you are used to working with SAN budgets that exceed our entire annual operating budget, but this was big savings for us!).


After storage is nailed down, the next big pain point of clustering (especially with Hyper-V) tends to be figuring out the best networking configuration.  Spending a lot of money and effort on a cluster to improve uptime does little good if something as simple as a failed ethernet cable or switch port can bring down a VM.  Strangely enough, this is exactly the case – a nic failure or switch failure that causes connectivity failure in a VM is not something a Hyper-V cluster can detect and work around.  So your options either become scripting elaborate routines to check network connectivity and migrate the affected virtual machines, or to turn to NIC teaming.

The Sorrowful Saga of NIC Teaming and Hyper-V

To the surprise of many, Windows has never had any way to aggregate NICs.  Microsoft has always left teaming solutions (and support of them) to the hardware vendors.  Both Broadcom and Intel have stepped in, with varying levels of success.  Typically teaming is very easy to set up, but virtualization can introduce some quirks that need to be understood and worked around.  At this time, both Broadcom and Intel offer support for teaming solutions under Hyper-V.  As always, however, your mileage may vary.  I have read way too many horror stories about both vendor solutions, but as of this writing Intel seems to have the edge and has worked out the kinks (revision 15 ProSet drivers and later).  Intel also offers a special teaming mode called Virtual Machine Load Balancing which is very handy, providing both load balancing and failover options at the VM level.

Our T710s came with the integrated quad port Broadcom nic, and a PCIe quad port Intel nic.  The cluster teaming plan called for more available ports for full redundancy, so each server was outfitted with an additional quad port Intel nic ($200 each on Ebay), as well as another Intel single-port PCIe nic, for a lucky total of 13 ports per node.

What could you possibly need all those NIC ports for?

The 2008 R2 revision of Hyper-V clustering introduces a number of important network-based features and services that increase the need for dedicated gigE interfaces.  The one most people have heard about is Live Migration – Microsoft’s answer to VMWare’s VMotion.  Live Migration allows you to seamlessly transition a running virtual machine from one cluster node to another.  This is obviously incredibly handy for managing server load, as well as shifting workloads off of physical hosts so that they can be taken offline for hardware or software maintenance.  Production-ready Live Migration should have a dedicated gigE nic (10gigE works great too if you have deep pockets).

Next is Cluster Shared Volumes – a handy feature in 2008 R2 that allows you to store multiple virtual machines on a single LUN (trying to provision per-VM LUNs is a headache I don’t even want to think about).  In our case, the use of CSVs allowed us to take our two physical disk arrays on the SAN (8-drive RAID10 and 5-drive RAID5) and format each as a single LUN.  I won’t go into too much detail here on CSV as there are million blogs out there with great information.  CSV should also have a dedicated NIC available to it, as it can be used to relay storage commands in case a node loses its connection to the shared storage (there must be a performance penalty for this, but I suppose it is good enough to limp along until you can live migrate the workloads to another fully functional node – which is why LM and CSV should have separate nics to work with).

The nodes themselves need a NIC for basic network connectivity and administration, so now we’re up to 3 NIC ports per node and we haven’t even talked about the virtual machines yet!  As previously mentioned, the cluster is pointless if a failed NIC or switch port can bring down a VM, so we decided to implement Intel nic teaming using the VMLB mode.  We carved up our two intel quad port nics into three teams (one for each of our aforementioned networks): TeamV for Vault (4 ports), TeamT for Trust (2ports), and TeamD for DMZ (2 ports).  This setup provides at least two ports per network to guard against both nic or switch failure, and avoids the need for configuring VLANs (which is a whole other mess for another blog) – we do have VLANs configured on our cisco switches, but only for trunking purposes – I like to avoid complicating the Hyper-V configuration as it is far easier to train administration staff without having to have them learn the ins and out of networking at that level.  If you prefer you could certainly cable all ports into a giant team and use VLANs to separate out the traffic.  At any rate, we’re now up to 11 NIC ports used (LM, CSV, host administration, and 8 teamed VM ports).

Not all VMs need be highly available

While most of our VMs are fairly critical and will made highly available across the cluster, certain machines are not – either because they just aren’t as important, or because the services they provide offer high availability in other ways.  For example, each of our nodes has a domain controller Hyper-V guest configured to run from local storage (not part of the cluster).  As long as domain clients are configured with both DCs in their DNS settings, they can deal with one or the other going down for a while.  Similarly, each of our nodes has an Exchange 2010 mailbox server (running in a DAG) and an Exchange 2010 CAS/HT server.  This way, the failure of an entire physical node does not derail DC or Exchange services.  It also allows us to play around with fancy storage options for the Exchange servers, but I will save that for another post…

I mention Exchange because we use Windows Network Load Balancing to provide high availability across our two CAS servers – and NLB creates port flooding on switches (by design).  By dedicating one of our NIC ports to each node’s CAS server, we are able to limit the port flooding on the cisco switches to only the ports those dedicated NICs plug into.  There are likely more sophisticated solutions to this problem, but this is what we came up with (had to use those spare broadcom ports for SOMETHING!).

Our count is now up to 12 NICs, but I happened to have a couple of spare Intel PCIe NICs laying around, so I figured it wouldn’t hurt to get an extra in there.  This would allow me to set up a broadcom team to provide switch redundancy on the nodes’ admin IPs, which could always be reconfigured and set up to take over in the event of one of the primary cluster links going down – so 13 it is!  Below is a picture of the color coded NIC cabling (the two short orange cables are crossovers for the LM and CSV links) – I have not attacked it with zip ties yet!  You can also see each node’s two SAS adapters/cables in the photo.

Putting it all together

Now that the hardware part of things was all set, it was time to adjust the software side.  The procedure basically broke down like this:

  1. Join each node to the domain (they were previously not domain members to avoid dependence on the virtual domain controllers), configuring DNS to point first to the other node’s virtual DC and second to an offsite domain controller with a reliable WAN link.
  2. Install latest NIC drivers and configure the teams.  Configure private networks as desired (for example I have my crossover networks set to 10.10.10.x and 10.10.11.x).
  3. Remove all the VHDs from VMs to be made highly available, then export their configurations and copy the VHDs to cluster storage (this step can be done many ways but I prefer manually copying the VHDs instead of having them tied to the Hyper-V export process; saves time if you end up having to re-do the import for some reason).
  4. Delete existing virtual networks and create new ones using the nic team interfaces, ensuring that all virtual networks to be used for highly available VMs are named identically across the nodes.
  5. Install the Failover Clustering feature and use its administration tool to run the Cluster Validation wizard – hopefully this will pass!
  6. Go ahead and create the cluster, and the wizard should be smart enough to set up the proper cluster networks (three of them in my case, the two privately addressed interfaces and the admin interface) – the virtual networks have no protocols bound to them other than the Hyper-V protocol so they are ignored by the cluster wizard.
  7. Add any storage disks that were not automatically added by the wizard (mine added all three of mine).
  8. Using the Failover Cluster management tool, enable CSVs and turn the appropriate storage disks into CSVs.
  9. Using the Hyper-V management tool, import the VM configurations you exported onto the CSVs as desired, and copy/move the VHDs into place and re-add them into the configurations, as well as specifying the appropriate virtual network(s).
  10. In Failover Clustering management, configure a service and select Virtual Machine, and select all the VMs you want to make highly available – they should pass validation!
  11. Fire up the VMs, and then attempt to Live Migrate one or two over to the other node.  This worked like a charm for me…  The only quirk I ran into was a Live Migration failure after editing a VM config from the Hyper-V management tool – it would appear that any settings changes need to be made from within Failover Clustering and not the Hyper-V tool to avoid this problem!

The entire procedure took roughly 13 hours, however a decent chunk of that time was spent simply waiting for large VHDs to copy, and/or dealing with unexpected surprises such as the fact that the MD3000 had the wrong rail kit with it (despite having the proper part # on the box) – nothing a pair of pliers wasn’t able to fix…  The only single point of failure left is the ethernet drop the colo provides – and in the near future we may add a second drop to remove that last potential failure point:

Thanks for reading!

Create an all-in-one x86+x64 Win7/Vista/Server 2008/R2 USB stick

One of the great things about the Vista and post-Vista operating systems is that the installer subsystem allows you a great deal of flexibility when it comes to installing multiple operating systems. It is fairly easy to put together a single installation DVD or USB stick that will allow you to install Vista, Windows 7, Server 2008 and Server 2008 R2 – in a variety of x86 and x64 flavors.

A couple of things to note about this guide:

  • I do not use Vista — I have never really used Vista and now with Windows 7 out there really is no reason to; however the steps here should work just fine with Vista installation sources
  • I have only tested this using a USB stick and will only cover that method here – it’s much more flexible (and faster) for installation than using DVD — but DVDs should work just fine: you can find plenty of tutorials on the web that will tell you how to use oscdimg.exe to take the files we create here and turn them into a burnable ISO


  • USB stick, at least 4 GB (perhaps larger depending on how many OSes you plan to integrate, I use a 16 GB) that has been properly formatted using Steps 1 and 2 here
  • ISOs or DVDs of the operating systems that you plan to integrate
  • imagex.exe (can be found for download on the web, or you can download the full 1.7 GB WAIK here)
  • computer running Vista or newer operating system (XP should work but I have not tested it)

STEP 1: Extract the operating system files

Take your various ISO files or DVDs and copy their contents into subfolders on your hard drive – in my case I have Windows 7 x86, Windows 7 x64, and Windows 2008 R2, so I created folders called e:\7×86, e:\7×64, and e:\2008r2    (E is the letter of my hard drive, not the USB drive).

STEP 2: Browse the WIM files and extract the desired editions

Open a WAIK command prompt, or browse to whatever folder you downloaded imagex.exe to within an administrative level command prompt.  Start by running the following command:

imagex.exe /info e:\7×86\sources\install.wim

This will display a big verbose mess that, once you parse through it, lists out all the editions embedded in the WIM file (and their associated index #).  Most default Microsoft WIMs will have multiple editions – in this case the ones in the Windows 7 x86 WIM boil down to:

  • 1 Starter
  • 2 Home Basic
  • 3 Home Premium
  • 4 Professional
  • 5 Ultimate

Since Starter edition is basically worthless, I only want editions 2-5 in my custom WIM file, so I run these commands one after another:

imagex.exe /export e:\7×86\sources\install.wim 2 e:\install.wim “Windows 7 HOMEBASIC (x86)” /compress maximum

imagex.exe /export e:\7×86\sources\install.wim 3 e:\install.wim “Windows 7 HOMEPREMIUM (x86)” /compress maximum

imagex.exe /export e:\7×86\sources\install.wim 4 e:\install.wim “Windows 7 PROFESSIONAL (x86)” /compress maximum

imagex.exe /export e:\7×86\sources\install.wim 5 e:\install.wim “Windows 7 ULTIMATE (x86)” /compress maximum

I now have an install.wim file in the root of my E drive that contains only the editions I specified in these commands.  Next,  I want to integrate Windows 7 x64 so I repeat the above steps using the 7×64 path instead of 7×86.  You will notice that the x64 version of Windows 7 has no Starter edition, so the index numbers are not the same as the x86 source!  Also, since just about any machine new enough to run x64 is likely new enough to handle Aero graphics, I don’t bother with integrating the Home Basic version of x64 into my WIM – so I only add Home Premium, Professional, and Ultimate.

I then repeat these steps again using the Windows 2008 R2 source (here again I discriminate – I only pull Standard, Enterprise, Standard Core, and Enterprise Core because I have no use for the DataCenter or Web versions).

You can of course integrate any OSes and editions you like!  When you are done, you will have an install.wim file of varying size (for those of you planning on burning to DVD, you may have issues with WIM files that are larger than 4 GB…  To get around this see this link)

STEP 3: Finalize the USB stick

You must now choose which OS you want to use as the boot environment – this needs to be an x86 operating system if you plan to install any x86 OSes (because x86 OSes cannot be installed from x64 boot environments).  You also want to use the newest operating system you can, because you may encounter issues if you try to deploy an OS that is older than the boot environment – in my case I use Windows 7 x86 to ensure I can deploy any operating system I like:

  • take the custom install.wim file (from E:\) and replace the install.wim file in e:\7×86\sources
  • copy all contents from e:\7×86 to your USB stick
  • ***EDIT*** please see JAG’s comment below for a link to the extra step required to get the server OS to work properly!

Now boot from the USB stick and you should see all your OSes, like below!

osinstall1 osinstall2