Virtualising Servers on a Mac ESXi Host

21st Oct 2015 | Opinion, Reviews

I’m writing this one because I’ve faced quite a few gotchas, and I’m still not there yet. The basic theory is to take a whole bunch of servers that I currently run and make them into virtual machines that can be moved around and generally make more efficient use of the infrastructure we currently have. None of the current servers have a heavy workload so it makes sense to do this.

Also it might be my last chance to do it in a sensible way with Macs- since Apple introduced the new Mac Mini with only a dual core i5, your hardware choices are becoming more limited. So I purchased 2 older 2012 Mac Minis- one which already had an SSD and 16GB of RAM, and another busted one that I think I can bring back to life. These machines have a quad core i7 processor, so 8 threads to use. I’ve also got one or two more of these machines, so I should be able to put a fair bit of grunt into the project if needed.

The first issue is learning a whole new set of software and nomenclature. VMWare has a bunch of products that range from free to you-can’t-afford-it. They’ll give you a free version of ESXi, but they won’t let you use the web management console, you need to use the (somewhat deprecated) Windows client.

So I lined up my hardware, purchased a copy of VMWare Fusion 8 Pro for Mac, downloaded ESXi v6.0.0 and started work. The ESXi install went really well- it recognised the Mac Mini 6,2 hardware straight up, did not require any extra drivers and gave me no trouble at all.

So, time to move some servers to it- how hard can that be? The next 48 hours (of the weekend) were an exercise in extreme frustration. Recognising the need for a copy of Windows, I figured out that an Intel Compute Stick @ $179 was cheaper than a copy of Windows 10 @ $220. Even though the stick comes with Windows 8.1, there’s a free upgrade to Windows 10. So I went out and bought one of those, to be frustrated by 2 problems- firstly, it doesn’t have an ethernet port and when I connected it to the wifi network it couldn’t see my ESXi host. This turned out to be a Meraki router issue and rather than figuring it out I connected it to a different SSID and it suddenly worked…. the second issue was with VMWare Fusion- I had imagined that I was buying a Windows license and I could virtualise this Intel Compute Stick. In fact Fusion even provides a tool to enable you to pull across the data and virtualise Windows on your Mac. Unfortunately after hours of trying to make it work I googled the issues and found that it will only virtualise Non OEM versions of Windows. Guess which type I have? Dear VMWare- how about putting that information in your error message? (Yes I do accept that I was a bit dumb assuming this would work, but error messages can be informative you know?)

So, here’s a few things I hate about Windows 10-

  1. Required me to log in with a Windows Live ID. No option to create a local account and use that (I’ve since found out you can do this, and I have- but it certainly isn’t offered to you upon setup)
  2. My Windows Live ID was horribly complex, and it tuns out that with 2 different keyboards it could not type an @ symbol. Meaning I could not log in
  3. So I had to reduce the security by logging in on another machine and simplifying the password
  4. The stick does not have an ethernet port and would not recognise my USB-ethernet adaptor. OK, maybe not a Windows issue!
  5. As a Mac guy, it’s a very alien experience using Windows….

Next it was time to test if we could virtualise one of the servers. I suppose pointing Fusion at a disk image and waving a wand would be too much to ask for? Yes, as it turns out. While Fusion can import a Windows machine (well SOME Windows machines), it requires you to do some bizarre yoga to allow it to boot existing Mac OS X- here’s 2 methods-

  1. Extract the installer ‘InstallESD.dmg’ from the downloaded Mac OS X bundle, go ‘Add New’ from ‘Install from Disc or Image’ and install a brand new copy of El Capitan etc. Then plug in a disk that contains the image you want to virtualise, and use Disk Utility in that virtual machine to copy that into a virtual disk image that you created with ‘Create a custom Virtual Machine’. From here
  2. If you follow the above method but simply use the downloaded installer bundle, Fusion will boot into the recovery partition, allowing you to access Disk Utility and you can then copy your image into the currently booted ‘recovery partition’. This will virtualise your machine but it will obviously contain the installer that you booted from, so your image will suddenly be 6GB bigger. You can delete this later.

Since Fusion Pro 7 (well technically the beta, which was apparently publicly available) you’ve been able to upload VM’s to an ESXi host, so surely it’s all working well now under Fusion 8? Ah, well no. After verifying that it boots in the Fusion 8 environment I used the new tool to transfer it to the ESXi host. And it wouldn’t boot. I kind of expected minor issues- Apple’s licensing for virtualising Mac OS X explicitly states that the VM should run on Mac hardware only. So the way VMWare deals with this is to add an emulation for the SMC (system management controller) on a Mac VM. Then when you move the VM to ESXi it strips out that part of the config from the setup doc that goes with the VM. In this case however, all was good- I checked for SMC= True in the file and it existed before and after transferring. So what’s happening? I got a clue by having a look at the machines listed in Fusion- the one on my Mac that would boot had a type of ‘OS X 10.10’ and on the ESXi host it was listed as ‘Other (32-bit)’. You can manually edit this- it’s not clear or Mac-like, but that didn’t work either. Interestingly, it did help a little bit- it seems that ESXi goes through a normal boot process of trying to boot from a list of preferred sources, SATA, optical disk, network etc. And it did actually boot from a netboot image of 10.7.5 that I have on the network. And that’s where we are stuck.

According to the comments at the bottom of this page from clindsey-

‘Here’s what I’ve found so far. Uploading a working VM seems to leave behind the .vmdk. I haven’t found a work around for this. If I create a new VM on the remoter server through Fusion Pro 8. The .vmdk (Virtual Hard Disk) IS created but remains unformatted. Booting to a Casper Imaging.nbi and presumably any other bootable network drive, and using DiskUtility to format the drive results in a VM that is ready for imaging/Installation.’

The big issue here is that with the free version of ESXi, you’re not allowed to write to the host without using the Windows app. You can install and use it, but many operations will fail. To be fair, this is only a recent addition to the stable of tools, you can see how rapid the development is by the fact they got v2 out in about 2 weeks. So I’m going back to Windows which I had hoped to avoid.

Possible solutions include using the conversion tool within Windows to convert and upload my Mac VMs, or as listed above, OP appears to be saying that creating a disk image and restoring to them VM may work. I’ll figure out a way around this and update here.

(fast forward to 2 weeks later)

Aaaaaand I’ve found about 1123 other ways of doing this that ALSO don’t work.

When uploading a vm to the ESXi 6.0.0 host from Fusion v8.0.2 host
1. Application Fusion is using the wrong kind of dialog box to show the progress bar. Minor annoyance, but this is a Mac interface issue and has nothing to do with my problem
2. Progress bar disappears way too quickly- within about 5 seconds- you can see that the ESXi Datastore space decreases by the correct amount, but you cannot see any progress of the VM being transferred. I accidentally did this 3 times before figuring out that the VM had been transferred 3 times. Strangely, vSphere client on Windows 10 appears to do this too.
3. When the transfer is finished, the VM does not appear in the Library of the ESXi host in Fusion- you have to quit and re open Fusion for the VMs to appear. Very odd.
4. The VMX key smc.present = “TRUE” IS transferred properly by this setup, no changes needed to the VMX file
5.  The VMX key smbios.reflectHost = “TRUE” does not appear in the Fusion VM, the transferred VM or even in the EXSi hosted VM that works properly. I conclude that this setting is not needed. The doc you supplied was from Fusion v6 and ESXi v5.5.
When I added smbios.reflectHost = “TRUE” to the non working ESXi hosted VM it did not help.
6. The doc VMWare support supplied had a very different issue to me- in the doc, the user was experiencing the ‘spinning beach ball of death’. In my case, the VM is failing to boot as ESXi cannot find a boot drive- mine is failing much sooner. So no beachball.
7. Save VM from Fusion as .ovf and upload to ESXi via vSphere client on Windows 10- same result. VM fails to find a boot volume
8. One thing that does need changing when uploading the VM to ESXi is the OS type- it always comes across as ‘(other) 32 bit’. Changing this to the correct ‘Mac OS X 64 Bit’ makes no difference. Covered above, but seems to be an ongoing issue
I have had some success though- I was able to use a standard 10.10.5 InstallESD to create a bootable VM within Fusion. And I could transfer it to the ESXi host. And it would boot- yay!
But nothing at all would convince the infernal machine to boot a disk image from a working machine instead of one wholly conceived in a virtual world.
I found out a number of things that don’t work here either. But let’s look at what does work-
If you start with your bootable clean install, you can shut it down, go to Settings and provision another 40GB disk for it to use. Boot it up again, and you can use Disk Utility to copy the contents of a disk image to the new virtual drive. That virtual drive must be connected via the ‘Shared folder’ feature within Fusion though. If you think you can simply attach the external drive and wait for the message to come up from Fusion ‘should I attach this drive to the VM or to the Mac?’
hey sure, attach it to the VM, that would be an easy way of copying my disk image. Except it won’t work. Something about that feature absolutely craps itself when you try. Every time.
So, use the Shared Folder feature, make your copy. Then set Startup Disk to the newly copied ex-physical machine and reboot to check. Working! now shut it down and delete the old drive. test again? Working! now transfer it to the ESXi host- Not working!
And that’s where we are stuck my friends.