Replacing VMware ESXi with Proxmox, KVM and LXC

And so it has come at last — my removal and replacement of VMware ESXi at home, with something better, known as Proxmox.  This post seeks to document the journey a bit, and give a tip or two along the way.

My reasons

I had a few drivers behind this move:

• My existing, single host was showing it’s age (a HP MicroServer N54L), and was never that powerful to begin with.  This was known and expected; I didn’t need or want a Boeing 747 in my house.  Due to it’s CPU (an AMD Turion dual core @ 2Ghz), it didn’t do particularly well handling Ethernet wire speeds, or general VM load for that matter.  This wasn’t such an issue having crappy DSL-speed internet, and I could deal with slowish speeds in the local LAN (eg to/from the NAS)
• I moved house, and this meant getting a far more tasty internet service (100mbps); suddenly, my VMs weren’t appearing to cope as well, including my firewall which is itself a VM.  I could download at internet “wire” speed, but it wasn’t consistent, and the host CPU would take a fair hammering
• I was, and am, just so sick and tired of the crap that VMware is putting out.  My key gripes:
  • No thin client?  Really?  Still?!?  Oh wait, yes there is, but I’ll have to upgrade to 6.5, which my hardware probably won’t run on, or be supported by
  • Pre 6.5 (only released in the last month or so.. 2016), and Adobe Flash?  Again, really?!?  I got super excited when VMware bought Zimbra, and then I saw hints of a similarity in web client; and then about a year later, VMware sold it
  • The HCL; I get that VMware needs one, but I’m also tired of endless Google searching to confirm if hardware I’m about to buy will even work with it, making custom ISOs, blah blah
  • Patching.  Sweet mother of the goat lord, the pain of VMware patching.  Add to that the confusion around how to use VIBs, etc
  • Want anything remotely cool or useful?  Pay.  Run vCenter.  Spec said server or appliance with half the capacity of your available gear

Typically late to the game, I discovered Proxmox, which pretty much addresses everything I list above, all for free, and all on commodity hardware.  Happy days.

The environment, old and new

The environment to be upgraded as mentioned above, is a single ESXi host, using a combination of local SSD storage, and iSCSI to a QNAP NAS.  The NAS is to stay (hopefully not for too much longer, as that’s something else I need to get a better brand of).

After much fussing, here’s the kit list I ended up with:

I originally only wanted the non-K CPU, but due to various stuff-ups and stupid shit from the supplier I ordered from (never again!), I ended up with the K model for the price of the non-K, and an Intel stock cooler on it.

Of particular note on the mobo side is the Intel C236 chipset, and the two glorious Intel I210 Gigabit NICs onboard, which was one of the most important elements in my build (ie. decent NICs, not Realtek crap).

The install of Proxmox was pretty standard, and I went with ZFS locally.  ZFS and I have been distant lovers for many years (since my Solaris days), and it would now seem to be quite usable and stable on Linux.  Brilliant.  Bye, confusing versions of VMFS with various annoying limitations and oddities (remind you of stupid vSphere alarms telling you to upgrade your VMFS version, when in fact it already is upgraded?)

Other, slower storage would be to the QNAP again, but this time using NFS.  General opinions is that NFS and iSCSI to QNAP is about the same performance-wise, and NFS makes accessibility of those datastores easier than “raw disk” as provided by iSCSI (and VMFS).

Once Proxmox was installed (a Debian variant, basically), waddya know?  All hardware just recognised and ready to go.  No stuffing around.  In fact, I installed it using a SD card reader, as I had no spare USB sticks.

My VM migration method

First step, create the new VM(s) on the PM box.  You can just follow a ton of existing info out there here.  The skinny basically is create the VM with all the right specs.  If this is a Linux VM (and it should be ;)) remember to use the virtio driver anywhere you can!  Set it for both network and disk, because for years now, all good variants of Linux have shipped with this driver baked in.

Yes, this means no stupid VMware tools install, no stupid perl scripts, and no multiple reboots to make it all happen, just to get accelerated network and disk controllers.

At this point, I also made sure that the display was set to SPICE if I knew X would be part of the deal, and in Options, turned on Qemu Agent for the benefits that brings.

For the disk, the following is what I did, based on storage type.

NFS destination

I created a temporary 1GB disk on the VM.  Some people then just copy the disk over the top of the existing one defined and created; I didn’t like that idea, but creating the temp disk populated the right images/<ID> directory, and I don’t know if it would have happened when creating the VM, disk or no disk.

Later on, this will be replaced with the actual disk.

ZFS destination

Being that this is block level storage, which will just appear as ZFS volumes (complete with /dev entries), I created the disk(s) the actual size they were over on the ESXi side, ready to be dd‘d to.  Feeling risky, I didn’t bother working out exact sizes, and rounding etc… just stuck with GB values.

Copying the disks

This was the interesting part.  A lot of PM doco and commentary talked about moving the VMDKs around and converting them to the qcow2 format using qemu-img.  In my ESXi 5.1 environment, without any thin provisioning (I don’t do thin, ever, for a few reasons), they were all “flat”.  This once meant something slightly different to me, rather than merely meaning “raw”.

So, using qemu-img with the vmdk option was pointless, as they were already raw.  This was confirmed by using file on the VMDK (as hinted here):

root@pm:/var/lib/vz/images# file somedisk-flat.vmdk
somedisk-flat.vmdk: DOS/MBR boot sector

iSCSI to NFS

As I was having to copy out of an iSCSI-backed VMFS LUN, I had three choices for copying (all having to involve the ESXi hypervisor):

1. Use the VMware fat clients datastore browser, and copy/pasta between the LUN and the NFS store
2. Use cp at the hypervisor CLI between stores
3. Use some form of scp to and from the hypervisor (either from the new PM host, or the NAS)

As most would know, ESXi doesn’t give hypervisor-level transfers like this the highest priority in the stack, so dependant on disk sizes, NAS etc, this can take a while.  Add to that any encryption (scp), and the poor little host gets a tad bogged down.  I ended up just using the fat client, at least for transfers from iSCSI to NFS (which, if I recall correctly, probably still uses SSL and hence encryption anyway).

Copy the disks into the right images/<ID> directory, and remember to rename to <whatever>.raw.

Now, here’s the bit that took me a little while, being used to having a GUI option available to rescan storage… you then need to run qm rescan, and a few seconds later, you’ll see the “unused” disk appear in the list of hardware for the VM!

Once done, then remove the existing 1GB disk (this needs to be done twice, as the first time just releases the disk from the VM config, the second will actually nuke it).  Then simply highlight the new disk, click Edit, and attach.

iSCSI to local ZFS

The second type of transfer I needed to do was from iSCSI to the PM box’s local ZFS storage, and for that, the quickest way was in fact to use wget and ESXi’s direct file browsing capability.  To get a usable URL, I just used a normal browser, hit my ESXi host, and clicked “Browse datastores in this host’s inventory” over on the right.  After finding the right VMDK, copy/pasta the whole URL, and use that with wget, curl or whatever you have on your PM host.

The next interesting part on ZFS is that, of course, it’s using ZFS datasets/volumes, and not a filesystem.  This means you have to direct-copy your raw disk images straight into specific Linux devices.  More on that below.

In the interim, I just copied them to somewhere local with enough room, temporarily.

Once copied, I used dd to write to the appropriate zpool /dev entry:

root@pm:/# dd if=somedisk-flat.vmdk of=/dev/zvol/rpool/data/vm-106-disk-1 bs=1M

Too easy!  Don’t forget to do a qm rescan.

Bring the awesome

At this point, I was simply able to power on each VM, and thanks to virtio drivers, and other Unixy tricks, they just booted!  Being timid about how easy that was, I did boot a few single user and run some fsck‘s, and everything was fine.

Things to note:

• If using LVM, devmapper will just magically map existing devices and just work
• If using UUID’s in /etc/fstab, things will just magically work
• If using legacy /dev/sdX entries, things will probably break, and you’ll need to edit /etc/fstab to point to virtio’s /dev/vdaX entries.  There might be additional pain for your boot or root disk, and having to rebuild your initrd stuff

Conclusion

So far, so good, a few weeks in.  Best decision I’ve made in years with the home network stuff.

That said, I’ve encountered some rather showstopping issues around ZFS on Linux, and swap, that has caused the host to blindly melt and reboot twice.  I’ve made some tweaks, and hopefully that issue is no more, and will be something for a future blog.

I also have a bizarre issue with one of my two LXC containers, whereby when cleanly shutdown, the host ZFS subvol goes readonly; haven’t had any luck tracking that down yet.

I hope this blog was useful to someone.  Let me know if so, and if you have any suggestions, corrections or questions, post away!