Techno, tech, and life

Replacing VMware ESXi with Proxmox, KVM and LXC


And so it has come at last — my removal and replacement of VMware ESXi at home, with something better, known as Proxmox.  This post seeks to document the journey a bit, and give a tip or two along the way.

My reasons

I had a few drivers behind this move:

I was, and am, just so sick and tired of the crap that VMware is putting out.

  • My existing, single host was showing it’s age (a HP MicroServer N54L), and was never that powerful to begin with.  This was known and expected; I didn’t need or want a Boeing 747 in my house.  Due to it’s CPU (an AMD Turion dual core @ 2Ghz), it didn’t do particularly well handling Ethernet wire speeds, or general VM load for that matter.  This wasn’t such an issue having crappy DSL-speed internet, and I could deal with slowish speeds in the local LAN (eg to/from the NAS)
  • I moved house, and this meant getting a far more tasty internet service (100mbps); suddenly, my VMs weren’t appearing to cope as well, including my firewall which is itself a VM.  I could download at internet “wire” speed, but it wasn’t consistent, and the host CPU would take a fair hammering
  • I was, and am, just so sick and tired of the crap that VMware is putting out.  My key gripes:
    • No thin client?  Really?  Still?!?  Oh wait, yes there is, but I’ll have to upgrade to 6.5, which my hardware probably won’t run on, or be supported by
    • Pre 6.5 (only released in the last month or so.. 2016), and Adobe Flash?  Again, really?!?  I got super excited when VMware bought Zimbra, and then I saw hints of a similarity in web client; and then about a year later, VMware sold it
    • The HCL; I get that VMware needs one, but I’m also tired of endless Google searching to confirm if hardware I’m about to buy will even work with it, making custom ISOs, blah blah
    • Patching.  Sweet mother of the goat lord, the pain of VMware patching.  Add to that the confusion around how to use VIBs, etc
    • Want anything remotely cool or useful?  Pay.  Run vCenter.  Spec said server or appliance with half the capacity of your available gear

Typically late to the game, I discovered Proxmox, which pretty much addresses everything I list above, all for free, and all on commodity hardware.  Happy days.

The environment, old and new

The environment to be upgraded as mentioned above, is a single ESXi host, using a combination of local SSD storage, and iSCSI to a QNAP NAS.  The NAS is to stay (hopefully not for too much longer, as that’s something else I need to get a better brand of).

After much fussing, here’s the kit list I ended up with:

CPUIntel Core i7-6700K 4.0GHz Quad-Core Processor$457.00 @ Shopping Express
MotherboardAsus P10S-M WS Micro ATX LGA1151 Motherboard$377.98 @ Mwave Australia
MemoryKingston FURY 32GB (2 x 16GB) DDR4-2133 Memory$299.00 @ PCCaseGear
StorageSamsung 850 EVO-Series 500GB 2.5″ Solid State Drive$218.00 @ Shopping Express
CaseFractal Design Define Mini C MicroATX Mid Tower Case$125.00 @ Mwave Australia
 Prices include shipping, taxes, rebates, and discounts 
 Generated by PCPartPicker 2016-12-31 16:10 AEDT+1100 

I originally only wanted the non-K CPU, but due to various stuff-ups and stupid shit from the supplier I ordered from (never again!), I ended up with the K model for the price of the non-K, and an Intel stock cooler on it.

Of particular note on the mobo side is the Intel C236 chipset, and the two glorious Intel I210 Gigabit NICs onboard.

Of particular note on the mobo side is the Intel C236 chipset, and the two glorious Intel I210 Gigabit NICs onboard, which was one of the most important elements in my build (ie. decent NICs, not Realtek crap).

The install of Proxmox was pretty standard, and I went with ZFS locally.  ZFS and I have been distant lovers for many years (since my Solaris days), and it would now seem to be quite usable and stable on Linux.  Brilliant.  Bye, confusing versions of VMFS with various annoying limitations and oddities (remind you of stupid vSphere alarms telling you to upgrade your VMFS version, when in fact it already is upgraded?)

Other, slower storage would be to the QNAP again, but this time using NFS.  General opinions is that NFS and iSCSI to QNAP is about the same performance-wise, and NFS makes accessibility of those datastores easier than “raw disk” as provided by iSCSI (and VMFS).

All hardware just recognised and ready to go.  No stuffing around.

Once Proxmox was installed (a Debian variant, basically), waddya know?  All hardware just recognised and ready to go.  No stuffing around.  In fact, I installed it using a SD card reader, as I had no spare USB sticks.

My VM migration method

First step, create the new VM(s) on the PM box.  You can just follow a ton of existing info out there here.  The skinny basically is create the VM with all the right specs.  If this is a Linux VM (and it should be ;)) remember to use the virtio driver anywhere you can!  Set it for both network and disk, because for years now, all good variants of Linux have shipped with this driver baked in.

Yes, this means no stupid VMware tools install, no stupid perl scripts, and no multiple reboots.

Yes, this means no stupid VMware tools install, no stupid perl scripts, and no multiple reboots to make it all happen, just to get accelerated network and disk controllers.

At this point, I also made sure that the display was set to SPICE if I knew X would be part of the deal, and in Options, turned on Qemu Agent for the benefits that brings.

For the disk, the following is what I did, based on storage type.

NFS destination

I created a temporary 1GB disk on the VM.  Some people then just copy the disk over the top of the existing one defined and created; I didn’t like that idea, but creating the temp disk populated the right images/<ID> directory, and I don’t know if it would have happened when creating the VM, disk or no disk.

Later on, this will be replaced with the actual disk.

ZFS destination

Being that this is block level storage, which will just appear as ZFS volumes (complete with /dev entries), I created the disk(s) the actual size they were over on the ESXi side, ready to be dd‘d to.  Feeling risky, I didn’t bother working out exact sizes, and rounding etc… just stuck with GB values.

Copying the disks

In my ESXi 5.1 environment, without any thin provisioning, they were all ‘flat’.

This was the interesting part.  A lot of PM doco and commentary talked about moving the VMDKs around and converting them to the qcow2 format using qemu-img.  In my ESXi 5.1 environment, without any thin provisioning (I don’t do thin, ever, for a few reasons), they were all “flat”.  This once meant something slightly different to me, rather than merely meaning “raw”.

So, using qemu-img with the vmdk option was pointless, as they were already raw.  This was confirmed by using file on the VMDK (as hinted here):

root@pm:/var/lib/vz/images# file somedisk-flat.vmdk
somedisk-flat.vmdk: DOS/MBR boot sector


As I was having to copy out of an iSCSI-backed VMFS LUN, I had three choices for copying (all having to involve the ESXi hypervisor):

  1. Use the VMware fat clients datastore browser, and copy/pasta between the LUN and the NFS store
  2. Use cp at the hypervisor CLI between stores
  3. Use some form of scp to and from the hypervisor (either from the new PM host, or the NAS)

As most would know, ESXi doesn’t give hypervisor-level transfers like this the highest priority in the stack, so dependant on disk sizes, NAS etc, this can take a while.  Add to that any encryption (scp), and the poor little host gets a tad bogged down.  I ended up just using the fat client, at least for transfers from iSCSI to NFS (which, if I recall correctly, probably still uses SSL and hence encryption anyway).

Copy the disks into the right images/<ID> directory, and remember to rename to <whatever>.raw.

You then need to run ‘qm rescan’, and a few seconds later, you’ll see the ‘unused’ disk appear.

Now, here’s the bit that took me a little while, being used to having a GUI option available to rescan storage… you then need to run qm rescan, and a few seconds later, you’ll see the “unused” disk appear in the list of hardware for the VM!

Once done, then remove the existing 1GB disk (this needs to be done twice, as the first time just releases the disk from the VM config, the second will actually nuke it).  Then simply highlight the new disk, click Edit, and attach.

iSCSI to local ZFS

The second type of transfer I needed to do was from iSCSI to the PM box’s local ZFS storage, and for that, the quickest way was in fact to use wget and ESXi’s direct file browsing capability.  To get a usable URL, I just used a normal browser, hit my ESXi host, and clicked “Browse datastores in this host’s inventory” over on the right.  After finding the right VMDK, copy/pasta the whole URL, and use that with wget, curl or whatever you have on your PM host.

The next interesting part on ZFS is that, of course, it’s using ZFS datasets/volumes, and not a filesystem.  This means you have to direct-copy your raw disk images straight into specific Linux devices.  More on that below.

In the interim, I just copied them to somewhere local with enough room, temporarily.

Once copied, I used dd to write to the appropriate zpool /dev entry:

root@pm:/# dd if=somedisk-flat.vmdk of=/dev/zvol/rpool/data/vm-106-disk-1 bs=1M

Too easy!  Don’t forget to do a qm rescan.

Bring the awesome

At this point, I was simply able to power on each VM, and thanks to virtio drivers, and other Unixy tricks, they just booted!  Being timid about how easy that was, I did boot a few single user and run some fsck‘s, and everything was fine.

Things to note:

  • If using LVM, devmapper will just magically map existing devices and just work
  • If using UUID’s in /etc/fstab, things will just magically work
  • If using legacy /dev/sdX entries, things will probably break, and you’ll need to edit /etc/fstab to point to virtio’s /dev/vdaX entries.  There might be additional pain for your boot or root disk, and having to rebuild your initrd stuff


So far, so good, a few weeks in.  Best decision I’ve made in years with the home network stuff.

That said, I’ve encountered some rather showstopping issues around ZFS on Linux, and swap.

That said, I’ve encountered some rather showstopping issues around ZFS on Linux, and swap, that has caused the host to blindly melt and reboot twice.  I’ve made some tweaks, and hopefully that issue is no more, and will be something for a future blog.

I also have a bizarre issue with one of my two LXC containers, whereby when cleanly shutdown, the host ZFS subvol goes readonly; haven’t had any luck tracking that down yet.

I hope this blog was useful to someone.  Let me know if so, and if you have any suggestions, corrections or questions, post away!


PS4, dynamic port forwards and my MikroTik


X11 forwarding and sudo


  1. With ZFS, it is HEAVILY recommended to use ECC memory. Otherwise, you’ll experience crazy weird storage issues. ~just a thought

    • Yeah, good point. I didn’t go that route with my build, so hopefully no issues…

    • You could make that statement (‘recommended to use ECC memory’) with pretty much any filesystem. The only difference with ZFS is that it will detect, and possibly even be able to correct, the corruption.

      • Well, I think it’s a little more specific than that. If I understand right, the correction is actually on reads, not writes. Due to that, this implies that ZFS needs to know, on writes, that things went 100% smoothly.

  2. Chris

    What’s your impression of network configs with the proxmox setup?

    With vmware it’s absolutely trivial and the only thing I really like about my esxi setup. 5 clicks and I have an in-memory vlan enabled switch, which talks to my storage over a separate VLAN, separating the VM-to-VM traffic from the disk IO. Or likewise to segment my internet facing VMs from the internal ones by another virtual switch.

    Last I checked, KVM and the veneer on top of it had nothing in this regard. It was still mucking around with low level text file editing or scripting, for even the most trivial network related task (iptables, manual bridge config, standard linux kernel configs for routing etc). Complexity galore and high risk of making mistakes if one’s not a certified “cisco + RHEL admin”.

    I’m a bit concerned about the linux-zfs issues you’ve encountered. My current all-in-one esxi box has been rock solid on the storage front. Am using PCI pass-through to an old solaris (illumos) VM, which then surfaces storage to the hypervisor over nfs and iscsi. Many power outages and not a single issue, as one’s come to expect of zfs. If linux has problems with zfs, then I’ll look into porting the current scheme, run “solaris” in a K-VM with passthrough, as zfs on solaris is a proven option for data integrity. It naturally assumes however that proxmox can handle delayed storage surfacing like ESXi, which I haven’t looked into, and may not be the case.

    • Thanks for the thoughts Chris!

      Yeah, I agree, the networking side was painful. Only slightly, but a bit weird nonetheless. That said, though VMware’s is GUI, it’s pretty much just as complicated in my opinion. Many years of playing with their stuff, and I still don’t fully get the various parts (port groups etc).

      Once I had nailed the networking side on the host (using OVS), I trunk and from that point, VM networking was super simple.

      Your ZFS setup sounds sweet. I forgot to update this article — essentially, I just didn’t have enough RAM in the host. I took it up to 64GB and haven’t had an issue since. The thought of going back to Slowaris makes me shudder a little, and contemplate curling up in a ball on the floor 😉

  3. Mike

    Thanks for this post! I’ve been gradually migrating services to new VMs on Proxmox, but I still had one ESX host with VMs that would have been larger pains to rebuild than I felt like doing. I was going to look in to this process eventually but was rushed along when my ESX host crashed this morning. This post gave me the confidence to just migrate rather than rebuild ESX for the time being. The migration went excellently and everything is back up.

    For me, Proxmox has been perfect so far. I have three Intel NUCs running it with 32 GB RAM each. Mostly M2 drives, though one is SATA SSD. Also backing up to a Synology via NFS. Works great for the small home lab services I like to run.

    Anyway, thanks again!

    • Thanks heaps for taking the time to comment, appreciate it Mike 🙂

      Yeah, overall, I’m pretty happy with my setup, and Proxmox… but like anything in software these days, there are bugs and annoyances.

      My main issue currently is a terribly bad IO issue with the Samsumg 850 EVO 500GB. If I hammer it at all, the system goes into meltdown, high IO wait, and really bad IOPS. Not sure if I have a bung drive, or the ashift=12 that Proxmox chose when building the pool (I’ve read in one place that 13 is better for those drives).

      Of course, one can’t just change that in the pool — it has to be rebuilt, so haven’t gotten around to working out how I’m going to do that.

      • Funny, I have that same Samsung drive (using it for my desktop VM), and i’m getting really bad latency warnings on ESXi, too. Thinking of moving to a linux-based hypervisor. sounds like my drive is showing its age, and i might want to replace it first.

  4. Thanks Jason!

    I recently purchased a Protectli FW6C (

    It has a 2.5Ghz i5, 32GB DDR4, 2x 1TB 960 EVO SSDs and 6x Intel NICs.

    Initially thought VMware would be the ‘obvious’ choice to run on this for pfSense and various server vitualisation at the home lab.

    I couldn’t have been more wrong, VMware caused endless issues. What a pile of crap!

    Tried Proxmox, then found your post, which validates for me to stop wasting time trying to get VMware working and jump ship immediately to Proxmox. Saved me a lot of time from doing further testing or research to make that decision. Install, ZFS mirror the SSDs and should be awesome with no issues!

    • Thanks for stopping by 🙂 Yeah, one basically can’t go wrong, I reckon.

      I’ve actually since rebuilt my box, and took ZFS out. Just too many niggly issues for me, and I just feel ZoL just isn’t up to par enough yet.

  5. Jan

    How did you solve problem with having live migration and snapshots together?

    • Hi Jan, can you expand a bit on what you mean?

      • Jan

        Hi, yes I can try 🙂 I try to migrate VMware ESXi to proxmox too but I find it very difficult. VMFS type of file system has snapshot feature. I was looking for equivalent in proxmox but it’s hard to find a proper solution. My configuration is FC SAN + LUNs + LVM as a shared storage. Live migration works like a charm but I don’t have snapshot feature which I really need. That’s why I asked, what is your solution for having for features? Or maybe you don’t use snapshots…

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress & Theme by Anders Norén

%d bloggers like this: