Migrating ESXi to Proxmox VE with PCI Passthrough

I recently migrate one of the old servers running VMware Hypervisor to KVM-based solution.

Installing Proxmox Virtual Environment

Installing Proxmox Virtual Environment (Proxmox VE) is as easy as installing Debian. Just download the ISO image without any registration, create a bootable USB and boot. I actually use Ventoy because it claims to boot Debian and it worked seamlessly.

But I've got failure after some simple configuration - the installer told me:

hardisk /dev/sdk too small (8GB)

I installed VMware Hypervisor on a 8GB USB key and it has been working well. As I have other storage for VM data, there is no need for over 8GB for a minimal hypervisor system to store caches. I searched, and I found related topics blaming the limitation.

Install proxmox in a 8GB USB not possible
Hi there, I’m trying to install proxmox in a USB key. I tried using an 8GB key which is recognized as 7.2GB and proxmox (both 3.4 and 4.0) complain that it’s not big enough. This is weird because the proxmox 3.4 that I have installed at the moment in a hard drive is only using 1.3GB of disk...
Can you lower the installation disk limit from 8GB to 7GB ?
Hello, My request is related to this thread: http://forum.proxmox.com/threads/24718-Install-proxmox-in-a-8GB-USB-not-possible I tried installing proxmox in two different 8GB usb and I can’t because of this limit. Could you lower this limit to 7GB ? Thanks.

And I eventually solve this problem in a simple way: I found any other Sandisk Cruzer Fit but provides 16GB. This time the installation went well without any extra configuration.

Passing-through LSI HBA Card

First of all, enable iommu and setup VFIO for the host:

More to read:

Pci passthrough - Proxmox VE

For passing-through an LSI HBA Card, some options are required for the virtual machine in Proxmox VE:

...
hostpci0: 01:00.0,pcie=1,rombar=0
machine: pc-q35-3.1
....
Special options in <VMID>.conf
References: Bug 243640 - QEMU / KVM Q35 V4.X PCIe Virtual and Physical (Passthrough) Devices not detected  - Comment 5

Fix IOMMU Groups

After the storage VM is configured, I continued to boot and found the network lost. After some digging, I found that whenever I boot the VM the network connection is down.

Follow the "Verify IOMMU Isolation" part in the official wiki:

Pci passthrough - Proxmox VE

Fix CentOS Ramdisk Drivers

For RHEL, CentOS and other based distro, dracut will generate initramfs containing only the drivers for hardware on host. But we can use dracut to re-generate ramdisk with virtio drivers to make the system bootable:

sudo dracut --add-drivers "virtio_pci virtio_blk virtio_scsi virtio_net virtio_ring virtio" -f -v /boot/initramfs-`uname -r`.img `uname -r`

Then power off and start the VM off, the drivers should load and system will boot.

References: Re: SCSI generic driver not loading at boot
Sakuragawa Asaba

Sakuragawa Asaba

A designer and photographer. Yet interested in cloud and opensource solutions.