Tri-boot Windows, Linux and VMWare ESXi within a single system

Background:

  When it comes to virtualization there are several choices of hypervisors in the market. As a solution consultant, I often play (and work) around with different hypervisors. I have been working on a solution to let me install multiple hypervisors in the same system and change to it whenever necessary. Finally, I have come up with the way to install Windows Server 2016, Ubuntu Server 17.04 and VMWare vSphere 6.5 hypervisor within a PC and boot them whenever I need remotely.
  My solution has the below advantages over other straight-forward solutions:

1. No BIOS / UEFI-level changes are required, including boot sequence interruption during system POST.
2. Does not require support from remote console hardware (e.g. HPE iLO or similar technologies)

  This solution will have the below characteristics:

1. It is rely on GRUB 2 boot loader, and the "primary" operating system is the Ubuntu Server
2. It requires at least two physical hard drives. The first one (i.e. the first hard drive in the BIOS boot sequence and it is suggested to be the hard drive connected to the port labelled as SATA 0 on your motherboard) will be installed with Windows and Linux accordingly and the other one will be used entirely for VMWare ESXi (which is the behavior of VMWare). USB drive installation for ESXi should work but I did not test it.
3. As of writing this blog the solution is only tested on UEFI system with CSM support. All the OSes are installed in MBR partition table (instead GPT). I am not sure if a native UEFI system with GPT partition table will work or not.

Procedures:

Overview:

  The three operating systems should be installed in the order of Windows, Ubuntu and VMWare. This will allow Grub (the default Ubuntu boot loader) to detect the Windows installation. VMWare should be installed to a separate disk drive, otherwise it will wipe all existing partitions of the disk including your installed operating system. Since grub-update will not detect ESXi installation and the ESXi boot loader on the other disk will not affect Grub, the order of Ubuntu and VMWare installation isn't that important.  I use a USB drive as the installation media of all three operating systems and did not encounter any issues.

Task 1, Windows Installation

  Points to note for installing the first operating system, Windows:

• You may consider to create a MBR partition table first using third party tools (e.g. Linux live CD) first to ensure Windows is installed with MBR. In MBR partition table, Windows installer will create two partitions for MBR partition table. If GPT is used the installer will create four partitions and below tasks may fail.
• Remember to partition the disk before starting the installation

Task 2, Ubuntu Installation

• I would suggest to use manual mode for creating partitions. I used extended partitions for all mount points (including the root "/" partition and swap area).
• The Windows boot loader will be replaced with Grub and it can only be restored (say, in case you change your mind) using Windows recovery media.
• The Windows installation is labelled as "Windows Recovery Environment" but it will behave as a standard Windows instance. It is a auto-probing feature of Grub and customization of the name is not covered (be warned that you should not directly edit the grub.cfg file for changing the name, it will not work and may have other negative impact)
• After Ubuntu is successfully installed, please try to boot into both Windows and Ubuntu environment. This is a straight-forward setup and does not differ from common guidelines on the web. 
• If you are going to install additional drives for virtualization, please use UUID in your /etc/fstab for the sake of consistency

Task 3, ESXi Installation

• There is no need to physically unplug your Windows/Linux hard drive before continuing
• During the loading screen, press "Shift+O" (letter "Oh", not number zero) when prompt. Append "formatwithmbr" to the boot command so the full command should look:

      runweasel formatwithmbr

•  After installation the system will reboot and you will not see anything changed in the Grub menu. Test your ESXi installation by altering BIOS boot sequence to the hard drive you have chosen for ESXi installation and try to boot into ESXi. 

Task 4, Customizing Grub

  Please note that the below steps involve modifying your Grub boot loader settings and may fail booting to any operating system if you have typos. Please make a backup of grub.cfg and prepare for rescue procedures.

1. Boot to the Ubuntu instance, edit the custom grub entry by using vi, the file should already exists:
   

      # sudo vi /etc/grub.d/40_custom

2. Append the following lines at the end of the file:

      menuentry "ESXi 6.5" {
        search --set=root --file /mboot.c32
        chainloader +1
      }

The above text block will create a new menu entry "ESXi 6.5" (you can change this) by searching the partition with file /mboot.c32, which is the VMWare boot file under MBR architecture, and add the partition to the chainloader. Information on the Internet leads you to explicitly specify the drive and partition number which will fail booting ESXi with an error of "Invalid Signature". I don't know why this method will work but not other suggestions found on the Internet and hopefully someone can explain to me.

3. Remember to update Grub before continuing

         # sudo update-grub

4. If you have been following this guide the newly created entry (which is not shown as part of update-grub returned message) would be index 3 (the first menu entry is index 0) and you can run the following command directly to tell grub to boot to ESXi.

      # sudo grub-reboot 3 && sudo reboot

5. Now you can see the beauty of this solution being able to (remotely) instruct the computer to boot to other operating systems (including Windows Server and ESXi) by running a simple command. This is also the reason why we need Ubuntu and Grub being the choices of default operating system and boot loader.

Closing thoughts

 I can use this method on a Kernel VM  under Ubuntu, you may also try this in virtualized environment before implementing it on your physical hardware because any failure during the configuration would leads to excessive reboots which may damage / affect the lifespan of your hardware. 
 While this solution works well under MBR partition table, I did not test for GPT and hopefully it is possible to do the same with minor modification. It is a big step forward to me because I can now remotely instruct the computer to reboot into other operating systems. The PC I use is also fitted with four additional hard drives for storing virtual machines images from those hypervisors. If you also want to do the same I suggest you to create 3 partitions on each disk you add formatted with NTFS, ext4 and VMFS, so that you can utilize all spindles when you are in one of the available hypervisors.

Lazy sysprep without answer file

Background:

  There exist many ways to sysprep a system under a KVM system, but I only need a minimum set of configurations (like to Start Menu items, show all known file types by default and a default user account. There is a very simple way (without using an unattended.xml answer file) to do this.
This method is tested on a Windows 10 Enterprise Edition. It will sysprep the operating system but will skip the user account creation process during next First-Time installation wizard, so that the account you used during the installation process is still valid. I tried using Windows Server 2016 but no settings (including account information, start menu items and default views) were preserved after the initialization.

Procedures:

  First install the operating system as usual, then run the below command on an evaluated command prompt. Note the "/quit" switch different from most of the other guidelines on the web. We need to edit the registry so we still need the system up and running after sysprep.

sysprep /generalize /oobe /mode:vm /quit

Following the completion of the sysprep task, run "regedit" and add the below DWORD (32 bit) value. Set it to 1.

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\OOBE\UnattendCreatedUser

I am using KVM as the hypervisor, so after the guest is shut down I just copy the qcow2 disk file and dumpxml as the template xml. The creation of template can be done using:

virt-clone --original-xml /template/xml/path --name name_of_new_vm --file /new/qcow2/image/name