Tri-boot Windows, Linux and VMWare ESXi within a single system

Background:

  When it comes to virtualization there are several choices of hypervisors in the market. As a solution consultant, I often play (and work) around with different hypervisors. I have been working on a solution to let me install multiple hypervisors in the same system and change to it whenever necessary. Finally, I have come up with the way to install Windows Server 2016, Ubuntu Server 17.04 and VMWare vSphere 6.5 hypervisor within a PC and boot them whenever I need remotely.
  My solution has the below advantages over other straight-forward solutions:

1. No BIOS / UEFI-level changes are required, including boot sequence interruption during system POST.
2. Does not require support from remote console hardware (e.g. HPE iLO or similar technologies)

  This solution will have the below characteristics:

1. It is rely on GRUB 2 boot loader, and the "primary" operating system is the Ubuntu Server
2. It requires at least two physical hard drives. The first one (i.e. the first hard drive in the BIOS boot sequence and it is suggested to be the hard drive connected to the port labelled as SATA 0 on your motherboard) will be installed with Windows and Linux accordingly and the other one will be used entirely for VMWare ESXi (which is the behavior of VMWare). USB drive installation for ESXi should work but I did not test it.
3. As of writing this blog the solution is only tested on UEFI system with CSM support. All the OSes are installed in MBR partition table (instead GPT). I am not sure if a native UEFI system with GPT partition table will work or not.

Procedures:

Overview:

  The three operating systems should be installed in the order of Windows, Ubuntu and VMWare. This will allow Grub (the default Ubuntu boot loader) to detect the Windows installation. VMWare should be installed to a separate disk drive, otherwise it will wipe all existing partitions of the disk including your installed operating system. Since grub-update will not detect ESXi installation and the ESXi boot loader on the other disk will not affect Grub, the order of Ubuntu and VMWare installation isn't that important.  I use a USB drive as the installation media of all three operating systems and did not encounter any issues.

Task 1, Windows Installation

  Points to note for installing the first operating system, Windows:

• You may consider to create a MBR partition table first using third party tools (e.g. Linux live CD) first to ensure Windows is installed with MBR. In MBR partition table, Windows installer will create two partitions for MBR partition table. If GPT is used the installer will create four partitions and below tasks may fail.
• Remember to partition the disk before starting the installation

Task 2, Ubuntu Installation

• I would suggest to use manual mode for creating partitions. I used extended partitions for all mount points (including the root "/" partition and swap area).
• The Windows boot loader will be replaced with Grub and it can only be restored (say, in case you change your mind) using Windows recovery media.
• The Windows installation is labelled as "Windows Recovery Environment" but it will behave as a standard Windows instance. It is a auto-probing feature of Grub and customization of the name is not covered (be warned that you should not directly edit the grub.cfg file for changing the name, it will not work and may have other negative impact)
• After Ubuntu is successfully installed, please try to boot into both Windows and Ubuntu environment. This is a straight-forward setup and does not differ from common guidelines on the web. 
• If you are going to install additional drives for virtualization, please use UUID in your /etc/fstab for the sake of consistency

Task 3, ESXi Installation

• There is no need to physically unplug your Windows/Linux hard drive before continuing
• During the loading screen, press "Shift+O" (letter "Oh", not number zero) when prompt. Append "formatwithmbr" to the boot command so the full command should look:

      runweasel formatwithmbr

•  After installation the system will reboot and you will not see anything changed in the Grub menu. Test your ESXi installation by altering BIOS boot sequence to the hard drive you have chosen for ESXi installation and try to boot into ESXi. 

Task 4, Customizing Grub

  Please note that the below steps involve modifying your Grub boot loader settings and may fail booting to any operating system if you have typos. Please make a backup of grub.cfg and prepare for rescue procedures.

1. Boot to the Ubuntu instance, edit the custom grub entry by using vi, the file should already exists:
   

      # sudo vi /etc/grub.d/40_custom

2. Append the following lines at the end of the file:

      menuentry "ESXi 6.5" {
        search --set=root --file /mboot.c32
        chainloader +1
      }

The above text block will create a new menu entry "ESXi 6.5" (you can change this) by searching the partition with file /mboot.c32, which is the VMWare boot file under MBR architecture, and add the partition to the chainloader. Information on the Internet leads you to explicitly specify the drive and partition number which will fail booting ESXi with an error of "Invalid Signature". I don't know why this method will work but not other suggestions found on the Internet and hopefully someone can explain to me.

3. Remember to update Grub before continuing

         # sudo update-grub

4. If you have been following this guide the newly created entry (which is not shown as part of update-grub returned message) would be index 3 (the first menu entry is index 0) and you can run the following command directly to tell grub to boot to ESXi.

      # sudo grub-reboot 3 && sudo reboot

5. Now you can see the beauty of this solution being able to (remotely) instruct the computer to boot to other operating systems (including Windows Server and ESXi) by running a simple command. This is also the reason why we need Ubuntu and Grub being the choices of default operating system and boot loader.

Closing thoughts

 I can use this method on a Kernel VM  under Ubuntu, you may also try this in virtualized environment before implementing it on your physical hardware because any failure during the configuration would leads to excessive reboots which may damage / affect the lifespan of your hardware. 
 While this solution works well under MBR partition table, I did not test for GPT and hopefully it is possible to do the same with minor modification. It is a big step forward to me because I can now remotely instruct the computer to reboot into other operating systems. The PC I use is also fitted with four additional hard drives for storing virtual machines images from those hypervisors. If you also want to do the same I suggest you to create 3 partitions on each disk you add formatted with NTFS, ext4 and VMFS, so that you can utilize all spindles when you are in one of the available hypervisors.

Lazy sysprep without answer file

Background:

  There exist many ways to sysprep a system under a KVM system, but I only need a minimum set of configurations (like to Start Menu items, show all known file types by default and a default user account. There is a very simple way (without using an unattended.xml answer file) to do this.
This method is tested on a Windows 10 Enterprise Edition. It will sysprep the operating system but will skip the user account creation process during next First-Time installation wizard, so that the account you used during the installation process is still valid. I tried using Windows Server 2016 but no settings (including account information, start menu items and default views) were preserved after the initialization.

Procedures:

  First install the operating system as usual, then run the below command on an evaluated command prompt. Note the "/quit" switch different from most of the other guidelines on the web. We need to edit the registry so we still need the system up and running after sysprep.

sysprep /generalize /oobe /mode:vm /quit

Following the completion of the sysprep task, run "regedit" and add the below DWORD (32 bit) value. Set it to 1.

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\OOBE\UnattendCreatedUser

I am using KVM as the hypervisor, so after the guest is shut down I just copy the qcow2 disk file and dumpxml as the template xml. The creation of template can be done using:

virt-clone --original-xml /template/xml/path --name name_of_new_vm --file /new/qcow2/image/name

Bridge Unencrypted Mosquitto MQTT Broker to AWS IoT

Background:

  The AWS IoT implementation is very secure, which is protected by X.509 certificates. This makes testing with generic smartphone clients impossible because most of the generic MQTT clients supports user and password authentication with SSL only. I am not going to provide step by step guide on how to setup the Mosquitto server as well as connecting to AWS IoT device, please refer to the reference links (#1 and #2) to understand the procedures and use this guideline as a supplementary note.

Mosquitto server:

- I use AWS EC2 platform with Ubuntu AMI for installation of Mosquitto server, but it requires an authentication key pair to access the console by putty and uploading files. If you are not familiar with using EC2 platform, you may use other VPS services or your own infrastructure (requires inbound ports 1883 and 8883). Also I did not follow the steps in Reference #1, and refer to #2 for installation in the Ubuntu AMI.

- In my case the latest repository is required to use mosquitto_passwd command to create password for the user, simply run the two commands before installation Mosquitto as instructed in #2.

sudo add-apt-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update

- I skipped all the encryption steps in reference #2 as I only want to have a quick try. In fact I only follow Step 1 and Step 5 in the link.

Configuring the bridge:

- I follow #1 with minor changes to the suggested configuration file:
In the suggested config:

topic awsiot_to_localgateway in 1
topic localgateway_to_awsiot out 1
topic both_directions both 1

In my version, only one topic is forwarded and in both directions for easier troubleshooting:

topic test both 1
topic test/+ both 1

- I use FileZilla to transfer the certificates including the root CA cert. to the Mosquitto server, the key pair for EC2 instance access is required here.

- During the first run, I suggest to take out the "-d" switch because it is easier to troubleshoot.
Suggested command:

sudo mosquitto -c /etc/mosquitto/conf.d/bridge.conf –d

In my version:

sudo mosquitto -c /etc/mosquitto/conf.d/bridge.conf

Results:

Tested Android MQTT client
MQTT IoT Client: This is a simple client but will get the job done for the testing of the bridge.
Mqtt IoT: In case you want to play around with more controls.

Disclaimer: I have no connection with the developers of above apps and just randomly picked them for my testing. Other MQTT clients should work as long as they support user / password authentication to the Mosquitto server

Reference links:
Reference #1:https://aws.amazon.com/blogs/iot/how-to-bridge-mosquitto-mqtt-broker-to-aws-iot/
Reference #2:https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-the-mosquitto-mqtt-messaging-broker-on-ubuntu-16-04

Create Single Node Ceph Host with S3 connectivity

Step 1, create a Ubuntu virtual machine from the physical host (which also runs Ubuntu in my case)

- Note that a total of 4 disks are required for a single node Ceph installation. In my case, /vmdisk0, /vmdisk1 and /vmdisk2 are three mount points from different physical disks.
- According to the guide, 1TB is the minimum disk size allowed for Ceph installation, by adding sparse=true and --force options in the command over-committed virtual disk file will be created with warnings.(In fact /vmdisk1 and /vmdisk2 is backed by two 320GB hard drives only)
- Install Ubuntu as you would normally do, suggested to install OpenSSH server for easier installation at later stage.

sudo virt-install --name=ceph-single-node --vcpus=2 --cpu core2duo --ram=2048 \
--memballoon virtio --os-variant=ubuntu16.04 \
--boot hd,cdrom,menu=on --description "Guest created for Ceph S3 object storage" \
--disk path=/vmdisk0/ceph-single-node-d0.qcow2,device=disk,format=qcow2,bus=virtio,sparse=true,size=8 \
--disk path=/vmdisk0/ceph-single-node-d1.qcow2,device=disk,format=qcow2,bus=virtio,sparse=true,size=1024 \
--disk path=/vmdisk1/ceph-single-node-d2.qcow2,device=disk,format=qcow2,bus=virtio,sparse=true,size=1024 \
--disk path=/vmdisk2/ceph-single-node-d3.qcow2,device=disk,format=qcow2,bus=virtio,sparse=true,size=1024 --force \
--disk path=/share/sources/ubuntu.iso,device=cdrom,format=raw,bus=ide \
--network network=net_public,model=virtio \
--network network=net_private,model=virtio \
--graphics vnc,listen=0.0.0.0 --noautoconsole

Step 2, upgrade your guest with the latest patches. Remember to restart your guest after upgrade completes.

sudo apt update && sudo apt upgrade

Step 3, install the repository key and perform installation. You may want to use SSH to connect to your guest for easier copy and paste of below commands.

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
echo deb http://download.ceph.com/debian-jewel/ trusty main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update && sudo apt-get install ceph-deploy

Step 4, create a user ceph-deploy for use onward.

sudo useradd -m -s /bin/bash ceph-deploy
sudo passwd ceph-deploy

- enter the password for the new user "ceph-deploy". Consider a more complex password because this user will have full sudoer rights.

echo "ceph-deploy ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-deploy
sudo chmod 0440 /etc/sudoers.d/ceph-deploy
sudo su - ceph-deploy

Step 5, below commands are run under the newly created user "ceph-deploy". Just hit "Enter" key when you are asked to enter the directory and passphrase of the new ssh key.

ssh-keygen
ssh-copy-id ceph-deploy@ceph-single-node

cd ~
mkdir my-cluster
cd my-cluster

Step 6, this is not included in the reference. I need to edit the host file before continuing (otherwise it will result in error of not being able to resolve the hostname of "ceph-single-node").

sudo vi /etc/hosts

- Comment out by adding "#" in front of the line, like this:

# 127.0.1.1     ceph-single-node

Step 7, continue with deploying the single node ceph.

ceph-deploy new ceph-single-node

Step 8, this is also different from the reference, the filename was changed. Edit the file below:

vi ceph.conf

- Add the following two lines. The first line told the system to keep one copy of the replica. (The reference suggested 2, but I think 1 is better for testing purpose)
- The second line is required to allow Ceph to run on single node.

osd pool default size = 1
osd crush chooseleaf type = 0

Step 9, install the ceph binaries, this will take some time to download the required files. Create a monitor right after installing the binaries.

ceph-deploy install ceph-single-node
ceph-deploy mon create-initial

Step 10a, configure the three virtual disks for use with Ceph, note that all three disks are untouched until now and they should contain no partition tables.

ceph-deploy osd prepare ceph-single-node:vdb
ceph-deploy osd prepare ceph-single-node:vdc
ceph-deploy osd prepare ceph-single-node:vdd

Step 10b, activate the partitions.

ceph-deploy osd activate ceph-single-node:/dev/vdb1
ceph-deploy osd activate ceph-single-node:/dev/vdc1
ceph-deploy osd activate ceph-single-node:/dev/vdd1

Step 11, redistribute the config and keys and modify the access rights.

ceph-deploy admin ceph-single-node
sudo chmod +r /etc/ceph/ceph.client.admin.keyring

Step 12, test the health of the Ceph deployment

ceph -s

- If the health reports error, try setting pg_num and pgp_num to higher number. I cannot repeat the error I faced in the first time. Maybe it's some typos for me but just keep it as notes.

# ceph osd pool set rbd pg_num 90
# ceph osd pool set rbd pgp_num 90

Step 13, install object storage gateway and CephFS.

ceph-deploy rgw create ceph-single-node
ceph-deploy mds create ceph-single-node

Step 14, create a user for S3 connectivity. Take note on the access_key and secret_key generated.

sudo radosgw-admin user create --uid="testuser" --display-name="First User"

- Install s3cmd and configure it, you will be asked for the access_key and secret_key to proceed.

sudo apt-get install s3cmd
s3cmd --configure

Step 15, configure the s3cmd config file to point to our server.

vi .s3config

- Replace the IP address with your ceph-single-node ones. If you want to access it from the Internet, just create a TCP NAT rule to your IP with TCP port 7480.

host_base = 192.168.1.11:7480
host_bucket = %(bucket)s.192.168.1.11:7480

Step 16, you are done with configuration. Now we will do some testing.

s3cmd mb s3://testdrive

echo "Hello S3 World" > hello.txt
s3cmd put hello.txt s3://testdrive
s3cmd ls s3://testdrive

Reference: http://palmerville.github.io/2016/04/30/single-node-ceph-install.html

(Relatively) High CPU usage of pfSense within virtualized environment

Symptom:

- Apply to pfSense starting version 2.0 and still seen in version 2.3.3 (I have no idea about older builds)
- Issue occurs in Kernel-based VM and VirtualBox, I have not tested in other virtualization platform as well as a pfSense physical box - Relatively high CPU utilization is observed through the hypervisor level (e.g. Linux "top" command in Kernel-based VM)
- Depending on virtualization platform, the issue may not be easily noticed. The most significant scenario for me is a VirtualBox deployment within Windows 10 on an Intel Core i5-2510M which the pfSense will utilize a full core (25% CPU utilization observed from Windows Task Manager)
- Changing the virtualized hardware architecture doesn't help (e.g. changing CPU emulation type, changing between standard hardware or virtio drivers)

Causes:

- The system timer of the pfSense is drawing excessive CPU time, interestingly the utilization cannot be observed within the pfSense (either dashboard or other monitoring pages), it is only the hypervisor host who reports the high CPU utilization. This leads to a long troubleshooting time as one might not convinced that this is a guest-only issue.
- Seems the root cause relies on FreeBSD but I did not confirm this.

Resolution:

- Below resolution will resolve the high CPU utilization (reduce from 25% CPU utilization observed above in Core i5-2510M environment down to 3% with no additional changes made) but will result in lower response time. However I did not notice any difference in daily usage (as a normal home user).

- Add the below lines to /boot/loader.conf.local within the pfSense environment
(Enter the Shell by SSH to the pfSense machine with admin account and Enter "8")

# disable CPU frequency/voltage throttling control
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
# disable local APIC timers
hint.apic.0.clock=0
kern.hz=100
# saves 128 interrupts per second per core at the cost of reduced scheduling precision
hint.atrtc.0.clock=0

- The pfSense requires a restart to make the changes effective Reference: