Vmware ESXI Nested Virtualization

This post was originally published on September 9, 2015 and updated November 14, 2019.

One of the benefits of nested virtualization is running multiple hypervisors on the same host server. This includes ESXi hypervisors , which can be installed as a virtual machines. The solution, commonly referred to as Nested ESXi, opens the door to a lot of interesting use cases such as development and testing, virtual training, or virtual sales demos and POCs.

Although the concept of nested virtualization is not limited to VMware, it is by far the most popular hypervisor. For that reason I decided to focus my article on the VMware hypervisor.

In this article, I am going to answer some common questions, discuss performance and common use-cases, provide a short walkthrough on how to enable a Nested ESXi and provide a list of CloudShare’s preconfigured ESXi environments.

Common Questions

I bet that when you first encountered nested virtualization, some of the first questions to pop into your head were most likely along of the lines of:

– “Why would you ever want to do such a scary thing?”
– “Isn’t normal Virtualization good enough anymore?”
– “It’s going to be extremely slow, isn’t it?!”

Before we move forward, let me answer your [burning] questions:
– Great benefit in certain use cases!
– It’s great!! But not for all scenarios.
– Potentially it would be a little slower, but not much if you do it right, and I’ll explain more next.
Why should I use this solution? What use cases is nested virtualization good for?

Practice an Upgrade Procedure of your VSphere Infrastructure

Suppose you want to upgrade your entire production VSphere infrastructure. What’s the catch? You’ve never done this before. What better way of experimenting than doing this on a nested environment? Performing the upgrade on a nested environment could save you the trouble and downtime of the production infrastructure, especially if you don’t have ‘spare’ ESX hosts to play with. Plus, you can enjoy all the added features that come along with a cloud based virtual IT platform such as snapshotting current state and reverting back to previous state.

Test/Development Environments

Suppose you or your company is building a new solution and you need a virtual IT lab to rapidly create and provision environments for build verification, test automation and/or manual testing. You can deploy Nested ESXi as a test/development platform without spending money on hardware and without changing any of your in-house tools or processes. By creating Nested ESXi templates you can easily deploy and tear down vSphere instances as needed during development and test cycles.

Virtual Sales Demos

What better way of selling your VMware vSphere solution than showing a live virtual demo of your product? Your sales staff can deploy a Nested ESXi demo template while on site with the customer and showcase its features, without the costs of expensive hardware/datacenter cabling and gear. For a better support experience, your support staff can deploy a Nested ESXi lab environment to reproduce customer issues or resolve bugs within your software.

Virtual Education and Training

Once your developers have finished building the product, you will need a way to educate your staff on the VSphere product Family and its features. You can provide hands on virtual training environments using the same Nested ESXi template used during development. The template can be used for virtual classroom activities or labs and deployed on a per student basis.  Using CloudShare’s Virtual IT lab environments,  you can deploy the virtual classroom only when needed and pay only for what you use thus eliminating high costs on infrastructure. If you convert the cloud-based training for external use, you can utilize public IP addresses to enable your customers to access the course anywhere in the world, while enjoying all the VMware Features (Students can Snapshot/ revert / clone ETC)

Walkthrough

Let’s try it ourselves:ESXi 5.x Virtual machine inside a psychical ESXi 5.x.

There are two ways to enable nested virtualization support:

1.  Via the nested machine’s .vmx file:
o VSphere 5.0 = “vhv.allow = true” (or in /etc/vmware/config)
o VSphere 5.x = “vhv.enable = true”.

 

2. Via VCenter Web Client (the recommended way):

Also, to enable Network Communication, make sure to enable “promisc” on the nested machine’s nic.

This is done in order to overcome VMware’s default behavior, which by default only forwards packets from the v-switch to VM’s Mac address.

If you like, Optional VMware Tools for nested ESXi 5.x could be downloaded from VMware ‘flings’:
https://labs.vmware.com/flings/vmware-tools-for-nested-esxi

What about Performance?

Suppose you run a ‘regular’ Virtual Machine. Technically, you are running a process in the hypervisor. If you have 10 VMs in the hypervisor you have 10 processes, and so on. The more processes you have, the more time you spend on context-switching and physical processor scheduling and ready time will increase, which will result in potentially less performance.

Now imagine that one of those VM processes ‘hides’ inside 10 mores VMs (nested). Will it be slower? Yes.

Much slower? Not necessarily. It all depends on your devices and configurations.

To prove my point, I ran a small test, and the nested machine got 10-15% less ‘overall performance’ score, compared to its parent machine.  My conclusion? If you don’t plan on doing production on those nested machines, the benefits far exceed the minor performance price tag.

Pre-configured ESXi Environment templates

While each configuration is likely unique, most of the time and hassle is on the front end: meaning, setting up a base image with nested virtualization. We saved all this hassle for you and created a number of nested ESXi environments for development and testing, virtual training and virtual sales purposes. These templates are a huge time saver – the environment is set up once, and once it’s saved as a blueprint, you can run tests, and save each result as it spins up and down.

The available templates include:

• VMware ESXi server 6.0
  Windows Server 2008 R2 x64 with vSphere 6.0 client Unlicensed.
• ESXi 5.5.0U2 Env
  VMware ESXi server 5.5.0U2
  Windows Server 2008 R2 x64 with vSphere 5.5.0U2 client Unlicensed.
• ESXi 5.1 Env
  This ESXi 5.1 environment contains two machines: a VMware ESXi server 5.1 & a Windows Server 2008 R2 x64 with vSphere 5.1 client. vSphere is unlicensed.
• ESXi 5.5 vCenter
  Full vCenter Environment, 2 ESXi 5.5 – Unlicensed. 1 vCenter and vSphere 5.5 Client – Unlicensed. 1 Windows Active Directory 2012

The system image includes nested ESXi and VCenter, storage communication, and a top of the line Linux box with bridge controls.

How do you access these environments?

If you are not yet a CloudShare user, simply select the ESXI 6.0Env from the VM template library page, select your desired plan and… voila, your environment will be ready in a few minutes.
Or, you can create your account and select the pre-configured VM template from within the CloudShare virtual IT lab platform.

So nested virtualization? VERY COOL!

Editor’s Note – This post was originally published in and has been completely revamped and updated for accuracy and comprehensiveness.