Friday, September 30, 2011

vSphere 5: vCenter Evaluation license expire

So you have been running the evaluation license.  The evaluation period would be 60 days for ESXi server and 30 days for vCenter.

Say you like to run on evaluation while getting your license and you couldn't get the license in time and 30 days has reached and your vCenter is expired.

What happens after the evaluation period?

I did a test and these are my findings.
  1. vCenter remain to be operational.
  2. All features are still enabled.
  3. You will receive a notice below once in awhile.
  4. Any action that result in disconnection of the ESXi servers will not be able to join back to vCenter e.g. reboot of ESXi server.  A direct connection using vSphere client is required to manage the VMs on the host.
  5. Any changes in configuration will not be possible and can cause any licensed functions not to function.
  6. Powered Off VMs cannot be power on.  Refer to KB.
In summary, after license expired, all operations remains as it is and no changes even with simple things like power on VM will not be possible.  This is a soft limit and not a hard limit.

Update 15th Nov 2013
Added point 6 and summary.

vSphere 5: HA Demonstration

Some people asked me this question:

Will HA be activated if datastore is lost on a host due to iSCSI connection failure or FC connection on just one host without losing the host server from the management network?

Though I know the answer however I was curious myself and decide to find out by performing this on a iSCSI connection to see the results.

From my understanding, HA protects hosts failure however as long the host primary detection method (management network) is still alive, the host is considered alive in such, even though the VMs that the host is hosting is "gone" due to perhaps a nic failure for iSCSI connection or a HBA failure for FC connection, HA will not be activated even though datastore heartbeat (only in vSphere 5.x and its a secondary detection if primary failed) is lost.

I realize vCenter will initialed a migration of the VM however this will get the process halt.  We have to kill the process and wait for a timeout in order to take over the VM again else no options will be available for control.

In conclusion, HA protects hosts failure and not device failure on host.

Troubleshooting virtual machines that loses disk access

Update 14 Nov 2011
What if you meet with a power outage?  All hosts were shutdown?  The answer is posted on Yellow Bricks.

Basically if it was not powered off gracefully it will be power on the VMs base on the protected list after the election process of electing a Master is completed.

Thursday, September 29, 2011

vSphere 5: Cluster with mixed ESX/ESXi version

Many have asked me whether if a cluster can be mixed with different versions of ESX/ESXi servers.  The answer is yes even down to ESX/ESXi 3.5, however for version 3.5, you would need to cater the legacy license server.  Please refer to the documentation here.

Here is a demo of a setup I did with ESX 4.1 and ESXi 5.0 in the same cluster with HA and DRS enabled managed using a vCenter 5.0.

Please note that upgrading VMware Tools to the latest version will still be supported on lower version of ESX/ESXi servers as it only update the OS drivers.  However, upgrading the virtual hardware will only allow it to be supported on the latest ESXi server.

In such, that in consideration when doing your migration and upgrading of your vSphere environment and perform the virtual hardware upgrade last if possible unless you have enough resources to cater for your HA.

Update 19th Apr 2013
Refresh the video as first half has missing audio.

Saturday, September 24, 2011

Promotion code for certification

Promo code for 25% discount.
Valid till 30th Nov



Thursday, September 22, 2011

vSphere 5 whitebox: ESXi upgrade with USB installer

Finally took some time to upgrade my ESXi whitebox which is running on ESXi 4.1 with custom driver to support the onboard Marvell LAN chipset.

The installation screen was different in this case and I took some pictures using my mobile.  My CD Rom happen not to work and I need to create a ESXi installer onto an UDB Drive to install on another USB Drive.

First I would just go through the tools I used to create a USB installer.

I use unetbootin version 5.55 and use the ESXi 5.0 installable image to create the USB installer.
This time it works.  Unlike in ESXi 4.1, it has to be using version 4.94, any higher version breaks the installation.

My ESXi 4.1 was installed on a 2GB USB disk.  So with ESXi 5.0 which required 4GB scratch disk, it is definitely not possible.

I did some google of cause and found USBIT.  What it does is, it creates a backup (clone image) of my 2GB disk and restore to my 8GB disk.  That boots up proper.  But my 2GB disk cannot be used as it is only seen with 4MB of space.  I have to use a HP USB Storage Format tool to format back to a normal 2GB disk.

Here you can see my 8GB USB disk purchased just to install ESXi 5.0 as the scratch disk required 4GB.

Choosing install will start the gathering information process.  After which I will be presented with either to Install or Force Migrate*.  This happens when you have custom VIBs on your current ESX server when performing upgrade.
After choose Force Migrate*, it will display the customer VIBs warning which there is no replacement in the ESXi 5.0 installer I am using. (You will not encounter this if you have create a new ESXi 5.0 image with customer VIBs using image creator)

A confirmation screen will be shown again that you cannot downgrade from ESXi 5.0 (why would I upgrade if I want to downgrade? Hmm...)

After upgrade installation is complete I was shown with the screen and everything works fine and I proceed to upgrade my vCenter using the steps from my previous post as my vCenter is a VM sitting on top of this whitebox, so I cannot use Update Manager to upgrade my ESX server.

Just for fun.  Do you like the default yellow and black display or a white and black display?
Use the keyboard F4, to play around.

Update 23 Jan 2013
Did my upgrade from ESXi 5.0 to 5.1.  UNetbootin 5.83 can be used to place the ESXi 5.1 Installer into a USB Drive.

Saturday, September 17, 2011

vSphere 5: VAAI features round up

VMware Blogs actually have a 4 parts series on the Storage features in vSphere 5.

Was reading up the VAAI thin provisioning alert and reclaimation and the differences on VMFS volumes and guest portion have me came across this article by Hitachi.

It explain in very simple terms which I felt was good to understand.  Those who are not as storage trained like myself would appreciate this portion.

Just to recap where we are in the lifecycle of VAAI, vSphere 4.1 introduced three primitives: XCOPY, ATS (Atomic Test & Set) & WRITESAME. vSphere 5.0 adds two new primitives: Thin Provisioning-STUN (TP-STUN) and UNMAP, bringing us to a total of 5 primitives.

Here is more detail about these new primitives and what they mean to the VMware & storage admins:

Thin Provisioning STUN – This is an Error Code to report “Out of Space” for a thin volume. If a storage array supports this SCSI command, then when a datastore reaches 100% capacity and any of the VM’s require additional blocks of storage, the ESXi host will receive the alert and will pause those VM’s while the other VM’s will continue to run. When using a Hitachi Dynamic Provisioning pool, however, administrators get thresholds and alerts within our array to prevent this from even happening, so in a real life situation, our customers would not get into this state. Consider this a second level of protection in a Hitachi storage array infrastructure.

UNMAP –When used with an array’s thin provisioned volume and the WRITESAME command, this new primitive allows the ESXi Host to reclaim space for a VMDK that had its file format converted (e.g. converted from a “zeroedthick” to “eagerzeroedthick” or deleted within the datastore). The biggest advantage is that now a VMware admin can see the actual disk space available within the data store and also have far greater efficiency for disk capacity management.

NAS Support is also extended in vSphere 5 which was previously only for FC in vSphere 4.1.  You can refer to VMware Blog.
Full clone - Allow load to perform the block copy to be done by NAS device.
Native Snapshot support - offload of load to NAS array.
Extended Statistics - visibility of NAS device space usage.
Reserved Space - Allow creation of thick disk instead of thin previously.

For vSphere 4.1 three VAAI Features, it was clearly explain by Brain here.

Full Copy – So you’re probably wondering how this feature is going to help me. I can think to two VMware functions that this VAAI feature provides upwards of 10x speed improvements in. The first would be when you are deploying a VM from a template. We will say for example that you are going to deploy a 50 GB VB. When the VM is deployed vSphere is going to read the entire 50 GB and then write the 50 GB for a total of 100 GB of I/O. With VAAI enabled and a storage array that supports VAAI this process creates very little I/O at the host. The vSphere host will send a command to the storage array that say make a copy of this VM and name it this. The copy will be done locally on the storage array and results in very little I/O between the host and array. Once completed the array will send a notice to the host to let it know the works was completed.
The second VMware feature to benefit from this is a Storage vMotion. I feel that this is where this really pays off because you are most likely moving a larger chunk of data with this command. For example sake let’s say we are going to move a 100 GB Virtual Machine from one disk to another. To do this in the past this would have caused 200 GB of I/O on the host. With VAAI the burden on the host is almost nothing as this work is done on the storage array. 
Hardware assisted locking – Too allow multiple hosts in your cluster to talk to the same storage volume VMware would lock the volume when one of the VM’s needed to write to it. This locking is to prevent another host from trying to write to the same blocks. This was not a large issue If you were using smaller volumes with only a handful of virtual machines on them.  Now with VAAI the locking has been offloaded to the storage array and it’s now possible to lock only the blocks that are being written to. This opens up the possibility to use larger volumes and increase the amount of VM’s that can be run on a single volume.
Block Zeroing – This feature is saving vSphere from having to send redundant commands to the array for writes. The host can simple tell the storage array which blocks are zeros and move on. The storage device will handle the work without needing to receive repetitive write commands from the host.

I was also going through the VMFS volumes and Guest.  Apparently all these only for VMFS volumes so do not mixed it up.  I must admit I got it confused badly.  I manage to chanced on a post by Duncan Epping and asked and him and he replied and cleared my doubts.

To put it simple, when moving between datastore with different block size, the native fsdm  is used.  Though been the slowest datamover, it is the only one that goes through to the Application layer thus reclaiming space from guest.

For both fsdm and fs3dm, both reclaim is done on the VMFS volume only.  The only advantage of fs3dm is its faster since it does not goes through the application layer and it comes in hardware and software type depending if your array supports it.

vSphere 5: Memory share calculation for mem.minfreepct

While I was reading up the vSphere 5 Clustering Technical Deep Dive book by Duncan Epping and Frank Denneman, I came across this section which took me to awhile to understand the calculation for the math part.

This section on memory share which has been changed in vSphere 5 which is also written in Frank Denneman, Mem.MinFreePct sliding scale function.

The example is as below:
Let’s use an example to explore the savings of the sliding scale technique. On a server configured with 96GB RAM, the MinFreePct threshold will be set at 1597.6MB, opposed to 5898.24MB if 6% was used for the complete range 96GB.
Free memory state Threshold Range Result
High 6% 0-4GB 245.96MB

4% 4-12GB 327.84MB

2% 12-28GB 696.32MB

1% Remaining memory 696.32MB
Total High Threshold


I was trying to get the calculation for the Result column and thought to elaborate it on a clearer picture which a believe some are confused like myself.

The first 0-4GB will be on 6%, followed by from 4GB-8GB will be 4%(which is total of 4GB), then the next 12GB-28GB on 2% (which is 16GB) and lastly the remaining memory minus off from total on 1% (which is 68GB).  When you add those up would give you the Threshold.

Hope that helps how the math was derived.

Friday, September 16, 2011

vSphere 5: vMotion enhancement

I was reading through this article vMotion Architecture, Performance, and Best Practices in VMware vSphere 5.

I was not aware (perhaps only myself) that ESXi 5 introduces virtual NUMA (vNUMA).  What this means that in terms of performance, the ESXi is able to know which is the most efficient way to access the memory.  This was not possible in ESXi 4.x.

On reading further, this brought something to me especially for environment who enable EVC with mixed of hardware differences.

ESXi 5 introduces virtual NUMA (vNUMA), which exposes the ESXi host’s NUMA topology to the guest operating systems. When using this feature, apply vMotion to move virtual machines between clusters that are composed of hosts with matching NUMA architectures. This is because the very first time a vNUMA-enabled virtual machine is powered on, its vNUMA topology is set, based in part on the NUMA topology of the underlying physical host on which it is running. After a virtual machine’s vNUMA topology is initialized, it doesn’t change. This means that if a vNUMA virtual machine is migrated to a host with a different NUMA topology, the virtual machine’s vNUMA topology might no longer be optimal for the underlying physical NUMA topology, potentially resulting in reduced performance.

This means, if you are to vMotion across a hardware different, the original NUMA architecture from the source host remains on the VM.  And the VM will not be aware of the change unless a power cycle is performed (not reboot) on the VM for performance.

Something interesting to know.  Don't you think?

Thursday, September 15, 2011

View 5.0 Upgrade from View 4.6

I am running vSphere 4.1 in my home lab with View 4.6.  We will be conducting an upgrade of the View to 5.0 which was release.
Do not this is my first time doing this upgrade.  Below are the links to download the technical documents on performing the upgrade.

The View 5.0 documentation can be found here.  Refer to the View Upgrades document on performing the upgrading.  Make sure all backup are done on the servers/VMs before performing any upgrade.

Base on the upgrade document, the process of upgrading is as follows:
  1. Connection Server
  2. Security Server
  3. Transfer Server
  4. View Composer
  5. View Agent

Part 1
In this video, I upgraded the View Manager/Connection server follow by the security server.
Make sure you have backup your View Manager configuration and settings.
Use the vdmexport.exe utility to back up the View 4.0.x, 4.5.x, or 4.6.x LDAP database.
Instructions can be found in the View 4.6 administration guide.

Part 2
In part 2, we carry on to upgrade the Transfer Server. This upgrade is a breeze. Nothing serious came up.

Part 3
In this video, we upgraded the View Composer component on the vCenter.
Make sure you know your database password for your DSN which will be used during the installation.
Make sure no connection to the vCenter and Connection server administration console is done during this process. I encounter an error when my connection server and vCenter is still left connected.

Part 4
Last but not least, we upgrade the View Agent on the master desktop image.  Do note after the upgrade, remember to create a snapshot after which create desktop pool with just one VM to test out the functionality of the snapshot.

View 4.6 through Security Server via internet on 3G

This is a test on View 4.6 on a laptop via a 3G network on the internet to access the desktop.

View 4.6 through Connection Server

This video show the connection on a laptop accessing the desktop in the internal network through the connection server.

View 4.6 Android through Connection Server

Here is a test of the performance using an Android tablet connected in the internal network accessing the Desktop via the Connection server.

Apology for the lousy video as I did this using my digital camera in order to see the whole desktop experience on the screen.

Wednesday, September 14, 2011

VMware View 5 GA with Workstation 8

Yes the GA binaries of VMware View 5 is out.  It is also released with workstation 8.

From the release notes, it glad to see it is compatible with vSphere 4.0 Update 3 and above.

Compatibility Notes

  • VMware View 5.0 is supported on the following versions of vSphere:
    • vSphere 5.0
    • vSphere 4.1 Update 1 or later
    • vSphere 4.0 Update 3 or later
  • For system requirements, see the VMware View Installation documentation.

The technical documentation can be found here.

The binaries can be located here.

Will be doing the upgrade on my View 4.6 to 5 on vSphere 4.1.  Will be doing the recording as I try this on the first time tonight.

So do look back for the new videos!

VCP 510 Passed!

Just passed my VCP 5 today.  Didn't score really well but a pass is a pass!

Before my start sharing my experience, I will answer to many who question how to be certified.  In order to be certified, you would need to take the vSphere x: Install, Configure and Manage course and the exam.  Completing either of which will not make you certified.  If your course is on vSphere 4, you would need to take the VCP 410 exam and VCP 510.  It has to be the same.

VMware always introduce a What's New course for the previous VCP to attend in order to take the next VCP exam.  However promotion period will always be there so this allow previous VCP to be able to just take the new VCP exam without having to attend the What's New course.  After the promotion period, previous VCP will need to take the What's New course before attempting the exam.

Now to my experience. One thing I must comment is the exam no longer test you on memorizing work e.g. in VCP3, VCP4 which did test you on the configure maximums.  Please do not ask me for any of the questions, just like any other exams, there is a Non-Disclosure Agreement (NDA) you have to accept.

Though this is good it also make the exam more tough as technically wise it has gone way deep.  It test you on day to day administration and troubleshooting as well as identifying performance issue.

As a consultant, I am at the all losing end having that administration was not my day to day job.  I only implement on best practice and once I stop touching it on the hands on level, I tends to forget about the default settings and often need to revert back.

The exam is close to VCAP-DCA (which I failed) I must say but in multiple choice form so you still have a 50-50 to get it right.  This exam has made it more valuable and prestige.

What got me through.
  • Setup a vSphere 5 Lab at home and practice though I didn't spend much time other than doing those recording in my youtube channel.
  • Read the bluerpint go through what you are unsure of and also the terms used.
  • Go through the VTSP5 online modules (helps for new features only).
Those who are keen to go VCP5, I encourage you not to go in there without the above tasks done unless you have in-depth experience in vSphere 4 administration (which will cover you about 80%) and know the new features in vSphere 5 do.

To be honest I did refer to dumbs when I first touch VMware back in 2008 for my VCP310 which I failed on my first attempt.  However for VCP510, I went in without second thought and I only did my studies in the night before!

Study and practice well, drop the dumps they are useless (have gone through a recent dump release after my exam, out of 400+ questions and less than 5 are accurate).

Good Luck everyone!

Monday, September 12, 2011

Consideration vSphere 5 Auto Deploy: Stateless ESXi

I was doing my VCP5 studies and was reading the Installation and Setup Guide and came across the below on Page 62.

The boot process proceeds as follows.
1 The administrator reboots the host.
2 As the host boots up, Auto Deploy provisions the host with its image and host profile, which are stored
in vCenter Server.
3 Virtual machines are brought up or migrated to the host based on the settings of the host.
n Standalone host. Virtual machines are powered on according to autostart rules defined on the host.
n DRS cluster host. Virtual machines that were successfully migrated to other hosts stay there. Virtual
machines for which no host had enough resources are registered to the rebooted host.
If the vCenter Server system is unavailable, the host contacts the Auto Deploy server for image profiles and
host profiles and the host reboots. However, Auto Deploy cannot set up vSphere distributed switches if vCenter Server is unavailable, and virtual machines are assigned to hosts only if they participate in an HA cluster. Until the host is reconnected to vCenter Server and the host profile is applied, the switch cannot be created and,
because the host is in maintenance mode, virtual machines cannot start. See “Reprovision Hosts with Simple Reboot Operations,” on page 77.

The word I would like to bring out is maintenance mode.  Without a vCenter any reboot of the host will result in one less host participated in the cluster.

I see this as a consideration to take note when using Auto Deploy.  Without vCenter, the vDS and HA will be not configured on the host been the fact that the database with the host profile provided by vCenter is not present.  (HA agent, FDM will not start as every reboot is equal to a new server joining to the cluster and in such HA will start to configure on the host.)

If this was on a stateful host, HA and vDS will not be lost as the ESXi server is able to start up the FDM agent and vDS setting is present.

Whether to use Auto Deploy or not is a big consideration and not as simple as it seems.  I once thought its was a good tool not ware of the above.  Though stateless ESXi has it advantages like easy update of ESXi state instead of updating all the servers one at a time or even security measure where data are sensitive and do not want to be present on the harddisk.

It has show that vCenter has taken a very important role in every single vSphere upgrade.  If you are to use Auto Deploy ESXi servers, do note of the importance of vCenter and having vCenter with redundancy like vCenter Heartbeat and a regular backup that is valid is important.

I mention "valid" backup, as often I realize organization never perform any check on whether their backup are valid and only discover during Disaster Recovery were they to know their backup cannot be used.

Backup Restore ESXi Configuration

Export ESXi configuration file

After you configured an ESXi host, you can back up the host configuration data. Always back up your host configuration after you change the configuration or upgrade the ESXi image.

IMPORTANT The vicfg-cfgbackup command is available only for ESXi hosts. The command is not available for ESX hosts and is not available through a vCenter Server system connection.

Requirements: VMware vSphere CLI installed on workstation

  1. Open VMware vSphere CLI Command Prompt
  2. Type -s [backup_file_name]
    -–server [server_name_or_ip] --username [user_id]

e.g. –s c:\esx1_20082011.txt –-server –-username root

Restore configuration file

After you configured an ESXi host, you can back up the host configuration data. Always back up your host configuration after you change the configuration or upgrade the ESXi image.

IMPORTANT The vicfg-cfgbackup command is available only for ESXi hosts. The command is not available for ESX hosts and is not available through a vCenter Server system connection.

Requirements: VMware vSphere CLI installed on workstation

  1. Open VMware vSphere CLI Command Prompt
  2. Run -l [backup_file_name] -–server [server_name_or_ip] --username [user_id]

E.g. –l c:\esx1_20082011.txt –-server –-username root

The host will be rebooted.

Installing drivers or Update using vCLI

At time where you need to install update or install drivers on ESXi/ESX servers where new devices like Network cards or HBA drivers are not embedded in the base image.

In vSphere 5 you can of cause recompile your own image.  However for existing servers, we would need to learn the commands.

Here I have attached the screen shots with steps to do the installation using the VMware vCLI.

Requirements: VMware vSphere CLI installed on your workstation which can be found here under
VMware vSphere 4.0 CLI
VMware vSphere 4.1 CLI
VMware vSphere 5 CLI

Open VMware vSphere  CLI Command Prompt on workstation.

 Check the current Maintenance Mode of the ESXi server by using: –operation info –-server   –-username

e.g. –operation info –-server –-username root 

Enter password when prompted.

Put ESXi server into maintenance mode by using: –operation enter -–server [server_name_or_ip] --username [user_id]

e.g. –operation enter –-server –-username root

Enter password when prompted. 


Check the current Maintenance Mode of the ESXi server by using: –operation info -–server [server_name_or_ip] --username [user_id]
e.g. –operation info –-server –-username root 

Enter password when prompted.


Install the driver or patch by using: –b -I -–server [server_name_or_ip] --username [user_id]

e.g. –b c:\ -I -–server --username root

Enter password when prompted. 


Exit ESXi server from maintenance mode by using: –operation exit -–server [server_name_or_ip] --username [user_id]

e.g. –operation exit -–server –username root 

Enter password when prompted.

Check the current Maintenance Mode of the ESXi server by using: –operation info -–server [server_name_or_ip] --username [user_id]
e.g. –operation info -–server –-username root

Enter password when prompted. 

Restart the server to take effect using: –operation reboot –force -–server [server_name_or_ip] --username [user_id]

e.g. –operation reboot –force –-server --username root 

Enter password when promoted.  

 Note: Notice --force was used as the server is no longer in maintenance mode.  If server is in maintenance mode, --force can be excluded.  It is recommended to reboot a server in maintenance mode to prevent DRS moving in VMs.

Friday, September 9, 2011

VCP5 Certification Upgrade Path

VMware has release a new upgrade path for those on VCP3.  There is no need to be VCP4 and you can go direct to VCP5 by going through the What's New for vSphere 5.

Instead of going to the Install, Configure and Manage course for vSphere 5 and taking the exam to be certified VCP5.

For VCP4, you are allow to take the VCP5 exam directly to be upgraded.

Both upgrade path are only valid till 29 Feb 2012.  So hurry take this opportunity and upgrade your skillset credential to VCP5!


Monday, September 5, 2011

vCenter 5 Web Client Server Installation and Configuration

Today will do an installation of the web client server on the vCenter.  I will also compare the vCenter 4.1 web access with the new web client access to see the improvement that was done.

Do note that the requirements for the web client access would require the following below.  I have also include the links to the offline package for Adobe Flash which many have problem locating it.

Internet Explorer 7,8
Mozilla Firefox 3.5, 3.6
Adobe Flash 10.2
Offline package for Latest Adobe Flash

1 Mar 2012 Update:
Alternatively, you can make use of the free vCenter vApp and used it as a web client.
Michael Webster has a post on the steps. 

You can install the web client on a separate Windows VM and it is able to register to multiple vCenters to enable the web client access to the respective registered vCenter when user points their browser to the web client server and select the vCenter from the drop down list. 

Other related posts:
vSphere 5 vRAM Licensing
vSphere 5 ESXi 5 Installation
Upgrade vSphere vCenter 4.x to 5.0
Upgrade ESX/ESXi 4.x to ESXi 5.0 using Update Manager
Upgrade VMware Tools and virtual hardware with Update Manager

vSphere Assessment Tool (vSAT)

If you are not aware, there is a new tool announced during VMworld US 2019. That is the vSphere Assessment Tool (vSAT). To access the sit...