VMware has just updated their security hardening guide, which provides recommendations for hardening a VI3 environment.

In addition to the updates for virtual machines and the ESX Service Console, they have now added new recommendations for ESXi, VirtualCenter Add-on components (plug-ins) and for Client components.

Here’s a brief overview of the recommendations for VMs and ESX hosts that have been added to the guide. No new recommendations were made for VirtualCenter except for the Plug-in ones.

Virtual machines:

• Disable copy and paste operations between the guest operating system and remote console
• Do not use nonpersistent disks
• Ensure unauthorized devices are not connected
• Prevent unauthorized removal or connection of devices
• Avoid Denial of Service (DoS) caused by virtual disk modification operations
• Specify the guest operating system correctly
• Verify proper file permissions for virtual machine files

ESX Service Console:

• Secure the SNMP configuration
• Protect against the root file system filling up
• Disable automatic mounting of USB devices

There are some general recommendations when using plug-ins and some specific ones when using Update Manager, Converter and Guided Consolidation. The guide recommends that the Update Manager and Converter plug-ins not be installed on the VirtualCenter server but should instead be installed on a separate server or virtual machine.

Also added is a section on client components. The guide recommends against the use of Linux-based clients when using the RCLI, VI Perl Toolkit scripts, VM console access initiated from a web access browser session and programs written using the VI SDK. The reason for this is that communications with Linux clients are vulnerable to man-in-the-middle attacks because the Linux versions of these components do not perform certificate validation. This risk can be partially mitigated by ensuring that the management interfaces (ESX Service Console and VirtualCenter) are on trusted, isolated networks.

The guide suggests that client components are to verify the VI Client integrity because of the VI Client extensibility framework that was introduced into VirtualCenter 2.5 which provides the ability to extend the VI Client. It also recommends that one monitor the usage of the VI Client instances by inspecting log files on client systems. Both of these tasks can be quite difficult to do because there are no native methods for doing this.

Finally a section was added for securing the host-level management in ESXi. Many of the recommendations for ESXi are the same ones that were made for ESX. Some unique recommendations for ESXi include ensuring secure access to CIM (the hardware management api’s). Also, admins may want to audit or disable the special technical support mode which is designed to be used in case of an emergency but is sometimes used by administrators to access specific functions in ESXi.

You can read the updated guide in its entirety here.

A general rule of thumb in virtual environments is to always treat virtual machines the same as you would physical servers. While this rule holds true in many cases, IT administrators should be aware of some exceptions to this rule. Let’s go over some reasons that you would not treat your virtual machines like physical servers:

• Patching – You should apply all the same operating system and application patches to a virtual machine as you would a physical server. However it is best to stagger your patch deployments so you do not patch and restart all of your virtual machines at the same time. If you did this concurrently you can cause excessive resource utilization on your host servers which could impact other virtual machines running on the host.
• Securing – Secure the virtual machine operating system as you would physical servers, in addition you should ensure that you have proper security setup on the host server’s management console that allows access to the VM as well as on the virtual machine files located on the host server’s disk system. It does no good to have tight security inside your VM and have weak security outside.
• System Monitoring – This is one area that can be very different for virtual servers. There is no need to monitor virtual machine hardware, if you have converted physical servers to virtual machines you should make sure you un-install any hardware management agents from them. In addition virtual machines boot much faster then physical servers. Because of this, many monitoring systems will not detect server re-boots because the boot process happens quicker then the monitoring interval. You may find that you need to adjust your polling interval for virtual servers so you can detect the faster re-boots.
• Performance Monitoring – Another area that is very different from physical servers. Traditional operating system performance reporting tools are often inaccurate when used on virtual machines because they are unaware of the virtualization layer and the underlying physical hardware. You should always use virtual server specific reporting tools to accurately measure performance on virtual machines.
• Anti-virus – Make sure you install anti-virus software on all your virtual machines the same as physical servers. Again one thing to be careful of is to stagger any on-demand scans and definition updates as to not overwhelm the host server. Having all your VMs running a full scan at the same time can completely bog down a host server.
• Backups – It’s OK to backup your virtual machines using traditional operating system backup agents. Always make sure you do not backup too many VMs on a single host at the same time. There are more efficient ways to perform backups in a virtual environment that you may look into to either complement or replace traditional backup methods.
• Disk defragging – You should periodically defrag virtual machine disks using traditional operating system tools for maximum performance. However be careful not to defrag a VM that has a snapshot running, doing this can cause the snapshots rapidly grow in size and degrade host performance. As usual do not defrag more then one VM on a host at a single time because of all the excessive disk activity that is causes.

Be careful not to do too many of the same operations concurrently. With physical servers, only a single server is effected, but in virtual environments many other servers running on a host server can be impacted.

Now that Microsoft has finally delivered Hyper-V, everyone is waiting to see how many VMware shops will make the switch. Are there any compelling reasons for a company that already has a large investment in VMware products to switch to another product? Here are some reasons why companies may or may not make the switch from VMware to Hyper-V:

Some reasons why companies may choose Microsoft Hyper-V:

• It’s Microsoft. Companies that mainly use Microsoft products could switch to get better support for running their products running on virtual hosts and to not have to rely on a separate vendor for virtualization.

• Cost. It’s definitely cheaper then ESX, but I’m a firm believer that you get what you pay for. Yes, Hyper-V is a lot cheaper then ESX but it lacks the maturity and high-end features that ESX has. It’s probably just a matter of time though before VMware lowers its cost for large enterprises as they have already done with the SMB market with its bundled foundation acceleration kits.

• Versatility. Hyper-V will pretty much run on any hardware that Windows will run on. ESX only supports a very specific set of hardware. VMware has recently expanded their hardware support and will continue to do so.

Some reasons why companies stick with VMware ESX:

• Cost (again). Companies with a lot of in-house VMware experience will have to re-train staff to learn Hyper-V and basically start from scratch. There is a large pool of skilled and experienced VMware architects and administrators available today as well as many VMware consulting firms and business partners.
• Less features. ESX and VirtualCenter have a very rich tool set including vMotion, DRS and HA. Hyper-V lacks the ability to team NICs on vSwitches and their Quick Migration feature requires downtime.
• Less third-party products. A large number of 3^rd party products and add-on’s are available for ESX to enhance it. It will take time for vendors to release products for Hyper-V.
• It’s VMware. ESX is a mature, stable product that has been around for many years, Hyper-V is a 1.0 product that will take to develop and get all the bugs out of it.

Will I make the switch? Probably not anytime soon. I’ll definitely be looking at Hyper-V and will make my own comparisons, but the lack of certain features is a show stopper for me right now. I’ll keep an eye on Hyper-V to see how it develops, re-evaluating it later as new versions are released.

The competition is going to be great in the virtualization market, as it helps to drive down costs and force vendors to innovate. The race is on between VMware and Microsoft with VMware already miles ahead. Nevertheless, Microsoft has a lot of money and the determination to be on top (take Lotus Domino, Novell Netware and Netscape as examples). Expect Microsoft to slowly whittle away at VMware’s dominance as their product matures and to see VMware to do whatever they can to maintain superiority in the virtualization market.

Should you assign a virtual machine (VM) more than one virtual processor or not? It’s common for admins to configure virtual symmetric multiprocessing, or VMs with multiple CPUs, whether it is needed or not.The decision to use more then one virtual processor in a VM should be based on an actual requirement by the applications installed on the VM and not simply because two processors are better then one. Many physical servers commonly have multiple CPUs regardless if the applications running require them. While being wasteful of server resources, this does not negatively impact a physical server but most VMs will usually run better with one virtual processor and can actually run slower when more than one is assigned to it.

The reason for this is the hypervisor’s CPU scheduler must find simultaneous cores available equal to the number assigned to the VM. So a four VCPU VM will need to have four free cores available on the host for every CPU request that is made by the VM. If there are not four cores available because other VMs are using them then the VM must wait until the cores become available. Single VCPU VMs have a much easier time because they only need there to be a single core available for the scheduler to process CPU requests for it.

Here are some tips on assigning VCPUs to VMs:

• Limit the number of VSMP VMs on your hosts. The less you have, the better your VMs will perform.
• Assign a VM multiple VCPUs only if you are running an application that requires it and will make use of them.
• Don’t assign a VM the same amount of VCPUs as your host system has total cores available.
• If you are going to use VSMP have at least twice (preferably three or four times) the number of cores available on your host system then that of your VM with the most VCPUs. So if you have a four VCPU VM, have at least eight cores available on your host server and preferably 16.
• If you are converting a multi-CPU physical Windows server to a single VCPU VM, make sure you change the HAL from multiprocessor to uniprocessor.
• Don’t use CPU affinity as it restricts the scheduler and makes it harder to process CPU requests. The scheduler is very good at what it does, so let it do its job.

You may hear the term SCSI reservations frequently when dealing with VMware servers that utilize shared storage. SCSI reservations are used to ensure exclusive access to disk-based resources when multiple hosts are accessing the same shared storage resources. In addition to being used by VMware hosts, SCSI reservations are also used by Microsoft Cluster Server.

SCSI reservations are only used for specific operations when metadata changes are made and are necessary to prevent multiple hosts from concurrently writing to the metadata to avoid data corruption. Once the operation completes the reservation is released and other operations can continue. Because of this exclusive lock, it is important to minimize the concurrent number of reservations that are made. When too many reservations are being made at once, you may receive I/O failures because a host is unable to make a reservation to complete an operation because another host has locked the logical unit number (LUN). When a host is unable to make a reservation because of a conflict with another host, it will continue to retry at random intervals until it is successful; however, if too many attempts are made the operation will fail.

Some examples of operations that require metadata updates include:

• Creating or deleting a VMFS datastore
• Expanding a VMFS datastore onto additional extents
• Powering on or off a VM
• Acquiring or releasing a lock on a file
• Creating or deleting a file
• Creating a template
• Deploying a VM from a template
• Creating a new VM
• Migrating a VM with VMotion
• Growing a file (e.g., a Snapshot file or a thin provisioned Virtual Disk)

Having a minimal amount of reservation conflicts is generally unavoidable and will not have a big impact on your hosts and VMs. To avoid having too many conflicts, try to limit the number of operations that can cause reservations and stagger them so too many are not happening simultaneously. All reservation errors are logged to the /var/log/vmkernel log file on each ESX host. To reduce the amount of conflicts:

• Limit the number of snapshots you have running, as snapshots grow in 16MB increments and every time they grow they cause SCSI reservations.
• Only vMotion a single VM per LUN at any one time.
• Only cold migrate a single VM per LUN at any one time.
• Do not power on/off too many VMs simultaneously.
• Limit VM/template creations and deployments to a single VM per LUN at any one time.
• Consider using smaller LUN sizes (<600GB) and do not use extents to extend a VMFS volume

A number of tools are available for migrating physical servers to virtual machines. Which tool to choose will depend on the source operating system, the target virtualization platform and the type of server being migrated. All the available tools support converting physical servers running Microsoft operating systems but only a few support Linux server conversions. I’ve compiled a brief list of some of the tools and options available for converting physical servers to virtual machines.

Platespin PowerConvert – A commercial product that has the broadest operating system support. Supported source operating systems include Windows NT, 2000, XP, 2003 and Red Hat and SUSE. Virtual platform support includes VMware Server/Workstation/ESX, Microsoft Virtual Server, XenSource and Virtual Iron. The Acronis True Image, Symantec Ghost and Live state image formats are supported as well. Platespin’s product also converts virtual machines to physical machines.

Ultimate P2V – A cheaper alternative with some freeware tools such as BartPE, but this product requires an imaging tool such as Symantec Ghost to perform the imaging. It also requires some work to build the boot CD and can be a bit complicated.

Vizioncore vConverter – Another very robust commercial product that has broad platform support. Optimized to work with VMware ESX but also supports Microsoft Virtual Server, XenSource and Virtual Iron as target formats. Only supports Windows 2000, 2003 and Vista source operating systems.

Microsoft Virtual Server 2005 Migration Toolkit – Only supports Windows NT, 2000 and 2003 source operating systems and only supports Microsoft’s virtual machine as a target format. Free but requires the Automated Deployment Services add-on to Windows Server 2003 Enterprise Edition. The conversion process is somewhat complicated and may not be a good choice for most users.

VMware Converter – A free tool (standard version) provided by VMware to convert physical servers running Windows NT, 2000, XP, 2003 and Linux servers to VMware virtual machines. Also supports Symantec Ghost, Livestate, Backup Exec, Acronis True Image and StorageCraft ShadowProtect image formats.

HP Server Migration Pack – Supports converting Windows 2000 & 2003 servers to VMware ESX, XenServer and Microsoft Virtual Server target formats running HP Proliant hardware. Also supports virtual to physical conversions.

Leostream P>V Direct – I thought I would mention this tool even though Leostream recently dropped this product to focus on other areas. Supports conversion of Windows NT, 2000, XP and 2003 source servers to VMware Server/Workstation/ESX, Microsoft VirtualServer and Xen target servers.

Other useful tools:

Robocopy - A free tool from Microsoft that is included in the Windows Server 2003 Resource Kit. Simply create a new virtual machine and copy the data from the physical server to the virtual server. Robocopy preserves all the date/time stamps and security when copying data from server to server. This is useful for migrating non-application, file servers where you only care about the data files on the server.Sysprep - A Microsoft tool to prepare an existing server for cloning and restoration from a disk image.NewSID - A Sysinternals utility to change a Windows server’s SID (security identifier) after a server has been cloned.

Imaging tools like Symantec Ghost and Acronis True Image create images of your physical servers. Once completed, you can create a new VM and restore the image to the virtual machine. VMTS.net has a good write-up on the procedure here.

You can always do a P2V by backing up a physical server using your traditional backup software, creating a new virtual machine, installing the same operating system on the VM and a backup agent, and then performing a restore from the backup taken of the physical server. Not the cleanest method for doing a P2V but can be used as a cheap alternative to other tools.

But for certain servers, it is less risky and just as easy to build a new server and migrate the data to it. Examples of this include SQL Servers and Domain Controllers. It’s a simple process to build a new virtual machine, install the operating system, dcpromo it and then shutdown your physical domain controllers. Likewise, building a new virtual SQL server then detaching the databases from a physical SQL server and attaching them to the virtual one is an easy process and reduces the risk of data corruption that can occur during the P2V cloning process.

VMware’s Virtual Appliance Marketplace has over 800 appliances available for download over a wide range of categories that can be used in your VMware environment. For those who may not be familiar, virtual appliances are pre-built, pre-configured virtual machines for use in virtual environments that are built to serve specific functions.

Some of the type of appliances available with VMware include anti-spam, database/app/web servers, firewalls, network monitoring, operating systems and administration tools. There is even an appliance for running DOS-based games from the early 90s. Most of the appliances are free to download and use except for some of the certified appliances from vendors such as IBM, Symantec, VMSight, Bluelane and Bea which must be purchased. Almost all of the appliances run various distributions of Linux to avoid operating system licensing costs and many utilize free open-source applications.

These appliances are compatible with any of VMware’s products including Player, Workstation, Server, Fusion and ESX. Appliances range in size from a few megabytes for some of the small router or firewall appliances to a few gigabytes for some of the bigger, more featured applications. A typical appliance download will usually include the virtual disk (vmdk) file(s), configuration (vmx or ovf) file and usually a few companion files. Once you locate an appliance that you want to use, simply download it and copy the files to your VMware server or workstation. After adding the VM via your management interface, you’re ready to power it on and start using it. Most appliances are pre-configured to use DHCP to automatically assign an IP address to the VM but they will usually allow you to manually configure a static IP address if needed. A new feature in VirtualCenter 2.5 allows you to automatically download and import ovf file-format appliances via a simple wizard interface. You can also use VMware Converter to import virtual appliances into ESX.

Below are a couple of notable appliances that you might want to check out:

ESX Deployment Appliance – (free) Makes deploying new ESX servers simple and fast

X-M0n0wall – (free) A great little firewall for protecting virtual networks

Network Security Toolkit – (free) Contains many open-source network security applications

NagiosVMA – (free) All-in-one open-source host and network monitoring system

LAMP Appliance – (free) A complete web environment including web server (Apache), database (MySQL) and scripting language (PHP)

Remote CLI – (free) Remote command line utility for managing ESXi servers

Browser Appliance – (free) Safely browse the internet inside a virtual machine to prevent malware from infecting your desktop.

LeftHand Virtual SAN Appliance – ($) Converts internal storage of VI3 servers into a iSCSI SAN

SpamTitan – ($) A full-featured email security appliance

VM Sight – ($) Provides virtual network reporting and analytics

VMware Infrastructure Perl Toolkit 1.5 – (free) Provides a Perl interface to manage and control a VI3 environment

vKernel – ($) ESX resource monitoring and reporting including chargeback reports

As this long running thread in the VMware forums indicates, many users are frustrated with VMware’s lack of support for a Linux-based Virtual Infrastructure client to manage VI3 environments. Currently, the VI Client will run only under Windows (as it’s written in .NET), so Linux shops are forced to purchase and install Windows to run it. An alternative web interface does exist; however, it can only manage virtual machine operations and not the ESX hosts which severely limits its usefulness to VMware administrators.

While VMware has not officially announced any plans to develop cross-platform versions of the VI Client or any of its other Windows-only applications, the above-mentioned thread includes one response from a VMware employee who hints that VMware may eventually release a Linux version. A Linux version of a VI Client would be considered a welcomed addition by many VMware customers, if not as an essential feature for those that are using ESX servers in non-Windows environments.

Many customers have also been wanting a Linux version of VirtualCenter, VMware’s centralized management product for ESX,  and support for open source databases like MySQL. VirtualCenter will only install on a Windows server and its required database only supports Microsoft SQL Server or Oracle databases. You can also use SQL Express with VirtualCenter, but it is not recommended or supported for production environments. Because of this limitation, customers that wish to use VirtualCenter must also plan on the additional expense of Windows operating systems licenses for the VirtualCenter server as well as a database license if they do not already have an existing SQL/Oracle database server that they can use for the VirtualCenter database.

Unless more customers speak up and request that VMware produce cross-platform versions of their current Windows-only applications, they will probably not end up developing them. If the demand exists, there’s a better likelihood of it happening. Having Linux versions would also help VMware compete in an increasingly competitive virtualization market. If you would like to see VMware develop a Linux version of the VI Client and other applications, contact your VMware sales representatives and let them know.

The page that displays by default when you enter the hostname of an ESX or VirtualCenter server has links that some administrators like to suppress. In addition to the option to log into the server, there are also links to download the installation program for the VI Client application and the VI SDK and also to log into the scripted installer. Because authentication is not required for the default web page of the server, anybody can download the VI Client to the chagrin of administrators.

Having only the web access login page displayed instead is the preferred way. To get to the login page, you have to click the link from the default page or add “/ui” to your URL.

You can modify the default web page on both the ESX and VirtualCenter servers so the download links are not displayed or simply force a re-direct to the login page instead. To do this, simply follow the steps below. Please note that future upgrades to VirtualCenter or ESX will usually overwrite your changes and you will need to do this again after any upgrades.

For ESX servers, you will need to modify the index.html file that is located in the /var/lib/vmware/hostd/docroot directory on the ESX server. Be sure and make a backup of this file before editing it. You can log into the Service Console and edit it with Nano or use an application like WinSCP to connect to the server and edit the file. For VirtualCenter servers, you will need to modify the index.html file located in the C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\docRoot directory.

If you simply want to hide the download links, you can comment out those lines of code in the index.html file. The comment tags tell the browser to not execute a section of code and are usually used to add documentation to programming code. The start tag is “” (without the quotes). Add “” on a new line after the second tag to comment out the two download links under getting started. Then save the file and refresh your browser and the links should be gone. Alternatively, you can remove the code completely so the links will not be visible if you viewed the source of the web page.

If you wish to automatically re-direct the default welcome page to the login page, add the following line at the beginning of the index.html file: If you do not want the main page to show before the redirect, either delete all the other code in the file or add a “” at the very bottom of the file. This comments out all the code in the file except the first line.

When system administrators receive new servers, they are often anxious to get them unpacked, in the rack and loaded up with ESX so they can start creating virtual machines. But an important first step should be done before proceeding with virtualization software installation on the server: always burn-in the memory to test for defective memory modules.

Defective memory will usually be unnoticed in a newly-deployed server and it may be months before signs of defective memory start to show. In one group of five HP servers, I had to replace seven memory DIMMs over an 18 month time period. Most of these were eventually detected by HP’s Insight Manager agents that reside on the server, but two of them caused hard server crashes of VMware ESX servers commonly known as a PSOD (Purple Screen of Death). A PSOD on one of your production servers, loaded up with important virtual machines, is never a good thing. You can reduce your chances of this happening by burning in your memory.

Most servers do a brief memory test on startup as part of their POST procedure. This is not a very good test and will only detect the most obvious of memory problems. A more thorough test checks the interaction of adjacent memory cells to ensure that writing to one cell does not overwrite an adjacent cell.

A good, free memory test utility is available, called Memtest86+, that performs many different tests to thoroughly test your servers memory. You can download it as a small 2MB ISO file that can be burned to a CD and booted on your new server. Let the memory burn-in for at least 24 hours (the longer the better though). Memtest86+ will run indefinitely and the pass counter will increment as all of the tests are run. The more RAM you have in your system, the longer it will take to complete one pass. A system with 32GB will generally take about one day to complete. Memtest86+ not only tests your system’s RAM but also the CPU L1 and L2 caches. Should it detect an error, the easiest way to identify the memory module that caused it is to simply remove a DIMM and run the test again and repeat until it passes. Documentation on Memtest86+ includes troubleshooting methods, detailed test descriptions and the causes of errors.

If you already have ESX servers running and want to test their memory, you can use the little known Ramcheck service to do this while ESX is running. This service is non-disruptive and runs in the background consuming minimal CPU cycles.

The extra time you spending testing memory before deploying servers helps eliminate potential problems down the road.