VMware has just updated their security hardening guide, which provides recommendations for hardening a VI3 environment.

In addition to the updates for virtual machines and the ESX Service Console, they have now added new recommendations for ESXi, VirtualCenter Add-on components (plug-ins) and for Client components.

Here’s a brief overview of the recommendations for VMs and ESX hosts that have been added to the guide. No new recommendations were made for VirtualCenter except for the Plug-in ones.

Virtual machines:

• Disable copy and paste operations between the guest operating system and remote console
• Do not use nonpersistent disks
• Ensure unauthorized devices are not connected
• Prevent unauthorized removal or connection of devices
• Avoid Denial of Service (DoS) caused by virtual disk modification operations
• Specify the guest operating system correctly
• Verify proper file permissions for virtual machine files

ESX Service Console:

• Secure the SNMP configuration
• Protect against the root file system filling up
• Disable automatic mounting of USB devices

There are some general recommendations when using plug-ins and some specific ones when using Update Manager, Converter and Guided Consolidation. The guide recommends that the Update Manager and Converter plug-ins not be installed on the VirtualCenter server but should instead be installed on a separate server or virtual machine.

Also added is a section on client components. The guide recommends against the use of Linux-based clients when using the RCLI, VI Perl Toolkit scripts, VM console access initiated from a web access browser session and programs written using the VI SDK. The reason for this is that communications with Linux clients are vulnerable to man-in-the-middle attacks because the Linux versions of these components do not perform certificate validation. This risk can be partially mitigated by ensuring that the management interfaces (ESX Service Console and VirtualCenter) are on trusted, isolated networks.

The guide suggests that client components are to verify the VI Client integrity because of the VI Client extensibility framework that was introduced into VirtualCenter 2.5 which provides the ability to extend the VI Client. It also recommends that one monitor the usage of the VI Client instances by inspecting log files on client systems. Both of these tasks can be quite difficult to do because there are no native methods for doing this.

Finally a section was added for securing the host-level management in ESXi. Many of the recommendations for ESXi are the same ones that were made for ESX. Some unique recommendations for ESXi include ensuring secure access to CIM (the hardware management api’s). Also, admins may want to audit or disable the special technical support mode which is designed to be used in case of an emergency but is sometimes used by administrators to access specific functions in ESXi.

You can read the updated guide in its entirety here.

A general rule of thumb in virtual environments is to always treat virtual machines the same as you would physical servers. While this rule holds true in many cases, IT administrators should be aware of some exceptions to this rule. Let’s go over some reasons that you would not treat your virtual machines like physical servers:

• Patching – You should apply all the same operating system and application patches to a virtual machine as you would a physical server. However it is best to stagger your patch deployments so you do not patch and restart all of your virtual machines at the same time. If you did this concurrently you can cause excessive resource utilization on your host servers which could impact other virtual machines running on the host.
• Securing – Secure the virtual machine operating system as you would physical servers, in addition you should ensure that you have proper security setup on the host server’s management console that allows access to the VM as well as on the virtual machine files located on the host server’s disk system. It does no good to have tight security inside your VM and have weak security outside.
• System Monitoring – This is one area that can be very different for virtual servers. There is no need to monitor virtual machine hardware, if you have converted physical servers to virtual machines you should make sure you un-install any hardware management agents from them. In addition virtual machines boot much faster then physical servers. Because of this, many monitoring systems will not detect server re-boots because the boot process happens quicker then the monitoring interval. You may find that you need to adjust your polling interval for virtual servers so you can detect the faster re-boots.
• Performance Monitoring – Another area that is very different from physical servers. Traditional operating system performance reporting tools are often inaccurate when used on virtual machines because they are unaware of the virtualization layer and the underlying physical hardware. You should always use virtual server specific reporting tools to accurately measure performance on virtual machines.
• Anti-virus – Make sure you install anti-virus software on all your virtual machines the same as physical servers. Again one thing to be careful of is to stagger any on-demand scans and definition updates as to not overwhelm the host server. Having all your VMs running a full scan at the same time can completely bog down a host server.
• Backups – It’s OK to backup your virtual machines using traditional operating system backup agents. Always make sure you do not backup too many VMs on a single host at the same time. There are more efficient ways to perform backups in a virtual environment that you may look into to either complement or replace traditional backup methods.
• Disk defragging – You should periodically defrag virtual machine disks using traditional operating system tools for maximum performance. However be careful not to defrag a VM that has a snapshot running, doing this can cause the snapshots rapidly grow in size and degrade host performance. As usual do not defrag more then one VM on a host at a single time because of all the excessive disk activity that is causes.

Be careful not to do too many of the same operations concurrently. With physical servers, only a single server is effected, but in virtual environments many other servers running on a host server can be impacted.

So, earlier this week I wrote a blog about Clabby Analytics Analyst Joe Clabby’s report spelling out a handful of reasons why Microsoft’s Hyper-V is going to take the lead in the virtualization market away from VMware Inc. over the next five years.

I received a lot of feedback on this blog from people defending VMware, and thought, why not get some Hyper-V users to talk to me about the product - how it performs, its related management tools, features, etc. I asked Microsoft’s press team to send some users my way for interviews, and about a week later Microsoft’s “Rapid Response” team sent me a couple of links to case studies.

Thanks, but I would like to interview some users myself, outside of Microsoft filters. How about at least sending me the contact info for the users profiled in these case studies?

Microsoft’s response was, “Unfortunately regarding direct contact information for the Hyper-V case studies, we have no further information to share.”

What? Really?

This strikes me as odd because Microsoft’s competition, VMware and even smaller virtualization companies like Virtual Iron refer me to real users to interview about their products.

Does this mean that Microsoft doesn’t have the same level of product confidence as the competition? VMware has offered plenty of customer references, and while those users do complain about the acquisition cost of VMware’s software, I don’t think I’ve heard any serious gripes about the product itself.

So I am interested in hearing from Hyper-V users about its performance, because as users and analysts have said, Microsoft won’t sail past VMware on price alone.

Speculations are overflowing within the virtualization community following Diane Greene’s resignation from VMware. As a glass-half-full kind of guy, I’d like to offer my reasons why VMware may thrive in the next several years.

First and foremost, I feel that VMware’s technology has the potential to continue to be superior to the competition. While price is among the more important decision points, the superior product will hold its own in the marketplace despite the higher price. The standing example in this arena is enterprise databases. Oracle is a better database platform than Microsoft’s offerings, yet both hold good position in the market place. A certain amount of normalization of market share for VMware is to be expected as other hypervisors and management products enter the market, but more organizations have yet to enter the market as a customer.

Storage integration will continue to drive the richest virtualization platforms. The most underrated technology that VMware has ever produced is the virtual machine file system or VMFS. VMware’s implementation of this technology will improve over time, and the competition is not yet there in this space.

VMware will have a lower host hardware cost per VM for the same performance deliverables. While this is incredibly difficult to precisely quantify, my experience is that VMware can run more virtual machines than Hyper-V on the same hardware. Again, because price is among the most important decision points, this point may help VMware as hardware becomes more capable for virtualization technologies.

VMware can continue to innovate within the virtualization space. VMware has the virtualization expertise to provide new products into the market, and among the major players in the field they would be the most suited to innovate at this point.

It is a given that the other platforms will make gains in market share with the relative flood of products into the space. But considering VMware’s proven ability to innovate in this space, they have the chance to retain their lead and keep going in the correct direction. We will see!

Clabby Analytics analyst Joe Clabby is 100% convinced that Microsoft’s Hyper-V will take over VMware in market share over the next three to five years, and makes some strong points for this in his recent report, Six Reasons Why Microsoft’s Hyper-V will Overtake VMware to Become the Major Player in the x86 Server Virtualization Marketplace.

The report came out prior to the shake-up at VMware on July 8, when the company announced that its Board of Directors replaced VMware co-founder and CEO Diane Greene was being replaced, and then lowered its revenue forecast.

VMware had the vision to see the value of virtualization and took the technology to the top unchallenged due to strategy, innovation and sales execution, but that ride is about to come to an end, Clabby said.

“With the introduction of Hyper-V by Microsoft, VMware is about to experience some very serious competition from a vendor with deep pockets, with a massive worldwide marketing and sales organization, with major market penetration across Fortune 500 and small and medium business markets, and with extensive and complementary infrastructure and management product depth,” Clabby reported.

Among the reasons Clabby believes Microsoft will crush VMware are that Microsoft already has an expansive installed base, a mammoth network of direct sales and indirect business partners, and is offering lower prices alternatives to VMware’s hypervisor and related infrastructure/management software products.

Unfortunately, I have to agree. History tends to repeat itself, and this has been Microsoft’s strategy for a very long time: see a great technology, copy it, and outprice the rest of the market.

Vanity Fair’s July issue had a great article that illustrates this, called “How the Web was Won” that looks at the eveolution of the Internet over the past 50 years, including details of how Microsoft took over Netscape Navigator by developing Internet Explorer.

The computer programmer known for founding Netscape Communications, Lou Montulli, told Vanity Fair, “From a scientific point of view none of us really respected Microsoft. There was definitely a sense of: They’ve put out of business three or four major companies, and they did it simply by copying what they did and outpricing or outmaneuvering them in the market. This is a general feeling of computer scientists everywhere, that Microsoft doesn’t tend to innovate as much and really just enters the market late, takes it over, and then stays at the top.”

Pricing aside, Microsoft already has a massive installed base.

“It will leverage this installed base, and price its products to out-function/undercut VMware’s pricing,” Clabby wrote. “The computing industry saw this same situation arise when Citrix built a leadership base for its terminal server products — only to have Microsoft enter the market and claim significant marketshare after Citrix pioneered the terminal server marke umbrella. Almost the exact same situation is about to happen again — this time between VMware and Microsoft.”

Microsoft also has a packaging advantage with its Hyper-V hypervisor, as it can be delivered with every single version of 64-bit Windows Server 2008, and installing Hyper-V is a cake walk, according to Clabby.

“A box simply needs to be checked during installation and Hyper-V becomes active. By not requiring IT buyers to find/acquire/download additional virtualization software, the job of deploying and testing virtualization within a Windows Server 2008 is greatly simplified. VMware cannot counter this packaging advantage,” Clabby wrote.

The most damning problem for VMware, according to Clabby, is product depth.

Though VMware has the advantage of technologies like VMotion, to move live VMs, and all of the handy add-on management and infrastructure software integrated into VMware, Clabby said Microsoft’s management and infrastructure is far deeper.

Microsoft’s Systems Center product portfolio inlcludes systems management tools like Configuration Manager; Operations Manager; Data Protection Manager; Virtual Machine Manager; System Center Essentials; Capacity Planner, and the list goes on, ad nauseum.

Besides all of those points, Microsoft is a $51 billion dollar software company and VMware’s revenue is just over $1 billion.

In short, given its deep pockets, large installed base and virtualization strategy, it is safe to say Microsoft will, once again, be laughing all the way to the bank.

Now that Microsoft has finally delivered Hyper-V, everyone is waiting to see how many VMware shops will make the switch. Are there any compelling reasons for a company that already has a large investment in VMware products to switch to another product? Here are some reasons why companies may or may not make the switch from VMware to Hyper-V:

Some reasons why companies may choose Microsoft Hyper-V:

• It’s Microsoft. Companies that mainly use Microsoft products could switch to get better support for running their products running on virtual hosts and to not have to rely on a separate vendor for virtualization.

• Cost. It’s definitely cheaper then ESX, but I’m a firm believer that you get what you pay for. Yes, Hyper-V is a lot cheaper then ESX but it lacks the maturity and high-end features that ESX has. It’s probably just a matter of time though before VMware lowers its cost for large enterprises as they have already done with the SMB market with its bundled foundation acceleration kits.

• Versatility. Hyper-V will pretty much run on any hardware that Windows will run on. ESX only supports a very specific set of hardware. VMware has recently expanded their hardware support and will continue to do so.

Some reasons why companies stick with VMware ESX:

• Cost (again). Companies with a lot of in-house VMware experience will have to re-train staff to learn Hyper-V and basically start from scratch. There is a large pool of skilled and experienced VMware architects and administrators available today as well as many VMware consulting firms and business partners.
• Less features. ESX and VirtualCenter have a very rich tool set including vMotion, DRS and HA. Hyper-V lacks the ability to team NICs on vSwitches and their Quick Migration feature requires downtime.
• Less third-party products. A large number of 3^rd party products and add-on’s are available for ESX to enhance it. It will take time for vendors to release products for Hyper-V.
• It’s VMware. ESX is a mature, stable product that has been around for many years, Hyper-V is a 1.0 product that will take to develop and get all the bugs out of it.

Will I make the switch? Probably not anytime soon. I’ll definitely be looking at Hyper-V and will make my own comparisons, but the lack of certain features is a show stopper for me right now. I’ll keep an eye on Hyper-V to see how it develops, re-evaluating it later as new versions are released.

The competition is going to be great in the virtualization market, as it helps to drive down costs and force vendors to innovate. The race is on between VMware and Microsoft with VMware already miles ahead. Nevertheless, Microsoft has a lot of money and the determination to be on top (take Lotus Domino, Novell Netware and Netscape as examples). Expect Microsoft to slowly whittle away at VMware’s dominance as their product matures and to see VMware to do whatever they can to maintain superiority in the virtualization market.

Word has it that Microsoft is finally getting it together and releasing Hyper-V, putting the tech world on notice that it is now safe to exhale.

Phew, we were all about to turn blue.

Has someone ever told you a story about some aging celebrity, and your first thought is, “Wait, you mean they’re not dead yet?’ I probably shouldn’t admit this, but when I read that Hyper-V was coming out, I thought, ‘What do you mean, it’s being released? I thought that already happened!”

My mistake, I had confused the release with another important Microsoft — ahem, milestone — in March: the Hyper-V release candidate (RC).

Excuse me for being flip, but I was bored to tears by this whole Viridian-cum-Hyper-V saga long ago. Two years ago, when I first started covering virtualization, the big news was that Microsoft had made Virtual Server 2005 available for free. Immediately thereafter, VMware returned the volley and made its hosted virtualization platform VMware Server free too, eliminating any real advantage Virtual Server 2005 may have had over the better-established GSX. So much for that story line.

Since then, we’ve lived through name changes, (Viridian to Hyper-V), release candidates, pricing announcements (why $28 dollars, why not $25? $29.99?), delays (will Microsoft meet its 180-days-after-Longhorn deadline? Will it beat it?), feature cuts, feature clarifications (“Quick migration” anyone?), and countless press articles with VMware cast as David to Microsoft’s Goliath — or is it the other way around?

Everything except an actually shipping, nonbeta, nonrelease candidate product.

Until now.

As a journalist, I’m just happy that the wait is over, and we can all stop walking around on tenterhooks, expected to drop everything every time Microsoft comes knocking at our inbox with some virtualization-related announcement that may or may not pertain to the release of Hyper-V.

Now we can all get on with our job of waiting for Microsoft to update us on the status of all the product features that it excised from Hyper-V last year: quick migration, hot add of system resources, increased numbers of CPUs, etc. What a relief!

Attention, college students: your tuition may soon decrease!

Well, maybe not. However, VMware Inc. reported today that 900 universities including top tier schools such as Harvard and Yale are saving big bucks using VMware Inc. virtualization.

Many renowned universities that have deployed VMware to reduce capital and operating costs, increase application and system uptime, decrease power consumption and improve disaster preparedness include Cambridge, Princeton, Stanford, Purdue, the University of Maryland, the University of Auckland, and the University of California campuses at Berkeley, Los Angeles and San Diego.

These schools and hundreds more around the world are running their mission-critical enterprise applications, database systems, and education-specific applications such as CollegeNET and the Blackboard Academic Suite in VMware virtualized environments, the company reported.

Others are using VMware for disaster recovery (DR).

Bowdoin College in Maine partnered with Los Angeles-based Loyola Marymount University to build a co-located datacenter for cross-country DR. By partnering and using VMware to create back-up systems, the schools have achieved higher availability and better load balancing, with more than 70% of their environment virtualized and more than 100 virtual machines (VM). They are saving $15,000 in annual server maintenance and have avoided $500,000 in hardware costs, according to VMware.

Ohio State University has been a VMware virtualizatiton customer since 2003 when the College of Humanities needed to upgrade its IT infrastructure and found there was not enough room to expand. After deploying VMware virtualization, the College was able to meet its upgrade needs with 54 VMs running on three physical host servers. The college avoided $160,000 in hardware costs and cut server provisioning time down from three weeks to five minutes, and the IT staff can now manage all of its VMware VMs from a single console.

Clearly, the education sector is a strong market for VMware, as there are now 900 universities and colleges using the virtualization platform. Because of this, VMware created a free online tool called VMware Academic Program staffed with IT professionals from higher education facilities to answer questions about overall IT best practices. In addition to these experts, the site also includes case studies to help understand how others have implemented VMware.

Should you assign a virtual machine (VM) more than one virtual processor or not? It’s common for admins to configure virtual symmetric multiprocessing, or VMs with multiple CPUs, whether it is needed or not.The decision to use more then one virtual processor in a VM should be based on an actual requirement by the applications installed on the VM and not simply because two processors are better then one. Many physical servers commonly have multiple CPUs regardless if the applications running require them. While being wasteful of server resources, this does not negatively impact a physical server but most VMs will usually run better with one virtual processor and can actually run slower when more than one is assigned to it.

The reason for this is the hypervisor’s CPU scheduler must find simultaneous cores available equal to the number assigned to the VM. So a four VCPU VM will need to have four free cores available on the host for every CPU request that is made by the VM. If there are not four cores available because other VMs are using them then the VM must wait until the cores become available. Single VCPU VMs have a much easier time because they only need there to be a single core available for the scheduler to process CPU requests for it.

Here are some tips on assigning VCPUs to VMs:

• Limit the number of VSMP VMs on your hosts. The less you have, the better your VMs will perform.
• Assign a VM multiple VCPUs only if you are running an application that requires it and will make use of them.
• Don’t assign a VM the same amount of VCPUs as your host system has total cores available.
• If you are going to use VSMP have at least twice (preferably three or four times) the number of cores available on your host system then that of your VM with the most VCPUs. So if you have a four VCPU VM, have at least eight cores available on your host server and preferably 16.
• If you are converting a multi-CPU physical Windows server to a single VCPU VM, make sure you change the HAL from multiprocessor to uniprocessor.
• Don’t use CPU affinity as it restricts the scheduler and makes it harder to process CPU requests. The scheduler is very good at what it does, so let it do its job.

You may hear the term SCSI reservations frequently when dealing with VMware servers that utilize shared storage. SCSI reservations are used to ensure exclusive access to disk-based resources when multiple hosts are accessing the same shared storage resources. In addition to being used by VMware hosts, SCSI reservations are also used by Microsoft Cluster Server.

SCSI reservations are only used for specific operations when metadata changes are made and are necessary to prevent multiple hosts from concurrently writing to the metadata to avoid data corruption. Once the operation completes the reservation is released and other operations can continue. Because of this exclusive lock, it is important to minimize the concurrent number of reservations that are made. When too many reservations are being made at once, you may receive I/O failures because a host is unable to make a reservation to complete an operation because another host has locked the logical unit number (LUN). When a host is unable to make a reservation because of a conflict with another host, it will continue to retry at random intervals until it is successful; however, if too many attempts are made the operation will fail.

Some examples of operations that require metadata updates include:

• Creating or deleting a VMFS datastore
• Expanding a VMFS datastore onto additional extents
• Powering on or off a VM
• Acquiring or releasing a lock on a file
• Creating or deleting a file
• Creating a template
• Deploying a VM from a template
• Creating a new VM
• Migrating a VM with VMotion
• Growing a file (e.g., a Snapshot file or a thin provisioned Virtual Disk)

Having a minimal amount of reservation conflicts is generally unavoidable and will not have a big impact on your hosts and VMs. To avoid having too many conflicts, try to limit the number of operations that can cause reservations and stagger them so too many are not happening simultaneously. All reservation errors are logged to the /var/log/vmkernel log file on each ESX host. To reduce the amount of conflicts:

• Limit the number of snapshots you have running, as snapshots grow in 16MB increments and every time they grow they cause SCSI reservations.
• Only vMotion a single VM per LUN at any one time.
• Only cold migrate a single VM per LUN at any one time.
• Do not power on/off too many VMs simultaneously.
• Limit VM/template creations and deployments to a single VM per LUN at any one time.
• Consider using smaller LUN sizes (<600GB) and do not use extents to extend a VMFS volume