A general rule of thumb in virtual environments is to always treat virtual machines the same as you would physical servers. While this rule holds true in many cases, IT administrators should be aware of some exceptions to this rule. Let’s go over some reasons that you would not treat your virtual machines like physical servers:

• Patching – You should apply all the same operating system and application patches to a virtual machine as you would a physical server. However it is best to stagger your patch deployments so you do not patch and restart all of your virtual machines at the same time. If you did this concurrently you can cause excessive resource utilization on your host servers which could impact other virtual machines running on the host.
• Securing – Secure the virtual machine operating system as you would physical servers, in addition you should ensure that you have proper security setup on the host server’s management console that allows access to the VM as well as on the virtual machine files located on the host server’s disk system. It does no good to have tight security inside your VM and have weak security outside.
• System Monitoring – This is one area that can be very different for virtual servers. There is no need to monitor virtual machine hardware, if you have converted physical servers to virtual machines you should make sure you un-install any hardware management agents from them. In addition virtual machines boot much faster then physical servers. Because of this, many monitoring systems will not detect server re-boots because the boot process happens quicker then the monitoring interval. You may find that you need to adjust your polling interval for virtual servers so you can detect the faster re-boots.
• Performance Monitoring – Another area that is very different from physical servers. Traditional operating system performance reporting tools are often inaccurate when used on virtual machines because they are unaware of the virtualization layer and the underlying physical hardware. You should always use virtual server specific reporting tools to accurately measure performance on virtual machines.
• Anti-virus – Make sure you install anti-virus software on all your virtual machines the same as physical servers. Again one thing to be careful of is to stagger any on-demand scans and definition updates as to not overwhelm the host server. Having all your VMs running a full scan at the same time can completely bog down a host server.
• Backups – It’s OK to backup your virtual machines using traditional operating system backup agents. Always make sure you do not backup too many VMs on a single host at the same time. There are more efficient ways to perform backups in a virtual environment that you may look into to either complement or replace traditional backup methods.
• Disk defragging – You should periodically defrag virtual machine disks using traditional operating system tools for maximum performance. However be careful not to defrag a VM that has a snapshot running, doing this can cause the snapshots rapidly grow in size and degrade host performance. As usual do not defrag more then one VM on a host at a single time because of all the excessive disk activity that is causes.

Be careful not to do too many of the same operations concurrently. With physical servers, only a single server is effected, but in virtual environments many other servers running on a host server can be impacted.

Clabby Analytics analyst Joe Clabby is 100% convinced that Microsoft’s Hyper-V will take over VMware in market share over the next three to five years, and makes some strong points for this in his recent report, Six Reasons Why Microsoft’s Hyper-V will Overtake VMware to Become the Major Player in the x86 Server Virtualization Marketplace.

The report came out prior to the shake-up at VMware on July 8, when the company announced that its Board of Directors replaced VMware co-founder and CEO Diane Greene was being replaced, and then lowered its revenue forecast.

VMware had the vision to see the value of virtualization and took the technology to the top unchallenged due to strategy, innovation and sales execution, but that ride is about to come to an end, Clabby said.

“With the introduction of Hyper-V by Microsoft, VMware is about to experience some very serious competition from a vendor with deep pockets, with a massive worldwide marketing and sales organization, with major market penetration across Fortune 500 and small and medium business markets, and with extensive and complementary infrastructure and management product depth,” Clabby reported.

Among the reasons Clabby believes Microsoft will crush VMware are that Microsoft already has an expansive installed base, a mammoth network of direct sales and indirect business partners, and is offering lower prices alternatives to VMware’s hypervisor and related infrastructure/management software products.

Unfortunately, I have to agree. History tends to repeat itself, and this has been Microsoft’s strategy for a very long time: see a great technology, copy it, and outprice the rest of the market.

Vanity Fair’s July issue had a great article that illustrates this, called “How the Web was Won” that looks at the eveolution of the Internet over the past 50 years, including details of how Microsoft took over Netscape Navigator by developing Internet Explorer.

The computer programmer known for founding Netscape Communications, Lou Montulli, told Vanity Fair, “From a scientific point of view none of us really respected Microsoft. There was definitely a sense of: They’ve put out of business three or four major companies, and they did it simply by copying what they did and outpricing or outmaneuvering them in the market. This is a general feeling of computer scientists everywhere, that Microsoft doesn’t tend to innovate as much and really just enters the market late, takes it over, and then stays at the top.”

Pricing aside, Microsoft already has a massive installed base.

“It will leverage this installed base, and price its products to out-function/undercut VMware’s pricing,” Clabby wrote. “The computing industry saw this same situation arise when Citrix built a leadership base for its terminal server products — only to have Microsoft enter the market and claim significant marketshare after Citrix pioneered the terminal server marke umbrella. Almost the exact same situation is about to happen again — this time between VMware and Microsoft.”

Microsoft also has a packaging advantage with its Hyper-V hypervisor, as it can be delivered with every single version of 64-bit Windows Server 2008, and installing Hyper-V is a cake walk, according to Clabby.

“A box simply needs to be checked during installation and Hyper-V becomes active. By not requiring IT buyers to find/acquire/download additional virtualization software, the job of deploying and testing virtualization within a Windows Server 2008 is greatly simplified. VMware cannot counter this packaging advantage,” Clabby wrote.

The most damning problem for VMware, according to Clabby, is product depth.

Though VMware has the advantage of technologies like VMotion, to move live VMs, and all of the handy add-on management and infrastructure software integrated into VMware, Clabby said Microsoft’s management and infrastructure is far deeper.

Microsoft’s Systems Center product portfolio inlcludes systems management tools like Configuration Manager; Operations Manager; Data Protection Manager; Virtual Machine Manager; System Center Essentials; Capacity Planner, and the list goes on, ad nauseum.

Besides all of those points, Microsoft is a $51 billion dollar software company and VMware’s revenue is just over $1 billion.

In short, given its deep pockets, large installed base and virtualization strategy, it is safe to say Microsoft will, once again, be laughing all the way to the bank.

Anyone with five minutes of IT experience knows that vendors sometimes publish bogus “benchmarks” that portray their products in the best of all possible lights. Virtualization guru and Burton Group analyst Chris Wolf recently uncovered a particularly spectacular example of this, courtesy of QLogic and Microsoft.

In a release, QLogic Corp., a networking technology provider, said it tested virtual machines running on Windows Server 2008 Hyper-V and attached to a storage area network (SAN) via its 8 Gbps Fibre Channel (FC) host bus adapters, and saw near-native performance of 200,000 I/O operations per second (IOPS).

But, as Wolf discovered, what QLogic failed to mention was that it ran its tests against a very unusual SAN array: the Texas Memory RamSan 325 FC, which uses solid-state storage. Further, the benchmark used block sizes of just 512 bytes, compared with a more real-world block size of 8 K or 16 K.

This left Wolf feeling duped and betrayed:

If I was watching an Olympic event, this would be the moment where after thinking I witnessed an incredible athletic event, I learned that the athlete tested positive for steroids.

Wolf ran this benchmark by a colleague, who calculated that had the same benchmark been performed using “real disks” with latency of 7 milliseconds, it would have limited throughput to a much less impressive 9,142 IOPS. Hardly anything to write home about.

Thanks to Wolf for taking the time to look into this.

I recently came across an article revealing that 1 out of 3 IT administrators have used their elevated privileges to snoop on confidential information. It’s always possible to lock out administrators to sensitive data through operating system access controls, however, a virtual environment opens up other avenues for exposing sensitive data.

With physical servers, the task of imaging a server’s hard drive for offline examination is not always easy. An administrator of a virtual environment can easily and stealthily snapshot a virtual machine to temporarily suspend writes to disk file, make a file system copy of the VM’s disk file from the host server while it is running and then take that copy to a workstation where they can mount it and attempt to gain access to information to which they would normally not have access.

Either by mounting the disk file to an existing VM then adding an additional hard drive to access the information on the drive, or creating a new VM and mounting a live CD to utilize hacking utilities to defeat the operating system security, admins can bypass operating system level controls to gain access to the data simply by making a copy of the disk file and mounting it elsewhere .

Virtual servers open up additional attack vectors over physical servers, illustrating why proper security measures must be utilized to ensure that sensitive data is adequately protected in virtual environments. In addition to properly securing host servers, auditing and logging should also be in place to track all logins and activities on host servers. Administrators typically need access to sensitive data to be able to do there jobs but this access should be limited as much as possible to only what they actually need.

Many administrators snoop because they know they can get away with it. By restricting access and logging events, the 2/3rds of IT administrators who set the better example make snooping more difficult for nosey admins.

Word has it that Microsoft is finally getting it together and releasing Hyper-V, putting the tech world on notice that it is now safe to exhale.

Phew, we were all about to turn blue.

Has someone ever told you a story about some aging celebrity, and your first thought is, “Wait, you mean they’re not dead yet?’ I probably shouldn’t admit this, but when I read that Hyper-V was coming out, I thought, ‘What do you mean, it’s being released? I thought that already happened!”

My mistake, I had confused the release with another important Microsoft — ahem, milestone — in March: the Hyper-V release candidate (RC).

Excuse me for being flip, but I was bored to tears by this whole Viridian-cum-Hyper-V saga long ago. Two years ago, when I first started covering virtualization, the big news was that Microsoft had made Virtual Server 2005 available for free. Immediately thereafter, VMware returned the volley and made its hosted virtualization platform VMware Server free too, eliminating any real advantage Virtual Server 2005 may have had over the better-established GSX. So much for that story line.

Since then, we’ve lived through name changes, (Viridian to Hyper-V), release candidates, pricing announcements (why $28 dollars, why not $25? $29.99?), delays (will Microsoft meet its 180-days-after-Longhorn deadline? Will it beat it?), feature cuts, feature clarifications (“Quick migration” anyone?), and countless press articles with VMware cast as David to Microsoft’s Goliath — or is it the other way around?

Everything except an actually shipping, nonbeta, nonrelease candidate product.

Until now.

As a journalist, I’m just happy that the wait is over, and we can all stop walking around on tenterhooks, expected to drop everything every time Microsoft comes knocking at our inbox with some virtualization-related announcement that may or may not pertain to the release of Hyper-V.

Now we can all get on with our job of waiting for Microsoft to update us on the status of all the product features that it excised from Hyper-V last year: quick migration, hot add of system resources, increased numbers of CPUs, etc. What a relief!

Desktop virtualization packages rely on snapshots and virtual drive functionality. The de facto functionality standard here is found in VMware Workstation and VMware Server, but the tools in Sun’s VirtualBox may be setting a new standard. Let’s take a quick look at how snapshots and virtual drives work within Sun xVM VirtualBox.

VirtualBox snapshot technology provides the same basic functionality as the VMware products in that they can be taken while the virtual machine (VM) is running or offline.  The snapshots are taken from two different places depending on the state of the VM. For a running VM, the snapshot is taken from the running console as shown in the figure below.

When a VM is powered off, snapshots may be taken in the properties of the VM. This difference is a slight inconvenience, but is an easy learning curve to overcome. Further, if a VM needs to revert to a saved snapshot, this same location is where the VM would be reverted. VirtualBox gives the option to build from the snapshots, so there can be multiple point-in-time restores for a single VM. Snapshots in VirtualBox are kept in the .VirtualBox\Machines\VMName\Snapshots location by default, and are a collection of .VDI and .SAV files. The figure below shows three point-in-time restores for a single VM:

As with all snapshot restores, you should be sure that you want to restore as the reverting process is authoritative to the VM. Reverting to a VirtualBox snapshot taken while the system is running reverts precisely to that point with the VM running, rather than a powered off state. Overall, the functionality inventory of VirtualBox snapshot functions as advertised and brings another positive view to this exciting virtualization platform.

More information on the VirtualBox 1.6.x product can be found in the online user guide.

Attention, college students: your tuition may soon decrease!

Well, maybe not. However, VMware Inc. reported today that 900 universities including top tier schools such as Harvard and Yale are saving big bucks using VMware Inc. virtualization.

Many renowned universities that have deployed VMware to reduce capital and operating costs, increase application and system uptime, decrease power consumption and improve disaster preparedness include Cambridge, Princeton, Stanford, Purdue, the University of Maryland, the University of Auckland, and the University of California campuses at Berkeley, Los Angeles and San Diego.

These schools and hundreds more around the world are running their mission-critical enterprise applications, database systems, and education-specific applications such as CollegeNET and the Blackboard Academic Suite in VMware virtualized environments, the company reported.

Others are using VMware for disaster recovery (DR).

Bowdoin College in Maine partnered with Los Angeles-based Loyola Marymount University to build a co-located datacenter for cross-country DR. By partnering and using VMware to create back-up systems, the schools have achieved higher availability and better load balancing, with more than 70% of their environment virtualized and more than 100 virtual machines (VM). They are saving $15,000 in annual server maintenance and have avoided $500,000 in hardware costs, according to VMware.

Ohio State University has been a VMware virtualizatiton customer since 2003 when the College of Humanities needed to upgrade its IT infrastructure and found there was not enough room to expand. After deploying VMware virtualization, the College was able to meet its upgrade needs with 54 VMs running on three physical host servers. The college avoided $160,000 in hardware costs and cut server provisioning time down from three weeks to five minutes, and the IT staff can now manage all of its VMware VMs from a single console.

Clearly, the education sector is a strong market for VMware, as there are now 900 universities and colleges using the virtualization platform. Because of this, VMware created a free online tool called VMware Academic Program staffed with IT professionals from higher education facilities to answer questions about overall IT best practices. In addition to these experts, the site also includes case studies to help understand how others have implemented VMware.

Should you assign a virtual machine (VM) more than one virtual processor or not? It’s common for admins to configure virtual symmetric multiprocessing, or VMs with multiple CPUs, whether it is needed or not.The decision to use more then one virtual processor in a VM should be based on an actual requirement by the applications installed on the VM and not simply because two processors are better then one. Many physical servers commonly have multiple CPUs regardless if the applications running require them. While being wasteful of server resources, this does not negatively impact a physical server but most VMs will usually run better with one virtual processor and can actually run slower when more than one is assigned to it.

The reason for this is the hypervisor’s CPU scheduler must find simultaneous cores available equal to the number assigned to the VM. So a four VCPU VM will need to have four free cores available on the host for every CPU request that is made by the VM. If there are not four cores available because other VMs are using them then the VM must wait until the cores become available. Single VCPU VMs have a much easier time because they only need there to be a single core available for the scheduler to process CPU requests for it.

Here are some tips on assigning VCPUs to VMs:

• Limit the number of VSMP VMs on your hosts. The less you have, the better your VMs will perform.
• Assign a VM multiple VCPUs only if you are running an application that requires it and will make use of them.
• Don’t assign a VM the same amount of VCPUs as your host system has total cores available.
• If you are going to use VSMP have at least twice (preferably three or four times) the number of cores available on your host system then that of your VM with the most VCPUs. So if you have a four VCPU VM, have at least eight cores available on your host server and preferably 16.
• If you are converting a multi-CPU physical Windows server to a single VCPU VM, make sure you change the HAL from multiprocessor to uniprocessor.
• Don’t use CPU affinity as it restricts the scheduler and makes it harder to process CPU requests. The scheduler is very good at what it does, so let it do its job.

The virtualization world is still waiting for the official release of the Open Virtual Machine File Format, or OVF, once Distributed Management Task Force (DMTF) puts the finishing touches on what will be an industry standard virtual machine (VM) format. According to DMTF’s Christy Leung the organization plans to announce the release of OVF over the next couple of months.

OVF frees users from platform dependence in virtual environments, enabling them to mix and match platforms without incurring interoperability problems. Despite the clear benefits of a common format in a multiplatform virtualization landscape, a universal format has encountered some roadblocks.

Since late 2007, DMTF has worked on OVF when Dell, HP, IBM, Microsoft, VMware and XenSource submitted a proposal for a standardized format for VMs. At the upcoming Burton Group Catalyst conference later this month, DMTF member organizations — including VMware, Citrix, and Novell — will demonstrate OVF interoperability publicly for the first time. According to Burton Group analyst Chris Wolf, “some vendors moved OVF support higher up on their development roadmap in order to have it ready in time to demonstrate at the Catalyst conference.”

Wolf says that OVF is worth the wait — and the investment in the long term. “OVF has a nice long-term goal of standardizing the way hypervisors mount and run VMs,” says Wolf, “but its immediate use is primarily in importing VMs and standardize how VM metadata is managed.”

Wolf goes on to say that while OVF VMs will soon be able to load onto any hypervisor, a virtual hard disk conversion may be required as part of the import process because of the presence of two primary virtual hard disk formats in play: Virtual Machine Disk Format for VMware and Virtual Hard Disk for Microsoft and Xen. “OVF would have even more value if all vendors could agree to use a single standardized virtual hard disk format,” according to Wolf. “Thus far, the reasons for not having a single virtual hard disk format are more political than technological.”

When DTMF finishes its work, OVF will greatly improve the functionality of virtual machines. “OVF metadata is extensible, so any software vendor could use OVF to embed their management metadata inside VMs, regardless of hypervisor,” says Wolf.

“That is a big deal, as vendors could have a consistent management methodology regardless of hypervisor.”

You may hear the term SCSI reservations frequently when dealing with VMware servers that utilize shared storage. SCSI reservations are used to ensure exclusive access to disk-based resources when multiple hosts are accessing the same shared storage resources. In addition to being used by VMware hosts, SCSI reservations are also used by Microsoft Cluster Server.

SCSI reservations are only used for specific operations when metadata changes are made and are necessary to prevent multiple hosts from concurrently writing to the metadata to avoid data corruption. Once the operation completes the reservation is released and other operations can continue. Because of this exclusive lock, it is important to minimize the concurrent number of reservations that are made. When too many reservations are being made at once, you may receive I/O failures because a host is unable to make a reservation to complete an operation because another host has locked the logical unit number (LUN). When a host is unable to make a reservation because of a conflict with another host, it will continue to retry at random intervals until it is successful; however, if too many attempts are made the operation will fail.

Some examples of operations that require metadata updates include:

• Creating or deleting a VMFS datastore
• Expanding a VMFS datastore onto additional extents
• Powering on or off a VM
• Acquiring or releasing a lock on a file
• Creating or deleting a file
• Creating a template
• Deploying a VM from a template
• Creating a new VM
• Migrating a VM with VMotion
• Growing a file (e.g., a Snapshot file or a thin provisioned Virtual Disk)

Having a minimal amount of reservation conflicts is generally unavoidable and will not have a big impact on your hosts and VMs. To avoid having too many conflicts, try to limit the number of operations that can cause reservations and stagger them so too many are not happening simultaneously. All reservation errors are logged to the /var/log/vmkernel log file on each ESX host. To reduce the amount of conflicts:

• Limit the number of snapshots you have running, as snapshots grow in 16MB increments and every time they grow they cause SCSI reservations.
• Only vMotion a single VM per LUN at any one time.
• Only cold migrate a single VM per LUN at any one time.
• Do not power on/off too many VMs simultaneously.
• Limit VM/template creations and deployments to a single VM per LUN at any one time.
• Consider using smaller LUN sizes (<600GB) and do not use extents to extend a VMFS volume