One of the more overlooked placement discussions that happen within the design or re-engineering phases of virtualization projects involves systems that are on an external network.

The placement of external systems can be addressed many different ways, including the use of virtual private network (VPN) authentication servers, web servers or remediation systems for network access control. Consider the following architecture diagram where larger virtualization hosts contain all types of systems within the virtualized environment:

While the networking of these virtual machines may be configured with the same protections as their physical counterparts, there are some concerns with this configuration. This can become even more of a concern in the event where the firewall is a virtual machine as well in the same environment. An architecture that can better protect the internal and external workloads would be to have a separate environment with connectivity and workloads only to the external interfaces. Consider the figure below for the same workload:

In this manner, more hosts may be needed for the same workload to account for maintenance mode and other factors when separated. These additional hosts may be configured with smaller hosts and smaller processor inventory to not incur any additional costs or licensing for anything that is licensed by processor.

If firewall or other core network appliances are virtualized, their placement requires a little more thought because they may have a footprint on both the internal and external networks. In the case of shared resources of internal and external workloads, an outbreak type event on an external system may have resources consumed at the expense of the internal workload. By having the internal and external workloads separated, the risk of attacks within the operating system or an attack that targets virtual machines would be initially contained by internal and external workloads.

This strategy can be applied to all virtualization products, and can also be applied more specifically to network and storage configurations to protect in the same fashion. 

A general rule of thumb in virtual environments is to always treat virtual machines the same as you would physical servers. While this rule holds true in many cases, IT administrators should be aware of some exceptions to this rule. Let’s go over some reasons that you would not treat your virtual machines like physical servers:

• Patching – You should apply all the same operating system and application patches to a virtual machine as you would a physical server. However it is best to stagger your patch deployments so you do not patch and restart all of your virtual machines at the same time. If you did this concurrently you can cause excessive resource utilization on your host servers which could impact other virtual machines running on the host.
• Securing – Secure the virtual machine operating system as you would physical servers, in addition you should ensure that you have proper security setup on the host server’s management console that allows access to the VM as well as on the virtual machine files located on the host server’s disk system. It does no good to have tight security inside your VM and have weak security outside.
• System Monitoring – This is one area that can be very different for virtual servers. There is no need to monitor virtual machine hardware, if you have converted physical servers to virtual machines you should make sure you un-install any hardware management agents from them. In addition virtual machines boot much faster then physical servers. Because of this, many monitoring systems will not detect server re-boots because the boot process happens quicker then the monitoring interval. You may find that you need to adjust your polling interval for virtual servers so you can detect the faster re-boots.
• Performance Monitoring – Another area that is very different from physical servers. Traditional operating system performance reporting tools are often inaccurate when used on virtual machines because they are unaware of the virtualization layer and the underlying physical hardware. You should always use virtual server specific reporting tools to accurately measure performance on virtual machines.
• Anti-virus – Make sure you install anti-virus software on all your virtual machines the same as physical servers. Again one thing to be careful of is to stagger any on-demand scans and definition updates as to not overwhelm the host server. Having all your VMs running a full scan at the same time can completely bog down a host server.
• Backups – It’s OK to backup your virtual machines using traditional operating system backup agents. Always make sure you do not backup too many VMs on a single host at the same time. There are more efficient ways to perform backups in a virtual environment that you may look into to either complement or replace traditional backup methods.
• Disk defragging – You should periodically defrag virtual machine disks using traditional operating system tools for maximum performance. However be careful not to defrag a VM that has a snapshot running, doing this can cause the snapshots rapidly grow in size and degrade host performance. As usual do not defrag more then one VM on a host at a single time because of all the excessive disk activity that is causes.

Be careful not to do too many of the same operations concurrently. With physical servers, only a single server is effected, but in virtual environments many other servers running on a host server can be impacted.

So, earlier this week I wrote a blog about Clabby Analytics Analyst Joe Clabby’s report spelling out a handful of reasons why Microsoft’s Hyper-V is going to take the lead in the virtualization market away from VMware Inc. over the next five years.

I received a lot of feedback on this blog from people defending VMware, and thought, why not get some Hyper-V users to talk to me about the product - how it performs, its related management tools, features, etc. I asked Microsoft’s press team to send some users my way for interviews, and about a week later Microsoft’s “Rapid Response” team sent me a couple of links to case studies.

Thanks, but I would like to interview some users myself, outside of Microsoft filters. How about at least sending me the contact info for the users profiled in these case studies?

Microsoft’s response was, “Unfortunately regarding direct contact information for the Hyper-V case studies, we have no further information to share.”

What? Really?

This strikes me as odd because Microsoft’s competition, VMware and even smaller virtualization companies like Virtual Iron refer me to real users to interview about their products.

Does this mean that Microsoft doesn’t have the same level of product confidence as the competition? VMware has offered plenty of customer references, and while those users do complain about the acquisition cost of VMware’s software, I don’t think I’ve heard any serious gripes about the product itself.

So I am interested in hearing from Hyper-V users about its performance, because as users and analysts have said, Microsoft won’t sail past VMware on price alone.

Clabby Analytics analyst Joe Clabby is 100% convinced that Microsoft’s Hyper-V will take over VMware in market share over the next three to five years, and makes some strong points for this in his recent report, Six Reasons Why Microsoft’s Hyper-V will Overtake VMware to Become the Major Player in the x86 Server Virtualization Marketplace.

The report came out prior to the shake-up at VMware on July 8, when the company announced that its Board of Directors replaced VMware co-founder and CEO Diane Greene was being replaced, and then lowered its revenue forecast.

VMware had the vision to see the value of virtualization and took the technology to the top unchallenged due to strategy, innovation and sales execution, but that ride is about to come to an end, Clabby said.

“With the introduction of Hyper-V by Microsoft, VMware is about to experience some very serious competition from a vendor with deep pockets, with a massive worldwide marketing and sales organization, with major market penetration across Fortune 500 and small and medium business markets, and with extensive and complementary infrastructure and management product depth,” Clabby reported.

Among the reasons Clabby believes Microsoft will crush VMware are that Microsoft already has an expansive installed base, a mammoth network of direct sales and indirect business partners, and is offering lower prices alternatives to VMware’s hypervisor and related infrastructure/management software products.

Unfortunately, I have to agree. History tends to repeat itself, and this has been Microsoft’s strategy for a very long time: see a great technology, copy it, and outprice the rest of the market.

Vanity Fair’s July issue had a great article that illustrates this, called “How the Web was Won” that looks at the eveolution of the Internet over the past 50 years, including details of how Microsoft took over Netscape Navigator by developing Internet Explorer.

The computer programmer known for founding Netscape Communications, Lou Montulli, told Vanity Fair, “From a scientific point of view none of us really respected Microsoft. There was definitely a sense of: They’ve put out of business three or four major companies, and they did it simply by copying what they did and outpricing or outmaneuvering them in the market. This is a general feeling of computer scientists everywhere, that Microsoft doesn’t tend to innovate as much and really just enters the market late, takes it over, and then stays at the top.”

Pricing aside, Microsoft already has a massive installed base.

“It will leverage this installed base, and price its products to out-function/undercut VMware’s pricing,” Clabby wrote. “The computing industry saw this same situation arise when Citrix built a leadership base for its terminal server products — only to have Microsoft enter the market and claim significant marketshare after Citrix pioneered the terminal server marke umbrella. Almost the exact same situation is about to happen again — this time between VMware and Microsoft.”

Microsoft also has a packaging advantage with its Hyper-V hypervisor, as it can be delivered with every single version of 64-bit Windows Server 2008, and installing Hyper-V is a cake walk, according to Clabby.

“A box simply needs to be checked during installation and Hyper-V becomes active. By not requiring IT buyers to find/acquire/download additional virtualization software, the job of deploying and testing virtualization within a Windows Server 2008 is greatly simplified. VMware cannot counter this packaging advantage,” Clabby wrote.

The most damning problem for VMware, according to Clabby, is product depth.

Though VMware has the advantage of technologies like VMotion, to move live VMs, and all of the handy add-on management and infrastructure software integrated into VMware, Clabby said Microsoft’s management and infrastructure is far deeper.

Microsoft’s Systems Center product portfolio inlcludes systems management tools like Configuration Manager; Operations Manager; Data Protection Manager; Virtual Machine Manager; System Center Essentials; Capacity Planner, and the list goes on, ad nauseum.

Besides all of those points, Microsoft is a $51 billion dollar software company and VMware’s revenue is just over $1 billion.

In short, given its deep pockets, large installed base and virtualization strategy, it is safe to say Microsoft will, once again, be laughing all the way to the bank.

Given that virtual environments for x86 servers are relatively new, most lack direct experience in performing major in-place upgrades. While there are many ways to approach a key upgrade to a virtual environment, we’ll take a look at one example of a server virtualization upgrade: VMware ESX 3.5 and VirtualCenter 2.5 to the Update 1 release of both products. This release resolved some major issues, putting the spotlight back on the new features of ESX 3.5, namely Storage VMotion.

Maintaining version control on a virtualization platform is in the best interest of ongoing administration. With VMware environments, this situation is illustrated by the sequential upgrade tasks with older versions of ESX and VirtualCenter. The first step in making a successful upgrade is to go through the release notes and scour the Internet for existing resources that can make this task less daunting. One particularly helpful resource is the RTFM Education ESX and VirtualCenter upgrade guide by Mike Laverick which goes through many scenarios with specific, step-by-step guides on almost every topic of the upgrade.

Having all of the resources in the world may still not be enough to ensure a smooth upgrade of the virtual environment. This is where a test environment for the upgrades can prove critical to a successful project. Provisioning an accurate test environment can become increasingly expensive, but can provide a beneficial test ground to ensure there are no surprises during the upgrade. Consider the test environment shown in the figure below:

This test environment is a smaller, yet representative environment of the larger environment in that it may have the same storage system, base drivers on the host systems yet simply providing a smaller workload. This environment can be an adequate test environment for all of the basic functions involved with an upgrade. As for provisioning the environment, there are some tricks available such as using the systems in an unlicensed or evaluation mode, reducing processor inventories or taking resources from the live environment if the loss can be sustained.

Planning and testing are the best defenses against an upgrade failure. Furthermore, because the scope of a virtual environment is so broad, the investment in testing and planning should be a no-brainer.

If you have not noticed, I have been on a Sun xVM VirtualBox kick recently. I think it is beneficial to virtualization administrators and managers to be familiar with at least two hypervisors — so why not learn more about xVM?

VirtualBox has a smooth interface for a version 1 release, but the one area that would require the most adjustment is the virtual networking. Let’s take a closer look at network functionality in VirtualBox.

Virtual networking on VirtualBox has a few key differences that VMware users would need to develop an understanding about before fully utilizing the potential of the product. The first difference is the concept of the virtual networking hardware. VirtualBox allows a virtual machine (VM) to have one of four network interface cards virtually assigned. These are the AMD PCNet PCI II, AMD PCNet FAST III, Intel Pro/1000 T and the Intel Pro/1000 MT. This array of virtual adapters allows a VM to have broad support for multiple operating systems, but the corresponding bridging functionality may make network administrators a little uneasy.

Spanning Tree
For Windows systems, VirtualBox uses a spanning tree algorithm from the native operating system bridging that may cause issues on systems with multiple interfaces in managed network environments. The bridged network functionality puts the VMs on the same physical network as the VirtualBox host system. In this fashion, a VM would be able to retrieve a DHCP network from the physical network and interact as if it were placed on the network parallel to the host. Windows XP and Server 2003 products’ bridging functionality is explained on the TechNet website.

Another key difference is that in order for a VM to use the bridged network is the addition of a bridging interface. Adding an interface is fairly straight forward with the use of the VBoxManage command. The following command would add a bridging interface named “VM-Bridge”:

VBoxManage createhostif "VM-Bridge"

Once this command is completed, the VM-Bridge interface is now present in the network connections inventory of the Windows control panel. Then a VM can be configured to use bridged networking with the newly created interface as shown in the figure below:

At this point, the VM-Bridge interface can transparently place the VM on the same network as the host when the Windows bridged connections are correctly configured. Note also that in the network configuration you can fully edit the MAC address of the VM. While exceptionally convenient, this can introduce risk for some environments and situations.

Now that we have gone through a quick look at VirtualBox’s implementation of bridging network connections for VMs, I would have to nudge the VMware products to be a little more seamless in the category of bridged networking. By having the VMware bridge protocol binding used instead of a separate series of adapters for the same purpose, VMware’s bridging fits better for most environments.

Make no mistake, the comprehensive VirtualBox networking implementation is fully competitive with VMware. There is much more to the VirtualBox networking implementation available for download in the online user guide in section 6.

Anyone with five minutes of IT experience knows that vendors sometimes publish bogus “benchmarks” that portray their products in the best of all possible lights. Virtualization guru and Burton Group analyst Chris Wolf recently uncovered a particularly spectacular example of this, courtesy of QLogic and Microsoft.

In a release, QLogic Corp., a networking technology provider, said it tested virtual machines running on Windows Server 2008 Hyper-V and attached to a storage area network (SAN) via its 8 Gbps Fibre Channel (FC) host bus adapters, and saw near-native performance of 200,000 I/O operations per second (IOPS).

But, as Wolf discovered, what QLogic failed to mention was that it ran its tests against a very unusual SAN array: the Texas Memory RamSan 325 FC, which uses solid-state storage. Further, the benchmark used block sizes of just 512 bytes, compared with a more real-world block size of 8 K or 16 K.

This left Wolf feeling duped and betrayed:

If I was watching an Olympic event, this would be the moment where after thinking I witnessed an incredible athletic event, I learned that the athlete tested positive for steroids.

Wolf ran this benchmark by a colleague, who calculated that had the same benchmark been performed using “real disks” with latency of 7 milliseconds, it would have limited throughput to a much less impressive 9,142 IOPS. Hardly anything to write home about.

Thanks to Wolf for taking the time to look into this.

Traditionally, developing and testing applications is a labor-intensive and time-consuming process that requires IT departments to create testing environments that mirror production environments. Once a testing environment is created—with production operating systems, network configurations and the like all painstakingly recreated—the test-and-development crew may need the machines only for a few days before the environment is scrapped. For IT operations, creating and tearing down test environments is just one more activity in already overtaxed schedules.

Virtualization technology – with its inherent ability to quickly create virtual machines – has been widely embraced for test-and-dev applications. Now virtual lab management software further helps IT administrators by automating and consolidating the processes required to establish lab IT infrastructure. Many virtualization proponents view these tools as the perfect antidote to the legwork required to set up and break down lab environments.

Easing IT’s burden
Providers VMlogix, Surgient and, naturally, VMware offer virtual lab management products designed to make the build-and-tear-down process required for test and development faster and easier. (VMware Lab Manager works only with VMware environments.) The software typically enables the configuration of multiple VMs in multiple environments and integrates with third-party quality assurance and testing tools, such as HP Quality Center, Borland SilkTest, IBM Rational Build Forge and IBM Rational ClearQuest, among others. For test-and-dev folks, the payoff of such tools is faster testing and development. For IT operations, the value of such tools has more to do with labor savings and cost overhead.

For about two years, Brian Boresi, manager of client engineering at Sisters of Mercy Health System, has used Surgient’s Virtual QA/Test Lab Management System (QTMS) test applications as part of an enterprise desktop refresh.

Before getting the tool, a subject matter expert would spend more than a week in a central lab testing a new system against core applications. Today, that process has been whittled down to about four hours. “An SME creates testing scripts based on a onetime visit to the lab,” Boresi said. “The virtual test tool automates the scripts which we run in a test environment on a VMware ESX server.”

Theresa Lanowitz, president of voke Inc. , an IT research firm, has studied the benefits of virtual lab management technology and said that such results as Boresi’s are fairly typical. With virtualized lab environments, Lanowitz said, “developers want to test in an environment as close to production as possible, and operations don’t have to set up a lab.”

Self-service provisioning
At Vignette Corp. , a software company, virtual lab technology enables developers and QA testers to provision their own test environments. The company uses LabManager from VMLogix, which includes self-service automation technology, allowing end users to create their own VMs without the intervention of IT operations. “Users now log in and self-service images for themselves,” said Rob O’Neill, Vignette’s senior manager of IT. “With automated workflows, users can check out machines, run them for testing, and then tear them down once they are finished.” The turnaround time for creating test environments ranges from about five minutes to 20 minutes, O’Neill said.

While VM sprawl has become an issue in production environments, it’s also a challenge for test and development. Bart Burkhard, manager of engineering for Overwatch Systems, a provider of software for military command and business information analysis, is currently piloting VMLogix’s LabManager in part to contain VM sprawl. “We have a number of disconnected labs and data centers used by developers and testers,” Burkhard said. “The disconnected labs and parallel projects make physical resource allocation and discovery difficult for us.”

Saving money, improving access to resources
For this reason, Overwatch opted to move test and dev from a physical to a virtual environment, Burkhard said, but the company was wary of the sprawl that could result. With LabManager, Overwatch now maintains a single repository of VMs that track how they are utilized by the company’s test and development staff. “As leases come up for various desktops in the labs, we’ll incrementally replace physical machines with VMs.”

From Burkhard’s perspective, the benefits of using a lab management environment are twofold. From a business perspective, it helps save money on items such as leases, power and cooling because it facilitates the move from physical to virtual environments. For end users, the use of lab management software is getting them access to resources faster. “The time we spent to allocate a machine into a lab with any disk size and memory based on the VMs we have is down from three days to one hour,” Burkhard said.

Desktop virtualization packages rely on snapshots and virtual drive functionality. The de facto functionality standard here is found in VMware Workstation and VMware Server, but the tools in Sun’s VirtualBox may be setting a new standard. Let’s take a quick look at how snapshots and virtual drives work within Sun xVM VirtualBox.

VirtualBox snapshot technology provides the same basic functionality as the VMware products in that they can be taken while the virtual machine (VM) is running or offline.  The snapshots are taken from two different places depending on the state of the VM. For a running VM, the snapshot is taken from the running console as shown in the figure below.

When a VM is powered off, snapshots may be taken in the properties of the VM. This difference is a slight inconvenience, but is an easy learning curve to overcome. Further, if a VM needs to revert to a saved snapshot, this same location is where the VM would be reverted. VirtualBox gives the option to build from the snapshots, so there can be multiple point-in-time restores for a single VM. Snapshots in VirtualBox are kept in the .VirtualBox\Machines\VMName\Snapshots location by default, and are a collection of .VDI and .SAV files. The figure below shows three point-in-time restores for a single VM:

As with all snapshot restores, you should be sure that you want to restore as the reverting process is authoritative to the VM. Reverting to a VirtualBox snapshot taken while the system is running reverts precisely to that point with the VM running, rather than a powered off state. Overall, the functionality inventory of VirtualBox snapshot functions as advertised and brings another positive view to this exciting virtualization platform.

More information on the VirtualBox 1.6.x product can be found in the online user guide.

As server virtualization technology makes its way from test environments into production, IT organizations are struggling to keep up with the inherent management challenges involved in dealing with virtual environments.

The ease with which VMs are created makes it that much easier for VMs to be launched and moved willy-nilly regardless of the security and software licensing cost issues, just to name two common problems. Vendors of course have been hip to these challenges. This month, Embotics Corp. released version 2.0 of its V-Commander management software designed to automatically nip virtual sprawl in the bud. One way the software does this is by automatically enforcing policy dictating such things as VM expiration dates and through role-based security access that defines just who can do what in terms of VM creation and migration.

Also this month, Netuitive Inc. revamped its Service Analyzer business service management (BSM) software to include virtualization management capabilities. Nick Sanna, Netuitive’s president and CEO, said the company’s self-learning correlation software can monitor the status of applications across the environment, whether they are physical or virtual. “The idea [behind Service Analyzer],” said Sanna, is to eliminate IT management silos by automating performance management and providing end-to-end visibility into business services.”