Given that virtual environments for x86 servers are relatively new, most lack direct experience in performing major in-place upgrades. While there are many ways to approach a key upgrade to a virtual environment, we’ll take a look at one example of a server virtualization upgrade: VMware ESX 3.5 and VirtualCenter 2.5 to the Update 1 release of both products. This release resolved some major issues, putting the spotlight back on the new features of ESX 3.5, namely Storage VMotion.

Maintaining version control on a virtualization platform is in the best interest of ongoing administration. With VMware environments, this situation is illustrated by the sequential upgrade tasks with older versions of ESX and VirtualCenter. The first step in making a successful upgrade is to go through the release notes and scour the Internet for existing resources that can make this task less daunting. One particularly helpful resource is the RTFM Education ESX and VirtualCenter upgrade guide by Mike Laverick which goes through many scenarios with specific, step-by-step guides on almost every topic of the upgrade.

Having all of the resources in the world may still not be enough to ensure a smooth upgrade of the virtual environment. This is where a test environment for the upgrades can prove critical to a successful project. Provisioning an accurate test environment can become increasingly expensive, but can provide a beneficial test ground to ensure there are no surprises during the upgrade. Consider the test environment shown in the figure below:

This test environment is a smaller, yet representative environment of the larger environment in that it may have the same storage system, base drivers on the host systems yet simply providing a smaller workload. This environment can be an adequate test environment for all of the basic functions involved with an upgrade. As for provisioning the environment, there are some tricks available such as using the systems in an unlicensed or evaluation mode, reducing processor inventories or taking resources from the live environment if the loss can be sustained.

Planning and testing are the best defenses against an upgrade failure. Furthermore, because the scope of a virtual environment is so broad, the investment in testing and planning should be a no-brainer.

Traditionally, developing and testing applications is a labor-intensive and time-consuming process that requires IT departments to create testing environments that mirror production environments. Once a testing environment is created—with production operating systems, network configurations and the like all painstakingly recreated—the test-and-development crew may need the machines only for a few days before the environment is scrapped. For IT operations, creating and tearing down test environments is just one more activity in already overtaxed schedules.

Virtualization technology – with its inherent ability to quickly create virtual machines – has been widely embraced for test-and-dev applications. Now virtual lab management software further helps IT administrators by automating and consolidating the processes required to establish lab IT infrastructure. Many virtualization proponents view these tools as the perfect antidote to the legwork required to set up and break down lab environments.

Easing IT’s burden
Providers VMlogix, Surgient and, naturally, VMware offer virtual lab management products designed to make the build-and-tear-down process required for test and development faster and easier. (VMware Lab Manager works only with VMware environments.) The software typically enables the configuration of multiple VMs in multiple environments and integrates with third-party quality assurance and testing tools, such as HP Quality Center, Borland SilkTest, IBM Rational Build Forge and IBM Rational ClearQuest, among others. For test-and-dev folks, the payoff of such tools is faster testing and development. For IT operations, the value of such tools has more to do with labor savings and cost overhead.

For about two years, Brian Boresi, manager of client engineering at Sisters of Mercy Health System, has used Surgient’s Virtual QA/Test Lab Management System (QTMS) test applications as part of an enterprise desktop refresh.

Before getting the tool, a subject matter expert would spend more than a week in a central lab testing a new system against core applications. Today, that process has been whittled down to about four hours. “An SME creates testing scripts based on a onetime visit to the lab,” Boresi said. “The virtual test tool automates the scripts which we run in a test environment on a VMware ESX server.”

Theresa Lanowitz, president of voke Inc. , an IT research firm, has studied the benefits of virtual lab management technology and said that such results as Boresi’s are fairly typical. With virtualized lab environments, Lanowitz said, “developers want to test in an environment as close to production as possible, and operations don’t have to set up a lab.”

Self-service provisioning
At Vignette Corp. , a software company, virtual lab technology enables developers and QA testers to provision their own test environments. The company uses LabManager from VMLogix, which includes self-service automation technology, allowing end users to create their own VMs without the intervention of IT operations. “Users now log in and self-service images for themselves,” said Rob O’Neill, Vignette’s senior manager of IT. “With automated workflows, users can check out machines, run them for testing, and then tear them down once they are finished.” The turnaround time for creating test environments ranges from about five minutes to 20 minutes, O’Neill said.

While VM sprawl has become an issue in production environments, it’s also a challenge for test and development. Bart Burkhard, manager of engineering for Overwatch Systems, a provider of software for military command and business information analysis, is currently piloting VMLogix’s LabManager in part to contain VM sprawl. “We have a number of disconnected labs and data centers used by developers and testers,” Burkhard said. “The disconnected labs and parallel projects make physical resource allocation and discovery difficult for us.”

Saving money, improving access to resources
For this reason, Overwatch opted to move test and dev from a physical to a virtual environment, Burkhard said, but the company was wary of the sprawl that could result. With LabManager, Overwatch now maintains a single repository of VMs that track how they are utilized by the company’s test and development staff. “As leases come up for various desktops in the labs, we’ll incrementally replace physical machines with VMs.”

From Burkhard’s perspective, the benefits of using a lab management environment are twofold. From a business perspective, it helps save money on items such as leases, power and cooling because it facilitates the move from physical to virtual environments. For end users, the use of lab management software is getting them access to resources faster. “The time we spent to allocate a machine into a lab with any disk size and memory based on the VMs we have is down from three days to one hour,” Burkhard said.

Almost every virtualization admin that I interact with has materially changed their strategy at some point with their first generation of server virtualization before the entire project is complete. Among the strategy changes are those related to network zoning, which will become a more important consideration as organizations approach higher levels of virtualization.

Specifically, the placement of external facing systems on the same virtual host as systems which house internal systems can put both sides of the network at risk if a compromise is made to the hypervisor from the external facing systems. This becomes especially important as the virtual appliance space allows organizations to easily consider firewall, intrusion detection, VPN and other external facing roles to be placed into the virtual environment as well as the frequent goal to virtualize everything.  

A more isolating strategy creates a separate environment with hosts dedicated to hosting virtual machines (VMs) that are external facing and not simultaneously host VMs on the internal network. While the hosts may be connected both to the internal and external networks in a DMZ network role, a compromise to the hypervisor or host system would not have as direct of an impact to the VMs running only on the internal networks. This also helps in emergency remediation by allowing a virtual host to be fully isolated or powered off until the issue is identified without impacting the internal network VMs.

When planning your next generation of server-side virtualization, consider the risks of placing internal and external network zones on resources that may contain external facing and internal only VMs. This type of architecture can bake in some inherent security into your environment that may save the day in the event of a zero-day vulnerability situation that affects the guest operating system or the virtualization hypervisor.

Palo Alto, Calif.-based VMware, Inc. announced the general availability of VMware Lifecycle Manager, which was first announced and covered by SearchServerVirtualization.com in February.

The product is VMware’s attempt to control virtual machine sprawl by showing who owns a virtual machine, when it was requested, who approved it, where it is deployed, how long it has been in operation and when it is scheduled to be decommissioned.

VMware Lifecycle Manager also gives IT managers the ability to measure and chargeback the use of virtual machines to individual department owners.

VMware offers other lifecycle management products, like VMware Stage Manager  and VMware Lab Manager, that are designed for specific functions and differ from the new LifeCycle Manager product.

So while VMware Lifecycle manager helps manage the creation, operation and decommissioning of virtual machines in compliance with company policies and standards, VMware Stage Manager helps to transition application stacks of multiple virtual machines through the integration and staging process prior to production. VMware Lab Manager helps with provisioning and allows manageability over the entire virtual environment.

There are a number of companies that offer virtual machine lifecycle management software similar to VMware’s, and the majority of them are based on VMware virtualization. They have the pleasure of selling against VMware while also supporting the virtualization giant’s product.

Some virtual machine lifecycle management vendors to consider if you are on the market for one of these tools include a new Alpharetta, GA-based company called vmSight that offers software for application performance, capacity planning, VM Sprawl control, billing and chargeback and regulatory compliance for VMware virtualization. Similarly, products from Buffalo Grove, IL.-based vizioncore Inc. with vCharter and Portsmouth, NH-based vKernel, which received kudos from Gartner Inc. for its VKernel Virtual Appliance Suite for Systems Management, offer virtualization lifecycle management products that go head to head with VMware’s offering.

VMware Lifecycle Manager is now available for purchase through VMware’s network of distributors, resellers and OEMs. VMware Lifecycle Manager is purchased a la carte and requires the purchase of the standard product offering per processor. There is also a customization option (one per VMware Lifecycle Manager server) that will allow customers to tailor fit VMware Lifecycle Manager into existing organizational tools and processes.

We all know that test environments are an important part of the quality process within IT. Unfortunately, we may not always follow through and provide good test environments. Virtualization changes all of that. This tip will share some of the strategies that I have found to truly enable robust test procedures beyond the basic testing on virtual machines.

100% Representative environment

With physical to virtual (P2V) conversion tools, I have been successful in creating test environments that are exact duplicates of what I am testing. A good example is a Windows server functioning as an Active Directory Services domain controller. Generally, I consider it a bad practice to perform a P2V conversion on a domain controller for continued use. But, in the case of a test environment, a P2V conversion is a great way to get a “copy” of your Active Directory domain into your test environment. For those of you wondering about the connectivity, of course the networking needs to be isolated from the live network.

With a P2V performed on a domain controller, I have had a great environment to test top-level security configurations, major group policy changes, and Active Directory schema updates. Outside of this type of test environment, these types of changes are usually difficult to simulate well. Sure we can create a development Active Directory domain, but it would not be fully representative of the live environment.

Performance testing

For many people that are new to virtualized environments, there may be some skepticism on virtual system performance. Providing test environments on virtual systems is nothing new, but our challenge is to make the test environments fully equivalent of the planned configuration. One strategy that can be implemented with VMware Infrastructure 3 is to have a small quantity of ESX hosts that are fully licensed and configured like the rest of the servers in the environment and refer to that as a development cluster or quality environment. In the development cluster, you can showcase high availability functionality, virtual machine migration, and fault tolerance to get the support of the business owners. Further, if the perfomance of the development environment is comparable to that of the live environment, the confidence of the virtual system is increased.

Caution factor

With this added functionality, it is also a little easier to cause issues with the live systems. With the example of having a fully-functional Windows domain controller, serious issues could result if that system is accidentally connected to the live network. Because of this risk, a good practice is to make virtual networks that are completely isolated. This goes beyond simply creating a test network within the virtual environment, but to configure the virtual host system to not permit any virtual machine connectivity to the live network. This makes file exchanges a little more difficult, but there are plenty of tools to assist in this space.

Readers, I invite you to share your virtualization test environment strategies. What has helped you deliver a better test procedure by using a virtual test environment?

Test-and-development environments that want to see how software runs on Citrix Systems Inc.’s XenServer virtual machines can now do so, thanks to VMLogix, which has added Citrix XenServer to the list of platforms supported by its LabManager offering.

Citrix XenServer joins a comprehensive list of virtualization platforms supported by VMLogix, including VMware ESX Server, VMware Server, and Microsoft Virtual Server; support for Oracle VM and Sun xVM is also forthcoming, said CEO Sameer Dholokia. For the time being, Dholokia said, the company has seen “a fair bit of interest in testing VMLogix on Citrix XenServer.”

The VMLogix offering competes directly with VMware’s Lab Manager and conceivably with the newly announced Stage Manager. In fact, Dholokia claims that VMLogix’s offering already includes much of the functionality included in Stage Manager and said that VMware customers may not understand the distinction between the Lab and Stage Manager products. “It will be interesting to see how they manage the confusion factor: ‘When do I use [VMware] Lab Manager, when do I use Stage Manager?’” Dholokia said.

Pricing for VMLogix LabManager is $25,000, plus a $2,500 agent fee per two-CPU server.