One of the more overlooked placement discussions that happen within the design or re-engineering phases of virtualization projects involves systems that are on an external network.

The placement of external systems can be addressed many different ways, including the use of virtual private network (VPN) authentication servers, web servers or remediation systems for network access control. Consider the following architecture diagram where larger virtualization hosts contain all types of systems within the virtualized environment:

While the networking of these virtual machines may be configured with the same protections as their physical counterparts, there are some concerns with this configuration. This can become even more of a concern in the event where the firewall is a virtual machine as well in the same environment. An architecture that can better protect the internal and external workloads would be to have a separate environment with connectivity and workloads only to the external interfaces. Consider the figure below for the same workload:

In this manner, more hosts may be needed for the same workload to account for maintenance mode and other factors when separated. These additional hosts may be configured with smaller hosts and smaller processor inventory to not incur any additional costs or licensing for anything that is licensed by processor.

If firewall or other core network appliances are virtualized, their placement requires a little more thought because they may have a footprint on both the internal and external networks. In the case of shared resources of internal and external workloads, an outbreak type event on an external system may have resources consumed at the expense of the internal workload. By having the internal and external workloads separated, the risk of attacks within the operating system or an attack that targets virtual machines would be initially contained by internal and external workloads.

This strategy can be applied to all virtualization products, and can also be applied more specifically to network and storage configurations to protect in the same fashion. 

Given that virtual environments for x86 servers are relatively new, most lack direct experience in performing major in-place upgrades. While there are many ways to approach a key upgrade to a virtual environment, we’ll take a look at one example of a server virtualization upgrade: VMware ESX 3.5 and VirtualCenter 2.5 to the Update 1 release of both products. This release resolved some major issues, putting the spotlight back on the new features of ESX 3.5, namely Storage VMotion.

Maintaining version control on a virtualization platform is in the best interest of ongoing administration. With VMware environments, this situation is illustrated by the sequential upgrade tasks with older versions of ESX and VirtualCenter. The first step in making a successful upgrade is to go through the release notes and scour the Internet for existing resources that can make this task less daunting. One particularly helpful resource is the RTFM Education ESX and VirtualCenter upgrade guide by Mike Laverick which goes through many scenarios with specific, step-by-step guides on almost every topic of the upgrade.

Having all of the resources in the world may still not be enough to ensure a smooth upgrade of the virtual environment. This is where a test environment for the upgrades can prove critical to a successful project. Provisioning an accurate test environment can become increasingly expensive, but can provide a beneficial test ground to ensure there are no surprises during the upgrade. Consider the test environment shown in the figure below:

This test environment is a smaller, yet representative environment of the larger environment in that it may have the same storage system, base drivers on the host systems yet simply providing a smaller workload. This environment can be an adequate test environment for all of the basic functions involved with an upgrade. As for provisioning the environment, there are some tricks available such as using the systems in an unlicensed or evaluation mode, reducing processor inventories or taking resources from the live environment if the loss can be sustained.

Planning and testing are the best defenses against an upgrade failure. Furthermore, because the scope of a virtual environment is so broad, the investment in testing and planning should be a no-brainer.

New vendors, strategies, technologies and capabilities seem to present themselves daily to the virtualization administrator and manager. One resource that can help is the Intel Premier IT Professional (IPIP) community.

Today I had the opportunity to attend the IPIP event here in Columbus, Ohio. The meeting provided a great vendor-independent view of virtualization products that revolve around Intel technologies. Planning your virtualization hardware environment is critical to the decisions that will be made in your current and future virtualization implementations.

Between now and the end of the year, Intel is conducting ten more of these events throughout North America. The agenda of these events includes sessions in the following areas:

• Intel product roadmap
• Client virtualization strategies
• Consolidation efficiencies through virtualization
• Application virtualization strategies

One important advantage to attending the events is that you can have access to non-disclosure information about the processor product line, a key planning part of virtual environments. But the live events are only the tip of the iceberg. On the IPIP website, members can access case studies, presentations, videos and white papers anytime. Also, every page on the IPIP site has a popularity tag that content of all types can be viewed from the tags.

The best part of these resources is that they are free. Check out the Intel Premier IT Professional website and register for an event in your area.

Big players in the virtualization world griped about the absence of performance benchmarks for virtual machines on CIO Talk Radio yesterday and discussed some of the issues surrounding virtualization standards.

Guests on the show included: Simon Crosby, Chief Technology Officer of the Virtualization and Management Division of Citrix; Tom Bishop, Chief Technology Officer, of BMC Software; Dr. Tim Marsland, Sun Fellow, Chief Technology Officer, for the Software Organization at Sun Microsystems Inc.; and Brian Stevens, Chief Technology Officer and Vice President of Engineering at Red Hat.

The glaring ommission in this lineup: VMware, Inc.

The panelists on CIO Talk Radio didn’t mention VMware by name, but did complain that some companies aren’t being open with their performance data, thus prohibiting the virtualization industry from publishing comparative performance data.

VMware’s licensing agreement for ESX allows users to conduct internal performance testing and benchmarking studies, and allows those users (and not unauthorized third parties) to publish or publicly disseminate the data provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study.

Users that have published benchmark data, like Sr. Systems Engineer Mark Foster did on his blog, have had to unpublish results because of VMware’s stipulations.

VMware introduced its own free benchmarking tool, VMmark, last year for certain applications.

Meanwhile, the SPEC Virtualization Committee has been working to create standard benchmarks for VMs. The committee’s goals are to deliver a benchmark that will model server consolidation of commonly virtualized systems such as application servers, web servers and file servers; provide a means to compare server performance while running a number of VMs; and produce a benchmark designed to scale across a wide range of systems.

SPEC expects these benchmarks to be available by the end of this year, but the timeline is not set in stone, according to the website.

Sun’s Marsland said benchmarking progress has been slow because there isn’t an easy way to define a workload, and a large number of benchmarks are required.

“We are talking about a virtual computer, with lots of aspects that need to be benchmarked,” Marsland said. “Every component that gets virtualized needs to be benchmarked.”

Having an open, standardized way of benchmarking is expected to push virtualization further into the mainstream because it will eliminate false perceptions about performance, panelists said. For instance, “there is the thought that I/O intensive workloads can not be virtualized, and the absence of benchmarks prevents us from proving otherwise. It is important for us to have good benchmarks out there,” one panelist on the show said.

Though users look at benchmarks, this type of data is most useful to vendors and OEMs who can use the performance standards to improve the technology, and of course, market their products.

“More open scrutiny of performance results will help us to improve as an industry overall,” Bishop said. “There are ways to measure performance in non-virtual environments, and people are adapting those techniques to get the most out of their virtualized environments.”

In terms of application performance in virtual environments, the issues differ depending on the data center infrastructure. The network, the servers and the storage all affect performance, said Stevens of RedHat.

“The areas that have to progress are around I/O. Intel and AMD are improving around page tables, and we will see improvements around I/O adapters soon,” Stevens said.

Another problem with virtualization? There are support challenges. If an application running in a VM starts acting wacky, the application vendor may not support it, Crosby said.

Licensing and support in virtual environments has been a major gripe with Oracle, for example, which does not support running its applications with VMware.

“It is a reasonable concern…right now there is irrational market based control. Some folks are abstaining from supporting certain apps [in virtual envionments]. As customers demand support, things will hopefully get rational, by next year I hope,” Crosby said.

I was recently asked, “do you have any visibility of the storge utilization you provide your virtual machines?” I stopped, thought about it and said “no”. However, in my situation, this is not yet a problem.

A pitfall for most enterprise server virtualization strategies is in a reservation for storage, regardless of what the virtual machine has written on the virtualized filesystem to the defined maximums. For example, if I have a base installation of a Windows Server 2003 system, the footprint as I do my server builds will be around 5 GB. My standard build allocation is 32 GB. This makes this system only 15.6% utilized from inception. This rule of thumb applies to most servers, and a standard build has 32 GB as an accepted footprint per system. 

Excluding backend storage virtualization and de-duplication strategies, what about systems that have a storage footprint larger than 32 GB? Well, luckily we’ve been down this path before:

The storage is the storage, virtual or physical.

Managing the percentage of utilization for shared storage should be a task of continuing diligence. I don’t (yet) have a large number of virtual servers with a footprint above the standard build, these systems face the same battles we have had for years with general purpose servers.  As an example, take a main file and print server that is 2 TB on a general purpose server: It will be about 2 TB on a virtual server as well from the storage perspective. For large storage footprints using iSCSI or storage-area network (SAN) technologies, the difference in configuration is minimal.

However, how do we address the first question about under-utilized storage footprints for the virtualized systems? Is it best to look only at operating system metrics? That may be an adequate solution for each operating system, but the aggregation will be from different sources and outputs. What are you doing to address storage utilization when you are not using storage virtualization?

Almost every virtualization admin that I interact with has materially changed their strategy at some point with their first generation of server virtualization before the entire project is complete. Among the strategy changes are those related to network zoning, which will become a more important consideration as organizations approach higher levels of virtualization.

Specifically, the placement of external facing systems on the same virtual host as systems which house internal systems can put both sides of the network at risk if a compromise is made to the hypervisor from the external facing systems. This becomes especially important as the virtual appliance space allows organizations to easily consider firewall, intrusion detection, VPN and other external facing roles to be placed into the virtual environment as well as the frequent goal to virtualize everything.  

A more isolating strategy creates a separate environment with hosts dedicated to hosting virtual machines (VMs) that are external facing and not simultaneously host VMs on the internal network. While the hosts may be connected both to the internal and external networks in a DMZ network role, a compromise to the hypervisor or host system would not have as direct of an impact to the VMs running only on the internal networks. This also helps in emergency remediation by allowing a virtual host to be fully isolated or powered off until the issue is identified without impacting the internal network VMs.

When planning your next generation of server-side virtualization, consider the risks of placing internal and external network zones on resources that may contain external facing and internal only VMs. This type of architecture can bake in some inherent security into your environment that may save the day in the event of a zero-day vulnerability situation that affects the guest operating system or the virtualization hypervisor.

VMware Inc. Senior Director of Enterprise Desktops Gerald Chen visited our office on Tuesday morning to discuss the different types of desktop virtualization and answer common questions about Virtual Desktop Infrastructure (VDI), for example, how it differs from terminal services and cost issues.

Here’s how VDI works: each end user gets a virtual machine (VM) that is deployed from a server in the data center directly to a PC, laptop or thin client computer. Each VM is customizable, so all of the user’s settings are saved and re-booted each time the user signs in, Chen said.

When a user logs off for the day, their VM goes idle, and wakes back up when the user logs into their system again, according to Chen. Chen believes that the advantage of VDI is that sensitive data is not being stored on desktops, which can easily be lost or stolen, and these virtual desktops are easier to manage than physical ones.

“VDI is great for industries like health care that are really concerned about information security and compliance. The real value though, is in management. All of the information is safe in the data center, and centrally managed through Virtual Infrastructure,” Chen said. “For instance, if you have 100 new employees who need desktops, you can deploy a VM for each of them in just minutes, and manage all of them centrally.”

VDI is different from Sever Based Computing (SBC) systems like Citrix Systems Inc.’s XenApp in that VDI is connects a single user to a single operating system (OS), instead of having multiple users share one OS.

“Not every application likes to share an OS, and there is also bad isolation; if one application crashes, everyone sharing that OS crashes as well. Those desktops can’t be customized either. It is a locked environment.”

Chen went on to explain that with VDI, four to ten VMs per server core are supported, so a server with one quad-core processor can, theoretically, house 40 VMs. Of course, that varies depending on things like workload, applications and memory. If the VMs become too heavy for the server to handle, management features in VI3 intervene. VMotion can move live VMs from one server to another when capacity issues arise, as can Dynamic Resource Scheduler, which allocates and balances computing resources as needed using VMotion.

Desktop virtualization case study
As VMware announced customer case studies in February, including one at Huntsville Hospital in Huntsville, Alabama.

The hospital needed to implement a new medical information application throughout its network while protecting HIPAA-related data. Deploying hosted desktops on VMware, the hospital could lock down sensitive patient data and reduce the cost and complexity of desktop management.

They used combinations of thin clients and blade servers to access the centralized virtual desktops, and in turn, reduced power consumption across the hospital by 78%, improved longevity with lower hardware maintenance needs and made wireless thin clients on wheeled carts available to hospital staff. Also, doctors can remotely access their VMs through the Internet using a web browser when necessary.

The downside to desktop virtualization
While the benefits are clear, there are some downsides to desktop virtualization: extra storage and initial cost.

Chen told SearchServerVirtualization.com that VMware is working on reducing image sizes and has designed a way to keep only one copy of files that are identical among many users, like icons and other graphics, to reduce the amount of storage necessary.

The cost of implementing desktop virtualization turns users off. According to Ars Open Forum blogger ‘Bright Wire,’ the cost and the magnitude of system upgrades required is not worth the benefits.

“The cost of deploying virtual desktops is massive,” Bright Wire wrote. “You will need to re-gear your existing desktops to run the virtual or you will need vendor equipment that costs twice as much as a new desktop. Either way, the cost is big in manpower. On top of that, your infrastructure will need serious review.”

According to VMware’s product specifications, local desktop virtualization requires a 500 MHz or faster processor with recommended 256 MB of memory, though Forrester reports that PCs must be faster and have more RAM to work efficiently.

“In addition you need to look into the server infrastructure,” Bright Wire said. “You are talking about needing a lot of iron on the backside to handle the needs of the server to supply two to 16 desktops. All this adds up quickly and can easily swamp a datacenter.”

As for pricing complaints, VMware is used to hearing them and holds firm to the ‘you get what you pay for’ mantra, saying the management benefits are worth the price.

The company charges $150 per concurrent user plus additional costs for support, either Gold or Platinum levels. Both bundles include VMware Infrastructure Enterprise Edition for VDI (which consists of VMware ESX Server 3.5 and VirtualCenter 2.5) and the VMware Virtual Desktop Manager 2. The VMware VDI Starter Edition, which enables 10 virtual desktops, has a list price of $1,500. The VMware VDI Bundle 100 Pack, which enables 100 virtual desktops, has a list price of $15,000.

The market indicates a demand for desktop virtualization, as a number of other vendors also entered the desktop virtualization space including Sun Microsystems Inc., Citrix., Pano Logic Inc. and Symantec. Chen would argue that many customers come for reduction in hardware but stay for the management applications.

“Reducing hardware costs is not a reason to use VDI, it is management. We have customers who have seen 40% to 50% ROI in terms of management costs and the amount of time it frees up.”

When system administrators receive new servers, they are often anxious to get them unpacked, in the rack and loaded up with ESX so they can start creating virtual machines. But an important first step should be done before proceeding with virtualization software installation on the server: always burn-in the memory to test for defective memory modules.

Defective memory will usually be unnoticed in a newly-deployed server and it may be months before signs of defective memory start to show. In one group of five HP servers, I had to replace seven memory DIMMs over an 18 month time period. Most of these were eventually detected by HP’s Insight Manager agents that reside on the server, but two of them caused hard server crashes of VMware ESX servers commonly known as a PSOD (Purple Screen of Death). A PSOD on one of your production servers, loaded up with important virtual machines, is never a good thing. You can reduce your chances of this happening by burning in your memory.

Most servers do a brief memory test on startup as part of their POST procedure. This is not a very good test and will only detect the most obvious of memory problems. A more thorough test checks the interaction of adjacent memory cells to ensure that writing to one cell does not overwrite an adjacent cell.

A good, free memory test utility is available, called Memtest86+, that performs many different tests to thoroughly test your servers memory. You can download it as a small 2MB ISO file that can be burned to a CD and booted on your new server. Let the memory burn-in for at least 24 hours (the longer the better though). Memtest86+ will run indefinitely and the pass counter will increment as all of the tests are run. The more RAM you have in your system, the longer it will take to complete one pass. A system with 32GB will generally take about one day to complete. Memtest86+ not only tests your system’s RAM but also the CPU L1 and L2 caches. Should it detect an error, the easiest way to identify the memory module that caused it is to simply remove a DIMM and run the test again and repeat until it passes. Documentation on Memtest86+ includes troubleshooting methods, detailed test descriptions and the causes of errors.

If you already have ESX servers running and want to test their memory, you can use the little known Ramcheck service to do this while ESX is running. This service is non-disruptive and runs in the background consuming minimal CPU cycles.

The extra time you spending testing memory before deploying servers helps eliminate potential problems down the road.