Big players in the virtualization world griped about the absence of performance benchmarks for virtual machines on CIO Talk Radio yesterday and discussed some of the issues surrounding virtualization standards.

Guests on the show included: Simon Crosby, Chief Technology Officer of the Virtualization and Management Division of Citrix; Tom Bishop, Chief Technology Officer, of BMC Software; Dr. Tim Marsland, Sun Fellow, Chief Technology Officer, for the Software Organization at Sun Microsystems Inc.; and Brian Stevens, Chief Technology Officer and Vice President of Engineering at Red Hat.

The glaring ommission in this lineup: VMware, Inc.

The panelists on CIO Talk Radio didn’t mention VMware by name, but did complain that some companies aren’t being open with their performance data, thus prohibiting the virtualization industry from publishing comparative performance data.

VMware’s licensing agreement for ESX allows users to conduct internal performance testing and benchmarking studies, and allows those users (and not unauthorized third parties) to publish or publicly disseminate the data provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study.

Users that have published benchmark data, like Sr. Systems Engineer Mark Foster did on his blog, have had to unpublish results because of VMware’s stipulations.

VMware introduced its own free benchmarking tool, VMmark, last year for certain applications.

Meanwhile, the SPEC Virtualization Committee has been working to create standard benchmarks for VMs. The committee’s goals are to deliver a benchmark that will model server consolidation of commonly virtualized systems such as application servers, web servers and file servers; provide a means to compare server performance while running a number of VMs; and produce a benchmark designed to scale across a wide range of systems.

SPEC expects these benchmarks to be available by the end of this year, but the timeline is not set in stone, according to the website.

Sun’s Marsland said benchmarking progress has been slow because there isn’t an easy way to define a workload, and a large number of benchmarks are required.

“We are talking about a virtual computer, with lots of aspects that need to be benchmarked,” Marsland said. “Every component that gets virtualized needs to be benchmarked.”

Having an open, standardized way of benchmarking is expected to push virtualization further into the mainstream because it will eliminate false perceptions about performance, panelists said. For instance, “there is the thought that I/O intensive workloads can not be virtualized, and the absence of benchmarks prevents us from proving otherwise. It is important for us to have good benchmarks out there,” one panelist on the show said.

Though users look at benchmarks, this type of data is most useful to vendors and OEMs who can use the performance standards to improve the technology, and of course, market their products.

“More open scrutiny of performance results will help us to improve as an industry overall,” Bishop said. “There are ways to measure performance in non-virtual environments, and people are adapting those techniques to get the most out of their virtualized environments.”

In terms of application performance in virtual environments, the issues differ depending on the data center infrastructure. The network, the servers and the storage all affect performance, said Stevens of RedHat.

“The areas that have to progress are around I/O. Intel and AMD are improving around page tables, and we will see improvements around I/O adapters soon,” Stevens said.

Another problem with virtualization? There are support challenges. If an application running in a VM starts acting wacky, the application vendor may not support it, Crosby said.

Licensing and support in virtual environments has been a major gripe with Oracle, for example, which does not support running its applications with VMware.

“It is a reasonable concern…right now there is irrational market based control. Some folks are abstaining from supporting certain apps [in virtual envionments]. As customers demand support, things will hopefully get rational, by next year I hope,” Crosby said.

I recently received a press release from London-based TechNavio, the creator of a Web-based information and research tool, that outlines the top five virtualization trends. Here they are, along with my own thoughts on these trends:

1. Business process automation.
TechNavio’s take. “Virtualization is expected to speed up the wider movement toward business process automation and remote collaboration. The TechNavio findings appear to indicate that the market in general is expecting a major investment in this area within the next two to three years.”
My thoughts. On the subject of business process automation, if TechNavio means “scripting,” I can agree with this trend. SearchServerVirtualization.com contributor Andrew Kutz has received a few questions from readers about automation, which suggests that there are plenty of other IT pros with similar questions. Also, he increasingly writes tips about scripting for X or Y, often concerning disaster recovery or hot backups. Most recently I’ve seen questions about scripting virtual machines (VMs) to power on and off at a certain time.
Food for thought. If scripting VMs advances, what will happen to the number of system admins and data center managers needed to run a data center? Perhaps all you IT programmers should slow down the scripting process before you script yourself right out of a job!

On the subject of remote collaboration, I definitely agree with TechNavio. I wrote an article on emerging client-side desktop virtualization technologies. In response, I received comments from readers who said that they had found a surprising number of companies that are exploring client-side virtual desktop infrastructure (VDI) technologies for implementation in 2008. I think it’s due time for VDI; just consider the number of stolen or misplaced laptops, or CDs that went missing in the mail containing personal information. . . .I don’t know about you, but identity theft certainly isn’t on my holiday wish list. And I certainly would appreciate company investment in this kind of technology, considering the increasing mobility of technology.

2. Network-delivered computing.
TechNavio’s take. “Virtualization is also expected to boost the move toward network delivered computing or what is being termed PC-over-IP. This in turn will place vendors such as Cisco, NEC and Sun at the heart of the market, but interestingly leaves the door open for a host of innovative start-ups.” <br>
My thoughts. I would agree here as well. My aforementioned article discusses vThere, which focuses on primarily providing client-side virtual desktops via their own (i.e., third-party) servers that a client notebook would connect to when opening the virtual desktop. During interviews, my subjects all mentioned the trend of software vendors moving to providing their software via virtual machine. We have already seen a few virtualization companies provide beta versions of newer software via VM. As virtualization continues to grow in adoption, I can easily see all kinds of independent software vendors providing their products via virtual machine download.

3. Legacy applications and virtualization.
TechNavio’s take. “As application virtualization speeds up, applications development and maintenance or ADM, vendors have a real opportunity to grow into a new market defined as optimizing legacy applications for virtualization.”
My thoughts. We haven’t focused much on application virtualization on SearchServerVirtualization.com and SearchVMware.com, so I don’t have an informed opinion on this subject. Readers, do you?

4. Small and midsized businesses (SMBs).
TechNavio’s take. “The biggest long-term opportunity for virtualization vendors lies in the SMB space, specifically end-to-end solutions that allow SMBs to outsource and virtualize their entire network.”
My thoughts. I disagree here. Clearly. there is opportunity and space for virtualization in the SMB market, but to say it’s the biggest long-term opportunity? That’s a stretch. I doubt that larger businesses, once virtualized, will stop virtualizing. I think that a more accurate statement would be that virtualization vendors should target SMBs to further extend virtualization.

5. Labor market and skills.
TechNavio’s take. “As the market for server virtualization heats up, finding people with the right skills is set to get harder. With this environment TechNavio predicts that there will be increased opportunities for IT services companies as well as for IT staffing solutions providers.”
My thoughts. I don’t know if I agree that finding people with the right skills will become more difficult; it depends on the IT workers and their drive to stay on top of certifications that prove their worth. (Cough, the VMware Certified Professional (VCP) exam, cough, cough.) And whenever technology advances, desired skill sets change, so this prediction isn’t all that impressive. As far as increased opportunities for IT services companies, yes. It’s easier to go to a business and say, “Get me a sys admin with a VCP stamp of approval!” than it is to shuffle through résumés looking for those who are VCPs. And I definitely think that those who have the right credentials will find themselves in increasing demand: So stay on top of what you’re worth salary-wise given the move toward virtualizing mission-critical servers. Just because your current company doesn’t realize your worth, it doesn’t mean that Company Y — which has more virtualized servers and a greater need for those with virtual environment management experience — doesn’t.

TechNavio’s press release also included a quote after these “top five trends.” Co-founder of Chicago-based Infiniti Research S. Chand (who conducted the research for this report) said, “Currently the biggest beneficiaries of server virtualization are the enterprise users whose businesses tend to be dependent on running compute-heavy, high availability, application intensive data centers. These include: ISPs, hosting and managed service providers, bank’s trading divisions, gaming, online retailers and the like.”

So if you are looking to get the most (read: more money) from your virtualization experience, check job offers with companies that deal with these types of services.