VMware Inc. and Tripwire Inc. have co-developed a free, downloadable utility to address the leading security concern in virtual environments today: misconfiguration of the hypervisor.

Portland, Ore.-based Tripwire ConfigCheck is a free Windows and Linux based utility that assesses the security of VMware ESX 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines, which were released in February.

The Security Hardening guidelines explain in detail the security-related configuration options of the components of VMware Infrastructure 3 and how security affects certain capabilities.

Tripwire ConfigCheck makes sure ESX environments are properly configured according to these guidelines and lends insight into vulnerabilities in virtual environments. It also provides the necessary steps towards full remediation.

Dan Schoenbaum, senior vice president of marketing and business development for Tripwire
said the utility is being offered for free to encourage the proliferation of VMware’s Hardening guidelines and to increase virtual machine (VM) security.

Tripware hopes that by giving a taste of their technology for free, users will become familiar with them and invest in their software products with more security capabilities, Schoenbaum said.

Colorado Springs, Co.-based Configuresoft Inc. also provides a toolkit for compliance with VMware’s security hardening guidelines. The toolkit consists of a set of rule-based templates, reports and dashboards that plug into Configuresoft’s Enterprise Configuration Manager (ECM).

As this long running thread in the VMware forums indicates, many users are frustrated with VMware’s lack of support for a Linux-based Virtual Infrastructure client to manage VI3 environments. Currently, the VI Client will run only under Windows (as it’s written in .NET), so Linux shops are forced to purchase and install Windows to run it. An alternative web interface does exist; however, it can only manage virtual machine operations and not the ESX hosts which severely limits its usefulness to VMware administrators.

While VMware has not officially announced any plans to develop cross-platform versions of the VI Client or any of its other Windows-only applications, the above-mentioned thread includes one response from a VMware employee who hints that VMware may eventually release a Linux version. A Linux version of a VI Client would be considered a welcomed addition by many VMware customers, if not as an essential feature for those that are using ESX servers in non-Windows environments.

Many customers have also been wanting a Linux version of VirtualCenter, VMware’s centralized management product for ESX,  and support for open source databases like MySQL. VirtualCenter will only install on a Windows server and its required database only supports Microsoft SQL Server or Oracle databases. You can also use SQL Express with VirtualCenter, but it is not recommended or supported for production environments. Because of this limitation, customers that wish to use VirtualCenter must also plan on the additional expense of Windows operating systems licenses for the VirtualCenter server as well as a database license if they do not already have an existing SQL/Oracle database server that they can use for the VirtualCenter database.

Unless more customers speak up and request that VMware produce cross-platform versions of their current Windows-only applications, they will probably not end up developing them. If the demand exists, there’s a better likelihood of it happening. Having Linux versions would also help VMware compete in an increasingly competitive virtualization market. If you would like to see VMware develop a Linux version of the VI Client and other applications, contact your VMware sales representatives and let them know.

Sun Microsystems, Inc. announced this week it has added new features to its Virtual Desktop Infrastructure software, originally released at VMworld in September 2007, including Sun’s Virtual Desktop Connector (VDC).

Sun’s VDI 2.0 provides interfaces to PCs, mobile devices, and thin clients including Sun’s own Sun Ray thin client offering. With it, centralized desktops can be delivered through the LAN or WAN to Windows Vista, Windows XP, Mac OS X, Solaris or Linux on the desktop, which is fairly unique in the Windows-centric desktop market, said Chris Kawalek, Product Line Manager, Desktop & Virtualization Marketing, Sun Microsystems.

Sun’s VDC, meanwhile, is is more or less a connection broker that interfaces with ESX 3.5 and 3.0.x and Virtual Center Server 2.0.x and 2.5 (VMware infrastructure 3) to create pools of virtual machines that can be defined based on templates.

With Sun’s updated VDI offering, administrators can statically or dynamically assign users to specific VMs, either for a set number of days or indefinitely. Another feature is the ability to ‘reset’ end users’ virtual machines (VMs) if problems arise. For instance, if the user contracts a virus while on the web, the VM can be reset to a date before the issue occurred and operate as it did on that date, Kawalek said.

The tight integration with VMware virtualization software can be attributed to the OEM agreement Sun signed with VMware Inc. in February. Thus, with VDI 2.0, users can actively manage VMware virtual machines, but VMs from other vendors like Virtual Iron can only be statically created and assigned, Kawalek said.

Kawalek said Sun moved into the VDI space last year because it embodies Sun’s ‘the network is the computer’ message. Another reason? It’s the popular thing to do. “Everyone is very interested in centralizing their desktop environment, which is why vendors like Hewlett-Packard and VMware are in this space,” he said.

Sun’s VDI Version 2.0 became available March 18 at $149 per user, including one year of support. Sun Ray thin clients start at $249. Directions on how to install VDI 2.0 are available online, and a free trial can be downloaded from Sun’s website.

As a follow-up to my prior post on getting CentOS 5.1 (x64) to host VMware server, this is a short instruction on what to do to get the MUI installed on your 64-bit CentOS box. I didn’t mention it last time because it’s a separate download and install, and I don’t personally install the MUI in VMware server unless I have a compelling reason (on Windows or Linux hosts). This is documented on VMware’s website as well, but it bears some simplification from a two-pager to a six-liner.

1. Download the management interface (VMware-mui-1.0.4-56528.tar.gz) and extract (tar zxf VMware-mui-<xxxx>.tar.gz)
2. Update your campat-db (yum update compat-db.x86_64 0:4.2.52-5.1 or just yum update compat-db to get ‘em all) if you haven’t already.
3. Browse to the directory you extracted you mui setup files to and run the installer (./vmware-install.pl)
4. Accept / Change options as needed.
5. Start the http daemon (/etc/init.d/httpd.vmware start)
6. Browse to https://YOUR.HOSTNAME.HERE:8333
7. Enjoy.

This how to (and my prior post) should also work on other RHEL clones (like Whitebox, when they get WBEL 5 out). It should also work on RHEL.

Fortunately for me, my job never requires me to determine vendor awards. However, Alex Barrett and the SearchServerVirtualization.com staff aren’t so lucky. While it’s great to have the power to name Products of the Year, it also means that you’re stuck hearing complaints from everyone that wasn’t named. In case you missed it, Alex recently published the SearchServerVirtualization 2007 Products of the Year.

I think that Alex and the editorial staff did a great job with selecting products, but thought that I would take a moment to highlight some vendors with excellent products that did not make the list. After all, it’s just as much fun to debate the vendors that were not recognized as it is for those who were.

VMware

Yes, VMware’s on the list, but at the same time they’re not on the list. If you didn’t notice, VMware ESX Server 3.5 is nowhere to be found in the article. The SearchServerVirtualization.com editors informed me that ESX 3.5 missed the cutoff date for award consideration (November 30th), and therefore wasn’t eligible. Editors do need time to work with a released product in order to make a fair judgment, so I understand the reasoning for the cutoff. Still, ESX 3.5 was a significant release from VMware, with features such as Storage VMotion adding significant value to VMware deployments.

Novell

Novell quietly had a great 2007, from a virtualization product perspective. Novell was right behind Citrix/XenSource in achieving Microsoft support for their Xen-based virtualization platform, and was pushing the innovation envelope throughout the year. Novell was the very first virtualization vendor to demonstrate N_Port ID virtualization (NPIV) on their Xen platform. Novell was even showing their work with open virtual machine format (OVF) last September at their booth at VMWorld. When you factor in Novell’s work with their heterogeneous virtualization platform management tool, ZENworks Virtual Machine Manager, you’re left with a pretty nice virtualization package. The vendors mentioned in the virtualization platform category (VMware, Citrix/XenSource, SWsoft) are all worthy of recognition, and I think it’s equally fair to recognize Novell’s work in 2007 as well. Perhaps Novell’s heavy lifting in 2007 will result in recognition in 2008; however, it’s safe to say that Novell is going to have some stiff competition from VMware, Citrix/XenSource, Microsoft, Sun, Parallels, and Virtual Iron.

Symantec

I thinks it’s hard to leave Symantec Veritas NetBackup 6.5 out of the discussion. In fact, amongst backup products, I’d list them as first, right alongside CommVault. Symantec was the first major backup vendor to announce support for Citrix XenServer backup, while all other backup products officially supported one virtualization platform - VMware ESX Server. The NetBackup team was also very innovative with VMware Consolidated Backup (VCB), as NetBackup 6.5 includes the capability to perform file level recoveries of VCB image level backups. Typically, a backup product performs two VCB backup jobs - an image level backup for DR purposes, and a file level backup for day-to-day recovery tasks. NetBackup 6.5 provides the ability to do this in a single pass, which I found to be pretty innovative. Factor in Data-deduplication (extremely valuable considering the high degree of file redundancy on VM host systems), also available in NetBackup 6.5, and it’s hard to see how NetBackup could be ignored.

SteelEye

SteelEye is another vendor in the data protection category that I’m surprised did not make the list. VMware HA by itself will not detect an application failure and initiate a failover job as a result, as it’s primarily designed to monitor and react to hardware failures and some failures within the guest OS. SteelEye LifeKeeper, on the other hand, provides automated VM failover in response to application and service failures (in addition to guest OS and physical server failures). Many failures are software-specific, and products that can automate VM failover or restarts in response to software failures go far to improve the availability of VMs in production.I’m limiting my comments only to the award categories, hence I’m only listing some of the products I’ve worked with in 2007 that fit into one of the SSV categories. I hope that for the 2008 awards, we’ll see a higher number of award categories, so all products in the virtualization ecosystem are represented.

Do you agree with editors’ choice of winners? Which deserving vendors do you feel were left off the list? I’d love to hear your thoughts.

The virtualization world was very excited for the release of VMware ESX 3.5 and Virtual Center 2.5 last week. However, should everyone jump on to the new platforms quickly? I say no. To be fair to myself, I have performed a limited set of upgrades already within the week, and some are planned over the next few weeks. Yes there was a beta process. Yes VMware generally publishes good software. Yes I know these are not Microsoft products. But here is why I say no to ‘jumping’ onto an upgrade immediately for virtualization products:

Larger Scope

The inherent nature of virtualization reaches scope farther than one system as it has historically. With a single ESX server hosting upwards of 30 virtual machines, the magnitude is amplified should there be an issue. So, as with anything critical - a test environment is a must. The development environment may be a small number of ESX servers that hold non-critical virtual machines so you can accept any risks that may arise in your upgrade.

Cover Your Bases

Be sure you are able to execute all scenarios with great confidence before proceeding into the upgrades. One example I will deal with soon is I will have a large number of critical virtual machines hosted on ESX 3.02. If I take one server into maintenance mode, then upgrade it to ESX 3.5 can I migrate from the 3.02 to the 3.5 systems without issue while online? I did upgrade Virtual Center to 2.5, so that was a good starting point for my 3.02 to 3.5 upgrades. VMware has put out release notes with a list of Known Issues with ESX 3.5 and Virtual Center 2.5 that are a good starting point to identify your migration upward in the ESX and Virtual Center versioning.

Upgrade or New Install? You Decide

ESX is released as a full/new install (as a CD ISO) or an upgrade (tar file) installation. I personally will go for the new install mechanism rather than the upgrade. This is because I find the ESX install quite straightforward and easy and rebuilding an ESX host can be done in very little time at all. With the rebuild process very quick for ESX and most management and configuration elements configured from Virtual Center, ESX is unique in build time requirements.

Old School Wait and See

Many people offer the old adage “Wait six months before upgrading” or some other variable time frame when core updates or service packs are available. This is to let other people “work out the bugs” in software before you have to deal with them. There is little basis in virtualization for this logic, but many people have adopted it as a policy related to updates. VMware is unique as new core functionality like Storage VMotion is available with ESX 3.5 and I know I am very excited for this new functionality. This ultimately is your call, but the best advice before anything is to get informed on the product releases and the known issues instead of starting a blind installation.

Today marks an host-oric day, as the first virtual desktops are ready in the lab for my most forward-thinking users (and, as temporary machines, any who happen to suffer hardware failure). As my company is a mid-sized firm, taking this plunge is a bit of “bleeding edge” for us, but it’s too promising to pass up. The early test environment was pretty basic - a few desktops with souped up memory, CentOS 4, VMware Server, and our XP build. First a side-note on CentOS - I love CentOS because it’s almost 100% binary compatible with Red Hat Enterprise Linux. In fact, it’s compiled from their SRPMs, with the copyrighted materials (the logo, some artwork, etc.) removed. On the client-side, ThinStation or any of the other many thin-client linux distros meant to communicate via RDP will work just as well (perhaps better). The roots of this initiative lie in my wanting to have my XP desktop available from where I was — my Macbook Pro, My Freespire 2 desktop, or my Vista desktop. All have desktop virtualization on them, but since they don’t all have the “same” products, mounting a share somewhere wasn’t going to work — and performance might be a bit… underperforming.

The best route was to have it available via RDP. I also wanted to build virtual desktops rfor users. The result, to kill an old commercial’s memory, is that VMware got their peanut butter in my chocolate, or I got my chocolate in VMware’s peanut butter. Either way, I liked the results. It was simple enough to do, and it performed well under even the limted circumstances. Best of all, it’s not complicated to manage. ESX and VirtualCenter more than did the job (though I thinkg a fortune 500 would have need for enhanced management tools, if only for filtering and tracking users to desktops).

After that worked out well for me, I started trimming it back to a more common user-centric desktop build as opposed to the IT-Centric build, taking temporary desktop replacement as a start-in point. The big first was security, while limiting complexity ran a close second. Thanks to AD’s Group Policy handling profile and folder redirection, there’s really no perceivable difference between the user’s original computer and the server-hosted VD. When their PC is fixed, they get it back, and we move on to the next broken-box situation.

The VD solution proved its value there, beating our 2X application server thin-clients (which fared well, but less well than VDs because of the difference in user experience between a linux desktop running a full-screen browser and an XP PC). The next step is to see if we can make this permanent. So, a few IT-savvy first-adopter types are going to get some very old PCs with some very new tricks. I can’t wait…

As reported in a number of other places, Virtual Iron has been making some great deals lately. They’ve picked up a new CEO, received a large sum (13m) in their most recent round of financing, and have been releasing products fast enough to keep the buzz going even though some (including me) have questioned their viability in light of the Xen/Citrix merger. While there’s no clear word on VI’s strategy for dealing with the merger’s consquences to the codebase, it’s clear that they’re doing the rigjht thing - focussing less on the merger and more on continuing their campaign against VMware. Namely, they’ve been forging ahead with their partnership with Platespin. This partnership has interesting benefits - for those few unhappy VMware customers who are happy with virtualization but not with VMware itself, it’s quite easy to make the change to Virtual Iron VMs using Platespin. It also lends VI an enterprise-credibility because of Platespin’s pervasiveness in the enterprise P2V / V2P / P2P / V2V market.

Then there’s always the price-war Virtual Iron started with VMware. Virtual Iron is not kidding when they say their prices are 20% of the cost of VMware’s VI3 Enterprise. Couple this to the fact that VMware still can’t manage to get the SKU out for their Mid-Sized Acceleration Kit, and Virtual Iron has a strong chance of remaining a serious (if small) competitor to VMware over the long term. In the end, this can only be good for the consumer in the smaller enterprises that Virtual Iron targets. With the backing of Intel, AMD, Platespin, and the of OEM alliances VI has made (HP and IBM offer Virtual Iron and VMware on their hardware), Virtual Iron is looking strong in the face of all comers - Citrix and VMware included.

What about Viridian? I’m waiting on that… given what I think of Virtual Server (nice toy), Vista (insert expletives here), and Server 2k8 (hyper-hype), I’m not any near convinced that Microsoft will put out a real hypervisor to compete with VMware or Xen. Truthfully, I’m more interested in what Phoenix is doing… but that’s for another blog. Time will tell.

Is VMware a better product? Yes, it’s far more mature, and has a much greater support based,  it’s also not being limited the way Virtual Iron is by Xen’s requirement to have newer AMD or Intel virtualization-friendly CPUs to run Windows natively. I think real question is this - Is VMware a superior product? On that, I’d have to say no - the little Xengine That Could has caught up quickly, serves similar markets, and beats them on price.

How many of you believe that Apple’s 1.1.1 iPhone update accidentally bricked modded iPhones? Personally, I try to air on the side of optimism, but there are certainly many people out there that think Apple intentionally went after those individuals who took it upon themselves to jailbreak and unlock their shiny gadget-of-the-moment.

Here we are again, not even a month later, and the new Linux Kernel, 2.6.23 was released on 2007/10/09. The latest product of the world’s greatest hackers includes a bevy of new features, including increased support for Xen and KVM and two open source virtualization solutions. Users of those products are probably very happy today, eagerly awaiting the adoption of the new kernel by their favorite distribution in order to take advantage of the increased guest support that comes with it.

VMware Server users on the other hand are getting the proverbial shaft. Kernel 2.6.23 has one MAJOR change and one minor change that completely break VMware Server.

For purposes of dramatic effect, I will detail the minor change first. VMware Server inserts a driver module into the kernel called vmnet. It provides magical networking gnomes that help shuffle bits in and out of VMs to the wide world of webs. In one of its source files, driver.c on line 522, the vmnet driver makes a function call to “unregister_chrdev”, a function defined in the Kernel source file “fs/char_dev.c”. Prior to Kernel 2.6.23 the function “unregister_chrdev” returned an integer value; a return value that the vmnet driver keys on in order to determine whether or not to issue a warning. Kernel 2.6.23 changes the function signature of “unregister_chrdev” to return void instead of and integer. This really hoses the vmnet module source file since it expects an integer value to be returned, and thus the vmnet module will not compile when the “vmware-config.pl” script is run. Luckily there is an easy fix. It seems that the function “unregister_chrdev” has actually returned a value of “0″ despite what transpires in the function as far back as 2.6.20, a Kernel that VMware Server runs fine on. Thus the easy fix is to just edit the vmnet driver.c source file and re-run the VMware Server configuration script.

That is the minor problem that the new Kernel creates.

The major problem is a bit more cumbersome, since the fix involves either redacting a change that Linus (Torvalds) has approved for the 2.6.23 Kernel or lying and declaring that the vmmon module is GPL licensed.

But I’m getting ahead of myself. Let’s start at the beginning. A memory structure called mm_struct is defined in a Linux Kernel header file “linux/sched.h”. Prior to 2.6.23 this structure included a field called “dumpable” that would determine how memory was dumped, securely or not. Kernel 2.6.23 removes this field and lets two functions defined in “fs/exec.c” take its place: set_dumpable and get_dumpable. VMware Server uses the dumpable property in its memory management module vmmon: in the file driver.c to be exact. Since the dumpable property is no longer in the 2.6.23 kernel the vmmon module will not compile.

One might think that a quick fix would be to simply edit the vmmon source file to use the new set_dumpable function. In fact, this action will result in a vmmon module that compiles; however, it will not insert into the Kernel, and an error will occur that says the module contains an unknown symbol. A quick check of dmesg reveals that the unknown symbol is indeed set_dumpable. ‘What, what, whattttt,” you say. But the set_dumpable symbol IS in the kernel. That is verifiable by peeking in /proc/kallsyms.

Heh, heh. Hold on to your seats. This is where it gets fun.

The function set_dumpable is exported in 2.6.23 with the new EXPORT_SYMBOL_GPL, meaning that only modules that are GPL licensed can use it. More can be read about this decision on the Kernel mailing list.

VMware Server’s vmmon module cannot use set_dumpable because it is not GPL licensed. There are two solutions to this problem. The first solution is to edit the Kernel source file “fs/exec.c” so that “set_dumpable” is exported with EXPORT_SYMBOL instead of EXPORT_SYMBOL_GPL and compile a custom Kernel. Then, the vmmon module source file “driver.c” still needs to be edited such that the “dumpable” property is no longer used in favor of “set_dumpable”. The second solution is to edit the vmmon module source file the same way as in the first solution, but also using the macro “MODULE_LICENSE” to indicate that the vmmon module is licensed under the GPL.

Neither solution is nice, because the first one involves maintaining a custom Kernel and custom vmmon module, and the second solution involves changing the vmmon module license without permission. A long-term solution is needed where either the Kernel developers change set_dumpable to be exported out from underneath the aegis of the GPL, or VMware could license the vmmon module under the GPL or create some type of GPL-compatible shim module that in turn calls the proprietary code in vmmon.

Perhaps most interesting of all is the timing. The same Kernel that provides extended support for Xen and KVM also breaks VMware Server. Coincidence? Like I said, I try to err on the side of optimism. How about you?

Chris Wolf, Burton Group senior analyst, analyzed Citrix’s acquisition of XenSource in a recent Burton Group blog post. He sizes up the situation, saying:

“While having the technology is one thing, bringing it to market is an entirely separate issue. This is where the Citrix acquisition makes great sense for XenSource. Financially fueled by Citrix, XenSource now has the financial clout, sales, and channel resources to go after the large stake of unclaimed virtualization market share in the enterprise. Don’t get me wrong. This will not be easy, as Citrix and XenSource are competing against powerhouse vendors with strong sales, channel, and IHV partnerships. VMware, Microsoft, Red Hat, and Novell are well established in the enterprise, and are all looking to add to their share of the market. Virtual Iron has been making a lot of noise in the SMB space lately, and they should see the explosion of the XenSource sales channel as a serious threat.”

Wolf sees the acquisition as a win for Citrix and Xen and for users, too.

“In the coming months and years, we should expect to get enterprise-class virtualization technologies at lower costs, with more features, and a motivated group of vendors that are eager to push innovation to remain competitive.”

Read his blog in its entirety on the Burton Group Data Center Strategies blog.