While Citrix Systems’ Xen’s ubiquity may help the technology earn a legacy as the invisible hypervisor, it may also prove the most challenging next step for IT administrators and developers who want to find or develop software that leverages, supports or extends the Xen hypervisor.

To understand the problem that Xen faces, take Java as an example. Java is great, and I am committed to developing applications that are truly cross-platform using what I consider this fantastic creation. But in all the years that Java has been around, it has failed to gain traction that NET has achieved in less time. Why?

Although Java is slower, it offers a greater advantage than .NET in terms of portability; but Java still hasn’t managed to gain a majority mindshare of developers. This is because Java’s true worth is its portability, its ability to blend into any system. Java has succeeded so well at being invisible that it has lost the sexiness associated with languages used to construct desktop and Web applications. Every once in a while, something like the Google Web Toolkit comes along that makes people take a step back and re-evaluate Java’s usefulness for end-user applications. Ultimately, Java has been left to the obscurity of providing enterprise, back-end applications.

Is Xen is destined to a Java-like fate? While ultimately it may not prove difficult to develop cutting-edge technology compatible with the Xen hypervisor, it may prove so to market it. If you are in the business of selling virtualization add-on products, you want to ensure that your product is compatible with VMware Infrastructure, because that is where the sales are.

The marketplace has not been especially kind to Xen for two reasons: it was not first to market, which is an important factor for any industry, and Xen resellers do not have the power of the VMware PR machine. Also, all major virtualization vendors, including VMware, say that hypervisors should be ubiquitous — the difference is that the VMware CEO Diane Greene has been quoted on virtualizationreview.com and in person. VMware shouts the same thing everyone else is casually discussing and this makes headlines.

As Xen’s legacy may be to become the ubiquitous, embedded hypervisor for all to use, its strength may also be its greatest detriment to Xen-based virtualization platforms. Xen’s strength is its practical application as the invisible, reused, resold, embedded hypervisor, but invisibility just hasn’t worked in Citrix’s favor. Instead, it shields partners from building ecosystems around Xen and has marginalized the brand name.

Hypervisors and VMs are becoming commoditized, resulting in a shifting emphasis towards user interface and management tools. In other words, anyone can make a virtualization platform but the platform that survives Hyper-V’s entry into the virtualization space is the one that develops stand-out management features.

Virtualization pro Andrew Kutz discusses the components of an evolving virtualization ecosystem.

Warning: The following blog post contains biting sacarsm and marginally humorous commentary that may offend sensitive VMware executives. Reader discretion is advised.

An open letter to VMware:

Hey VMware, it’s me again. I know you’re probably still mad at me for last week. Well, I’m going out on a very public limb here to apologize for something that I did.

I’m sorry that I forgot your version.

Yes, you let everyone know that your version was coming up, but I forgot to create a calendar reminder for it and I just plain forgot. You know how that goes, right? 

Now I don’t mind owning up to my bad memory, but here’s the thing – you have sooo many versions! Most people just have one version per year, you have at least five. There’s the version for VMware Infrastructure (VI), currently at 3.5. ESX is already 3.5 versions old, and ESX 3i has its own version too. Then there is the VirtualCenter and the VI client at 2.5. VMware Consolidated Backup (VCB) is straggling behind at 1.1. I think the VI SDK is also 2.5 versions old, but with the VI Perl Toolkit at version 1.5 and the VI Toolkit (for Windows) in beta, it is hard to keep up.

VMware, your enterprise portfolio has expanded far beyond simply ESX, and none but two of the versions align. Therefore, with so many available products, it is fast becoming impossible to understand which version works with which. You should release minor point releases between major revisions in order to maintain a consistent major version number for your enterprise product offerings.

I know you’re a busy company, and it is hard to get everybody together on one day out of the year to celebrate your version, but I beg you, please try. Except for those closest to you, it is getting extremely difficult to remember your versions, or figure out which version we actually mean. Here’s an idea: for the rest of the year, skip all of your versions and then start over your versions all at once on a single day. Maybe even at VMworld? It can be your special version day. I’ll even bring party hats and cake (if you will invite me.)

VMware Infrastructure 4 (VI4) can include:

- ESX 4
- ESX 4i
- VirtualCenter 4
- VI SDK 4
- VI Perl 4
- VI Toolkit (for Windows) 4
- VCB 4

I know it will throw people off at first; your customers might think they missed some of your versions. However, I think in the end you’ll have a lot of people thanking you.

I feel real bad about missing your version, and I don’t want to let the announcement pass me by again. Maybe I should use Outlook?

I ran into problem recently where the network would not come up in a Dom-U after the initial domain creation. All subsequent reboots would result in an unprivileged domain without network connectivity. Here is an example of the error that would occur if I attempted to restart the network manually:

root@odin:~# /etc/init.d/networking restart
* Reconfiguring network interfaces…
eth0: ERROR while getting interface flags: No such device

SIOCSIFADDR: No such device

eth0: ERROR while getting interface flags: No such device

SIOCSIFNETMASK: No such device

eth0: ERROR while getting interface flags: No such device

Failed to bring up eth0.

So what is going on? Well, it turns out that udev was binding the domain’s NIC to the initial MAC address of the vswif (which changes on reboot by default). Hence when the domain comes online for a second (and third, and so forth) time with a new MAC address, the domain does not recognize the NIC.

The solution is quite simple: edit the udev rules that bind NICs to MAC addresses. In Ubuntu this is in two places. You need to edit “/etc/udev/rules.d/70-persistent-net.rules” and comment out any lines that look like this:

SUBSYSTEM==”net”, DRIVERS==”?*”, ATTRS{address}==”00:16:3e:2b:2e:7b”, NAME=”eth0″

And because the aforementioned file is automatically generated, you also need to edit “/etc/udev/rules.d/75-persistent-net-generator.rules” and comment out any lines that look like this:

SUBSYSTEMS==”xen”, ENV{COMMENT}=”Xen virtual device”

I also commented out the following line for good measure:

ACTION==”add”, SUBSYSTEM==”net”, KERNEL==”eth*|ath*|wlan*|ra*|sta*” NAME!=”?*”, DRIVERS==”?*”, GOTO=”persistent_net_generator_do”

This will prevent *any* generation. Anyway, once you make these changes and restart the Dom-U, the network will come up every time without a hitch.

Hope this helps!

ivi (pronounced eve-e) stands for Java Virtual Interface, and it is a project that aims to create a single, graphical, management interface for all the major virtualization products.

Implemented in Java+Swing, ivi is truly portable. Currently ivi uses the VMware Virtual Infrastructure 3 (VI3) software development kit (SDK) to communicate with VI servers and the XenApi to talk to Xen servers. Future plans include adding support for libvirt to allow communication with KVM and OpenVZ and, eventually, support for the Common Information Model (CIM) as a way to talk with VMware, Xen, and Microsoft all through one interface.

The number one barrier to a properly utilized datacenter is the lack of a single management tool to control a heterogeneous environment. The idea behind ivi is to create a single management application that can be used to control all of your datacenter’s virtualization solutions. With so many virtualization products available, using so many different types of architectures, it is more important than ever to possess a management tool that can provide IT professionals a single point of management.

You can read more about ivi at http://www.lostcreations.com/code/wiki/ivi.

A while back I reported on what was a sticky issue for many people: VMware Server 1.0.4 did not work with the latest Linux kernel (2.6.23+) because the VMware Server memory module used the dumpable bit which had been removed in 2.6.23+ in favor of the GPL v3 exported set_dumpable and get_dumpable functions. Because the VMware Server memory module is not GPL v3 compliant (it does not use the MODULE_LICENSE macro to declare itself such), either a kernel recompilation was required in order to redact the GPL v3 changes to the set and get dumpable functions or a vmmon module recompilation was required in order to lie about its license type. Unlike the writer’s strike, a compromise has been reached.

The Linux Kernel development team has not removed set and get dumpable’s GPL v3 requirements and VMware has not made the vmmon memory module GPL v3 compliant (which in turn would require VMware Server to be licensed under GPL v3). VMware did not even future proof themselves by creating a Kernel module shim licensed with LGPL. VMware simply access the dumpable bit directly with set_bit and clear_bit now. Lines 1663 of the vmmon source file driver.c begins with:

<code>
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 23) || defined(MMF_DUMPABLE)
/* Dump core, readable by user. */
set_bit(MMF_DUMPABLE, ¤t->mm->flags);
clear_bit(MMF_DUMP_SECURELY, ¤t->mm->flags);
</code>

While some may hail this change as a good thing, I do not. What happens next time when there is no work-a-round? Both VMware and the Linux Kernel Development team have a chance to showcase that closed-source and open-source can work together. That closed-source companies are open to listening to the reasons for things like GPL v3. And proponents of GPL v3 have an opportunity to show that they are not just zealots whose blind actions damage the usefulness of their software to end users.

I think it is great that VMware listened (whether or not it was to me) and fixed this issue. I just wish that the opportunity for two communities to come together could have been embraced instead of side-stepped.

Until next time.

They say the best laid plans of mice and men rarely succeed. It is clear to me then that some of the development team that VMware has working on Server must be what Mulder and Scully were searching for — not the truth, the other thing, human/alien, (wait, scratch that,) human/mouse hybrids. I figure if a double-negative makes a positive and Mars is in the orbit of Venus, then a human/mouse hybrid probably succeeds a little more than it fails. And that is my poetic, round-a-bout way of saying that at first blush, VMware Server 2.0 hits the mark more than it misses.It is nice that VMware Server 2.0’s installer attempts to uninstall VMware Server 1.0.x for you, except that 1.0.x’s uninstaller is famous for not working! It does not like to shut down VMs in a timely manner. I tried to manually shutdown the daemon, but the vmnet1 NIC arrived in some type of hung state. A reboot was eventually necessary as countless console messages prevented me from accessing the server even from the console. I know this is not indicative of a 2.0 problem, but it sure soured me to upgrading right off the bat.

However, once I finally resolved that issue, 2.0 installed like a champ! No problems at all. That is more than I can say for previous versions of VMware Server. VMware even bypassed the nasty problem on non-GPL3 compliance by not using the GPL3-ified version of set_dumpable in their vmmon memory module. Instead they call the set_bit function directly:
<code>
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 23) || defined(MMF_DUMPABLE)
/* Dump core, readable by user. */
set_bit(MMF_DUMPABLE, ¤t->mm->flags);
clear_bit(MMF_DUMP_SECURELY, ¤t->mm->flags);
</code>

It’s nice to see that someone listened to me (I’ll pretend someone did — most likely VMware just saw that this was a problem and fixed it — good on them!)

So far so good, VMware Server 2.0 installed great. But then it comes time to manage it. The very first thing I do is open a Web browser and point it to the Tomcat instance being used by Server 2.0. However, it does not ever authenticate me. I get a funky malformed URL error. That’s like, totally bogus, dude. I mean, cryptic error messages? Who does VMware think they are? Every other software developer in the world? VMware — we hold you to higher standards — better error messages please.

Luckily, Server 2.0 is manageable by the VirtualCenter client, which I happen to have. Unfortunately the Server 2.0 Website does not make that an available download for users without said client. . I started up the VI client and tried to connect to my Server 2.0 instance until I realized that the VI client is subject to the same issues connecting to Server 2.0 as it is with ESX — it does not accept pass phrases. My pass phrase is over 70 characters long and the VI client rejects it. Or it is using the trim function (if you know what it does you know I gave something away, otherwise, LOOK, a rainbow!). I ssh into my Linux box and change my password to something less secure and then attempt another VC connection and this time it works.

The VirtualCenter client is a great way to manage VMware Server. The VirtualCenter client is a terrible way to manage VMware Server. I am sensing some dichotomy here. I am glad we (do not) agree! While the VI client is a great improvement over the MUI (we finally get meaningful statistics!), it would have been nice to get a client version that did not constantly throw .NET errors about objects not initialized or null this and weak reference that just because VMware Server 2.0 does not fully implement everything that ESX does. That is annoyance number one. Oh, click “Continue” instead of “Quit” or watch the VI client close on you!

That brings me to annoyance number two, and this one is far worse. There is no intuitive way to add existing VMs into VMware Server 2.0! You have to double-click on a configured data store in order to explore its contents, navigate to the VM’s vmx file, and then click “Add to Inventory.” However, if you right-click on the data store you get an error. If you look for an “Explore data store” option you will not find one. There should be a “Search this server for VMs to import” option. At the very least, when the installer asks you where to store VMs (from which it creates the first data store), it should ask if you wish to import existing VMs.

Overall I am happy with VMware Server 2.0. It seems much faster and you can finally create more administrative users than just “root”. However, there is much spit and polish needed before VMware Server 2.0 is ready. Most of that focus needs to be on what VMware already knows — its management. The VIX API and increasingly integrated Virtual Infrastructure client functionality are a good first step, but VMware Server 2.0 is not there yet.

Chris Wolf and I were presenting Virtualization 101 in Seattle yesterday when something he said sparked an idea in my usually dormant brain. Okay, it’s not usually dormant, but Seattle is so cold I think half of my synapses aren’t firing! In the process of discussing virtual machines (VMs), Chris mentioned that each major virtualization solutions provider has registered itself with the Institute of Electrical and Electronics Engineers (IEEE) and received one or more Organizationally Unique Identifiers (OUIs). An OUI is 24-bit number that makes up the first half of all of the Media Access Control (MAC) addresses assigned by an organization to devices it produces. MAC addresses are most frequently associated with Ethernet adapters, so why are virtualization vendors registering with the IEEE to obtain OUIs?

Virtualization vendors also produce Ethernet adapters — virtual network interface cards (NICs). Most VMs would be rather useless if they could not access some sort of network, so virtualization vendors must create virtual NICs in order for the VMs to get on the big wide world of Webs. And since these virtual NICs have to participate on the network just as if they were physical, they must use MAC addresses. Because the first 24 bits of these MAC addresses, the OUI, is organization-specific, there is a real potential for network administrators to detect not only if a machine on the network is virtual by its MAC address, but also what type of virtual machine it is (what vendor’s software is hosting it). While best practices dictate that you do not change the MAC address of VMs, enterprise virtualization solutions do present this as an option, and, because of this, here is the scenario I see occurring.

One way to harden the Apache Web server is to use mod_security to alter the Web server’s signature. For example, you can fool clients into thinking that the Web server hosting their favorite videos is actually a Microsoft Internet Information Systems (IIS) 5.0 server instead of Apache 2.2. Administrators do this in order to fool attackers into attempting the wrong types of attack vectors. Even though best management practices dictate that administrators NOT alter their VMs’ MAC addresses, I forsee them doing so anyway in order to fool would-be hackers into attempting the incorrect attack vectors on VMs. For example, if a VM is hosted on ESX and its MAC address has an OUI registered by Microsoft, then a would-be attacker may try known Microsoft Virtual Server or Hyper-V exploits on the VM instead of ESX exploits.

Who knows? Twelve months from now altering a VM’s MAC address to be that of another vendor may be considered a best practice, but right now, with the already complex problem of managing virtual hardware, IT administrators are best served to leave their VM MAC addresses well enough alone.

Of course, that doesn’t stop the idea from being completely and utterly cool!

Hope this helps!

A recent discussion on the OpenBSD mailing list led to the assertion that virtualization decreases security. For those interested, a summary of the discussion is available on Kernel Trap. But proponents on both sides of the argument have taken to throwing about emotionally driven comments rather than thinking objectively about the subject. Of course, because the original comment labeled all those as”stupid” and “deluded” who think virtualization somehow contributes to security weaknesses, who can really blame people for getting a bit emotional? All the flame-war commentary aside, the question remains, does virtualization weaken security? The original argument that virtualization can diminish security was based on two points:

1. If software engineers cannot create an OS or application without bugs, what hope does a virtualization solution have to be bug-free?
2. x86 hardware is ill-suited for virtualization.

Bug-free software
The first point does two things: it lumps all software engineers, operating systems, and applications into one pool and assumes that it is possible to find bug-free code.

Addressing the sub-points in order, while it is true that software engineers are human (and we make mistakes) and that software in general has a track record of imperfections, it is also true that the world does not judge all software engineers or software to be the same. In fact, I would guess that a lot of members of the OpenBSD mailing list in fact prefer OpenBSD to, Windows, let’s say. However, there are many readers of this blog that may prefer Windows to OpenBSD, or Linux, or OS X, etc. The same preference could be applied to office suites (OpenOffice, StarOffice, MS Office, KOffice, etc.). The fact of the matter is that we all have our own preferences: we do not judge software to be the same.

Secondly, the first point argues that the community should expect a bug-free hypervisor, and anything less contributes to the decrease of the overall security of a server platform. This is a very lofty expectation indeed! A very long time ago I wrote to Slashdot, heck, almost seven years ago now, and asked the question why it was not possible for developers to spend more time on projects and produce bug-free software. Commander Taco (the ring-leader of Slashdot) himself replied to me and said that it was a foolish expectation: software is 1) written by humans and 2) is far too complex today to be without errors. However, people still judge some operating systems to be more secure than others. The same for kernels. How can such a judgment possibly be made if all software has bugs? The answer is “easily.” We observe the rapidity that bugs are discovered in software, the impact that they have on the IT infrastructure across the world, the speed at which operating system and independent software vendors (OSVs and ISVs) release patches, and how easily those patches are applied without affecting the rest of the server platform, and then we judge the security of a piece of software. Therefore we do not judge a piece of software to be secure because it contains no bugs, but rather by the history of its imperfections and how quickly blemishes are removed.

Notice that I did not say whether or not I agreed with Mr. Slashdot. I do not. I do believe that software designed be generally purposed, such as today’s OSes, is doomed to be bug-laden, simply because it lacks a specific purpose and too many conditions have to be accounted for. However, imagine if the same leeway were given to the software that runs our air-traffic control systems? Or military installations? Such software is held to a higher standard, and it can be in part because it is designed with a specific purpose. The same is true of hypervisors: they are designed specifically for one purpose. They do not yet have all the cruft and bloat sitting on top of them that today’s OSes do. Here’s hoping that the ISVs producing today’s leading virtualization solutions step up to the challenge.

In short, I believe that there is a reasonable expectation that hypervisors will be a lot more secure than general purpose applications written on top of general purpose OSes could ever hope to be.

x86 hardware
Yes, the x86 instruction set was never designed to be virtualized, but to say that the instruction set has not grown well above and beyond its original intentions is to do an injustice to the original minds at Intel who took part in creating one of the most persevering pieces of technology to date. With the first set of virtualization extensions, those created to solve the problems of ring compression and ring aliasing, the x86 instruction set was given a breath of new life. And with the latest extensions, enabling live migrations of virtual machines across multiple generations of processor versions, the staying power of the x86 instruction set in a world with virtualization has been increased even further.

My point is simply that just because the x86 instruction set was not designed with virtualization in mind does not mean that it cannot work, and work securely. That is the beauty of x86: it can be extended to do what we need it to do. History has spoken.

Conclusions
With both of the original arguments shown to be false, is the conclusion of the original argument then reversed? Not completely. While virtualization does not decrease security, the potential for it to do so is there. Hypervisors are software, and although they are a lot less likely to have bugs than a general-purpose piece of software, bugs can still occur. However, to blanketly state that virtualization decreases security is far too general as there are many different implementations of virtualization. For example, if VMware ESX was found to have a memory sharing bug that allowed one virtual machine to read and write the memory of another, does this mean that XenSource is immediately compromised? Of course not. So even when a bug in a hypervisor is found it does not immediately mean that all virtualization is suddenly subject to the same problem.

As I stated earlier, the security of any software package is judged by the number of bugs historically observed, their impact (potential or real), and how quickly the parties responsible for said software fix the bug. While the potential is there, it is far too soon to observe whether or not virtualization decreases security. Only time will tell.

What do you do when you are giving a session on virtualization migrations, and no one shows up? You sit down and blog about how you are giving a session on virtualization migrations and no one showed up! In all fairness, TechTarget is giving away a Mercedes-Benz right now, so I think people are eagerly awaiting that drawing, and my session is right after lunch, AND it *is* a repeat of one I did earlier today. Yes, those are the things that I will tell myself tonight as I curl into a fetal position and sob in a corner

DCD 2007 has been a natural extension of last year’s conference. 2006 saw virtualization come into its own in the enterprise, and 2007 has seen virtualization mature into a ready-to-use, ready-to-integrate solution for many of today’s data center related problems. One of those problems has been disaster recovery, business continuity, and business resumption. This year’s DCD conference has had no shortage of innovative and instructional sessions on how to create cost-effective BC solutions using virtualization. Two technologies that are paving the way for these SMB to enterprise BC solutions are iSCSI and 10 GbE. Virtualization requires shared storage to enable any of the features used in DR and BC scenarios (LiveMigration, Resource Scheduling, Power Management), but for the longest time shared storage has meant fibre channel SANs, an expensive proposition for many. The continued success of iSCSI and the eventual commoditization of 10 GbE will enable enterprise-class shared storage at a fraction of the cost for SMBs and enterprise data centers alike. Inexpensive, enterprise-class shared storage will result in the availability of virtualization’s high end features that contribute to DR and BC solutions for all organizations — small, large, and everything in between.

Once again, DCD 2007 was a natural extension of 2006 — the data center has continued to evolve around virtualization. What will next year bring? My best guess? To steal from a colleague, we are going to start hearing about the highly-utilized data center, the dynamic data center, the data center that can be managed and monitored from a single console. And who is to say that the data center management of the future will require any human interaction at all?

Dave?