It’s been six months since I posted about the value of the VMware Certified Professional (VCP) certificate, and I thought I’d provide an update.

As the image shows, courtesy of indeed.com, the VCP is as hot as ever.

Since I last covered this topic, the following shifts occured:

• The VCP gained $3,000
• The A+ climbed $6,000
• Network+ declined $1,000
• MCP gained $1,000
• MCSE gained $2,000
• CCA lost $2,000
• CCEA picked up $2,000
• RHCT picked up $3,000
• RHCE picked up$2,000
• RHCA lost $1,000

The big gain in VCP salaries over a period of less than six months shows that this technology is still very much an in-demand skill set and a hot certification to show off. It’s a new year and salaries did jump overall, so this is reflected in the data. As before, the international trend is also continuing, as the next two images (from itjobswatch.co.uk) show, in terms of salary and demand.

I intend to keep tracking these statistics every few quarters, so stay tuned. I’m also keeping my eye out for Citrix-sponsored Xen certifications and will be bringing an analysis of those to the blogosphere as soon as there’s some quantifiable information available.  And with VMware ramping up its certification programs, I expect to be adding second and third-tier VMware certifications.

What other certifications do you think should be compared? I’ve included a broad list of non-developer certs to show the variety and range in entry-level (MCP, A+) system admin certs through top-tier (CCEA, RHCA) certs to compare the VCP’s placement as a hot-technology. I’ve left off network, storage and many specialty certs because they may not be pervasive enough in the enterprise or may not be relevent topically. Since I’m one person with one view, I hope our readers will comment below and dictate to me what should be compared. So please fire away.

The Inquirer recently published a story on how Dell is considering giving away VMware VI 3i licensing on its PowerEdge servers. While I won’t rehash the details of the rumor here, I’ll add my opinion and analysis on why this bold move is being made, since it appears VMware is actively supporting this tactic after having said that hardware vendors will be free to choose what fees to charge customers for 3i, if any.

Hypervisors are destined to become a commodity item, even more so than other software, because everyone will be utilizing virtualization within the next few years. Dell and VMware aren’t reacting so much to competition from Virtual Iron, Hyper-Hype (oops, I mean Hyper-V) or Virtuozzo as they are to Phoenix’s Hyperspace and Xen’s Embedded offering. Hyperspace is the big target here, as it’s embedded virtualization from a BIOS manufacturer.

Putting a hypervisor at that level takes the old “I need to dual boot” equation from power-users who need access to Linux and Windows or Mac and Windows, etc., to another level in addition to being another virtualization offering. Virtualization originally took that need away for most people. I for one stopped dual booting and ran Linux in Windows, Windows in Linux and Windows and Linux in Mac via virtualization products from the minute I got my mitts on VMware Workstation all the way through Parallels Desktop. Now, Phoenix is turning the tables, and taking virtualization away from being built on top of the BIOS like a conventional OS and making a run to own the space from the board-level up. Phoenix is virtually going from the niche unthought-of product to an enteprise contender (no pun intended.)

VMware saw this coming. It anticipated the inevitable, embedded hypervisor, which is why 3i came out in the first place. It also knows that we, the computer-consuming public, don’t really consider the BIOS when we buy a computer (be it a personal machine or a server.) We don’t even realize that we pay for the BIOS, because BIOS builders charge chip- and board-level makers a licensing fee. That licensing fee per machine is passed on to us in the cost of the board, and it’s minimal.

I am convinced this is where virtualization is headed: Virtualization will be a commodity, practically free for all without needing much to install or configure after the fact. VMware is betting on this core hypervisor as a lead-in to its flagship products. I expect VMware to focus this new strategy of transitioning customers from base-level embedded hypervisor to high-end pricing on management, replication, storage, etc.

Dell is also wise to this trend. They see the advantage of the embedded hypervisor as much as they saw the advantage of selling VMware’s ESX product line pre-installed on their hardware. They see that sooner or later everything they sell will have virtualization built-in. I expect them to sell Hyperspace alongside 3i. I also expect that will need to make the price points equivalent, lest there be howls from customers buying Hyperspace wanting to upgrade to a higher-level of virtualization management.

Advanced features like VMotion, DRS, HA, CB, etc. are licensed at the license server level, making 3i as good a choice for virtualization as ESX 3.5. This comes from VMware’s own 3i announcement:

“VMware ESX Server 3i is the new architectural foundation for VMware Infrastructure 3, the most widely deployed virtualization software suite for optimizing and managing industry-standard IT environments. VMware customers will be able to easily implement the entire suite of VMware Infrastructure 3 products on top of this foundation, including VirtualCenter, VMotion, Distributed Resource Scheduler (DRS), High Availability (HA) and VMware Consolidated Backup (VCB).”

As it stands, 3i is cheap at around $500, so don’t expect this shift in pricing to impact VMware’s bottom line.

Since I just love to read white papers (n.b., sarcasm), I grabbed a copy of VMware’s Comparison of Storage Protocol Performance. Actually, I found it to be a good read. It’s short and to the point. This sums it up quite nicely:

“This paper demonstrates that the four network storage connection options available to ESX Server are all capable of reaching a level of performance limited only by the media and storage devices. And even with multiple virtual machines running concurrently on the same ESX Server host, the high performance is maintained. “

The big four storage connections are:

• Fibre Channel (2 GB was tested)
• Software iSCSI
• Hardware iSCSI
• NFS NAS

The paper infers that network file system (NFS) is perfectly valid for virtual machine (VM) storage, performing in all of the tests at a level comparable with software iSCSI, very close to hardware iSCSI and lagging behind 2 GB Fibre Channel (FC). This doesn’t surprise me one bit: I like NFS network-attached storage (NAS) for VM storage. I prefer storage area network, or SAN-based storage because I prefer to store on a virtual machine file system; but for low-criticality VMs, NAS’s price is right (well, as long as you don’t count Openfiler, IET, etc.) Also, it’s plausible to build out a virtual infrastructure storage architecture using nothing but Fedora Core and be supported.

I was particularly interested in the FC vs. iSCSI performance results presented in this VMware white paper. At the lowest end of the scale, iSCSI beat FC. Granted, the low end of the scale isn’t what will be seen in most production environments but it is  interesting data. What I liked most was that nowhere did 2GB FC truly outclass 1Gb iSCSI. It was faster in most of the higher I/O testing, but never did it double the performance. 2 GB FC did show a big performance improvement in the multiple VM scalability test but not double (about 185 MB per second vs. about 117 MB per second).

On to what I didn’t like in this white paper:

• No 4 GB FC comparisons. 4 GB FC is the sweet spot for high-performance enterprise SANs being put in place to support the big iron now being virtualized. It should have been covered, even if it is still a little bit of a nascient technology (well, not in terms of maturity but in terms of it’s market segment.)
• No 1 GB FC connections. (There are still plenty out there.)
• No NIC Teaming comparisons. I want to know how much additional CPU overhead is involved. I want to know how much performance is improved if you team NICs on your software iSCSI targets and initiators.
• No multipathed comparisons. This should have been done. Mutipathing is a way of life for anything as mission critical as a server that hosts multiple servers.
• No 10 GB Ethernet iSCSI comparisons. VI 3.5 is out. 10 GB Ethernet support is built into VI 3.5 (see the HCL, page 29.) Not to test this is a big oversight.
• No internal-disk storage was tested. Ok, maybe it’s not reasonable for me to expect this to be tested. Maybe I’m just grouching now.

I was surprised to see that software iSCSI got its tail handed to it in CPU workload testing. I’ve never done this testing but I knew there was a big overhead involved. I just didn’t expect it to be that big, especially compared to NAS, which I expected to be right there with iSCSI rather than much more CPU efficient (FC was the 1.0 baseline, NAS scored about 1.8-ish 1.9-ish, and SW iSCSI was about 2.75.) This means one thing: while performance is great across all protocols, plan on extra CPU power for software iSCSI.

I was pleasently surprised to see hardware iSCSI dead-even with 2 GB FC. I had expected some additional overhead even with dedicated hardware, but that wasn’t the case. I would expect to find that in a dedicated iSCSI solution–unless you’re using really cheap equpment like hooking up a couple of big drives to that old desktop–you won’t hit the CPU-use ceiling unless you fail badly at planning.

All of these protocols are perfectly valid. There could have been more meat in the paper, but it did a good job of accurately testing four of the most common storage architectures used with VMware’s products.

Overall, I give this white paper seven “pokers.” Why pokers? Because stars and 1-10 ratings are common. Pokers are mine. Because fireplace pokers can jar you into action if you get bit by one, seven pokers means you should read this paper if you have any responsibility for virtualization.

I have a confession to make: When it comes to Citrix’s XenServer/Presentation Server/MetaFrame/WinFrame product line, I’ve always been biased. I simply love it. I remember giving people JDE software in a midsize manufacturing company (that has since been swallowed by a large imperial juggernaut.) When I was a server admin there, I had only to deploy the Citrix client and some configs and the desktop admins loved me for it. At my prior company, I remember going desktop to desktop putting nasty, frequently-updated-due-to-crappy-design applications on hundreds of clients’ desktops. Thanks to Citrix, I was the office hero because staff could work at home when they needed to.

Then I discovered VMware and fell in love all over again. Now, I could deploy rich desktops without granting server access to the desktop. I could consolidate hundreds of servers, roll out emergency desktops in half the time, deploy servers from cloned templates with ease and backup entire systems without any agents. The only problem was that I never had Citrix run well on ESX 2 and 2.5, even though I was being told that Citrix and VMware go together like PB and J. If I were to anthropomorphize the whole thing (and I will) I’d say the two were jealous of each other and vied for my love and affection.

Putting Citrix on VMware
I had been advising people against using Citrix and VMware together; but should one insist, I have always recommended that they do some serious testing first. Then one day I broke down: After having read the VMware Performance Study and a great VMTN post, I figured it was about time I did my own testing. And like the aforementioned references, I got great results.

I officially rolled out Citrix Presentation Server on VI3.5 and the performance has been stellar. I don’t have a lot of users on the Presentation Servers, but I run them alongside other production servers hosting the server side of some medical applications (billing applications, etc.), effectively putting the client and server on the same hosts. I’ve done this for my own office and for a couple of clients now. You could say that I am officially backpedaling now and embracing Citrix on VMware.

Here are my suggestions if you decide to try this for yourself:

Disaster recovery services (DRS) - Use anti-affinity rules to keep your Citrix servers from bunching up together if you allow automatic allocations. While it’s unlikely that a large farm will wind up with all of its Citrix eggs in a few baskets and then lose all of those baskets, it’s a possibility that should be planned for.

Storage - Use the fastest storage you can use. Citrix directly affects the user experience and shouldn’t be skimped on. Slow Citrix equals unhappy masses, which equals poor perception of IT, which equals job troubles for you. If you have multiple storage area networks (SANs) to connect to, or even multiple logical unit numbers (LUNs) on the same SAN in different RAID groups, separate out the virtual machine disk files across your storage infrastructure to minimize the amount of disk I/O that Citrix boxes can generate (this is a good thing to do in any Citrix environment, not just a virtualized one.) Granted, I’m talking out of the side of my head here, since I run one of my Citrix farms on an iSCSI SAN, and it performs very well, but scalability may be an issue I don’t have to address to the same degree as the largest enterprises.

Benefits of a Citrix on VMware system
The net result I’m seeing is an average of 18-20 users on each Citrix box before performance starts to tank. I was getting the same performance on my physical boxes. I don’t need to schedule a reboot as frequently due to memory leaks (though we also redid the base Win2K3 install for R2, so I can’t definitely point to virtualization as benefactor here), and when I do reboot, the reboot time is, like any virtual system, much faster than a hardware reboot.

Since I now can put my Citrix disk on the SAN, I can do block-level backup of data stored on Citrix servers (which, as any Citrix admin can tell you, happens no matter what you do since users always find a way.) Having templates makes it easy to roll out new Citrix boxes as well, especially since PS4.5 makes adding a new Citrix box to your farm a breeze. Then there’s my favorite: snapshots. I’d take a lower user/server ration if I had to just to have this feature. Luckily, I don’t have to. I can take snapshots just before and after every new application is installed for publishing; before and after every app is patched; before and after updating Windows; and before and after updating Citrix. Being able to roll back with such ease is what makes me truly, deeply happy with Citrix on VMware.

So, same user/server ratio, shorter downtime periods, quicker deployment and snapshots: I call this a win-win.

If one thing annoys me to no end, it is unused capacity.

That’s why I like virtualization. And it’s also why I like grid computing. Heck, that’s why I like cheeseburgers (there’s no empty space in my stomach after a trip to In-and-Out.) Virtualization makes efficient use of existing hardware to control costs. VMware environments often have host servers with more than a small RAID1 array in them because they were existing servers retasked as ESX hosts. Sometimes this space gets used for ISO file storage, sometimes it gets used for virtual machine toys (so called gray- or black- boxes) and sometimes it’s used for production VMs. Labs are often set up on the local storage with machines that have no production value hosted, which makes great use of that space (but perhaps not as good of use of CPU or memory for production machines).

A case study in space, storage and VMs
Then there are times when I walk into sites like the one I saw last week. They had a virtual-machine iSCSI SAN set up on each ESX host homed to the local storage. This was in addition to their FC SAN, by the way. They even ran part of their production environment off of it, using the unused internal disk space on the ESX servers to store virtual appliances that ran iSCSI Targets in them, similar to what I described in an earlier post. What a great use of space. Kudos to them!

I was concerned about how they were to keep those virtual machines up in the event of host failure. First I was told that they put in a poor-man’s round robin, which is to say, host 1 has iSCSI SAN 1, host 2 has iSCSI SAN 2, 3 had 3, and 4 had 4, and they all replicated to each other; VMs hosted by ESX 1 were on SAN 2, those on ESX 2 were on SAN 3, those on ESX 3 were on SAN 4, and those on ESX4 were on SAN1. Then, anti-affinity rules were used to prevent VMotioned VMs from winding up on the same host as the SAN on which their files existed. The replication prevented any single SAN failure from becomming a nightmare. They hadn’t done anything unusual on the network side though, which bothered me (I would prefer dedicated physical NICs for the SAN VMs!), but their performance testing showed no need for the extra NICs to be added. It’s a little hard to follow-the-leader, but it worked reasonable well. This was done with a variety of open-source packages, some of which I had never seen before. I recognized IET right away, but the SAN appliances were all custom builds, and it took me some time to figure out what was what and what was going on where. It was not an efficient use of that internal disk space because of all the replication across servers being, essentially, mirrors of mirrors.

Ingenious? Yep. Complicated to track? Yep. Functional. Yep. Needing of a little less work to manage? Yep.

There are commercial products to do this, one of them is LeftHand Network’s Virtual Storage Appliance. I’ll post more about my experiences with this very soon.

Using Storage VMotion (SVM) from the command line is fairly straightforward. However, a lot of admins may not be comfortable with the inefficiency of taking extra steps to perform a task that can be better handled via an integrated GUI tool.

It’s a feature that was clearly lacking in the VI 3.5 / VC 2.5 product release. It’s one that should have been addressed by the product team but wasn’t (I suspect for reasons related to release dates more than anything.) I looked at a few of the graphical tools for managing SVM, and found a couple of them to be pretty robust. One comes from fellow TechTarget blogger Andrew Kutz, while another comes from developer Alexander Gaiswinkler.

Gaiswinkler’s application is pretty neat and does  good job outputting information. It takes the remote command line interface (CLI) utility for managing a VMware environment and adds a graphical layer to it. Setup is relatively simple and quick, taking a few short steps. I would expect, if a firm were to make a product around it, an installation package would be able to handle the process easily. Simply put, to install the SVM GUI, you need only follow the following:

• Install VMware’s remote CLI
• Save the vms.pl script to %PROGRAMFILES%\VMware\VMware VI Remote CLI\bin\
• Save the svmotiongui.exe to your PC (I put mine in %PROGRAMFILES%\VMware\SVMGUI\ (a directory I created manually, but it really doesn’t matter where you put it)
• Launch said exe file

In testing, I was able to SVM several vmdk files across my iSCSI data stores without incident. My only complaints about the application are that it’s seperate from the VI client interface and that it needs to be documented. But it’s a great tool even without that integration.

I also took Andrew Kutz’s plugin-based SVM GUI for a spin, and was equally happy with it. Because it’s an actual plugin for the VI console, integration was a given. The installation was smooth, just like installing any add-on. The steps are:

• Launch the server and client installers from your VC host and clients and follow the prompts
• Open your VI Client, go to Plugins, and the Available tab
• From the Click Download and Install Plugin
• Follow the prompts again
• From the Installed tab and check the box

I sent the same virtual machines flying around the iSCSI skies again without the slightest problem. While it’s not at a 1.0 release, I was so impressed that I’ve moved it into production. The one downside I’ve found is in the two seperate installers, but since that’s a relatively common practice for server and client-side applications that’s not much of a downside at all. I would personally like one installer which can detect the presense of the VC server, run the server installer if it is present, then run the client installer.

The seperate GUI application is solid. On a scale of 1-10, 10 being terrific, I’d have to give it an 8. The VC integatred client gets 9. Both are great tools, but Andrew’s plugin-based model is tops in my book. Both could use more documentation, and a more streamlined install processes, but neither one is badly in need of either. Both do a great job.

Bad luck sometimes follows you. Then again, I don’t believe in luck, or its evil twin Murphy’s Law.

Still, bad stuff happens, and in IT, you’re usually there to see it. This happened at an undisclosed location near a super secret military installation next to a classified roadside diner in an unidentified town west of an unnamed river. In other words, my office, the place where I’m both the head VMware and Citrix person and the IT Director (which may give me multiple personality disorder someday, but at least I won’t rust out!).

Arriving at 8:15AM, as I’m wont to do in order to get a jump start on the day, I get an angry mob assaulting me with torches and pitchforks before I’m even to my office. That usally means a big problem, and in this case, nothing’s running. No problem, probably just a power outage knocking things around again. We have frequent outages here, and since we’re not a 24×7 shop, sometimes the UPSes run out of juice and I sometimes have to restart things by hand. Normally everything shuts down gracefully, but on this occassion, the whole place was a mess.

A quick check of server health and I see that all of my physical boxes, including my SAN and VI hosts, are up and running and have been all night. I log into VC to see exactly what I had expected: lots of powered-off servers. Now had there been an event, these servers should have VMotioned off to other servers, and failing that, come back up automatically in a set order. They didn’t. That’s usually a SAN-related issue.

Ok, I check the SAN and see that it’s fine now, but showing uptime since a little past midnight. It’s looking more like the power outage was long enough to take down the UPS that the SAN was on, but not the VMware servers, which shouldn’t be the case unless there are battery problems I’m not being alerted to by APC’s software. Guess what? Yep, that’s it. Ok, lets restart some machines - Linux servers all come up wonderfully. Windows servers, not so much on some of them. The blue screens of death are visible as far as the eye can see — so blue, I thought I was in the Caribbean, except for the stress and lack of rum-based drinks.

Most of them are reporting disk and kernel related problems. Most of the error messages relate to a missing %WINDIR%\SYSTEM32\CONFIG\SYSTEM file. Another common one reports that NTOSKERNEL.EXE is missing. Great, that’s a huge part the registry and the system kernel. This is gonna take days to fix if I have to pull backups and restore from them. Well, maybe not if I’m lucky and its just some corrupted space on the virtual disks.

Treating them like physcial machines, the next step is to boot the recovery console from CD (in this case an ISO file) and run chkdsk with the /p and /f switches as a first step to troubleshooting. Except of course, that there’s no hard disk to be found by the Windows installer, which cause a brief, although painful heart flutter at the though of pulling from backup. It’s one thing to do a quarterly test, it’s another when it’s real. Successful tests or not, massive documentation and howtos or not, the worst starts to flash through your head. You question your methodology: no matter how thorough you tried to be, you begin to think that maybe the test methods were somehow flawed and the backups will fail. Ok, the problem at hand is that I can’t get the disks to be seen by the Windows installer CD. Time to focus and forget doubt.

And this is where it becomes about virtualization, in case you thought I was going off-topic.

Fear pushed aside, it’s time to look at this from a hardware point of view, but also to remember that the hardware is all virtual. A common pattern emerges: the machines in trouble are all converted machines from an existing VMware Server 1.x install that we P2Ved some time ago. It wasn’t something I noticed right away, as a few of those machines were never P2V-ed, some were P2Ved by hand (i.e., just rebuilt), and our Linux VMs from that same VS box are all fine.

The common denominator is that all of the problem boxes had the Bus Logic SCSI controller. None of them would see the virtual hard drives. Switching over to the LSI Logic controller and accepting the change allowed me to run the recovery console, as the Windows installer saw the disks. It was a quick fix: from the recovery console, I ran the disk check and recovered with no further steps needed.

So, I get the Windows boxes back up, curse Murphy for his Law, and vow that in all future conversions, I will change over the controller before the converted box goes into production. Oh, that and I will have some people in to look at the UPS situation. Now, where are those rum-based drinks?

Inspired by a great walkthrough on the LTB Blog about getting VI3 running on a mac via VMware Fusion, I decided to go ahead and give VI3.5 a try on my Macbook Pro using Parallels. However, I wound up disappointed in my effort. I love Parallels for my XP and Linux virtual machines, but ESX 3.5 was just too far out on the fringe for it to handle. Nevertheless, I will blog about the experience.

Here’s what I used in my setup:

1. VMware’s VI3 installation set, obtainable in demo form.
2. An Intel-powered Mac (the MacBook Pro 17 / 2GB is what I used for this demo) with at least 1GB of RAM, but preferably more.
3. Parallels Desktop for Mac.
4. Lots of disk space (I used a 250GB firewire external hd).

I built the base ESX server, ESXtest1 with only 768MB of RAM, as I am a bit RAM shy on this machine. I wanted to have another machine, a second ESX box, for VMotion, Storage VMotion, etc. VirtualCenter will be hosted on an external box, since it’s going to sit on XP and we already know Parallels can do XP beautifully. It was a straightforward build with very few changes to the default settings:

• The default location of the virtual machine files
• The amount of RAM
• The boot media (I used an ISO)
• The network type (I used bridged)

I didn’t get far. As soon as I booted up, I received the dreaded error, “The installer was unable to find any supported network devices.” This means one thing: VMware doesn’t support the NIC (a Realtek 8029 AS) that Parallels emulates and doesn’t have drivers for it. Parallels doesn’t have any alternative devices to use, even though they have a drop down box.

And thus endeth my travels into purely fun-testing land. Oh well.

OK, I had a foul up in my last post about my physical-to-virtual (P2V) migration journey. I used Vizioncore’s vConverter in my file server P2V migration, and it didn’t work. I then posted PlateSpin’s product name with Vizioncore’s product name (i.e. PlateSpin vConverter) as if there were some merger from the great beyond, rather than doing the simple thing and actually editing my own posts for accuracy. It’s all been edited out correctly now, but for full disclosure purposes, there was indeed a company name and product name mix up.

Now, that said, I’ve used both products on other OEM boxes and they went just fine, so take it for what it is: a singular experience and the nature of blogging (and working with editors… that “no thanks to” line was not mine).

I seem to have a hard time with company names… I previously used incorrect capitalization for eGenera (it’s actually Egenera) some time back, and I often refer to Openfiler as OpenFiler.

Back to the tale… The domain controller from hell - a Windows 2000 server with OEM disk drivers, OEM RAID controller management tools, OEM disk management tools, and OEM network management tools. A machine nearly as old as my oldest child, one that shipped with Windows 2000 before there were service packs. It has had patches, service packs, driver roll-ups, OEM driver updates, and (probably) chocolate sauce slathered onto it in it’s lifetime.

It has also had so many applications added and removed that I think I could actually hear grinding from worn spots in the registry and creaking from the drive platters. Needless to say, this DC was spiraling down into the vortex at the end of usefulness. That’s not to say it wasn’t a great machine for a long time, but with full disks and a gurgling CPU, the poor thing was about done. It still doesn’t beat the teenagers I decommissioned early in 2007 - pair of Novell 4.11 boxes that were old enough to drive (well, with a Learner’s Permit anyway). Gotta love longevity.

First order of business: a little cleanup. Backup. Backup again to a second location. Remove IIS (it’s not used anymore). Remove an extraneous antivirus management console (a competing product is in use, this one’s just dead weight). Dump the temp files. Wipe out randomly saved setup files for applications like Acrobat Reader. Compress what I can. Defragment.

Finally, enough free space is there to support the VMware Converter agent. Lets try that and see how it goes (often, it’s the only tool I need). Failure. Hours of waiting are gone, even though the conversion hit 100% and claimed success. Turns out there’s an invisible OEM partition sitting out there that the OEM tools don’t show, and said partition has hosed the boot device order on the new virtual machine (VM). What do I see - the Blue Screen of Death (BSOD), pointing me to INACCESSIBLE_BOOT_DEVICE.

Not a huge deal, right? Just edit the boot.ini, right? No need to worry about that missing partition, right? Nope. Sure, I try to repair it by mounting to a good VM and going into the disk to edit the ini file. No luck. Ok, lets get rid of the driver. We can see the partition. Done. Now lets try again.

Failure. Are we sensing a pattern here? Same BSOD. Just like the first time, Converter goes 100% and the box BSODs on boot again. So, now that the disk management tools are no longer hiding the OEM partition, I edit boot.ini to get rid of the partition, make sure that the partition is unchecked in Converter, and try again. It succeeds!

Kind of. It’s slower than molasses in the Minnesota winter, that kind of winter where all you want to do is sit inside by the fire and let the good folks out at — sorry, for a minute there I was channeling my inner Garrison Keillor. I’m back. It’s drivers.

The OEM RAID drivers are still in there, but they are easy to strip. And it even boots up again and runs. There’s no network though. OEM NIC drivers get the strip next, but still no network (as expected). Reinstalling the VMware tools to replace the drivers doesn’t help. Next step is to shut down the VM, remove the NIC, boot again, and then add in a new NIC.

Hosed. Now the machine boots up, but it won’t let me log on. The OS is toast, and I’m not whipping out the recovery CD.

Time to pull out another product. Vizioncore’s vConverter, an acquisition from Invirtus that’s stable, robust, and more feature-rich than VMware’s offering. Redo the P2V with this tool. Same problems with the boot screen.

And there it sits for a day, in limbo, while I spend some time on Google, TechTarget, and VMware’s websites.

Finally, it’s all together… AD is corrupt. Somewhere in all that stripping of drivers, I’ve whacked Active Directory. Ok, lets fix that: Start over. Whack the VM. Re-backup. Run DCPROMO and demote the server so that it’s no longer a Domain Controller.

Time to P2V - I used vConverter again, but edited the VM before boot so that there’s no NIC. I boot it, and remove all the OEM drivers then add in the NIC. It boots. It runs. It flies. No need to robocopy. All the apps are in place and running. It just hums along happily and serves it’s purpose.

Murphy’s law: Whatever can go wrong, will.

Based on the volume of media coverage, explosive growth of iSCSI storage area network (SAN) companies and my own experiences, I think it’s safe to say that NFS and iSCSI SANs have begun to prove their worth as valid storage platforms with VMware’s Infrastructure 3, and both are also a lot cheaper than 2 GB and 4 GB Fibre Channel SANs. This has made it very interesting to those who are in the lower-tier of purchasing power - namely the many small and medium-sized companies that can’t afford the initial acquisition costs (the bulk of my clientele, by the way). That brings in the great equalizer - open source NFS and software iSCSI solutions, though given a choice I prefer iSCSI, because it can support VM partitions.

One of these OSS iSCSI packages is called IET, the iSCSI Enterprise Target, and it’s rolled into a Linux distro called Openfiler, a distribution focused on being a storage platform. Before we all rush out and use Openfiler for all our needs, IET itself has had some problems with supporting VMotion, so it’s earned a bit of a bad rap. In fact, there are often wails of protesters screaming about how IET is unsupported by the good folks at VMware because of the way two SCSI commands are structured and how that can hose a VMotion move. Luckily, there’s some good news on that front.

The truth be told, the good news there is only half-good. IET’s release 0.4.5 addresses those problems. The better news is that this update is rolled into the latest patches for OpenFiler (which has a self-update option, just like any other distro). There’s a nice link in a blog here. The reason it’s only half-good… IET is still not supported, patch or no patch. Still, I’ve been using it in the lab, and it works just fine for me. I’ve done 1, 2, and 10 VMotion moves by manipulating DRS, both manually and automatically, and haven’t hit a snag yet. That said, it isn’t supported.

Hitting an Openfiler target with VI3.5 is easy. First, install OpenFiler, ESX and VirtualCenter and configure them. I will skip most of the details of basic setup, as it’s superflous. Openfiler’s install guide is here, but the Petri IT Knowledgebase has a specific article that covers soup-to-nuts here. I’ll post another below that’s a little shorter and to the point. You’ll need to create the virtual machine on the local disk of the ESX host (otherwise, what’s the point?) and configure it to take up just enough, but not too much of that precious internal disk space. In my example, I built a virtual machine with two virtual disks (you can do it with one, if you want). The trip from an unallocated lump o’ disk to iSCSI-enabled volumes went like this:

Part One: Disk Management

Set up your physical storage from the get-go. During setup, I chose NOT to do anything to the second drive - which is to say, I didn’t let Disk Druid touch it at all. When Openfiler boots, and you have your basic settings done, things like ntp, the admin password, etc., go to the Volumes button, and click on Physical Storage Mgmt. From there, create the extended and logical partitions you will need to get your iSCSI volumes up. The steps, once Openfiler is up, look a little like this:

The basic setup. System, and nothing else:

Creating the Extended Partition:

Creating the Logical (Physical) Partition:

The Partitons on the Second Drive:

During creation, I ran into a problem on both machines I did this build on - in each case I had created the logical partition, but OF didn’t see it. I had to go delete it, and create it again.

Part 2: Creating the LUN

First up, enable the server to act as an iSCSI Target. This is done from the Services tab, under Enable/Disable. In this example, I’m only using iSCSI, but if you wanted to enable NFS, this is where you would do so. Mine looked like this:

Then, go back to Volumes, under Volume Group Mgmt, and create the iSCSI volume group. In this example, I’m using the whole disk, but you don’t have to.

The before and after looks like this:

Next, head over to Create New Volume (one tab left). Creating a volume is, like the rest of the process, relatively straightforward. I made one volume, comprising the entirety of my one Volume Group. Having chosen the name”san” (how creative of me) for the process, I kept with the same simplicity for the volume I created.

After that, you are forwarded to the List of Existing Volumes tab. Congratulations, you have an iSCSI SAN. Now, about configuring that SAN…

Part 3: Security

There are two ways to lock down your SAN. One is via the network, and choosing only to allow certain hosts or networks access to the SAN. From the List of Existing Volumes tab, select the Edit link under the Properties column. You will then be presented with a screen called Volume’s Properties giving you a number of options for securing the LUN. I won’t dwell too much on the network access restrictions, as my screen captures will be different from other networks. The long and short of it is that in the General tab there is a sub-tab called Local Networks. You can create individual hosts and subnets there, and then allow or deny access to these networks on the Volume’s Properties tab.Using MS-CHAP to lock down who can connect to your target is a much smarter plan.

Part 4: Getting VMware to See Your LUN

This is extremely straightforward, and is no different than any other iSCSI implementation. The only thing I’ll mention here is something that is left out of many, many documents. Open the ports for iSCSI on your ESX hosts! In 3.5, it looks like this:

There are a lot of considerations that need to be addressed about what you keep there, chiefly around reliability and performance. For example, if you put virtual machines in the iSCSI SAN on the physical host, when the physcial host dies, it takes the SAN with it, along with all those virtual machines. Another consideration is networking. The long and short of it is this - if you have a whole lot of unused disk space on your ESX hosts, you can use Openfiler to put stuff there. As to what stuff, that’s entirely up to you. I personally prefer iso files and other junk that doesn’t need to go anywhere or do anything, and isn’t going to hurt me when something fails. Of course, you could also use NFS, which Openfiler supports.

The end result, Openfiler gets a solid 8 pokers from me. I give it a ten for being a soup-to-nuts full-featured distro that I can use at almost any client site for almost any purpose. If this were a storage column, it would remain a ten. It drops back because they haven’t yet earned a nod from VMware, which is something I would have expected them to pursue, considering their growth and media attention lately. What makes it frustrating is that Xinit systems, the company behind the project, makes storage appliances. Since they’ve been producing Openfiler since 2003, I think it’s about time they got on the move with this!