VMware has just updated their security hardening guide, which provides recommendations for hardening a VI3 environment.

In addition to the updates for virtual machines and the ESX Service Console, they have now added new recommendations for ESXi, VirtualCenter Add-on components (plug-ins) and for Client components.

Here’s a brief overview of the recommendations for VMs and ESX hosts that have been added to the guide. No new recommendations were made for VirtualCenter except for the Plug-in ones.

Virtual machines:

• Disable copy and paste operations between the guest operating system and remote console
• Do not use nonpersistent disks
• Ensure unauthorized devices are not connected
• Prevent unauthorized removal or connection of devices
• Avoid Denial of Service (DoS) caused by virtual disk modification operations
• Specify the guest operating system correctly
• Verify proper file permissions for virtual machine files

ESX Service Console:

• Secure the SNMP configuration
• Protect against the root file system filling up
• Disable automatic mounting of USB devices

There are some general recommendations when using plug-ins and some specific ones when using Update Manager, Converter and Guided Consolidation. The guide recommends that the Update Manager and Converter plug-ins not be installed on the VirtualCenter server but should instead be installed on a separate server or virtual machine.

Also added is a section on client components. The guide recommends against the use of Linux-based clients when using the RCLI, VI Perl Toolkit scripts, VM console access initiated from a web access browser session and programs written using the VI SDK. The reason for this is that communications with Linux clients are vulnerable to man-in-the-middle attacks because the Linux versions of these components do not perform certificate validation. This risk can be partially mitigated by ensuring that the management interfaces (ESX Service Console and VirtualCenter) are on trusted, isolated networks.

The guide suggests that client components are to verify the VI Client integrity because of the VI Client extensibility framework that was introduced into VirtualCenter 2.5 which provides the ability to extend the VI Client. It also recommends that one monitor the usage of the VI Client instances by inspecting log files on client systems. Both of these tasks can be quite difficult to do because there are no native methods for doing this.

Finally a section was added for securing the host-level management in ESXi. Many of the recommendations for ESXi are the same ones that were made for ESX. Some unique recommendations for ESXi include ensuring secure access to CIM (the hardware management api’s). Also, admins may want to audit or disable the special technical support mode which is designed to be used in case of an emergency but is sometimes used by administrators to access specific functions in ESXi.

You can read the updated guide in its entirety here.

I recently came across an article revealing that 1 out of 3 IT administrators have used their elevated privileges to snoop on confidential information. It’s always possible to lock out administrators to sensitive data through operating system access controls, however, a virtual environment opens up other avenues for exposing sensitive data.

With physical servers, the task of imaging a server’s hard drive for offline examination is not always easy. An administrator of a virtual environment can easily and stealthily snapshot a virtual machine to temporarily suspend writes to disk file, make a file system copy of the VM’s disk file from the host server while it is running and then take that copy to a workstation where they can mount it and attempt to gain access to information to which they would normally not have access.

Either by mounting the disk file to an existing VM then adding an additional hard drive to access the information on the drive, or creating a new VM and mounting a live CD to utilize hacking utilities to defeat the operating system security, admins can bypass operating system level controls to gain access to the data simply by making a copy of the disk file and mounting it elsewhere .

Virtual servers open up additional attack vectors over physical servers, illustrating why proper security measures must be utilized to ensure that sensitive data is adequately protected in virtual environments. In addition to properly securing host servers, auditing and logging should also be in place to track all logins and activities on host servers. Administrators typically need access to sensitive data to be able to do there jobs but this access should be limited as much as possible to only what they actually need.

Many administrators snoop because they know they can get away with it. By restricting access and logging events, the 2/3rds of IT administrators who set the better example make snooping more difficult for nosey admins.

VMware Inc. and Tripwire Inc. have co-developed a free, downloadable utility to address the leading security concern in virtual environments today: misconfiguration of the hypervisor.

Portland, Ore.-based Tripwire ConfigCheck is a free Windows and Linux based utility that assesses the security of VMware ESX 3.5 hypervisor configurations compared to the VMware Infrastructure 3 Security Hardening guidelines, which were released in February.

The Security Hardening guidelines explain in detail the security-related configuration options of the components of VMware Infrastructure 3 and how security affects certain capabilities.

Tripwire ConfigCheck makes sure ESX environments are properly configured according to these guidelines and lends insight into vulnerabilities in virtual environments. It also provides the necessary steps towards full remediation.

Dan Schoenbaum, senior vice president of marketing and business development for Tripwire
said the utility is being offered for free to encourage the proliferation of VMware’s Hardening guidelines and to increase virtual machine (VM) security.

Tripware hopes that by giving a taste of their technology for free, users will become familiar with them and invest in their software products with more security capabilities, Schoenbaum said.

Colorado Springs, Co.-based Configuresoft Inc. also provides a toolkit for compliance with VMware’s security hardening guidelines. The toolkit consists of a set of rule-based templates, reports and dashboards that plug into Configuresoft’s Enterprise Configuration Manager (ECM).

VMware Inc. Senior Director of Enterprise Desktops Gerald Chen visited our office on Tuesday morning to discuss the different types of desktop virtualization and answer common questions about Virtual Desktop Infrastructure (VDI), for example, how it differs from terminal services and cost issues.

Here’s how VDI works: each end user gets a virtual machine (VM) that is deployed from a server in the data center directly to a PC, laptop or thin client computer. Each VM is customizable, so all of the user’s settings are saved and re-booted each time the user signs in, Chen said.

When a user logs off for the day, their VM goes idle, and wakes back up when the user logs into their system again, according to Chen. Chen believes that the advantage of VDI is that sensitive data is not being stored on desktops, which can easily be lost or stolen, and these virtual desktops are easier to manage than physical ones.

“VDI is great for industries like health care that are really concerned about information security and compliance. The real value though, is in management. All of the information is safe in the data center, and centrally managed through Virtual Infrastructure,” Chen said. “For instance, if you have 100 new employees who need desktops, you can deploy a VM for each of them in just minutes, and manage all of them centrally.”

VDI is different from Sever Based Computing (SBC) systems like Citrix Systems Inc.’s XenApp in that VDI is connects a single user to a single operating system (OS), instead of having multiple users share one OS.

“Not every application likes to share an OS, and there is also bad isolation; if one application crashes, everyone sharing that OS crashes as well. Those desktops can’t be customized either. It is a locked environment.”

Chen went on to explain that with VDI, four to ten VMs per server core are supported, so a server with one quad-core processor can, theoretically, house 40 VMs. Of course, that varies depending on things like workload, applications and memory. If the VMs become too heavy for the server to handle, management features in VI3 intervene. VMotion can move live VMs from one server to another when capacity issues arise, as can Dynamic Resource Scheduler, which allocates and balances computing resources as needed using VMotion.

Desktop virtualization case study
As VMware announced customer case studies in February, including one at Huntsville Hospital in Huntsville, Alabama.

The hospital needed to implement a new medical information application throughout its network while protecting HIPAA-related data. Deploying hosted desktops on VMware, the hospital could lock down sensitive patient data and reduce the cost and complexity of desktop management.

They used combinations of thin clients and blade servers to access the centralized virtual desktops, and in turn, reduced power consumption across the hospital by 78%, improved longevity with lower hardware maintenance needs and made wireless thin clients on wheeled carts available to hospital staff. Also, doctors can remotely access their VMs through the Internet using a web browser when necessary.

The downside to desktop virtualization
While the benefits are clear, there are some downsides to desktop virtualization: extra storage and initial cost.

Chen told SearchServerVirtualization.com that VMware is working on reducing image sizes and has designed a way to keep only one copy of files that are identical among many users, like icons and other graphics, to reduce the amount of storage necessary.

The cost of implementing desktop virtualization turns users off. According to Ars Open Forum blogger ‘Bright Wire,’ the cost and the magnitude of system upgrades required is not worth the benefits.

“The cost of deploying virtual desktops is massive,” Bright Wire wrote. “You will need to re-gear your existing desktops to run the virtual or you will need vendor equipment that costs twice as much as a new desktop. Either way, the cost is big in manpower. On top of that, your infrastructure will need serious review.”

According to VMware’s product specifications, local desktop virtualization requires a 500 MHz or faster processor with recommended 256 MB of memory, though Forrester reports that PCs must be faster and have more RAM to work efficiently.

“In addition you need to look into the server infrastructure,” Bright Wire said. “You are talking about needing a lot of iron on the backside to handle the needs of the server to supply two to 16 desktops. All this adds up quickly and can easily swamp a datacenter.”

As for pricing complaints, VMware is used to hearing them and holds firm to the ‘you get what you pay for’ mantra, saying the management benefits are worth the price.

The company charges $150 per concurrent user plus additional costs for support, either Gold or Platinum levels. Both bundles include VMware Infrastructure Enterprise Edition for VDI (which consists of VMware ESX Server 3.5 and VirtualCenter 2.5) and the VMware Virtual Desktop Manager 2. The VMware VDI Starter Edition, which enables 10 virtual desktops, has a list price of $1,500. The VMware VDI Bundle 100 Pack, which enables 100 virtual desktops, has a list price of $15,000.

The market indicates a demand for desktop virtualization, as a number of other vendors also entered the desktop virtualization space including Sun Microsystems Inc., Citrix., Pano Logic Inc. and Symantec. Chen would argue that many customers come for reduction in hardware but stay for the management applications.

“Reducing hardware costs is not a reason to use VDI, it is management. We have customers who have seen 40% to 50% ROI in terms of management costs and the amount of time it frees up.”

With all the talk about virtual security these days , you would think that people actually are addressing the concerns over security in virtual environments. However, many administrators resist implementing strict and proper security measures in their environments because of administration inconveniences that tighter security usually causes.

For example, the default settings of VMware ESX prevent users from using secure shell (SSH) to log into the server as the root user. Yet, the first thing many users do is to modify the SSH configuration to allow root access via SSH because this is a more convenient way to log into Service Console. The correct and more secure way to do it would be to setup a separate SSH user account and then use the SU – command to gain root privileges. Xtravirt has published a good step by step guide on how to do this here.

When you virtualize servers, additional security measures should be followed in addition to standard ones that you would use for physical servers. Most importantly, the host system must be protected at all costs: If someone gains control of the host server then all of the VMs that run on the host can be compromised. The Center for Internet Security (CIS) has published some security guidelines for ESX and virtual machines that I would recommend you read through and follow to ensure your environment is secure. Xtravirt has a great security assessment template that they’ve put together that you should look at also.

Virtual networking is another critical area for securing virtual hosts. Virtual switches differ from physical ones and must be properly configured to ensure secure host and virtual machine network traffic. Often, simple recommendations like isolating Service Console and vMotion traffic are not followed, which creates unnecessary risk and exposure of your hosts.

Are you willing to risk losing your data? Data breaches can result in negative press exposure, lawsuits and fines. I would encourage everyone to please take security seriously. Security may cause some administration inconveniences and headaches, but they are a small price to pay to ensure that your servers, and more importantly your company’s sensitive data, is well protected and safe.

To help you with this I’ve included a list of some good virtualization security blogs and websites that you should check out:

• SearchServerVirtualization Security blog (various contributors)
• Rational Survivability (Christofer Hoff)
• VMware Security Blog (VMware)
• Vmtn Security Community
• Security Focus
• Security Top 10 List and Security Links

If your company deals with credit cards, you are required to follow the Payment Card Industry’s data security standards (PCI DSS). The major credit card players – Visa, Mastercard, American Express and Discover — set forth these requirements in order to protect credit card data. If audits reveal that these regulations are not followed, fines or revocation of credit card processing privileges can result. Often, these audits force companies to implement basic security practices that should have already been in place; however, no virtualization-specific requirements have yet been put into practice.

Having just survived another annual PCI compliance audit, I was again surprised that the strict standards for securing servers that must be followed contain nothing specific concerning virtual hosts and networks. Our auditor focused on guest virtual machines (VMs), ensuring they had up-to-date patches, locked-down security settings and current anti-virus definitions. But ironically, the host server that the virtual machines were running on went completely ignored. If the host server was compromised, it wouldn’t matter how secure the VMs were because they could be easily accessed. Host servers should always be securely locked down to protect the VMs which are running on them.

It seems that much of the IT industry has yet to react to the virtualization trend, having been slow in changing procedures to adjust to some of the unconventional concepts that virtualization introduces. When I told our auditor that the servers were virtual, the only thing he wanted to see was some documentation stating that the remote console sessions to the VMs were secure. It’s probably just a matter of time before specific requirements for virtual servers are introduced. In fact, a recent webinar takes up this issue of whether or not virtualized servers can be considered compliant, addressing section 2.2.1 of the PCI DSS which states, “Implement only one primary function per server”; that is to say, web servers, database servers and DNS should be implemented on separate servers. Virtual servers typically have many functions running on a single physical server, which would make them noncompliant.

Looking at the PCI Knowledgebase, it seems many companies are confused on this and some are not implementing virtualization until this is cleared up. We’ll have to wait and see what develops and how the specification is modified to allow for virtual servers. It would be in the best interest of companies like VMware and Microsoft to work with the PCI to get this sorted out as soon as possible.

You can read the current PCI Compliance 1.1 specification here.

VMsafe, the new security technology created by VMware Inc., gives virtualization users the ability to monitor and secure virtual machine resources in ways never before possible.

After I wrote a short article on VMsafe last week, I received feedback from Burton Group analyst Chris Wolf, who was at the VMworld conference in Cannes, France. His comments weren’t included in the story, but they put things in perspective, so here they are:

“VMsafe is a very important technology in my opinion, as it changes how virtual environments are secured. Today, security appliance virtual machines (VMs) typically monitor other VMs by connecting to them over a virtual switch. The result is virtual network monitoring that resembles physical network monitoring,” Wolf said. “The current model is fine until VMs begin to dynamically move across a virtual infrastructure. Dependent security appliances always have to follow their associated VMs as a VM is relocated. This can complicate the live migration and restart processes.”

“With VMsafe, you would typically configure a security appliance per physical host, such as an [intrusion prevention system] virtual appliance. The security appliance vendor would leverage policies to determine what to monitor (such as by VM, virtual switch, subnet, etc). With VMsafe, the appliance can connect to any virtual switch by accessing it through the hypervisor; you no longer have to configure a special promiscuous port group for network monitoring,” Wolf said. “With security configured at the host level with no direct attachment to virtual networks, VMs can move freely about the virtual and physical infrastructure and still have their related security policies enforced.”

Wolf continued, “VMsafe also provides the framework for offloading many security activities to special-purpose security VMs, including roles such as antivirus monitoring. As we move to an automated or dynamic data center, having special-purpose security appliances that are capable of enforcing security policies at the hypervisor level can ease security management in an environment that will be constantly changing. Sure, it’s possible to enforce security policies with special purpose network-based appliances, but such configurations would be substantially more complex to deploy and manage than comparable solutions based on VMsafe technology.”

Chris Wolf and I were presenting Virtualization 101 in Seattle yesterday when something he said sparked an idea in my usually dormant brain. Okay, it’s not usually dormant, but Seattle is so cold I think half of my synapses aren’t firing! In the process of discussing virtual machines (VMs), Chris mentioned that each major virtualization solutions provider has registered itself with the Institute of Electrical and Electronics Engineers (IEEE) and received one or more Organizationally Unique Identifiers (OUIs). An OUI is 24-bit number that makes up the first half of all of the Media Access Control (MAC) addresses assigned by an organization to devices it produces. MAC addresses are most frequently associated with Ethernet adapters, so why are virtualization vendors registering with the IEEE to obtain OUIs?

Virtualization vendors also produce Ethernet adapters — virtual network interface cards (NICs). Most VMs would be rather useless if they could not access some sort of network, so virtualization vendors must create virtual NICs in order for the VMs to get on the big wide world of Webs. And since these virtual NICs have to participate on the network just as if they were physical, they must use MAC addresses. Because the first 24 bits of these MAC addresses, the OUI, is organization-specific, there is a real potential for network administrators to detect not only if a machine on the network is virtual by its MAC address, but also what type of virtual machine it is (what vendor’s software is hosting it). While best practices dictate that you do not change the MAC address of VMs, enterprise virtualization solutions do present this as an option, and, because of this, here is the scenario I see occurring.

One way to harden the Apache Web server is to use mod_security to alter the Web server’s signature. For example, you can fool clients into thinking that the Web server hosting their favorite videos is actually a Microsoft Internet Information Systems (IIS) 5.0 server instead of Apache 2.2. Administrators do this in order to fool attackers into attempting the wrong types of attack vectors. Even though best management practices dictate that administrators NOT alter their VMs’ MAC addresses, I forsee them doing so anyway in order to fool would-be hackers into attempting the incorrect attack vectors on VMs. For example, if a VM is hosted on ESX and its MAC address has an OUI registered by Microsoft, then a would-be attacker may try known Microsoft Virtual Server or Hyper-V exploits on the VM instead of ESX exploits.

Who knows? Twelve months from now altering a VM’s MAC address to be that of another vendor may be considered a best practice, but right now, with the already complex problem of managing virtual hardware, IT administrators are best served to leave their VM MAC addresses well enough alone.

Of course, that doesn’t stop the idea from being completely and utterly cool!

Hope this helps!

A recent discussion on the OpenBSD mailing list led to the assertion that virtualization decreases security. For those interested, a summary of the discussion is available on Kernel Trap. But proponents on both sides of the argument have taken to throwing about emotionally driven comments rather than thinking objectively about the subject. Of course, because the original comment labeled all those as”stupid” and “deluded” who think virtualization somehow contributes to security weaknesses, who can really blame people for getting a bit emotional? All the flame-war commentary aside, the question remains, does virtualization weaken security? The original argument that virtualization can diminish security was based on two points:

1. If software engineers cannot create an OS or application without bugs, what hope does a virtualization solution have to be bug-free?
2. x86 hardware is ill-suited for virtualization.

Bug-free software
The first point does two things: it lumps all software engineers, operating systems, and applications into one pool and assumes that it is possible to find bug-free code.

Addressing the sub-points in order, while it is true that software engineers are human (and we make mistakes) and that software in general has a track record of imperfections, it is also true that the world does not judge all software engineers or software to be the same. In fact, I would guess that a lot of members of the OpenBSD mailing list in fact prefer OpenBSD to, Windows, let’s say. However, there are many readers of this blog that may prefer Windows to OpenBSD, or Linux, or OS X, etc. The same preference could be applied to office suites (OpenOffice, StarOffice, MS Office, KOffice, etc.). The fact of the matter is that we all have our own preferences: we do not judge software to be the same.

Secondly, the first point argues that the community should expect a bug-free hypervisor, and anything less contributes to the decrease of the overall security of a server platform. This is a very lofty expectation indeed! A very long time ago I wrote to Slashdot, heck, almost seven years ago now, and asked the question why it was not possible for developers to spend more time on projects and produce bug-free software. Commander Taco (the ring-leader of Slashdot) himself replied to me and said that it was a foolish expectation: software is 1) written by humans and 2) is far too complex today to be without errors. However, people still judge some operating systems to be more secure than others. The same for kernels. How can such a judgment possibly be made if all software has bugs? The answer is “easily.” We observe the rapidity that bugs are discovered in software, the impact that they have on the IT infrastructure across the world, the speed at which operating system and independent software vendors (OSVs and ISVs) release patches, and how easily those patches are applied without affecting the rest of the server platform, and then we judge the security of a piece of software. Therefore we do not judge a piece of software to be secure because it contains no bugs, but rather by the history of its imperfections and how quickly blemishes are removed.

Notice that I did not say whether or not I agreed with Mr. Slashdot. I do not. I do believe that software designed be generally purposed, such as today’s OSes, is doomed to be bug-laden, simply because it lacks a specific purpose and too many conditions have to be accounted for. However, imagine if the same leeway were given to the software that runs our air-traffic control systems? Or military installations? Such software is held to a higher standard, and it can be in part because it is designed with a specific purpose. The same is true of hypervisors: they are designed specifically for one purpose. They do not yet have all the cruft and bloat sitting on top of them that today’s OSes do. Here’s hoping that the ISVs producing today’s leading virtualization solutions step up to the challenge.

In short, I believe that there is a reasonable expectation that hypervisors will be a lot more secure than general purpose applications written on top of general purpose OSes could ever hope to be.

x86 hardware
Yes, the x86 instruction set was never designed to be virtualized, but to say that the instruction set has not grown well above and beyond its original intentions is to do an injustice to the original minds at Intel who took part in creating one of the most persevering pieces of technology to date. With the first set of virtualization extensions, those created to solve the problems of ring compression and ring aliasing, the x86 instruction set was given a breath of new life. And with the latest extensions, enabling live migrations of virtual machines across multiple generations of processor versions, the staying power of the x86 instruction set in a world with virtualization has been increased even further.

My point is simply that just because the x86 instruction set was not designed with virtualization in mind does not mean that it cannot work, and work securely. That is the beauty of x86: it can be extended to do what we need it to do. History has spoken.

Conclusions
With both of the original arguments shown to be false, is the conclusion of the original argument then reversed? Not completely. While virtualization does not decrease security, the potential for it to do so is there. Hypervisors are software, and although they are a lot less likely to have bugs than a general-purpose piece of software, bugs can still occur. However, to blanketly state that virtualization decreases security is far too general as there are many different implementations of virtualization. For example, if VMware ESX was found to have a memory sharing bug that allowed one virtual machine to read and write the memory of another, does this mean that XenSource is immediately compromised? Of course not. So even when a bug in a hypervisor is found it does not immediately mean that all virtualization is suddenly subject to the same problem.

As I stated earlier, the security of any software package is judged by the number of bugs historically observed, their impact (potential or real), and how quickly the parties responsible for said software fix the bug. While the potential is there, it is far too soon to observe whether or not virtualization decreases security. Only time will tell.

What do you do when you are giving a session on virtualization migrations, and no one shows up? You sit down and blog about how you are giving a session on virtualization migrations and no one showed up! In all fairness, TechTarget is giving away a Mercedes-Benz right now, so I think people are eagerly awaiting that drawing, and my session is right after lunch, AND it *is* a repeat of one I did earlier today. Yes, those are the things that I will tell myself tonight as I curl into a fetal position and sob in a corner

DCD 2007 has been a natural extension of last year’s conference. 2006 saw virtualization come into its own in the enterprise, and 2007 has seen virtualization mature into a ready-to-use, ready-to-integrate solution for many of today’s data center related problems. One of those problems has been disaster recovery, business continuity, and business resumption. This year’s DCD conference has had no shortage of innovative and instructional sessions on how to create cost-effective BC solutions using virtualization. Two technologies that are paving the way for these SMB to enterprise BC solutions are iSCSI and 10 GbE. Virtualization requires shared storage to enable any of the features used in DR and BC scenarios (LiveMigration, Resource Scheduling, Power Management), but for the longest time shared storage has meant fibre channel SANs, an expensive proposition for many. The continued success of iSCSI and the eventual commoditization of 10 GbE will enable enterprise-class shared storage at a fraction of the cost for SMBs and enterprise data centers alike. Inexpensive, enterprise-class shared storage will result in the availability of virtualization’s high end features that contribute to DR and BC solutions for all organizations — small, large, and everything in between.

Once again, DCD 2007 was a natural extension of 2006 — the data center has continued to evolve around virtualization. What will next year bring? My best guess? To steal from a colleague, we are going to start hearing about the highly-utilized data center, the dynamic data center, the data center that can be managed and monitored from a single console. And who is to say that the data center management of the future will require any human interaction at all?

Dave?