What we have learned about Linux and Virtualization….

To virtualize or not to virtualize, that is the question.  A simple question, but one with a difficult answer.  The process for determining whether any environment should be virtualized is not always as simple as it seems.  In this brief, we will discuss the steps we use to determine whether or not something is a good fit for virtualization.  We go over some general rules of thumb for making this type of decision.  We will also discuss which types of tools you should use to complete the assessment.

It seems like every time I enter into a discussion with any of my peers these days, the topic of virtualization is surely going to come up.  A lot of the reason behind this is because I work in an environment that has gone from almost no virtualization five years ago, to nearly ninety percent(90%) virtualization today.  We aren’t alone though, most of the large companies I have worked with in the past have done pretty much the same thing.  With large inexpensive blade servers and the ability to put a large number of CPU Cores in an Xseries box these days make it almost impossible to pass up.  What my peers and I have discovered as we have been asked to race to a 100% virtualized environment finish line, is that it’s not always the right move.  There are a lot of situations where a completely virtualized environment makes sense.  However, it’s the few edge cases that will keep you from reaching that goal.

Software licensing is one of the easiest ways to increase the ROI of any virtualization project.  In the beginning, that savings alone could pay for the whole project.  As businesses themselves, our vendors though have started to catch up and figure out how to keep their profits up, while the virtualization wave keeps rolling in.  In some cases, these changes have removed some, if not all of the savings we once would have gotten.  The fact is that the ever faster Moore’s Law races to better performance keeps forcing prices down while increasing the CPU cores per machine.  To give you an idea of how things have changed, let’s take these two old school models and show you how they might have been charged or licensed.  Any resemblance to how a specific company charges for software is purely coincidence, this is just an example.  Software A is sold at a per CPU basis prior to the virtualization race.  Software B is sold per physical machine.  In some cases, the companies selling Software A had to redefine what a CPU meant.  Did it mean virtual CPU or physical CPU?  The best model we have seen is a a lowest count model.  So if the number of virtual machine cores is less than the total number of physical CPU cores in the machine, you pay for the lower number of CPU’s.  Other companies are using a more confusing model where they convert CPU Cores to a mathematically created number.  This number is then used to price against.  This method can get confusing quickly and may or may not save you any money.  Software licensed under Software B per machine model, have largely left that mode in place.  They tend to choose to license at the virtual machine level instead of coming up with a new formula.  With either method though, you need to read your contracts ahead of time and most likely ask a sales representative to stop over and explain it to you.  This is most often a large part of the decision process I use when advising about whether or not it will be worth virtualizing any piece of closed source software.  Even some open source support models get into the act, so read all of your contracts carefully and ask your sales representatives how they handle licensing under a virtualized environment.

Disk I/O or applications with a lot of reading and writing to the disk are a questionable fit.  The reason is that you only have so much throughput to work with going to the disk.  New disk technologies are coming on all the time that will help to eliminate this issue in the future.  Until then we only have the option of solutions like direct attached storage dedicated to the virtual machine.  This can still suffer from the same bandwidth issues, but for now it is the best option even with a higher cost of implementation for the cards and other equipment needed.  These direct attached options also have other problems that can limit things like automated migrations and issues with backing up these same disks.  

Where virtualization shines, is in CPU and memory intensive applications.  So while a syslog server might not be a great choice for virtualization because of all the disk writes it will need to do.  Completing analytics on those logs though, might work wonderfully.  Applications mixed on the virtualization host are also great ways to exploit more of the potential of the hardware that you purchase.  So if you have a web application that your customer care representatives use to support customers from 8am to 5pm, you will be able to share that capacity with your billing processing application from 5PM to 8AM.  At the same time, a large group of low CPU but High memory consuming apps can very effectively be combined with high CPU/low memory applications.

The hardest of all of the values of virtualization to show in a ROI is the ability to build, tear down, and restore to a snapshot are amazing.  One of the scariest parts of doing an in place upgrade of any software, from the OS to the web browser,  is the possibility of complete corruption.  When you put this software on a virtual machine you can take a snapshot of the machine before you attempt the upgrade.  Then no matter what, within seconds, you can revert to that snapshot and reboot.  Everything is back to the state it was in prior to you starting.  That restore point could be from yesterday or last year when you clicked the button to take a snapshot, is the only time based restriction.  Here is an example of what we mean:

A restaurant has a point of sale system that could be run on a virtual machine.  The vendor that supplies this software sends the restaurant a new upgrade.  The upgrade has to be done to the existing system in place on the server because of database updates.  The restaurant can take a snapshot of the virtual machine first to save a copy of what it looked like before they begin.  Once that is complete, they can then run the vendors update software.  If at any point they decided that the update either isn’t working as expected or just isn’t working at all, they just tell the virtualization software to go back to the way it was before they began.  They can also make copies of the virtual machine and do test runs of the upgrade on the copy.  Then if the update doesn’t go well they can keep trying it until they work out the complete and final process with no interruption to the other functions.  They come out of the testing processes with a tested process and can be much more confident with the release of the updates.  

Having participated in several releases of new software like this, I can honestly say doing things on physical hardware is incredibly more stressful and demanding.  When you don’t have to worry about making a mistake you tend not to miss as many steps or cause yourself other issues.  This by no means guarantees that issues won’t happen that you didn’t see in your tests.  With this type of process you can back out and try again until you are successful.  

The next big feature of virtual machines is the ability to migrate them from one piece of hardware to another.  In a recent FLOSS Weekly episode the guys from Virtual Box discussed a test they do by pushing a virtual machine around the office.  The test isn’t successful until they have pushed the machine from all of the major operating systems in a big circle.  This is a silly example, but if you had a server in one data-center that needed to be shut down for maintenance you could easily push the virtual machine either across the data-center or around the world.  This technology makes it easy to get the machine to a safely running system with no issues.

So we have the list of reasons showing when to do virtualization including what steps and tools we use to make the final decision.
1) Collect as much data as makes reasonable sense.  If you have a monitoring solution like Zenoss or GWOS, create some reports there.  If you don’t, then in Linux there is a package and tool called SAR.  SAR can be set up to run and collect data about how the system is performing.  You can then use tools like kSar to display the output and create pretty graphs.(Pretty graphs always help tell the story.  A picture really is worth a thousand words in this situation)

2) Determine the servers that seem most likely to be able to co-exist on the same hardware.  Try to come up with a scheme that can be easily explained to others.  Focus on your balance between CPU and Memory.  Shy away from things that consume a lot of Disk I/O.

3) Determine the hardware needs of your organization based on the data gathered and your costs.  For instance, do not spend big money on memory if the applications that will use it won’t migrate for a year when the prices will have dropped.

4) Verify your design with any internal developers and other support team members.  Does everyone agree with your assessment and plan?  How do they think you did?  What builds do they have for you?

We know this seems to be very simplistic and it is because keeping it simple and straight forward is what we have found works best.  Do not over think the decisions, just give it a try and see what works.  Some of our best plans based on the most data have blown up in our face because things just acted differently in a virtual world than our predictions.  The biggest problem we have faced have been around putting to many a a certian workload type together.  While allocating more Virtual CPU’s than you have physical ones generally works doing the same with Memory does not.   Remember that you can almost always migrate the hosts to other hosts to balance them out if you make a grave mistake in this area as long as you have machines to do it with.

We couldn’t live without virtualization at this point.  The cost savings are smaller than we had hoped but the productivity gains have been massive.  The confidence level of our admins is also rising on top of the snapshot abilities and quick cloning of a machine.  If you haven’t started yet, don’t wait, it’s simple, easy and can cost your company very little to get started.

What is the best Linux Distribution for the Enterprise?

What is the best Linux Version for the Enterprise?

While Linux is taking the world by storm, I often get asked what the best version of Linux is for enterprises of all sizes.  Red Hat, Suse and Ubuntu are always top notch distributions but then so are Fedora, CentOS, Mint, OpenSuse, and SlackWare.  Which distro is the best depends largely on what software you will be running on your Linux installation.  For most companies, standardizing on a distribution seems like it should be the final goal.  As with most things we discuss, the answer is maybe as well as it depends.  We are going to limit our scope in this CTO Brief to three of the top Business/Enterprise backed distributions.

A Distribution
Before we begin let us review what we mean by a distribution or distro.  At their core, a distro is a collection of software packages and a Linux Kernel.  At this point, almost every package available in one distribution is available in the others.  Each tends to use and support either custom tools or tools that the leadership of the distribution believes are the best for the end user.  The other noticeable difference between them is the UI or user interface.  This can be as simple as the major color choice for the distribution or as intricate as including the path to get to your favorite applications in the app menus, or what side the minimize, maximize and close buttons are on.  What really makes something an enterprise class package of Linux is the type and length of support you get from the distribution.  Red Hat, SuSE and Ubuntu all offer between 2 and 5 years of support for certain versions of their respective distributions.  This means that they will track, build, package and update all of the packages you get with the initial release.  This often means that the new shiny features are not present.  In turn this generally means that the system is better tested and more stable.  You always have the option to upgrade any software to the latest and greatest versions.  That does not mean however that when you call for support they will support you.  With these long term releases they are making certain guarantees and because of that, they need to control as many variables as possible.

For this article we are going to focus on the three top corporate backed and supported distributions which are Red Hat, Suse and Ubuntu.  These three are the broadly accepted as the leaders in this space.  This does not mean that, for instance, FreeBSD or CentOS aren’t enterprise grade.  I know several people that have been betting their businesses on them for the last few years with no regrets or major issues.  So we will have a follow up discussion on these distribution’s next time with FreeBSD, CentOS, and Fedora/OpenSuse.  We will try to point out how they differ.  But for now, let us break down the first three distros and shine a light on what we believe are each of their strengths and weaknesses. 

Desktop vs Servers

As you continue reading, we want to take a moment and point out that the focus is on Linux used on servers.  The bias here is author based, not reality based.  The focus in my career has been using Linux on servers.  In the last three to five years, amazing changes have taken place in the world of desktop Linux.  Some of the things that are happening will probably amaze even the most hardened Microsoft or Apple fan boy.  The server side of almost any of the Linux distributions we are talking about in this article or the follow up next time has been production ready for years.  Servers though do not have to have users typing on their keyboards or 16 USB Devices all hooked up through 2 ports on a PCI card.  So while a distribution may excel in one area like servers, another may excel on the desktop.  Red Hat and Suse are the ones in this article that do servers best.  Ubuntu on the other hand shows us all what can be done with Linux running as a desktop operating system. 

Red Hat
Red Hat was one of the first distro’s to do mass distribution right.  While not the first, they have proven over the years that you can make money with free software.  Red Hat’s leading position has ushered in several strategic partnerships with companies like Dell, IBM and HP.  They have, over the years, used these relationships to build up a level of support from third party vendors that is the envy of all of the other distributions.  When Linux is mentioned as a supported platform, most companies list Red Hat first, then any others they will also support.  Red Hat has also realized and built update and deployment models that work well within most businesses.  A new version of Red Hat may come out every year or so, but when you buy it, they will support it and provide security updates to the included software for at least the next five years.  This stability has served Red Hat well and allowed them to again attract third part vendors to develop for Linux and specifically for Red Hat Linux.

Red Hat’s pluses are pretty simple and clear.  They have a proven, tested and knowledgeable team of support people.  They have by far the most experienced enterprise class support and have generally been the first distro to be supported by non-Linux development teams porting their applications over.  It is rare that a developer or company will not support an application they say runs on Red Hat.

Red Hat does have a few negatives.  The biggest negative is its price and pricing structure.  Red Hat charges not just for the instance of Linux you installed, but there will be an additional fee for having a server with multiple cores. This can make the solution extremely expensive by comparison to Suse or Ubuntu.  Red Hat is also extremely slow to adopt new features of the OS and other software.  This may seem like a small thing until you want the latest and most stable version of an application.  When a new file system like the btrfs(butter fs), with its cool new features that give teams the ability to roll back a file or entire system in seconds, Red Had users will have to wait for the next major release to see if it is included.

Suse Linux
Suse was originally developed in Germany and various other countries in Europe to be the distribution for system administrators.  In the early days, the administrative tools created by this distro were the best of the bunch.  While still a leader in this area, several others have gained ground.  Like Red Hat, Novell has used its connections to focus on large enterprise companies like IBM to create some custom solutions.  For instance, they offer a special set of tools to help with Lotus Notes.  Novell is also a sponsor of the Mono Project.  Mono is a Linux compatible framework for Microsoft’s .net framework.  There is even Moonlight, which is the counterpart to Silverlight, Microsoft’s answer to Adobe’s Flash tool set. 

The stability of the distribution and its focus on providing enterprise solutions in line with those of its parent company Novell shine just as brightly.  The integration with Novell’s Management Platform called Zenworks makes administering large numbers of servers very simplified.  If you are just starting out with Linux, Novell offers a server creation tool called Suse Studio.  This tool is a web based product that lets you create and test a server.  Once you have all the packages and configuration set, you can then download a virtual machine or Anaconda file.  This lets you get up and running quickly, while at the same time helping new users bypass a part of the learning curve associated in switching to Linux.  The pricing on Suse Support is much more affordable than Red Hat.  The packages that include the Zenworks Management product are still priced less than a hundred dollars.  The pricing does change as you go above 16 CPU’s in a machine.

Suse has been picked on over the last few years because of its strong focus on attracting corporate or enterprise customers.  They have made deals with Microsoft, they strongly support Mono Development, and they have developed tools to make other third party closed source tools work better like Lotus Notes.  The other major concern with Novell at the moment is that they seem to be up for sale.  While this is still a rumor, the big money is betting that a buyer will most likely split the divisions up and sell off the parts.  This could mean that your support could slow or stop all together.  The chance of it stopping seems unlikely as IBM, HP, or others will offer consulting as a replacement for the support you would get from Novell.  The chance of being without updates or future enhancements is also small because of the work done on the OpenSuse project and the nature of Linux itself.  Unlike Windows, remember Linux distributions can be forked/split and everyone can see the source code when it is published and released to the public.  That prevents your investment in Suse from becoming useless.  The community will rally around the old distribution and create a new one.

We at linuxinstall.net attribute a large part of the maturation of Linux as a Desktop Operating system to Mark Shuttleworth and the team at Canonical, the shepherding company of Ubuntu.  While Novell and Red Hat have both provided extremely valuable contributions to Linux as a desktop operating system, Ubuntu has been the driving force behind the enhancements.  Ubuntu was the first desktop distribution that I ever installed on a laptop with a wireless card where everything just worked.  Where the server version of Ubuntu is concerned, the limitation they create during the install process can make it more time consuming to use in larger scale environments.  This is largely due to having to install several features, like a non-standard Mysql database, or getting things like ISCSI to work, which takes a little more effort than on Red Hat or Suse.  Ubuntu does seem to have heard the voice of the system admin’s in the crowd and are supposed to be bringing many new server focused tools and features to the next release due out in October 2010.

What makes Ubuntu unique is their attention to detail and approach to users.  They have a meet up after each release to plan and discuss what should be in the next release.  It is the little inputs from users that keeps Ubuntu ahead of the pack.  With the latest release, one of the focus items was on Linux start-up speed.  The goal was less than 10 seconds from powered off to up and connected to the Internet.  This was accomplished and we as users are the ones that benefit.  The details of how and why can be found here.  They also provide all of their releases for free.  The only time you have to pay Canonical is if you purchase their Ubuntu Advantage program.  The real advantage is the fit, finish and polish on the desktop environment.  They continuously focus on how to make users lives better.The Ubuntu Advantage programs are right in the middle and available from Canonical starting at $105 for desktop support and $380 for server support.  Their pricing goes up from there depending on support hours desired and whether you want three years of support.

The problem with all of the polish and finish on the desktop side is that they have not spent a similar amount of time working on the server focused tools.  If they stick to the recently discussed plans, they seem to be comfortable with where they are on the desktop.  This should allow them to start to focus on the server tools they need starting with this Falls October 10, 2010 release.  This is not to say that they have no tools for managing servers or workstations for that matter.  Like Novell and Red Hat, they offer their own management service called LaunchPad. This is a great tool for smaller businesses that would probably be comfortable with cloud storage of the configs and such.  However, not having it available as a purchasable software you can install on your hardware, it is probably not going to be wildly accepted for some security minded companies like Banks and Insurance companies.  

The Final Verdict

We are going to rate each of them in 4 areas:
1) Software included – What software do they include in the package and is it current and stable?

They all turned out to be equal which is what we believe everyone expected.

2) Do they offer support and how competitive is it?

Red Hat has a slight advantage here because of how long they have been at it.  We have no direct experience with Ubuntu’s support, but most of the reviews we found on the Internet seemed happy with the support.

3) Deployment and Management Tools – How we rated the tools they offer after using them.

Deployment and Management tools are getting better all the time.  SuSe is taking first place only because they offer their tool for self hosting.  Ubuntu only lost because their tool still needs some polishing and minor features that are missing.

4) Third Part Software Support – How many vendors support them and what level of support do they get from non-Open Source solution providers?

Red Hat is the leader in this arena.  Suse is a close second because companies like IBM and HP decided when they started to support Linux that they needed to support two distributions just in case once of them did not survive.  So almost everything that is stated to be supported on Red Hat is also supported on SuSe.


Distrobution Software included Support Available Deployment/Management Tools Third Party Software Support Average
Red Hat 5 4.5 4 5 4.625
SuSe 5 4 4.5 4 4.375
Ubuntu 5 4 3 2 3.5

With the right team administering your Linux machines, any of these distributions should be a great addition to your data center.  While most companies want a goal of settling on just one, the reality is that for both cost and best of bread solutions, a mix is almost every one’s destiny.  The experience of the contributors and friends of LinuxInstall.net is that forcing any third party company, even IBM or HP, to support a given distribution that is not their recommendation often ends in pain if not failure.  While the differences are subtle with certain applications, those differences can determine whether or not you successfully deploy.  What we are trying to say is to expect a mix of at least two of these distributions.