Do you do centralized logging?

One of the things that can really save you or your Admin’s time is a centralized logging server.  I came across this article about how to set one up on either GenToo or Ubuntu.  It’s a great how to that would get you started.  If you are looking for something really powerful I would suggest looking at Splunk.  Splunk is a  great tool to help you group, filter and analyze your logs.  There are very few other solutions.  If you have a solution for managing your logs please post it in the comments.  If I can get more than just Splunk I will try to do a more in-depth review of the options everyone suggests.

What is the best Linux Distribution for the Enterprise?

What is the best Linux Version for the Enterprise?

While Linux is taking the world by storm, I often get asked what the best version of Linux is for enterprises of all sizes.  Red Hat, Suse and Ubuntu are always top notch distributions but then so are Fedora, CentOS, Mint, OpenSuse, and SlackWare.  Which distro is the best depends largely on what software you will be running on your Linux installation.  For most companies, standardizing on a distribution seems like it should be the final goal.  As with most things we discuss, the answer is maybe as well as it depends.  We are going to limit our scope in this CTO Brief to three of the top Business/Enterprise backed distributions.

A Distribution
Before we begin let us review what we mean by a distribution or distro.  At their core, a distro is a collection of software packages and a Linux Kernel.  At this point, almost every package available in one distribution is available in the others.  Each tends to use and support either custom tools or tools that the leadership of the distribution believes are the best for the end user.  The other noticeable difference between them is the UI or user interface.  This can be as simple as the major color choice for the distribution or as intricate as including the path to get to your favorite applications in the app menus, or what side the minimize, maximize and close buttons are on.  What really makes something an enterprise class package of Linux is the type and length of support you get from the distribution.  Red Hat, SuSE and Ubuntu all offer between 2 and 5 years of support for certain versions of their respective distributions.  This means that they will track, build, package and update all of the packages you get with the initial release.  This often means that the new shiny features are not present.  In turn this generally means that the system is better tested and more stable.  You always have the option to upgrade any software to the latest and greatest versions.  That does not mean however that when you call for support they will support you.  With these long term releases they are making certain guarantees and because of that, they need to control as many variables as possible.

For this article we are going to focus on the three top corporate backed and supported distributions which are Red Hat, Suse and Ubuntu.  These three are the broadly accepted as the leaders in this space.  This does not mean that, for instance, FreeBSD or CentOS aren’t enterprise grade.  I know several people that have been betting their businesses on them for the last few years with no regrets or major issues.  So we will have a follow up discussion on these distribution’s next time with FreeBSD, CentOS, and Fedora/OpenSuse.  We will try to point out how they differ.  But for now, let us break down the first three distros and shine a light on what we believe are each of their strengths and weaknesses. 

Desktop vs Servers

As you continue reading, we want to take a moment and point out that the focus is on Linux used on servers.  The bias here is author based, not reality based.  The focus in my career has been using Linux on servers.  In the last three to five years, amazing changes have taken place in the world of desktop Linux.  Some of the things that are happening will probably amaze even the most hardened Microsoft or Apple fan boy.  The server side of almost any of the Linux distributions we are talking about in this article or the follow up next time has been production ready for years.  Servers though do not have to have users typing on their keyboards or 16 USB Devices all hooked up through 2 ports on a PCI card.  So while a distribution may excel in one area like servers, another may excel on the desktop.  Red Hat and Suse are the ones in this article that do servers best.  Ubuntu on the other hand shows us all what can be done with Linux running as a desktop operating system. 

Red Hat
Red Hat was one of the first distro’s to do mass distribution right.  While not the first, they have proven over the years that you can make money with free software.  Red Hat’s leading position has ushered in several strategic partnerships with companies like Dell, IBM and HP.  They have, over the years, used these relationships to build up a level of support from third party vendors that is the envy of all of the other distributions.  When Linux is mentioned as a supported platform, most companies list Red Hat first, then any others they will also support.  Red Hat has also realized and built update and deployment models that work well within most businesses.  A new version of Red Hat may come out every year or so, but when you buy it, they will support it and provide security updates to the included software for at least the next five years.  This stability has served Red Hat well and allowed them to again attract third part vendors to develop for Linux and specifically for Red Hat Linux.

Red Hat’s pluses are pretty simple and clear.  They have a proven, tested and knowledgeable team of support people.  They have by far the most experienced enterprise class support and have generally been the first distro to be supported by non-Linux development teams porting their applications over.  It is rare that a developer or company will not support an application they say runs on Red Hat.

Red Hat does have a few negatives.  The biggest negative is its price and pricing structure.  Red Hat charges not just for the instance of Linux you installed, but there will be an additional fee for having a server with multiple cores. This can make the solution extremely expensive by comparison to Suse or Ubuntu.  Red Hat is also extremely slow to adopt new features of the OS and other software.  This may seem like a small thing until you want the latest and most stable version of an application.  When a new file system like the btrfs(butter fs), with its cool new features that give teams the ability to roll back a file or entire system in seconds, Red Had users will have to wait for the next major release to see if it is included.

Suse Linux
Suse was originally developed in Germany and various other countries in Europe to be the distribution for system administrators.  In the early days, the administrative tools created by this distro were the best of the bunch.  While still a leader in this area, several others have gained ground.  Like Red Hat, Novell has used its connections to focus on large enterprise companies like IBM to create some custom solutions.  For instance, they offer a special set of tools to help with Lotus Notes.  Novell is also a sponsor of the Mono Project.  Mono is a Linux compatible framework for Microsoft’s .net framework.  There is even Moonlight, which is the counterpart to Silverlight, Microsoft’s answer to Adobe’s Flash tool set. 

The stability of the distribution and its focus on providing enterprise solutions in line with those of its parent company Novell shine just as brightly.  The integration with Novell’s Management Platform called Zenworks makes administering large numbers of servers very simplified.  If you are just starting out with Linux, Novell offers a server creation tool called Suse Studio.  This tool is a web based product that lets you create and test a server.  Once you have all the packages and configuration set, you can then download a virtual machine or Anaconda file.  This lets you get up and running quickly, while at the same time helping new users bypass a part of the learning curve associated in switching to Linux.  The pricing on Suse Support is much more affordable than Red Hat.  The packages that include the Zenworks Management product are still priced less than a hundred dollars.  The pricing does change as you go above 16 CPU’s in a machine.

Suse has been picked on over the last few years because of its strong focus on attracting corporate or enterprise customers.  They have made deals with Microsoft, they strongly support Mono Development, and they have developed tools to make other third party closed source tools work better like Lotus Notes.  The other major concern with Novell at the moment is that they seem to be up for sale.  While this is still a rumor, the big money is betting that a buyer will most likely split the divisions up and sell off the parts.  This could mean that your support could slow or stop all together.  The chance of it stopping seems unlikely as IBM, HP, or others will offer consulting as a replacement for the support you would get from Novell.  The chance of being without updates or future enhancements is also small because of the work done on the OpenSuse project and the nature of Linux itself.  Unlike Windows, remember Linux distributions can be forked/split and everyone can see the source code when it is published and released to the public.  That prevents your investment in Suse from becoming useless.  The community will rally around the old distribution and create a new one.

We at attribute a large part of the maturation of Linux as a Desktop Operating system to Mark Shuttleworth and the team at Canonical, the shepherding company of Ubuntu.  While Novell and Red Hat have both provided extremely valuable contributions to Linux as a desktop operating system, Ubuntu has been the driving force behind the enhancements.  Ubuntu was the first desktop distribution that I ever installed on a laptop with a wireless card where everything just worked.  Where the server version of Ubuntu is concerned, the limitation they create during the install process can make it more time consuming to use in larger scale environments.  This is largely due to having to install several features, like a non-standard Mysql database, or getting things like ISCSI to work, which takes a little more effort than on Red Hat or Suse.  Ubuntu does seem to have heard the voice of the system admin’s in the crowd and are supposed to be bringing many new server focused tools and features to the next release due out in October 2010.

What makes Ubuntu unique is their attention to detail and approach to users.  They have a meet up after each release to plan and discuss what should be in the next release.  It is the little inputs from users that keeps Ubuntu ahead of the pack.  With the latest release, one of the focus items was on Linux start-up speed.  The goal was less than 10 seconds from powered off to up and connected to the Internet.  This was accomplished and we as users are the ones that benefit.  The details of how and why can be found here.  They also provide all of their releases for free.  The only time you have to pay Canonical is if you purchase their Ubuntu Advantage program.  The real advantage is the fit, finish and polish on the desktop environment.  They continuously focus on how to make users lives better.The Ubuntu Advantage programs are right in the middle and available from Canonical starting at $105 for desktop support and $380 for server support.  Their pricing goes up from there depending on support hours desired and whether you want three years of support.

The problem with all of the polish and finish on the desktop side is that they have not spent a similar amount of time working on the server focused tools.  If they stick to the recently discussed plans, they seem to be comfortable with where they are on the desktop.  This should allow them to start to focus on the server tools they need starting with this Falls October 10, 2010 release.  This is not to say that they have no tools for managing servers or workstations for that matter.  Like Novell and Red Hat, they offer their own management service called LaunchPad. This is a great tool for smaller businesses that would probably be comfortable with cloud storage of the configs and such.  However, not having it available as a purchasable software you can install on your hardware, it is probably not going to be wildly accepted for some security minded companies like Banks and Insurance companies.  

The Final Verdict

We are going to rate each of them in 4 areas:
1) Software included – What software do they include in the package and is it current and stable?

They all turned out to be equal which is what we believe everyone expected.

2) Do they offer support and how competitive is it?

Red Hat has a slight advantage here because of how long they have been at it.  We have no direct experience with Ubuntu’s support, but most of the reviews we found on the Internet seemed happy with the support.

3) Deployment and Management Tools – How we rated the tools they offer after using them.

Deployment and Management tools are getting better all the time.  SuSe is taking first place only because they offer their tool for self hosting.  Ubuntu only lost because their tool still needs some polishing and minor features that are missing.

4) Third Part Software Support – How many vendors support them and what level of support do they get from non-Open Source solution providers?

Red Hat is the leader in this arena.  Suse is a close second because companies like IBM and HP decided when they started to support Linux that they needed to support two distributions just in case once of them did not survive.  So almost everything that is stated to be supported on Red Hat is also supported on SuSe.


Distrobution Software included Support Available Deployment/Management Tools Third Party Software Support Average
Red Hat 5 4.5 4 5 4.625
SuSe 5 4 4.5 4 4.375
Ubuntu 5 4 3 2 3.5

With the right team administering your Linux machines, any of these distributions should be a great addition to your data center.  While most companies want a goal of settling on just one, the reality is that for both cost and best of bread solutions, a mix is almost every one’s destiny.  The experience of the contributors and friends of is that forcing any third party company, even IBM or HP, to support a given distribution that is not their recommendation often ends in pain if not failure.  While the differences are subtle with certain applications, those differences can determine whether or not you successfully deploy.  What we are trying to say is to expect a mix of at least two of these distributions.

Are you monitoring your servers and Network?

In the last CTO-Brief we discussed building and managing a large number of servers.  The general response we received on reddit, LinkedIn, Twitter, and in E-Mail was that the article was informative but overlooked monitoring.  Let me assure you that we did not leave monitoring out on accident.  We thought it was too large a topic for one article.  Everyone who criticized us was absolutely right about saying that once you build it, you then have to monitor it.  The reasons you need to monitor are pretty simple.  Following is our list of top five reasons for monitoring:

  1. Keeping Customers Happy – You cannot fix what you do not know is broken.  Unless you are monitoring, you will have to rely on customers to tell you when something is down.  When you do have an outage, being able to tell your customers that you are already aware and working on the problem builds their confidence in your abilities to administer the systems.
  2. Proving that you are an AWESOME administrator and/or Administration Team – I have had more than one Director of Operations tell me that we need to “tell the story” of how good we are.  Unless you can demonstrate with data and confidence that you are meeting the Service Level expectations of your customers, there really is no story to tell.
  3. Getting a restful nights sleep after a major release or update to your systems – If you are monitoring and trust those systems to do their jobs then sleeping is easy the night of a big deployment or upgrade.
  4. Performance Management – Knowing when to buy that next system or when to shutdown a server or two, is best shown with data than without.  Getting new machines approved is far easier when you can show managers a graph of how the use of a system is growing and needs to be scaled to the next level.  If your plans including a migration to a Virtual Infrastructure, monitoring lets you easily pick off the first candidates for virtualization.  The machines with the least used CPU’s and Memory can be the ones to set your site on.
  5. Troubleshooting Application Issues – Both performance and troubleshooting, benefit from being able to see what was going on when the problems occurred.  Looking at a set of pretty graphs can save hours of time looking for errors in logs and running down the wrong path to a speedy resolution.

So now we know why to monitor. Next we need to know what to monitor.  To do that, we need to know what our goals and priorities are for monitoring.  The goals for monitoring do not tell you much about which tools to use, but they do tell you how far you need to go.  For instance, if all you want to monitor is whether a server is up and functional, your monitoring needs are quite less then if you want to monitor down to an application level.

The number of open source options in the area of monitoring generally gather information in one of two ways.  The first is by use of the Simple Network Management Protocol or SNMP.  The second is with a software agent, which is usually proprietary to the monitoring software.  The more advanced systems can sometimes take a hybrid approach of both.  There are advantages to both approaches.  SNMP is a very low resource consuming system.  SNMP is supported by nearly every network device and operating system.  If not configured properly though it can be extremely insecure.  Where security is concerned, agents are not guaranteed to be any better.  What they do offer though is tighter integration between the client and hosts.  One drawback to an agent thought can be the additional system resources that they consume, but this depends on the agent in question.

In our next article we will delve deeper into one monitoring project called Nagios which is the base of several other pieces of monitoring software.  Nagios is a wonderful open source project that is amazingly feature complete.  One of the most useful features are System Templating, Hours of operations for alerting, Outage Windows, Escalation Paths and reporting.  The big complaint with it though is how painful it is to configure.  It is not overly complicated, but setup can be very tedious.  To address this, several different projects have created web based user interfaces that abstract the configuration into an easy to use system of templates and other tools to make life with Nagios as close to perfection as possible.  These tools generally incorporate other tools with painful configuration files like MRTG and Cacti for performance and usage graphing.  Both of these reporting packages are awesome projects we have used on numerous projects to show off all kinds of facts about system performance and usage. 

In the future we plan to review Zenoss, GroundWork, and Hyperion HQ.  We know this is not a complete list, but we think it is a pretty good start.  Is there one you think we are crazy to leave off?  If so please let us know in the Comments.