So lately it is off to the races with new releases.  With Ubuntu, Fedora, Red Hat, just to name a few having recently put forth their latest offerings.  This will always be followed by the derivative distributions like Mint and Centos.  So what is so great about all this new stuff?  Well everything of course.  Don’t you want to be on the latest and greatest version of the Kernel?  Don’t you want access to new file systems like BRTFS?

The open source community is great at coming up with new and exciting tools all of the time.  There are no strict rules for the process of creating all of these new features, tools, or products.  Most projects though follow a process of having at least a “stable” and a “development” release of their code.  The developer, documenters and users work together to set the standards that need to be reached before for a new feature, tool or product can move from the development release to the stable release.  To get more real world testers for the new components, some projects migrate what could be called a late stage beta program into the Stable Release.   These additions will normally be marked with some type of experimental tag.  On production servers, you normally want to stay away from anything with this tag unless it is to correct an issue that you are currently dealing with.  At times though, you will need this one new feature or in some cases a driver to make something work.  Do you use the Experimental code?  For the most part our answer is yes as long as you do a lot of testing.  Here is a situation I found myself in, early on in my career:

“I was responsible for managing sendmail gateway servers going to the Internet for my company and we were sending out about 1GB in E-Mail a day.  Not bad, except that we would do mass e-mails to our shipping customers with status updates in the middle of the night.  The storage needed to handle these bursts was growing rapidly.  At the time this was happening, raid SCSI cards were costing us over $5,000 each.  We had 6 -1GB drives in the server, but needed them in a RAID array.  The corporate budget was tight and we could not convince management that a hardware RAID solution was a high enough priority to get into the budget.  At the time, the software raid solutions were just starting to be supported in the Linux kernel world.  The drivers were tagged “experimental”.  After hours of debate amongst the Solaris/Linux Admins, we decided to put this new software raid onto a test system and pump some messages through it.  Everything worked great on the test system.  We upgraded production with the new option and we we all crossed our fingers.  Everyone slept very little that night and we were surprised that it had just worked.  A few months later we had a hardware failure on one of the drives.  We marked the disk for replacement, swapped in a new drive, and rebuilt the array all with the system was processing messages without any downtime.  It was just like a hardware raid card.”

Since the experimental tag was there, we did a lot more research before we tried to deploy it.  We actually e-mailed the lead developer for the project, explained what we were thinking, and asked him if he thought it was ready to go.  He informed us that he had been running the latest version for three months and other than the day his cat danced on his keyboard, he had experienced no issues.  Try to do that with any hardware or software vendor out there today.  We received a personalized response from the developer who knows the code best, which  was impressive.  Nothing we would ever see from the likes of Microsoft or IBM.  The big companies keep their developers as far away from customers as possible, for several reasons.  Not the least of which being that they are all generally grumpy and occasionally frustrated with the way a piece of code is behaving.  (OK I am not saying that Admins like myself, are any better, just that customer service people are supposed to talk to the greater public.)  So we went for it and it worked.  That was pretty cool.  But I am not saying, “Let’s all go out and try some experimental software in a production environment.”  I am just saying that sometimes, just like Gmail and it’s beta tag, the experimental tags get left on a little longer than maybe they need too.  When you have a situation that calls for it, try it.  Remember that Linux and the other open source operating systems were and sometimes still are, experiments.  At the same time, both Windows and MacOSX have their unstable experimental parts also.

You should always use caution when using experimental packages, drivers and tools.  When ever you have a need though you should try the work out on a test server.  After your work is completed, remember to give feedback to the project and tell them why you did, or did not, choose to use their software.  Most people underestimate the power of even a small paragraph of positive encouragement or constructive criticism written to a developer or developers on a project.  Remember that your feedback is the reason that people create open source development in the first place.

Updates, experimental or otherwise, to any software can be stunning if you get what’s promised.  If you get hit by a bug though, it can make your life miserable.  If you are testing this type of software on a machine you are just playing around with, it may not be a big problem.  However, if you are hit with a bug on a critical server when it is in production at your job, it may lead to  an uncomfortable conversation with your boss.  If that happens often enough, you could even lose your job, so remember to practice the rules of being a safe admin.