Managing systems like a Puppet Master instead of a Server Servant

Recently I began playing around with free and open source tools that could be replacements for some closed source tools I have used to build and manage systems.  I specifically wanted to be able to build servers up to a base level so that I could test out tools like Hudson, several different Source Code Management solutions and different application server platforms.  I also run some systems in my home network that I find myself rebuilding often so I can be on the latest and greatest versions of my favorite distro’s.  In the past, this has meant building a new machine, installing and configuring the software, then swapping out the old machine for the new one.  That last step is almost always followed by several hours, if not days, of tweaking and trying to remember the settings I learned about the last time that made thing 1 work better with thing 2.  I was able to improve efficiency at work with a traditional Vendor Model(Closed Source) software, but until recently had not found the open source alternatives to be that compelling.  Considering that this is my hobby as well as my job, it has always just seemed faster and easier to set them up more or less by hand.  Now that is all about to change and I am ready to be a Puppet Master.

Puppet as described via the Puppet website is:
“an open source data center automation and configuration management framework. Puppet provides system administrators with a simplified platform that allows for consistent, transparent, and flexible systems management.”

Why not use Launchpad, FAI or Spacewalk you may be wondering?  I was looking for simple tools that do just what I require.  This was also a much easier and faster setup.  The time it took me just to read the documentation when I was looking into these solutions was more than I have spent installing and configuring Puppet.  I am sure there are a few features I am missing, like the ability to install an OS instance, but I do not do that too often and was just looking to manage my systems with some tool for this round of changes.  I have started playing with Novell’s Open Source Baracus(http://baracus-project.org/Site/Baracus.html) for doing OS Installs and updates.  I will leave that discussion for an upcoming article.

Before you download and start playing with Puppet, I strongly suggest setting up a Source Code Management system like Git or Mercurial, assuming you don’t already have one running.  I didn’t know which one I wanted to use so I downloaded the SCM machine from the guys at Turnkey Linux.  There is nothing worse than accidentally deleting a configuration line you spent hours searching to find to solve a major issue.  Except that is, saving the errant file into your Puppet repository, logging off and going home for the day expecting it to replicate while you sleep and solve a problem.  While you sleep the file is replicating and services are restarting putting you back to where you began or making things even worse for you the next morning.  If you have the file backed up and/or version controlled it will let you get back to where you thought you were when you left for home.  The last thing is a tool like etckeeper.  Etckeeper is a great little tool the folks at Puppet turned me on too.  I have made several attempts in the past to add the use of version control to my /etc/ directory.  If you don’t keep up with it or remember to commit your changes before an update or upgrade to software, you can often lose your most current configurations.  Etckeeper has hooks into apt, yum and pacman-g2, that allow it to use your favorite SCM tool, as long as they are not svn or cvs, to check in any changed files to /etc before the package that contains it are installed or removed.  I have only been using it for a couple of weeks and have already tested out it’s ability to correct my mistakes twice.

I use Ubuntu Servers at home and SuSE servers at work.  I have chosen Ubuntu at home so I can keep up my ability to flip between Debian based and RPM based systems.  One of the things I like most about Debian systems is the apt-get/dpkg package management system.  With only a few commands I had Puppet installed on the server and a client ready to receive files.  Once installed, another fifteen to thirty minutes and I had them talking and my first few files were under Puppet management.  I have now setup a few RPM based machines using YUM instead of just RPM.  It gave me a vary similar experience and the Puppet software just worked.  That shows the level of polish and unity the Puppet project has going on with things they consider complete.  

Having such good luck with the initial setup I decided to try setting up the Puppet Dashboard.  It was at this point that things came to a grinding halt.  The software installs well enough.  It’s figuring out how to configure it that seems to be impossible based on the documentation provided.  It’s all written in Ruby and while I could probably read the code and figure out what I am doing wrong, how many other people are going to do that?  In the corporate world, probably no one will attempt this while at their day job.  The almost complete lack of documentation on this part of the tool should have been a warning to me.  Instead I spent several hours trying to figure out what I had done wrong. (At the time of this writing I still haven’t.)  The good news is that this is a relatively new part of the package and the dashboard not running isn’t a showstopper.  Configuring several additional servers was about three steps per server.  No rebooting and only the services with related configurations needed to be restarted.  After a few more hours of work I have the most common files I update replicating across the network.  I then imported my DNS and DHCP related files and set them up to be managed with Puppet.  The great part of this addition is that I can make changes to these files and once updated, Puppet automatically restarts the appropriate services.  

Puppet is extremely flexible with how it is configured and where you can put the related files, so I decided that I would place them into a “special” directory.  I then wrote scripts to help me remember the SCM commands to check the files in periodically.  This now give me exactly what I have been wanting to do for years.  Etckeeper puts copies of my files into the SCM in a directory for each server.  Puppet then updates the files and with each update, etckeeper backs up the current file to the SCM.  

Here is a drawing of a basic network similar to what I have in my home network.

The Puppet software let’s you design systems with a base configuration and then group the servers that are similar in purpose and apply additional configurations.  So in the picture above, the Dark Blue lines represent the base configuration.  The Red Arrow between the Puppet Server and Mail Server represents the Mail server specific settings.  If you do have multiple servers that do a basic functions like DNS/DHCP above, you can even use templates to change the files or a set of the files it copies based on a server specific configurations.  Puppet can then change the server specific parts of the configuration you tell it to and the server it’s being applied to.  There are no real limits to what you can control and push out with Puppet, as long as it’s in a file and goes in a standard place.  This could easily be used to manage things like Web Sites, but probably isn’t the best solution for replicating your file servers data, even though it could.  

As you are setting Puppet up, one of the most interesting steps is that you are required to create certificates between the client servers and the Puppet Master Server.  If managed properly, this gives you a relatively secure and reliable way to know the server you are configuring is meant to be that type of server and with that type of configuration.  The documentation repeats in several places that while you can “mis-configure” the Puppet Master to accept any computers certificate on request, you should not.  If you ignore this advice, it would allow anyone to request a key with any servers configuration.  For example, I could stand up a copy of the company website and get the configuration then forge a whole new site.

Some other cool features is that the push to servers is staggered to keep them from all trying to pull the same file or files as soon as you update them.  For most new Admins, this seems like overkill and if you are on a Gigabit network then it probably is.  If however, you are on a mixed network with servers both local and remote, you may not have the bandwidth for a remote site to pull all of the files associated with changes to every server at the remote location at once.  Once configured, it will also track the changes progress.

If you are looking for an easy way to manage the configurations of 5 or 50,000 servers, this tool makes it simple.  The configuration of the tool is simple, the template language/format is great, and their future plans are only going to make it all better.  All in all, we here at linuxintall.net have to give this a “Go Install.”

Automation - Can to much of a good thing be bad?

Senior systems administrators on any platform know that automation is the single fastest way to improve the effectiveness of their team.  Scripts provide stability, repeatability and reduce the time spent on often repeated tasks.  If done correctly, automation will make everything more stable and manageable.  

However, scripts for managing systems can be a double edged sword.  On one hand, they make a team highly efficient.  They can help junior admins perform far above their experience level and free senior admins up to investigate more difficult problems.  On the other hand though, they can lead to a loss of knowledge.  The knowledge it took to create the scripts becomes locked inside of them.  So what do you do to strike the proper balance?  How can you keep the knowledge fresh in every-one's mind while still automating?  What steps can be taken to avoid knowledge erosion and worse the brain drain or vacuum that is left when people leave?

The first thing to remember is that there is no one thing that can be done to answer these questions.  Here we will provide you with some tips and ideas we have found to be useful and effective.  This is a short list and we hope that it will inspire you to think about what might work for you and your company. 

The first item is well documented scripts and procedures.  Taking 5 minutes to write up what you were thinking when you wrote the script can save you days trying to figure it out later.  As more object oriented scripting languages like Python, Ruby and Perl take hold, it becomes easier to break down complex scripts into much easier and digestable chunks.  These smaller chunks, like the core ideas behind Linux, should do one thing and do it well.  The names of the functions should describe what they do.  For instance, a function called createNewSSHKeys, should probably create new SSH keys.  This combined with an explanation of what you were trying to do inside the function will help you and others manage them.  When you get really good at this way of thinking, people should be able to take your function calls out and write a manual procedure that could replace your automation.  If that is your goal, then it only makes sense that starting with a well documented procedure to compare against when your done scripting makes sense.  It is unlikely that every procedure step will match a function or series of function calls.  Getting everything close does count though.

As much as self documenting scripts helps though,  documenting configuration files for your scripts can keep things fresh in peoples mind.  At the very least, if done correctly, it will give them a breadcrumb trail to follow to see if what they think is being set is set.  We recently began testing out Puppet, an automated way to manage server configuration files and other admin related tasks.  The configuration files for Puppet can be used as a great example.  They allow you to use a combination of intelligent names and comments to inform the person reading the file what will be changed.  They also include a description of where to look to verify that the changes are being done correctly.  This means that I don’t need to know Ruby, the language Puppet is written in, to figure out how or what its going to do.  The configuration file itself tells me everything I need to know.  When you write your own script, the time it takes to do this may not be warranted.  So at the very least, make sure that you have comments that tell people where to look for the output based on these configurations or what the configurations mean in the file.

Try to keep everyone with the sharp skills needed so they are ready to slice through problems as they arise. This also means internal training.  One of the things we have participated in on a regular basis is a short one hour refresher put on by the subject matter experts(SME) for each of the technologies we use.  Doing this accomplishes a few different things at once.  It helps the SME keep their documentation current.  It gives the SME an opportunity to share changes they want to make or have made in the environment.  Then it gives everyone supporting the environment a chance to ask questions about the technology when there is no pressure.  When possible, annual reviews of each area that a team supports, goes a long way towards elevating the teams ability to be as productive as possible.

While you can never completely prevent brain drain when a team member leaves, the steps above, if done correctly, can go a long way.  Having been the person transitioned to more than once, the better these steps are followed, the better we have felt about taking on the responsibility.  Another side effect of these approaches and others along the same thought process is that it allows people to migrate from one SME area to another.  This helps people stay fresh and keeps them from becoming bored and complaisant.  The more driven your team is to solve businesses problems, the more profitable you will be.