puppet

wags007

Managing systems like a Puppet Master instead of a Server Servant

Recently I began playing around with free and open source tools that could be replacements for some closed source tools I have used to build and manage systems.  I specifically wanted to be able to build servers up to a base level so that I could test out tools like Hudson, several different Source Code Management solutions and different application server platforms.  I also run some systems in my home network that I find myself rebuilding often so I can be on the latest and greatest versions of my favorite distro’s.  In the past, this has meant building a new machine, installing and configuring the software, then swapping out the old machine for the new one.  That last step is almost always followed by several hours, if not days, of tweaking and trying to remember the settings I learned about the last time that made thing 1 work better with thing 2.  I was able to improve efficiency at work with a traditional Vendor Model(Closed Source) software, but until recently had not found the open source alternatives to be that compelling.  Considering that this is my hobby as well as my job, it has always just seemed faster and easier to set them up more or less by hand.  Now that is all about to change and I am ready to be a Puppet Master.

Puppet as described via the Puppet website is:

“an open source data center automation and configuration management framework. Puppet provides system administrators with a simplified platform that allows for consistent, transparent, and flexible systems management.”

Why not use Launchpad, FAI or Spacewalk you may be wondering?  I was looking for simple tools that do just what I require.  This was also a much easier and faster setup.  The time it took me just to read the documentation when I was looking into these solutions was more than I have spent installing and configuring Puppet.  I am sure there are a few features I am missing, like the ability to install an OS instance, but I do not do that too often and was just looking to manage my systems with some tool for this round of changes.  I have started playing with Novell’s Open Source Baracus(http://baracus-project.org/Site/Baracus.html) for doing OS Installs and updates.  I will leave that discussion for an upcoming article.

Before you download and start playing with Puppet, I strongly suggest setting up a Source Code Management system like Git or Mercurial, assuming you don’t already have one running.  I didn’t know which one I wanted to use so I downloaded the SCM machine from the guys at

Turnkey Linux

.

 There is nothing worse than accidentally deleting a configuration line you spent hours searching to find to solve a major issue.  Except that is, saving the errant file into your Puppet repository, logging off and going home for the day expecting it to replicate while you sleep and solve a problem.  While you sleep the file is replicating and services are restarting putting you back to where you began or making things even worse for you the next morning.  If you have the file backed up and/or version controlled it will let you get back to where you thought you were when you left for home.  The last thing is a tool like

etckeeper

.  Etckeeper is a great little tool the folks at Puppet turned me on too.  I have made several attempts in the past to add the use of version control to my /etc/ directory.  If you don’t keep up with it or remember to commit your changes before an update or upgrade to software, you can often lose your most current configurations.  Etckeeper has hooks into apt, yum and pacman-g2, that allow it to use your favorite SCM tool, as long as they are not svn or cvs, to check in any changed files to /etc before the package that contains it are installed or removed.  I have only been using it for a couple of weeks and have already tested out it’s ability to correct my mistakes twice.

I use Ubuntu Servers at home and SuSE servers at work.  I have chosen Ubuntu at home so I can keep up my ability to flip between Debian based and RPM based systems.  One of the things I like most about Debian systems is the apt-get/dpkg package management system.  With only a few commands I had Puppet installed on the server and a client ready to receive files.  Once installed, another fifteen to thirty minutes and I had them talking and my first few files were under Puppet management.  I have now setup a few RPM based machines using YUM instead of just RPM.  It gave me a vary similar experience and the Puppet software just worked.  That shows the level of polish and unity the Puppet project has going on with things they consider complete.  

Having such good luck with the initial setup I decided to try setting up the Puppet Dashboard.  It was at this point that things came to a grinding halt.  The software installs well enough.  It’s figuring out how to configure it that seems to be impossible based on the documentation provided.  It’s all written in Ruby and while I could probably read the code and figure out what I am doing wrong, how many other people are going to do that?  In the corporate world, probably no one will attempt this while at their day job.  The almost complete lack of documentation on this part of the tool should have been a warning to me.  Instead I spent several hours trying to figure out what I had done wrong. (At the time of this writing I still haven’t.)  The good news is that this is a relatively new part of the package and the dashboard not running isn’t a showstopper.  Configuring several additional servers was about three steps per server.  No rebooting and only the services with related configurations needed to be restarted.  After a few more hours of work I have the most common files I update replicating across the network.  I then imported my DNS and DHCP related files and set them up to be managed with Puppet.  The great part of this addition is that I can make changes to these files and once updated, Puppet automatically restarts the appropriate services.  

Puppet is extremely flexible with how it is configured and where you can put the related files, so I decided that I would place them into a “special” directory.  I then wrote scripts to help me remember the SCM commands to check the files in periodically.  This now give me exactly what I have been wanting to do for years.  Etckeeper puts copies of my files into the SCM in a directory for each server.  Puppet then updates the files and with each update, etckeeper backs up the current file to the SCM.  

Here is a drawing of a basic network similar to what I have in my home network.

The Puppet software let’s you design systems with a base configuration and then group the servers that are similar in purpose and apply additional configurations.  So in the picture above, the Dark Blue lines represent the base configuration.  The Red Arrow between the Puppet Server and Mail Server represents the Mail server specific settings.  If you do have multiple servers that do a basic functions like DNS/DHCP above, you can even use templates to change the files or a set of the files it copies based on a server specific configurations.  Puppet can then change the server specific parts of the configuration you tell it to and the server it’s being applied to.  There are no real limits to what you can control and push out with Puppet, as long as it’s in a file and goes in a standard place.  This could easily be used to manage things like Web Sites, but probably isn’t the best solution for replicating your file servers data, even though it could.  

As you are setting Puppet up, one of the most interesting steps is that you are required to create certificates between the client servers and the Puppet Master Server.  If managed properly, this gives you a relatively secure and reliable way to know the server you are configuring is meant to be that type of server and with that type of configuration.  The documentation repeats in several places that while you can “mis-configure” the Puppet Master to accept any computers certificate on request, you should not.  If you ignore this advice, it would allow anyone to request a key with any servers configuration.  For example, I could stand up a copy of the company website and get the configuration then forge a whole new site.

Some other cool features is that the push to servers is staggered to keep them from all trying to pull the same file or files as soon as you update them.  For most new Admins, this seems like overkill and if you are on a Gigabit network then it probably is.  If however, you are on a mixed network with servers both local and remote, you may not have the bandwidth for a remote site to pull all of the files associated with changes to every server at the remote location at once.  Once configured, it will also track the changes progress.

If you are looking for an easy way to manage the configurations of 5 or 50,000 servers, this tool makes it simple.  The configuration of the tool is simple, the template language/format is great, and their future plans are only going to make it all better.  All in all, we here at

linuxintall.net

have to give this a “Go Install.”

wags007

Episode 40 – Puppet Master Interview

Running Time: 00:50:32
1) Introduction
Can you believe we have managed to keep this up for 40 episodes?

2) News
Virtual Box 4.0 is release cool early review
Fedora Project Hacked
Turnkey Linux takes it to 11 with Ubuntu 10.4 LTS

3) Topic of the night… Puppet RULES!!!
We wrote up a review and introduction of Puppet
Then we got to interview Luke Kanies the Founder and Puppet Master at Puppet Labs. (Check out another interview of Luke with Leo and Randall on FLOSS Weekly did back in November 2009)

4) Conclusion
Recommendations for People to interview
E-Mail us at podcast@linuxinstall.net
Go to the WebSite to call us via Google Voice
Facebook Fan Page
Follow us on Twitter and Identica as @linuxinstall
Look for us and comment on iTunes, odeo

http://player.wizzard.tv/player/o/j/x/129662536042/config/k-73b443e966a1409d/uuid/root/height/325/width/325/episode/k-dc905aadce503cc3.m4v

wags007

Automation – Can to much of a good thing be bad?

Senior  systems administrators on any platform know that automation is the  single fastest way to improve the effectiveness of their team.  Scripts  provide stability, repeatability and reduce the time spent on often  repeated tasks.  If done correctly, automation will make everything more  stable and manageable.  

However,  scripts for managing systems can be a double edged sword.  On one hand,  they make a team highly efficient.  They can help junior admins perform  far above their experience level and free senior admins up to  investigate more difficult problems.  On the other hand though, they can  lead to a loss of knowledge.  The knowledge it took to create the  scripts becomes locked inside of them.  So what do you do to strike the  proper balance?  How can you keep the knowledge fresh in every-one’s  mind while still automating?  What steps can be taken to avoid knowledge  erosion and worse the brain drain or vacuum that is left when people  leave?

The  first thing to remember is that there is no one thing that can be done  to answer these questions.  Here we will provide you with some tips and  ideas we have found to be useful and effective.  This is a short list  and we hope that it will inspire you to think about what might work for  you and your company. 

The  first item is well documented scripts and procedures.  Taking 5 minutes  to write up what you were thinking when you wrote the script can save  you days trying to figure it out later.  As more object oriented  scripting languages like Python, Ruby and Perl take hold, it becomes  easier to break down complex scripts into much easier and digestable  chunks.  These smaller chunks, like the core ideas behind Linux, should  do one thing and do it well.  The names of the functions should describe  what they do.  For instance, a function called createNewSSHKeys, should  probably create new SSH keys.  This combined with an explanation of  what you were trying to do inside the function will help you and others  manage them.  When you get really good at this way of thinking, people  should be able to take your function calls out and write a manual  procedure that could replace your automation.  If that is your goal,  then it only makes sense that starting with a well documented procedure  to compare against when your done scripting makes sense.  It is unlikely  that every procedure step will match a function or series of function  calls.  Getting everything close does count though.

As  much as self documenting scripts helps though,  documenting  configuration files for your scripts can keep things fresh in peoples  mind.  At the very least, if done correctly, it will give them a  breadcrumb trail to follow to see if what they think is being set is  set.  We recently began testing out Puppet, an automated way to manage  server configuration files and other admin related tasks.  The  configuration files for Puppet can be used as a great example.  They  allow you to use a combination of intelligent names and comments to  inform the person reading the file what will be changed.  They also  include a description of where to look to verify that the changes are  being done correctly.  This means that I don’t need to know Ruby, the  language Puppet is written in, to figure out how or what its going to  do.  The configuration file itself tells me everything I need to know.   When you write your own script, the time it takes to do this may not be  warranted.  So at the very least, make sure that you have comments that  tell people where to look for the output based on these configurations  or what the configurations mean in the file.

Try  to keep everyone with the sharp skills needed so they are ready to  slice through problems as they arise. This also means internal training.   One of the things we have participated in on a regular basis is a  short one hour refresher put on by the subject matter experts(SME) for  each of the technologies we use.  Doing this accomplishes a few  different things at once.  It helps the SME keep their documentation  current.  It gives the SME an opportunity to share changes they want to  make or have made in the environment.  Then it gives everyone supporting  the environment a chance to ask questions about the technology when  there is no pressure.  When possible, annual reviews of each area that a  team supports, goes a long way towards elevating the teams ability to  be as productive as possible.

While  you can never completely prevent brain drain when a team member leaves,  the steps above, if done correctly, can go a long way.  Having been the  person transitioned to more than once, the better these steps are  followed, the better we have felt about taking on the responsibility.   Another side effect of these approaches and others along the same  thought process is that it allows people to migrate from one SME area to  another.  This helps people stay fresh and keeps them from becoming  bored and complaisant.  The more driven your team is to solve businesses  problems, the more profitable you will be.