Should the Baracus Project be a member of your A-Team?

As stated in the Puppet article, we have been investigating alternative solutions to the closed source server build and management options on the market.  Novell has been sponsoring a project called Baracus.  Baracus is an open source project that is trying to become the next generation system for booting, building and managing power used by systems.  It seems like something we should check out so we did.  The project was announced to the public on November 19, 2010.  How good or bad could it really be?

For testing the system out, we choose to download the projects SuSe Studio created VM.  Being  a Novell project, it’s based on OpenSuse 11.2, with all of the setup steps in the documentation done for you.  A few normal admin tasks to change passwords, setup my account and the other normal system stuff got the system up.  To make everything work, there are a few other things you will need to do like setup a DHCP server.  While not required, a DNS server would make life easier also, so keep that in mind when starting out.  With almost any distribution you can set both of these up pretty easily.  I suggest installing webmin if you have never done it before.  It will help you get them up and running, with clean configuration files at least.  The documentation from the project is already at a point where the setup instructions are amazingly complete.  I was able to login to the Web GUI without any issue and start roaming around.  The system is very well thought out.  The look, feel and options included show that this system was built by system administrators for system administratos.   It’s not the prettiest looking interface, but it works and things are grouped logically.  They give you the option of using the WebGUI or command line.  A few of the commands really have to be run via the command line at this point but they seem to be working on improvements in a timely fashion.  There have been two updates since I downloaded this in November, both with noticeable improvements.  

How does it work?
The system does a three step boot process.  The first boot interrogates the system hardware and builds a hardware profile and uploads it to the Baracus Server.  Once uploaded, or registered, you will see the servers MAC Address in a list and be able to view what step the process is at.  The second boot will either boot and bring the server to a halt, allowing you time to choose the configuration options, or start building it if you already have that setup.  The third, fourth if you paused to configure, boot sets the server to boot off of either the local disk or network boot location depending on your choice of configuration.    When we say, “configuration options” here, we mean that you set up almost anything you want to do to a server.  This can be from upgrades/patches to turning it into a net booted system.

The first thing that impressed me with this system was that it’s not just a Suse build system.  As of the writing of this article you can build Debian, OpenSuse, SuSe Linux Enterprise Desktop and Server, Fedora, Red Hat Enterprise Linux, Ubuntu, OpenSolaris, ESX 4.x, Windows 7 and 2008 server, and Xenserver.  There are examples of silent configuration files available for most if not all of the systems listed.  Updating these files and adding them to the database used by Baracus is easy and took a few minutes.

The Virtual Machines only come with OpenSuse pre-installed.  So I set off to figure out how to add Ubuntu.  It turns out it was one command.  Here it is, “basource add –isos –distro ubuntu-10.10-x86_64”.  That’s it.  It goes out and downloads the ISO, puts it in the proper location, creates the needed mount points, and adds it to the database so that you can build servers from it.  If you want to do a silent install of the supported OSes all you have to do is make your modifications to the appropriately named file and issue another command to add that configuration to the database.  In just longer than the time it took the system to download the ISO, I was ready and testing my first build of Ubuntu 10.10 over a network connection with a silent install.  Having spent hours or nights in the past setting up systems to be able to build and boot off of the network, this was pretty impressive.  I have now setup Fedora and multiple OpenSuse versions.

On the network here we built an Ubuntu 10.10 VMWare system in roughly 15 minutes.  We set up custom disk partitions, setup our users, groups and additional software packages.  With a few more changes we had a script setup to update the repos and patch the system.  Then finally set up some scripts to automatically configure Puppet.  Now in less than 20 minutes we can take a raw VMWare Server and have it completely configured and up to date.  Having done all of this in the WebGUI, I tried doing it from the command line. It worked just as well and was actually a little faster.

So it’s so green what is wrong with it?
Really there are very few misses in the WebGUI, documentation and command line.  A few things we believe to be either documentation errata or bugs.  These did not show themselves however until I tried to bend it to what I wanted.  The problems with the WebGUI are mostly that we would like to see better errors to the user in a few odd spots and more AJAX like behaviors.  Having the assigned machine names instead of MAC addresses would be really helpful, as would some other views of the systems.  The groups functionality seemed too hard to use and isn’t offering enough right now.  Most of the documentation seems complete but more documentation on errors and what to do about them needs to be flushed out.  Where we had problems though, it didn’t take long to find and fix the problem.

Our Conclusion
Baracus is a great system that should be an amazing system with just a few cleanup and documentation fixes.  At this time I am not sure it’s really ready for people to use in production.  So we here are say try it but don’t rely on it just yet.

Did we not answer your question?  Please ask it in the comments.


How to setup and Manage a Web Development Environment Properly….

So what does it take to reliably build Web Applications? The answer really depends on two factors, the size of your company and more importantly the size of your development and testing teams. Over or under building your development environment can end up being every bit as expensive to your company’s bottom. If you under build it, you can impact testing teams or end up running too few tests and miss critical bugs. Over build the environment, and you will end up with a lot of servers wasting electricity, cooling, licensing and hardware resources as well as additional payroll costs to administer the servers.

Even if you are a boutique shop that has less than a dozen developers you should have at least two development environments. One for developers and one for testers. The developers will use the development environment to merge their code and unit test it as a complete unit with all of the other developers. The testing environment will be used to do functional and performance testing. So why do we define what the environments should be used for? The answer is simple, more often than not, most companies admit testers to start spending more time in the developers environment than they do in the testing environment. Why is that bad? This keeps the developers from checking code as they should because their space for doing that testing is now tied up with people doing “functional” testing. The development environment is also the place that your system administrators should be making the first attempts at updates of critical infrastructure. So having rules around the use of these environments and enforcing them is as critical as having them.

As your company’s Web Development team grows you will need to assess and adapt to the growth. As you start dedicating resources to performance testing you will need an environment for those resources to use. As you grow still further you will likely add a pre-production environment that mimics your production environment that we generally refer to as staging. A staging environment is really for the operations teams to test their final deployment procedures. A staging environment should not be sized the same as production. For instance, if you have four(4) Web Servers, six(6) application Servers, and three(3) LDAP servers in your production environment your staging environment might have just two(2) web servers, two(2) Application Servers and a pair of LDAP servers. The point of this environment is just to make sure that you will have a close proximity to your production environment that will let you test for single server failures, verify that your processes work across multiple servers, and give you an environment to test production problems in until the next deployment is prepared. This builds confidence and assures themselves and others that they have all of the steps needed in the complex environments that represent today’s application environments.

How large is too large a development environment? The best way to determine this is by looking at the usage of each of the environments. Simple monitoring can be done with log analyzing tools like webalizer. This type of tool will show you how much the systems are getting used but will consume small amounts of resources. I have recently used this type of data to convince several development teams that we could and should reduce our development server count by almost twenty(20) percent. Remember that not every development effort needs to be on its own environment or servers. Use things like Web Server Virtual hosts and Virtualization technology like VMWare and VirtualBox to condense the relatively low utilization servers. By doing this, a small team of developers can share a larger piece of hardware that will be more fully utilized. The easiest way to figure out if your environments are being underused is through monitoring. If you have any environments that are getting zero, or nearly zero usage at some point in the development cycle, you probably want to look at why. The situation of too many environments generally happens when development pulls back. The recent economic downturn and recession has placed many companies in this situation. In some instances I have seen companies, especially those using virtualization, end up with a one server per project team for either or both development and test. This leads to massive overbuilding. No web technology exists today that needs a one to one relationship. If you do find one, start looking for something else.

So what about Technology like IBM’s Virtual Enterprise, VMWare’s Lab Manager Product, or something similar? If you aren’t familiar with this class of application they are the ones around automatic provisioning of systems. These are sold with the promise of better utilization and control over your environment. For instance if you have a critical application that suddenly got hit with a Slashdot or Digg style event, something that draws an unusual amount of traffic to your site, the systems could build and bring up additional instances to help with the load. When the crisis passes, then the servers could be destroyed and the resources that had been allocated returned to the pool of available resources. This sounds like exactly what you need for your development environment right? Only as long as you have strong rules around how it can be used which will assure that systems at some point in time will get destroyed and the resources returned, then it is beneficial. Be careful with this idea though, and make sure you have someone who is ready to be the enforcer, or get ready to pay big bucks buying the resources needed to support an environment gone wild.

When building a development environment, web or otherwise, keeping it under control is the key. Uncontrolled growth with or without a good tool, will eventually crush the support teams. At the other extreme, too small and rigorous an environment will cost developers time and leave testers and admins impacted while they wait for changes to get to them. Automation of tools to build and remove environments sounds like a great time saving solution, but they have other hidden costs in the area of setup and maintenance that may well whip out any savings. So keep that all in mind. Let me know what your latest Web App is in the comments.

Questions of the week:

How many environments do you have at your company for how many developers?

What are your biggest challenges in your development environment?