So what does it take to reliably build Web Applications? The answer really depends on two factors, the size of your company and more importantly the size of your development and testing teams. Over or under building your development environment can end up being every bit as expensive to your company’s bottom. If you under build it, you can impact testing teams or end up running too few tests and miss critical bugs. Over build the environment, and you will end up with a lot of servers wasting electricity, cooling, licensing and hardware resources as well as additional payroll costs to administer the servers.

Even if you are a boutique shop that has less than a dozen developers you should have at least two development environments. One for developers and one for testers. The developers will use the development environment to merge their code and unit test it as a complete unit with all of the other developers. The testing environment will be used to do functional and performance testing. So why do we define what the environments should be used for? The answer is simple, more often than not, most companies admit testers to start spending more time in the developers environment than they do in the testing environment. Why is that bad? This keeps the developers from checking code as they should because their space for doing that testing is now tied up with people doing “functional” testing. The development environment is also the place that your system administrators should be making the first attempts at updates of critical infrastructure. So having rules around the use of these environments and enforcing them is as critical as having them.

As your company’s Web Development team grows you will need to assess and adapt to the growth. As you start dedicating resources to performance testing you will need an environment for those resources to use. As you grow still further you will likely add a pre-production environment that mimics your production environment that we generally refer to as staging. A staging environment is really for the operations teams to test their final deployment procedures. A staging environment should not be sized the same as production. For instance, if you have four(4) Web Servers, six(6) application Servers, and three(3) LDAP servers in your production environment your staging environment might have just two(2) web servers, two(2) Application Servers and a pair of LDAP servers. The point of this environment is just to make sure that you will have a close proximity to your production environment that will let you test for single server failures, verify that your processes work across multiple servers, and give you an environment to test production problems in until the next deployment is prepared. This builds confidence and assures themselves and others that they have all of the steps needed in the complex environments that represent today’s application environments.

How large is too large a development environment? The best way to determine this is by looking at the usage of each of the environments. Simple monitoring can be done with log analyzing tools like webalizer. This type of tool will show you how much the systems are getting used but will consume small amounts of resources. I have recently used this type of data to convince several development teams that we could and should reduce our development server count by almost twenty(20) percent. Remember that not every development effort needs to be on its own environment or servers. Use things like Web Server Virtual hosts and Virtualization technology like VMWare and VirtualBox to condense the relatively low utilization servers. By doing this, a small team of developers can share a larger piece of hardware that will be more fully utilized. The easiest way to figure out if your environments are being underused is through monitoring. If you have any environments that are getting zero, or nearly zero usage at some point in the development cycle, you probably want to look at why. The situation of too many environments generally happens when development pulls back. The recent economic downturn and recession has placed many companies in this situation. In some instances I have seen companies, especially those using virtualization, end up with a one server per project team for either or both development and test. This leads to massive overbuilding. No web technology exists today that needs a one to one relationship. If you do find one, start looking for something else.

So what about Technology like IBM’s Virtual Enterprise, VMWare’s Lab Manager Product, or something similar? If you aren’t familiar with this class of application they are the ones around automatic provisioning of systems. These are sold with the promise of better utilization and control over your environment. For instance if you have a critical application that suddenly got hit with a Slashdot or Digg style event, something that draws an unusual amount of traffic to your site, the systems could build and bring up additional instances to help with the load. When the crisis passes, then the servers could be destroyed and the resources that had been allocated returned to the pool of available resources. This sounds like exactly what you need for your development environment right? Only as long as you have strong rules around how it can be used which will assure that systems at some point in time will get destroyed and the resources returned, then it is beneficial. Be careful with this idea though, and make sure you have someone who is ready to be the enforcer, or get ready to pay big bucks buying the resources needed to support an environment gone wild.

When building a development environment, web or otherwise, keeping it under control is the key. Uncontrolled growth with or without a good tool, will eventually crush the support teams. At the other extreme, too small and rigorous an environment will cost developers time and leave testers and admins impacted while they wait for changes to get to them. Automation of tools to build and remove environments sounds like a great time saving solution, but they have other hidden costs in the area of setup and maintenance that may well whip out any savings. So keep that all in mind. Let me know what your latest Web App is in the comments.

Questions of the week:

How many environments do you have at your company for how many developers?

What are your biggest challenges in your development environment?