Sunday 28 October 2012

This is Automation Sparta - Wildcard DNS

This post is the 3rd in a set of posts outlining Automation Sparta - a No Configuration setup for Web Development.

In this post I'll talk about the power of wildcard DNS and how you can use it to reduce configuration time in your web deployment.

In web development there are typically at least 3 environments (servers) in your Project Universe. Dev, Staging (QA), Live (Production). To test a web application we obviously need a domain to hit for each environment.

Some agencies use:
something-dev.yourcompany.com
something-staging.yourcompany.com
something-live.yourcompany.com

Some use:
something.yourcompanydev.com
something.yourcompanystaging.com
something.yourcompanylive.com

Others use:
something.dev.yourcompany.com
something.staging.yourcompany.com
something.live.yourcompany.com

Note: "something" is typically be a project code or project name.

Actually, to be honest, I've never seen an agency that has it's own Live domain. They typically wait until the client tells them what they want the live domain to be. This means that they often get as far as Staging in  deployment. They never reach a deployment to the Live environment until very late in the development cycle. This is a horrible horrible horrible practice! Anyway, let's pretend that all agencies have the good practice of testing their new websites on the live server even before the client tells them the desired domain...

If you take a look at the 2nd and 3rd example, you can see that for each project (web application) the only part of the domain that will change will be the prefix. This is where wildcard DNS can be applied. 

All you need to do is register the following wildcards with your DNS provider:

*.dev.yourcompany.com
*.staging.yourcompany.com
*.live.yourcompany.com

Just make sure that each wildcard points to the respective server. Now any prefix can be used for each environment. For example. thisissomethingrandom.dev.yourcompany.com will automatically point to your development environment.

The benefit of this is obvious. It removes the need for configuration of DNS for each project. You can just deploy your site to the respective server. All you need to do is create an IIS site. And this can be automated! This also means that you can deploy to your Live server early in the development process without having to wait for the client to register a domain (which seems to normally happen late in the game). These are 2 crucial components to a "no configuration" setup.
As an added bonus, you'll also notice that many software licenses are valid for sub-domains. This means that you can use the same license file for each environment. Of course you may still need a new license for the actual live domain. But at least you can still test your site on the live server before the real domain is registered/available.


Anyway, if your DNS provider does not support wildcard DNS then you definitely consider changing providers. Wildcard DNS is a massive time saver. It is just another DNS entry. It shouldn't cost you anything extra. For more information about wildcard DNS click here.

This is Automation Sparta - Multiple Deployment Repositories

This post is the 2nd in a set of posts outlining Automation Sparta - a No Configuration setup for Web Development.

Many months ago I read about multiple deployment repositories using Git in this post:
http://blog.beanstalkapp.com/post/754555821/deployment-best-practices-3-using-environment

Since then I've taken it a step further with Mercurial. In this post I'm going talk generally about separate repositories for each environment in your Project Universe (Dev, Staging, Live etc), how they can help you automate deployment, and get yourself of some deployment trouble.

When we start a project we normally have a single repository. In Subversion you may develop in the trunk repository. Many people deploy from the trunk. However it is often considered better practice to have a deployment branch. Having a deployment branch means that you merge into this branch when you're ready to deploy. Deployments happen only from this branch. This saves you having to change configurations to deploy from different branches. When the code has been made Live, the trunk must be merged into so that you have a reference to the current state of the Live environment. Note that in Mercurial, we can "clone" - branching is slightly different. 

This workflow can sometimes be a problem and is prone to human error. For example, what if a developer forgets to merge to the trunk? Now you've lost your reference to what is Live. What does the next developer do when they need to fix a bug? Also, how do you deal with a situation where a feature is waiting to be pushed live, but a bug fix must go up first?


Taking deployment repositories a step further...

Having separate deployment repositories means that your Live repository is always going to reference what is Live. Moreover, you know exactly what is on Dev, and Staging etc.

Some people think that multiple repositories is overkill and a lot of extra work. But having separate deployment repositories also means that you can now automate deployment to your different environments. All you need to do is set up a build trigger on your Continuous Integration server. For example, when a change in the Dev repository is detected, build and deploy. Automating your deployment is extremely powerful.

If it still sounds a bit overwhelming, consider these user stories:
"As a developer I want to drop my new code in a deployment box, so that when it gets dropped in there, an automatic build is triggered, and deployment to the development server occurs". 

"As a developer I want to drop my new code in a deployment box, so that when it gets dropped in there, an automatic build is triggered, and deployment to the staging server occurs". 

So think of these deployment repositories NOT as repositories, but as boxes where your code goes to get deployed. The fact that these deployment boxes are actually full featured code repositories gives you much more power and more options. For example, Developer 1 has features sitting on Dev and Staging waiting to be deployed. Developer 2 has fixed some bugs but doesn't know about Developer 1's changes. When Developer 2 tries to push to the deployment box (repository) for deployment to the development server it will fail. This is a positive thing because Developer 2 now knows that there was actually other work being done, and that they should communicate with Developer 1. 

To deploy to the development server, Developer 2 will simply strip Developer 1's changes on the deployment repository before pushing the bug fix. Developer 2 will also encounter this on Staging, but not on Live because the features were not pushed Live yet. For Developer 1, they will encounter a similar issue when instructed to push the changes Live. Developer 1 now knows that they must merge the bug fix into the feature set before trying to deploy again. And of course going through a proper testing a deploy process to ensure stability of the merged bug fixes. Mercurial and Git make merging automatic (or at least very easy compared to SVN) as much more information is kept for each change set.

We've talked about deployment repositories, but where is it that you are doing your work? Ie. where is your working repository? It doesn't matter how you structure your working/feature/maintenance/main repositories. They can be anywhere you want. All you need to do is put your code into the deployment box for where you want to deploy to.

Hopefully you've been convinced that multiple deployment boxes (repositories) is much more powerful and actually makes your life easier when you couple it with automated build and deployment from a Continuous Integration server.

My controversial statement of the day: I'm sure you can do something similar using TFS and Subverion. But personally I think that if you're not using Mercurial or Git then you're living in the dark ages of version control. And if your company doesnt want to move to a distributed version control system you might consider changing companies, because they're probably holding you back as a developer.




Saturday 27 October 2012

This is Automation Sparta - Umbraco UK Festival 2012

Update: Due to unforeseen time constraints and it being that time of year, I'm going to have to consolidate these posts (mostly) to formal-ish posts on The Cogworks Blog. Those posts will be more business, less waffle any. I wanted to have some less formal personal blog posts regarding this topic, so I'll still be doing the odd one here and there. The Cogworks first post on Sparta starts here.


This post is the first in a set of posts outlining Automation Sparta - a No Configuration setup for Web Development.

"What I'm talking about is automation of the configuration of your automation"

I presented an automation setup yesterday at Umbraco UK Festival 2012, and that quote of mine got a lot of people's attention.

In this post I'm going to talk (very high level) about standardised naming, and how it lead to the realisation that setting up Web Development environments in general can be automated.

It all started earlier this year I joined The Cogworks. Compared to other agencies I have worked at, they had great processes for efficient development and deployment of web applications. They had a some really good templating and automation (eg. Team city build and deployment, and a solid Visual Studio template). This takes away human error, adds consistency yadda yadda. But I also noticed that there were a lot of manual tasks required to configure each process.

I came to realise that inconsistencies in naming between projects was slowing us down. Specifically repositories, databases, IIS sites, folder names, and DNS across the entire Project Universe (environments for Dev, Staging, Live etc).

I've seen this in every agency I've worked in. In any given project (depending on who the developer is), they choose one of these naming schemes: My Project, my project, MyProject, myProject, myproject, my_project, my-project. In one project the database may be called MyProject_Umbraco, in the next it could be Umbraco-myProject.

So the goal was to define and standardise naming for everything in the Project Universe.

When the naming of code repositories, databases, IIS sites, folder names, and DNS across each Project Universe is standardised, you start to realise that much more can be automated. In fact more than anyone in this industry has ever considered. You start to realise that you can automate the creation of your entire Project Universe.. in 1 click! Which is what I demonstrated.

But it doesn't stop there. Once you have automated the Project Universe, you can think about the development setup.

Normally before you even start to code you must:
  • Create a code repository (and check out)
  • Create a database (with login)
  • Create and configure a Visual Studio solution with connection strings, smtp etc.
This is the analogy I gave for the setup required before you even start to program:



Ok, now I can start to program....


Anyway...So now we've automated the creation of our code repository and the development database. We already know the connection string. So why cant we automate the insertion of the connection string into our web.config? Well we can. An MSBuild transform does that for us when we run a Team City build.

So that means we need to automate the insertion of the connection string into the MSBuild file. How do we do that? Easy! Use C# string replace. Template your MSBuild transforms, then use string replacement. Do this for all your environment transforms. You already know the database connection strings! I dont think I can overstate how powerful this one step is. Templating your build scripts opens up a world of possibilities with automation. Think about all the glorious project specific things you can automate now in a build script?

But why stop there? Since we have a Visual Studio template, why not automate project renaming? Ie project files, solution file, namespaces.

So configuring a Visual Studio solution from a template comes down to replacing strings in some files, and renaming a few files and folders.

The result is an application which creates your Project Universe, and spits out a Visual Studio solution with everything ready to go.

Now we're talking about a No Configuration Setup. Yes that's right. A No Configuration Setup for development and deployment of web applications. To be more precise... a setup which has automated configuration of development and deployment, based on standard consistent naming.

"What I'm talking about is automation of the configuration of your automation"

Our exact setup may not applicable to everyone. But from what I've seen in agencies, I think all agencies can adopt an approach that gives the same result. I.e. A No Configuration Setup.

I hope all of this information gives you food for thought about what you can and should automate.

Over the next few weeks I'm going to blog about exactly what is involved in each part of the automation. I'll even give code samples.

If you're interested, keep an eye on this blog and The Cogworks blog