OK, I wrote about this a while ago. I still see lots of people wondering about just how to do this. The local Boston django group has about four people who submitted ideas around project management and deployment.
In the abstract you really need three things: 1) a distributed version control system (I prefer git), 2) a database migration system (in the rails world, rake, in the django world south, in the php world you can use manual sql files for both up/down migrations with a simple DB table keeping track – it is essentially what rake and south do, only without an ORM), and 3) a system that gets you onto all of your servers to run the necessary commands/scripts (again, in the rails world capistrano, in the django world fabric, in the php world make some shell scripts).
I have found it easiest to create feature branches in git for new features. Having your main production server running off of master. To write new features you just create a branch off master. If these features need qa, or user acceptance testing, you can create a point release branch to push to those servers only the features which you need at any time (or if it is not that busy, just push the feature branch).
(Aside: I always work in a local branch. So if I am working on master features. I create a master_local. If I am working on feature_a which is a remote branch, I create feature_a_local. This enables you to rebase upstream changes from master/feature_a without messing up the distributed history. It also prevents the useless commits from when pull causes a merge.)
I am against a release branch. I am fine with QA/user acceptance testing branches. This is the difference. If the QA branch gets messed up – fine, create another one. If the branch for the client’s preview gets messed up – create another one. (If you’re squashing your commits it makes it even easier to keep track of things). But once something is ready for release you just don’t want to touch it. Only things you are absolutely positively sure of get on master. Nothing else. That is an easy rule to enforce. Master is production. You mess with master you’re in trouble. With point release branches you get the endless wondering of “wait, which version is production, and which is QA, and which is the clients?”. It is fine that a QA or client branch get messed up because someone merged to the wrong point branch. But when you have master checked out and you type `git merge` you better be doing the right thing. It is a lot harder to confuse `master` with `18.104.22.168` vs `22.214.171.124`.
Some people worry about reverting. With git it is easy to revert. If you are pushing lots of things you should be squashing your releases down to one commit (that is, when you merge your feature branch into master, squash it first). So if your release goes poorly you can migrate off whatever you need (or don’t), and then git checkout the patch before the release.
Some people worry about atomicity. Git is built from a filesystem perspective so there is no lag time where any files are out of version with one another. In any case, once you’re making DB changes in addition to your code pushes – any atomicity (in milliseconds) is ruined since the DB takes seconds (in the case of large indexes, perhaps even minutes) to get up to date. If you’re worried about atomicity stop the webserver gracefully. If you’re worried about down-time – you shouldn’t be. If you gracefully take apache down, and seconds later bring it back up it is unlikely any user will ever notice. If you’re worried about it – release when your system is least in use. (In the case of rails and django you have to stop the webserver when releasing to ensure that your updated source files get re-compiled.)
And I will add one more tip for client caching. Users ought to cache this content aggressively. As a developer it is your job to deliver the new application code to them. You will want to add a version to the querystring to all the CSS and JS that you send out in your application. So when you make a change to a statically served file, you change the version on that querystring (in a settings file) so that no user has this cached.
All in all I really don’t see the huge headache in this area. I hope that we (Active Frequency) can put together a presentation on the specifics of our django deployment process (fabric, south, git/mercurial). (We also use virtualenv to sandbox our pypi and django version dependencies.)