an obession with first principles

Opposite Incentives In Client/Vendor Relationships

Posted: Monday Sep 17th | Author: JohnO | Filed under: Leadership, Management | View Comments

I’ve been an individual contributor software consultant/vendor for roughly ten years now. In every industry vendor/client relationships require careful management and can easily fall apart. However, I think there is something uniquely difficult when it comes to software. The software industry is barely getting started when compared to other industries like commercial manufacturing (Henry Ford), or civil engineering (the Romans). That something is a massive knowledge gap. And in my opinion it eventually creates opposite incentives.

The reality that software has zero tangible presence to humans in this world is what creates this massive knowledge gap. We’re barely are the beginning of creating truly re-usable and truly working building blocks of the software industry. In construction you take for granted that you just have nails, screws, 2x4s, 4x4s, as well as building codes designed by mechanical engineers that say “You will not do …” and “You will do …” otherwise we slap your project with a failing grade.

We don’t have nails, screws, and 2x4s. We might have something called `Screw`, and within it there is every single possible option a screw could have (length, thread width and spacing, head type, finish, which metal it is) and every single time you want to use `Screw` you need to make all those decisions. Some you care about, some you don’t, and if you’ve ever wandered the halls of Home Depot some things you’ve never even heard of before!

When it comes to the building code your client decides what you’re building and whether you get a passing grade. But they have no idea what is involved with their project. After all, they’re the client and they’re paying you — if they could do it themselves they would! Knowledge gap. On the other hand, if they were looking at a mass produced children’s plastic toy, with some common sense, they’d roughly know how it was put together, how many pieces there are, and have a feel for how “cheap” or “sturdy” it may be based on how heavy and pliable it is.

The client, due to this knowledge gap, is generally unable to give you the most important pieces of information you’re going to need to make something that just works. This is why our software elders always say: “Build the first iteration knowing you’re going to throw it away.” Because they know this knowledge gap is real, and its better starting over with what you’ve learned that trying to fight against the grain of what you’ve built.

First Order Incentives

Clients pay you to just do what they want. They have zero incentive to understand what their vendors are doing. They just want results. Developers have every incentive to believe their clients, that they are a good and truthful people who understand their business, their processes, and what they want at the end of the day.

Second Order Incentives

When developers finally get to the underlying truth of the matter, they have every incentive to explain what is happening to their client. However, the client has no incentive to believe the developer understands their issues — just do what I say. Otherwise, it would be an admission that they were wrong. I’m not paying you to tell me I’m wrong about my own things! This isn’t a car where you can take out the broken water pump and show them, or point to the transmission fluid leaking out through a hole in the line. The client has every incentive to believe in forward progress just like every other industry. After you’ve instructed the factory to create one widget, making 1,000 widgets is a lot faster and easier right? There is no incentive to go back to the drawing board.


It comes down to trust, just like every other relationship. But money is involved here. With a giant knowledge gap and money flowing in one direction that trust can quickly erode. All of us find ourselves keeping up these relationships. Niceties and codes of conducts are absolutely necessary, but those cannot fix the structural imbalances in incentives. It is easy to pick on details and specific events for when trust was eroded. But if you don’t realize the game you’re playing you can only keep juggling for so long.

In my opinion, there is one more structure we need to give to this relationship. One that alters incentives. And that is a baseline understanding of being partners with your clients. Not just a vendor.

Versioning and Releasing

Posted: Thursday Jan 6th | Author: JohnO | Filed under: Leadership, Management, Programming | View Comments

OK, I wrote about this a while ago. I still see lots of people wondering about just how to do this. The local Boston django group has about four people who submitted ideas around project management and deployment.

In the abstract you really need three things: 1) a distributed version control system (I prefer git), 2) a database migration system (in the rails world, rake, in the django world south, in the php world you can use manual sql files for both up/down migrations with a simple DB table keeping track – it is essentially what rake and south do, only without an ORM), and 3) a system that gets you onto all of your servers to run the necessary commands/scripts (again, in the rails world capistrano, in the django world fabric, in the php world make some shell scripts).

I have found it easiest to create feature branches in git for new features. Having your main production server running off of master. To write new features you just create a branch off master. If these features need qa, or user acceptance testing, you can create a point release branch to push to those servers only the features which you need at any time (or if it is not that busy, just push the feature branch).

(Aside: I always work in a local branch. So if I am working on master features. I create a master_local. If I am working on feature_a which is a remote branch, I create feature_a_local. This enables you to rebase upstream changes from master/feature_a without messing up the distributed history. It also prevents the useless commits from when pull causes a merge.)

I am against a release branch. I am fine with QA/user acceptance testing branches. This is the difference. If the QA branch gets messed up – fine, create another one. If the branch for the client’s preview gets messed up – create another one. (If you’re squashing your commits it makes it even easier to keep track of things). But once something is ready for release you just don’t want to touch it. Only things you are absolutely positively sure of get on master. Nothing else. That is an easy rule to enforce. Master is production. You mess with master you’re in trouble. With point release branches you get the endless wondering of “wait, which version is production, and which is QA, and which is the clients?”. It is fine that a QA or client branch get messed up because someone merged to the wrong point branch. But when you have master checked out and you type `git merge` you better be doing the right thing. It is a lot harder to confuse `master` with `` vs ``.

Some people worry about reverting. With git it is easy to revert. If you are pushing lots of things you should be squashing your releases down to one commit (that is, when you merge your feature branch into master, squash it first). So if your release goes poorly you can migrate off whatever you need (or don’t), and then git checkout the patch before the release.

Some people worry about atomicity. Git is built from a filesystem perspective so there is no lag time where any files are out of version with one another. In any case, once you’re making DB changes in addition to your code pushes – any atomicity (in milliseconds) is ruined since the DB takes seconds (in the case of large indexes, perhaps even minutes) to get up to date. If you’re worried about atomicity stop the webserver gracefully. If you’re worried about down-time – you shouldn’t be. If you gracefully take apache down, and seconds later bring it back up it is unlikely any user will ever notice. If you’re worried about it – release when your system is least in use. (In the case of rails and django you have to stop the webserver when releasing to ensure that your updated source files get re-compiled.)

And I will add one more tip for client caching. Users ought to cache this content aggressively. As a developer it is your job to deliver the new application code to them. You will want to add a version to the querystring to all the CSS and JS that you send out in your application. So when you make a change to a statically served file, you change the version on that querystring (in a settings file) so that no user has this cached.

All in all I really don’t see the huge headache in this area. I hope that we (Active Frequency) can put together a presentation on the specifics of our django deployment process (fabric, south, git/mercurial). (We also use virtualenv to sandbox our pypi and django version dependencies.)