an obession with first principles

The Democratic Id: Then and Now

Posted: Saturday Dec 22nd | Author: JohnO | Filed under: Politics | View Comments

The Lady and I have been watching the Vietnam documentary by Ken Burns, and I was absolutely struck by something about LBJ’s attitude and outlook on the war. It matched, precisely, the leadership of the democratic party during the 2016 election. Let me explain what I saw

In 1968, with nearly half a million American soldiers in Vietnam, the party line coming out LBJ’s democratic administration was: “We are winning.” And if you looked at just the numbers of casualties, after you un-doctored them, maybe you were. His leadership team could not imagine they were not winning. That things were not improving. That the war would have to be over soon. That the enemy would capitulate and come to the bargaining table for peace. LBJ and his military advisors would tell the American people that by now we must have depleted half of the NVA and NLF(VC) forces. And their will to continue fighting was nearly spent. And then the Tet Offensive came. The Tet Offensive was actually a much worse loss for the NVA and VC than any fighting previously. They came out the jungles and into the cities. They no longer had the advantage of the knowledge of terrain where the US troops were entirely outclassed. Their losses were staggering. The documentary made a prescient political point; when you’ve been telling the people you’ve been winning and the end is near, and then the enemy unleashes an 80,000 strong force across the entire territory of South Vietnam (quite literally, they were hitting Saigon, Danang, the eastern shore line, as well as south of Saigon). Your citizens now know you are entirely full of shit.

The democratic party Id cannot see evidence that suggests the world is not operating as they believe and want it to. They refused to believe that body counts were inflated; they were. They refused to believe the enemy had a much stronger will to fight. They refused to believe their win conditions were valid (I’ve watched over half of the documentary, and its still not entirely clear to me what their win conditions actually were–besides killing as many people as possible).

I found this attitude to be a perfect comparison to the democratic leadership involved in the Clinton campaign in 2016. There were two main populist candidates in the two year campaign; Sanders, and Trump. Both insisted that the status quo was not a sustainable option. No other GOP candidate had found a clear message (other than an anti-democrat, anti-Obama message), and thus all faded away. Meanwhile, in the face of incredible mounting evidence on so many fronts the Clinton campaigned, quite firmly, on the notion that America, today, is doing great, does not need to be made great, because its already working incredibly well.

No evidence of inequality, no evidence of health disparities, no evidence of racial disparities, no evidence whatsoever could dislodge their central claim. Reading Clinton’s day-after reaction, as well as the campaign’s confidence in the week before the election is further evidence that no facts can pierce their Id.

If I could distill what I think that Id is it would be: when we are in charge, whatever happens is linear progress. Nothing bad happens on our watch. If it happened we’re going to find a way to say it was planned, and it was good. Even if it was tragic it was a sacrifice for the greater good.

The Signals of “Fine”

Posted: Monday Oct 8th | Author: JohnO | Filed under: Politics | View Comments

A friend mentioned to me that, his biggest worry about the Trump administration, is that “Everything seems fine”. And the “everything” he meant is that all the traditional sign-posts; the stock market, unemployment, GDP are “fine”. And, in my opinion, these are precisely what the elite and upper-middle class believe to be the signposts. It is absolutely not the signposts of everyone else who is suffering despite these green signal lights.

The stock market is going to continue rise, and then crash consolidating wealth by destroying value for tens of millions, and raking in untold millions for thousands. But all the arbiters and prophets on Wall St will continue to tell you everything is fine. The unemployment numbers are entirely cooked books, and have been that way for over ten years. But no one in the upper-class or elite cares. They just want their green signal light. The GDP isn’t a false number—its just irrelevant for the person making minimum wage with two jobs creating that profit for the owning classes.

It should give both myself, and him, pause. That we have very little connection to those actually suffering. Because we can certainly convince ourselves that everything is “fine enough”. We have a terrible president, and yet the world turns and everything is fine. There are millions who know its not fine. Every day. And if we imagine it is fine, we are deserving of their wrath when (not if) they raise up.

Myth of Arrival

Posted: Wednesday Sep 26th | Author: JohnO | Filed under: Philosophising | View Comments

I fear that my writing will become more philosophical; the Lady and I have been watching _The Good Place_ in the evening. My post here is inspired by this reflection on Huxley’s _A Brave New World_ by Harry J. Stead.

There are real world problems with the notion of “having a teleological end”. Much has been written about abuser/victim complexes within Christianity. These are allowed to exist because of Christianity’s teleological end vis-a-vis “Heaven” or the Kingdom of God. The eternal reward the believer expects allows themselves to suffer through to any treatment in this world. But this becomes quite literally abuse when this punishment is meted out by the hand of their husband, pastor, etc.

Huxley’s classic novel uses this concept of the teleological end in his narrative, but instead of placing it into the far distant future, he places it within the achievable now—if only we have the political will to do so. In this case the ends are happiness and comfort. And their civilization is willing to give up freedom, love, and excitement, banishing them medically with a drug “soma” (a perfect word choice; it is the ancient Greek word for fleshly body).

While we don’t (yet) live in dystopia, we often make this some days of our lives. We chase happiness, placing it as a teleological end. Rather than realizing happiness is a momentary state which we pass through. We cannot, could not, stay there, forever. It is not altogether virtuous. And anger is not altogether a vice (when properly directed). Our lives cannot operate focusing on an end, only. The arrival of the teleological end is a myth. It never arrives. Any orientation that presupposes its arrival as a conclusion to justify the means is specious.

“True happiness is to enjoy the present, without anxious dependence upon the future, not to amuse ourselves with either hopes or fears but to rest satisfied with what we have, which is sufficient, for he that is so wants nothing. The greatest blessings of mankind are within us and within our reach. A wise man is content with his lot, whatever it may be, without wishing for what he has not.” — Seneca

Opposite Incentives In Client/Vendor Relationships

Posted: Monday Sep 17th | Author: JohnO | Filed under: Leadership, Management | View Comments

I’ve been an individual contributor software consultant/vendor for roughly ten years now. In every industry vendor/client relationships require careful management and can easily fall apart. However, I think there is something uniquely difficult when it comes to software. The software industry is barely getting started when compared to other industries like commercial manufacturing (Henry Ford), or civil engineering (the Romans). That something is a massive knowledge gap. And in my opinion it eventually creates opposite incentives.

The reality that software has zero tangible presence to humans in this world is what creates this massive knowledge gap. We’re barely are the beginning of creating truly re-usable and truly working building blocks of the software industry. In construction you take for granted that you just have nails, screws, 2x4s, 4x4s, as well as building codes designed by mechanical engineers that say “You will not do …” and “You will do …” otherwise we slap your project with a failing grade.

We don’t have nails, screws, and 2x4s. We might have something called `Screw`, and within it there is every single possible option a screw could have (length, thread width and spacing, head type, finish, which metal it is) and every single time you want to use `Screw` you need to make all those decisions. Some you care about, some you don’t, and if you’ve ever wandered the halls of Home Depot some things you’ve never even heard of before!

When it comes to the building code your client decides what you’re building and whether you get a passing grade. But they have no idea what is involved with their project. After all, they’re the client and they’re paying you — if they could do it themselves they would! Knowledge gap. On the other hand, if they were looking at a mass produced children’s plastic toy, with some common sense, they’d roughly know how it was put together, how many pieces there are, and have a feel for how “cheap” or “sturdy” it may be based on how heavy and pliable it is.

The client, due to this knowledge gap, is generally unable to give you the most important pieces of information you’re going to need to make something that just works. This is why our software elders always say: “Build the first iteration knowing you’re going to throw it away.” Because they know this knowledge gap is real, and its better starting over with what you’ve learned that trying to fight against the grain of what you’ve built.

First Order Incentives

Clients pay you to just do what they want. They have zero incentive to understand what their vendors are doing. They just want results. Developers have every incentive to believe their clients, that they are a good and truthful people who understand their business, their processes, and what they want at the end of the day.

Second Order Incentives

When developers finally get to the underlying truth of the matter, they have every incentive to explain what is happening to their client. However, the client has no incentive to believe the developer understands their issues — just do what I say. Otherwise, it would be an admission that they were wrong. I’m not paying you to tell me I’m wrong about my own things! This isn’t a car where you can take out the broken water pump and show them, or point to the transmission fluid leaking out through a hole in the line. The client has every incentive to believe in forward progress just like every other industry. After you’ve instructed the factory to create one widget, making 1,000 widgets is a lot faster and easier right? There is no incentive to go back to the drawing board.


It comes down to trust, just like every other relationship. But money is involved here. With a giant knowledge gap and money flowing in one direction that trust can quickly erode. All of us find ourselves keeping up these relationships. Niceties and codes of conducts are absolutely necessary, but those cannot fix the structural imbalances in incentives. It is easy to pick on details and specific events for when trust was eroded. But if you don’t realize the game you’re playing you can only keep juggling for so long.

In my opinion, there is one more structure we need to give to this relationship. One that alters incentives. And that is a baseline understanding of being partners with your clients. Not just a vendor.

Similarities between writing and programming

Posted: Thursday Sep 13th | Author: JohnO | Filed under: Philosophising, Programming | View Comments

I’m Toby. My fiancee and I have been re-watching the West Wing to imagine a world where capable humans sit in those offices, and we’ve decided that I am Toby Ziegler. I try to be two steps ahead. I can be a curmudgeon in a group of working people, but I’ve always been a teddy bear at heart.

“Listen, when you get home tonight you’re going to be confronted by the instinct to drink alone. Trust that instinct. Manage the pain. Don’t try to be a hero.”

Can I assume from your total silence and blank faces that you’re all with me?

There is literally no one in the world I don’t hate right now

These are all sentiments I share. And they are phrases which, were I eloquent enough, I might utter in real life. Clearly, I am not.

We were watching the episode where Will Bailey (aside: I wrote “Josh Molina” originally because I feel he is always just playing himself I feel) joins him in writing the second inauguration speech. Toby is failing to write well and burning his pages. It caught me off-guard because I know that feeling. But I don’t work on paper so there is nothing for me to burn for catharsis. (Aside: Hey programmers! Can we solve that?, catharsis is a strong and worthwhile emotion.)

That feeling is the mental load of trying to grapple with complex, large, and sometimes dissonant ideas in your mind all once to come up with something elegant, simple, and coherent. In that way writing and programming are very, very similar. Undoubtedly there are times where you are working incrementally in small steps to achieve a quick goal or change. But there are many times you’ve got a large structure you’re turning over and over in your brain. And its not only when you’re creating something new.

Of course the hardest part can just be getting started. Which reminds me of the old Joel Spolsky article “Fire and Motion” where one of the hardest things to do can simply be to “launch the damn editor”. Your brain is realizing that “Hey, I’m not ready for all of that yet, let me warm up first.” Getting through that period of warming up can be hard. Especially when you’re tired. No one does their best writing, or programming, when exhausted. They are both creative acts.

Boston Mid-term Primaries: Capuano

Posted: Wednesday Sep 5th | Author: JohnO | Filed under: In the News, Politics | View Comments

Boston saw a lot of political change in the democratic party last night. I won’t recap the results, either you already know it, or you’re going to very soon. I want to explain my perspective on Capuano.

I live in Capuano’s district. I have been happy to have him as my representative. I don’t have any bones to pick with him, his policy platform, or his voting record. But I did not vote for him, and I’m glad Pressley won.

I was very curious to understand why he was being challenged in a primary by a Boston city councilor. Like many other, I wondered what they was to gain by replacing a person with seniority; specifically a member of the House Transportation committee when we need all the federal dollars we can get here to fix the numerous transportation problems we do have, with a junior member who would more-or-less vote precisely the same way he has voted, and would likely vote.

In my mind there are two arguments against him, and the first is rather simple. The MA-7th (he used to be the 8th prior to re-districting but his constituents remained mostly the same) is a majority-minority district and it would only make sense that a minority ought to represent them in the House. Pressley will be the very first minority to represent the entire state in the House (there was one previous black Senator). Many have made this argument before but, to many others, this doesn’t pack enough punch.

There is a second argument, that I have, that I have not seen anyone else put forward. And this argument is loosely tied to the first. Based on his response to a question about Kaepernick Capuano had this to say: “I thought that particular action divided America, because he chose to do it during the national anthem,”..”I just thought it could have been done in a way that would have [brought] more people into the discussion, rather than actually anger an awful lot of America”.

As a 20 year Congressman from Boston, a leader in the democratic party (and if we’re not calling him a leader, why isn’t he after 20 years — that is a different failure all-together), if he cannot get the answer to this question right he needs to lose his seat to someone who does get it right. Both he and the GOP share the same framing of the Kaepernick-issue, as well as who is important in this issue: white television watchers.

That is, simply put, not permissible from someone calling themselves progressive. This is not permissible from someone representing a majority-minority district. He lost because he should of. And, he lost the minority neighborhoods hard, while barely coming out ahead in upper-middle class white neighborhoods. This is why he conceded so quickly.

The issue at the heart of these primaries is one of culture. Capuano is not demonstrating a culture in the democratic party that is going to win again the GOP. And this has been a failure of the party for nearly four decades, and why they’ve been losing statehouse after statehouse and governor’s seat after seat. This is what needs to change. And I’m glad its starting.

Leftist Globalization

Posted: Thursday Jul 5th | Author: JohnO | Filed under: Anthropology, Philosophising, Politics | View Comments

I couldn’t even read Fukuyama’s book “The End of History”. It’s on my shelf, and the first few pages are so dripping with ideology and holding his hands over his eyes I had to stop. I firmly agree with Derrida’s critiques. Fukuyama’s largest crux is “democratic peace theory”, which, I believe, is wrongly stated. The reason that “western-style democracies rarely go to war” has nothing to do with democratic representation of the polis. It has everything to do with economics.

I’m reading “The Great Transformation” by Polanyi right now. And it has a simple thesis: market commodification is responsible for the first world war. The industrial revolution upended the entire makeup of human society. Polanyi likened the psychological effects recorded in England to those recorded by slaves kidnapped from their homeland and dropped into an entirely different world.

The governments actually were primarily responsible for the welfare of the poor (and in this time, “poor” meant anyone who had to work for a living, e.g. not landed gentry). All of the nationalistic responses to the commodifications of labor and goods was an effort to restore the results of the welfare government. All fell short, and sometimes had disastrous effects.

The gold standard that was adopted in order to facilitate the further commoditization of labor and goods resulted in nationalistic responses of checks and balances, tariffs, etc. His thesis (and keep in mind, Polanyi is an avowed socialist) is that the abundance of nationalistic responses to market capitalism is what created the conditions for a world war.

I find his thesis and arguments very compelling. We can say three things as a result of this kind of argument:

  1. Market commodification does reduce violence between nation-states.
  2. Nationalistic reactions, even well-intentioned, to preserve both the welfare of the poor, and the welfare of capital at the same time set the stage for war.
  3. Market commodification replaces the violence of war with the violence of the market/capital

In my opinion this has ramifications for how leftists battle the current regime of neoliberal economics (which can easily viewed as capital’s battle after WW1 to re-introduce favorable economic terms and doctrine for capital), and the current battle against neoliberalism seen in the Trump administration.

It’s clear due to his often conflicting economic policies, that he really does not have a handle on how international economics works at all. In ways he continues neoliberal doctrine of hollowing out any and every state apparatus and banishing it from having any interplay with “the free market”. But his international obsession with tariffs and protective measures (I honestly have no clue who he thinks he is protecting) are precisely what Polanyi observed during the Hundred Year Peace. And the retaliatory tariffs and policies being put in place by Canada, China, the coming reaction from the EU, and I would presume Mexico once the new administration is seated, is what Polanyi argues led directly to WW1.

If the traditional attempts to fight globalizing market forces, like isolationism, lead to war, then how should we address the violence that market forces deals out to all of us each day?

The constant refrain from the anti-WTO protests of the late nineties and early two thousands was that capital has been globalized (e.g. moves without any restraints across any and all borders), but citizens have consistently been prevented from the same freedoms. Indeed, this is a direct contradiction (ala David Harvey) of what a labor commodity market ought to stand for. And of course this is being wielded violently against the citizens qua workers. What would a truly globalized citizen look like?

The other aspect of Polanyi’s critique is that the welfare reforms that intended to help the poor rarely did. One reason they often cannot is because they must always first satisfy capital, either through a gold standard which is enforced harshly, or other mechanisms. What would a truly globalized welfare system look like that refused to satisfy capital? Which, instead, forced capital to compete after all competition is supposedly enshrined as a virtue?!

Data Architecture in Complex Projects

Posted: Wednesday Feb 22nd | Author: JohnO | Filed under: Design, Programming | View Comments

This post is an attempt to be more a comprehensive thought that I expressed here in a previous post. It’s also an attempt to think in-depth about something besides todays annoying politics.

Every project starts off as an MVP. Which means it starts off small in scope, and as simple as it can possibly be. The path of least resistance for every project is a normalized relational database accessed via an OOP ORM. The problem is that this works very well. When projects are simple and small this is undeniably the fastest and most bug-free approach. Some projects grow in size, but remain simple. There are just more UI elements, more records to be tracked. But in a real way, the project remains a simple implementation of simple business rules where there are few considerations, few conditional code paths, and little interdependence between elements of the system.

There are many different kinds of complexity in business systems.

Historical data is a complexity where normalized relational data fails the hardest. In these systems there is a core set of data around which all of the business rules operate. When you have complex business rules and unexpected results you’re trying to explain, if you do not have clear record of the operations performed on the historical data you are going to have a real hard time explaining, understanding, and ultimately fixing the unexpected results. Normalized relational data always retains the current value(s) of the data set, while discarding previous values. And you have to fight hard to try and keep previous values.

The complexity of multi-user interfaces are where OOP ORMs fail the hardest. Once more than one single user is operating on a object/record/dataset you instantly create a problem where the last update wins, and it crushes previous changes. These issues can create serious problems and unexpected results in your dataset without any clear path of resolution. Your solutions are inherently limited, and then further limited based on the constraints of the rest of your architecture. Getting a full eventually-consistent backend is highly unlikely because of the time and effort you’ll need to complete it.

A smaller complexity that does exist is around reporting. Every business process needs reporting. It’s never the first thing you write. It’s always the last. The performance needs of your operational system always come first. Optimizing your relational data, indexes, etc for daily operations means its never optimized for reporting.

Principles That Prevent Problems

Keep that core data your business rules rely around in your language’s native types (for me in Python that means, list, dictionaries, tuples, strings, ints, and Decimals). Doing this means that you can store it in all the various places you’ll need to. If you have to put it in a relational database, you can store it as a JSON encoded string which you can then parse with standard libraries into your language’s native types. Or if your database supports native JSON storage, even easier. If you have a document store available it will go there without a problem as well. Try to avoid storing your base data set as a set of related records in a database. It means that all if your main business rules are then *forced* to be run through your ORM layer code. In a complex system this is very limiting as I outlined above.

When you have recorded all inputs and operations and have the original data set you can always replay those operations. So if a serious business rule changes you can easily run simulations to see what the outcome of that change will be. And you don’t have to write extra/repeated code or migrations to change you data. You’ve already written it, in one place, and tested it. This architecture pattern allows you to write more functional code. Which is easier to write, read, and test. This architecture pattern allows your to move easily towards more micro-services/distributed servers when your load demands it.

Creating a second system for reporting where data is ported (either in batches or real time) into it and then reported is a necessary step. It also creates a good pattern that your reporting data, if it is ever found to be wrong/have bugs, you can always regenerate it–because its existence is based on a record elsewhere.

The Common Liberal’s Problem

Posted: Friday Dec 16th | Author: JohnO | Filed under: In the News, Politics, Power | View Comments

Most people in the US have pretty casual ethics. That means its generally seen as moral to just obey the law. And if you’ve transgressed in a less-than-major way its fine so long as you didn’t get caught. The law has always been seen as generally good and in the best interest of everyone. There have been rare instances where its been seen as moral to be civilly disobedient — and the disobedient actions which have been taken have been incredibly mild and non-violent (e.g. marching on roads or bridges, sitting at counters, boycotting services).

A “common liberal” believes this to be the height of moral structure. When I say common liberal I don’t mean “people who vote democrat”. I mean those people, and “moderate republicans”, as well as anyone who claims to be independent or “fiscally conservative but socially liberal”. I mean the people that don’t follow politics, but view voting as the height of their civic duty, and generally believe “everything will just be ok”.

Here is their problem. Those who have come to operate the majority of the Republican party no longer believe it is or morally correct to obey the law or the governing norms set down through tradition over the decades. The “liberal” continues to believe that mere statements or penned documents are going to stop the GOP from continuing to do whatever they want. Both these people, and the DNC (which is the bigger problem), believe that by stepping behind a microphone and stating that their political opponents are acting immorally and need to stop they are acting meaningfully.

They are wrong. And if the longer it takes for them to realize this the more ground they will lose. Here are three immediate events to show them they are wrong:

  1. The governor of Maine has directed his administrations department of labor to refuse to enforce a minimum wage bill passed by the state legislature. This is a direct assault on the separation of powers and the rule of law.
  2. The state legislature of North Carolina has placed every single appointment by the incoming democratic governor’s administration subject to the republican led legislature, and removed the possibility of the UNC to have any party majority, thus deadlocking its ability to take any actions. This is once again a direct assault on the separation of powers, and the functioning of the state education system.
  3. The president-elect is in direct violation of the emoluments clause of the U.S. Constitution and is refusing to take any action or even hold a press conference about the problem. And the GOP leadership in the Senate has declared this constitutional violation not a problem. Once again a direct assault on the rule of law.

Meanwhile no one in any position of leadership in the DNC has done anything more than talk into a microphone as re-assert that laws exist and legislative norms exist. There have been no lawsuits filed to my knowledge. There have been no injunctions filed to my knowledge. There have been no calls for protest to my knowledge. And the common liberal still thinks that calling your representative is a solution. Representatives are not the solution and they are showing us why.

RDBMS and Data

Posted: Wednesday Mar 23rd | Author: JohnO | Filed under: Programming | View Comments

I am coming to the realization that certain patterns and systems need a non-standard idea. Well, its not entirely non-standard, its becoming quite standard: the document store. The patterns that I am currently dealing with are interesting. Normally you would think they are perfect for RDBMS operations. And lots of the characteristics of the system really do. And those wouldn’t change. However, when it reaches a level of interdependence, where each piece of data can’t actually contain its own logic, because it has no meaning without other pieces of data, this is where RDBMS falls down for me. You have to fight against it to get good performance. All because there is one central piece of logic that needs all the pieces of data. And that means multiple JOINS, over many different indexes.

This central black box logic that you continually use to determine the state of your data will be run a lot. From a lot of different places. With a lot of different parameters. Because the business needs dictate it. This makes dealing with the limits of RDBMS problematic. It would make much greater sense to actually have a simple data structure of all this info. Sure that can get stored in a record of the RDBMS, but it is still flat there. The issue isn’t that the data is dense, sure there are lots of flags and details, but its relatively small. The issue is that there are lots of records. Iterating over them in memory is by far the better choice. I only wish I saw it sooner.

There is one more issue that moving it out of the relational system and into a document store solves: historicity. Both in terms of data and in terms of code. When you want to change the “present” object you are also stuck with changing all the past data that references it. This can have obviously negative side-effects. You can also operate on the presence of a key within the data store (e.g. new versions and releases will introduce new keys) rather than having to run data migration to update the new, now non-historic, RDBMS field accordingly.

I hope that in the future I get a project, and that I have enough foresight, to see this pattern coming and make the appropriate decisions in the beginning to more easily facilitate this kind of architecture.