an obession with first principles

RDBMS and Data

Posted: Wednesday Mar 23rd | Author: JohnO | Filed under: Programming | View Comments

I am coming to the realization that certain patterns and systems need a non-standard idea. Well, its not entirely non-standard, its becoming quite standard: the document store. The patterns that I am currently dealing with are interesting. Normally you would think they are perfect for RDBMS operations. And lots of the characteristics of the system really do. And those wouldn’t change. However, when it reaches a level of interdependence, where each piece of data can’t actually contain its own logic, because it has no meaning without other pieces of data, this is where RDBMS falls down for me. You have to fight against it to get good performance. All because there is one central piece of logic that needs all the pieces of data. And that means multiple JOINS, over many different indexes.

This central black box logic that you continually use to determine the state of your data will be run a lot. From a lot of different places. With a lot of different parameters. Because the business needs dictate it. This makes dealing with the limits of RDBMS problematic. It would make much greater sense to actually have a simple data structure of all this info. Sure that can get stored in a record of the RDBMS, but it is still flat there. The issue isn’t that the data is dense, sure there are lots of flags and details, but its relatively small. The issue is that there are lots of records. Iterating over them in memory is by far the better choice. I only wish I saw it sooner.

There is one more issue that moving it out of the relational system and into a document store solves: historicity. Both in terms of data and in terms of code. When you want to change the “present” object you are also stuck with changing all the past data that references it. This can have obviously negative side-effects. You can also operate on the presence of a key within the data store (e.g. new versions and releases will introduce new keys) rather than having to run data migration to update the new, now non-historic, RDBMS field accordingly.

I hope that in the future I get a project, and that I have enough foresight, to see this pattern coming and make the appropriate decisions in the beginning to more easily facilitate this kind of architecture.

blog comments powered by Disqus