I am leaving soon for Seedcamp week 2009 at London, where I will be mentoring as well as participating in a panel on Scaling and Performance (with Blaine Cook, Vijay Pandurangan, James Aylett, and Matt Biddulph). And this got me thinking.

I remember my first job interview. It was back at 1991, and I was interviewed by Ron Loewy for a developer job.

Ron asked me to tell him about things I’ve done before. Having done 3 commercial projects already as a freelancer self-taught developer, I described them. They all involved long term storage / query mechanisms of some sort, and Ron asked me to describe what I used for storage.

Now, the only data structure I knew at that point was Linked List. I also knew a bit of OO that I picked up from the Object Oriented Programming Guide booklet that shipped with Turbo Pascal 5.5. So I came up with FileLinkedList (TFileLinkedList in TP-speak), which descendant from my LinkedList class. And that is what I used for storage and query in these 3 projects.

Ron listened patiently while I described my incredibly versatile FileLinkedList, waited for me to finish, and then asked – so, ever heard of a database?

Sure, I said, I heard of the Paradox Engine, but it’s kinda expensive, isn’t it?

Somehow I got the job, and we went on to write software that connected to the most bizarre databases, from Oracle, DB2 and Sybase to DB2, ISAM and VSAM.

In the meantime, I also discovered SQL and RDBMSes, wrote a bunch of ORMs, and grew up believing its the best thing since sliced linked list. The only real question when starting a new project was which RDMBS to use – Oracle or MS SQL? InterBase or Ingres?

When we started creating Yedda late 2005 this was still an important question. But as Yedda matured and needed to scale up, the question changed. It became:

How do we architect the system so that we rarely have to use the database?

Following the AOL acquisition late 2007 this question became even more critical – growth was rapid and peaks were high and mostly unpredictable. But with the help of good friends like page output cache, distributed memory cache (memcached), queues [RabbitMQ], HTTP accelerators [varnish], CDNs and others, we were able to sustain this while also serving hundreds of millions of dynamic widgets per month using a single fully normalized RDBMS database instance – with plenty of room for growth.

We were lucky though – adding so many additional layers which were not designed into the original architecture could just as well have caused the system to collapse under its own weight and require a major rewrite. I’ve seen it happen before.

In a newer system we’ve designed at AOL post-Yedda, with considerably more demanding scale, volume and responsiveness requirements, it was clear from the start that we will never be able to use an RDBMS for the online queries. The RDBMS was quickly reduced to nothing more than a long term persistent storage, with the application relaying on mem cache for most of its read operations, with persistent storage in a key-value store and the writes always going through a queue mechanism, hoping for eventual consistency :)

Making sure that somewhere behind all this there is still a normalized RDBMS with all of our data required a non-trivial effort. But it does give me a warm and fuzzy feeling, knowing that it’s there.

Will there be a RDBMS somewhere behind the scenes in my next project? I’m not sure.

It depends on the project of course. Modern web applications accumulate data so fast that a RDBMS may simply be meaningless. Using it for long term persistency is a nice idea, but I already know that if I lose my key-value store there is no practical way to rebuild it from the RDBMS in any kind of reasonable timeframe. SQL and RDBMS are supported by so many query and reporting tools, but with that much data, a map-reduce technology like Disco quickly becomes the only practical reporting infrastructure at Yedda. And so on and on. 

So what’s the point.  I might as well reuse FileLinkedList :)

Update: Breaking news – I lost my voice! No panel for me. Oh well, I am sure it will be a fascinating discussion anyway :)