Skip to main content
Ben Nadel at Scotch On The Rocks (SOTR) 2011 (Edinburgh) with: Mark Mcaulay
Ben Nadel at Scotch On The Rocks (SOTR) 2011 (Edinburgh) with: Mark Mcaulay ( @mcaulay )

I Heart Logs: Event Data, Stream Processing, And Data Integration By Jay Kreps

Published in ,

Last week, after posting a review of Designing Data-Intensive Applications by Martin Kleppmann, fellow InVision engineer - Ben Darfler - recommended that I take a look at I Heart Logs by Jay Kreps. After making my way though Kleppmann's magnum opus, I Heart Logs was an easy and welcome read that I got through in just three sittings. While the book doesn't go into too much technical detail, it effectively paints the picture of an infrastructure that allows all facets of your organization to tap into the vast and diverse amounts of data that your organization collects. And, at the center of this scalable, transformable, real-time processing infrastructure is the re-playable, append-only log.


I Heart Logs by Jay Kreps, review by Ben Nadel.  

Historically, when I think of the term "logs", I think of the place that I record application errors. Or, perhaps the place where Apache / nginx records all of the incoming requests and HTTP response codes. And, while these are certainly logs, the focus of "I Heart Logs" is on the type of log known as a "commit log" or a "journal". These are append-only sequences of records order by time.

More importantly, these logs are the source of truth for the state of data across the entire organization. And, because this state is centralized, it can then become the source of truth for a horizontally scaled, ever-evolving, multi-faceted application architecture. Since this is a wildly new concept for me as web developer - who happens to love databases - one of the best descriptions that Kreps offers up is the view of a log-centric architecture though the lens of database indices:

Maybe if you squint a bit, you can see the whole of your organization's systems and data flows as a single very complicated distributed database. You can view all the individual query-oriented systems (Redis, SOLR, Hive tables, and so on) as just particular indexes on your data. You can view a stream processing system like Storm of Samza as just a very well-developed trigger-and-view materialization mechanism. Classical database people, I have noticed, like this view very much because it finally explains to them what on earth people are doing with all these different data systems - they are just different index types! (Kindle, Location 722)

This is fascinating! I have no practical experience with this stuff; I still have two feet very much planted in the world of monoliths. But, I have certainly considered using some sort of centralized messaging system as a means to do cache-busting across services. As it turns out, Kreps even describes this centralized log as a more comprehensive messaging system:

You can think of a log as acting as a kind of messaging system with durability guarantees and strong ordering semantics. In distributed systems, this model of communication sometimes goes by the (somewhat terrible) name of atomic broadcast. (Kindle, Location 295)

The book discusses how logs relate to producers and consumer, stream processing, intermediary logs / streams, and application integration; but, it doesn't go into much technical detail as to how this can all be accomplished. That said, Jay Kreps - as it turns out - is the original author of the tremendously successful open-source application, Kafka; and, he has used this approach (and Kafka) to create a logging system at LinkedIn that handles hundreds-of-billions of message each day. I think it's safe to say that it's a proven approach.

At InVision, we're going to start using Kafka as an event-bus for cross-system communication. After having read I Heart Logs, I am especially excited to dig into Kafka so that I can get a better sense of the future that Kreps describes. I know that an event-bus is not exactly what Kreps is discussing in the book; but, the more comfortable I get with a technology like Kafka - and the more hands-on experience that I get with distributed systems and microservices - the more I'll be able to wrap my head around the architecture he's recommending.

If this all sounds new to you - as it does to me - you should check out the book. I Heart Logs by Jay Kreps isn't a "how to" manual; but, it will help you see a different future. A future in which data is a real-time processing pipeline that any area of your organization can tap into and consume.

Reader Comments

I believe in love. I believe in compassion. I believe in human rights. I believe that we can afford to give more of these gifts to the world around us because it costs us nothing to be decent and kind and understanding. And, I want you to know that when you land on this site, you are accepted for who you are, no matter how you identify, what truths you live, or whatever kind of goofy shit makes you feel alive! Rock on with your bad self!
Ben Nadel