CFRedlock - My ColdFusion Implementation Of The Redlock Distributed Locking Algorithm From Redis
When you're working on a single server, using the CFLock tag is an easy way to synchronize code across various parts of your application. However, once you have two or more servers, CFLock loses its guarantees. In order to lock across a distributed system, you need to use a distributed lock. Recently, I stumbled across Redlock, which is a distributed locking algorithm - proposed by Redis - which uses a collection of Redis key stores in order to provide powerful guarantees around locking and safety. Since I already use Redis for [less robust] distributed locking, I thought it would be fun to implement the Redlock algorithm in ColdFusion.
Project: See the CFRedlock on my GitHub account.
Unlike other "vanilla" distributed locking systems, Redlock is intended to provide stronger guarantees around safety and "liveness" (from the documentation):
- Safety property: Mutual exclusion. At any given moment, only one client can hold a lock.
- Liveness property A: Deadlock free. Eventually it is always possible to acquire a lock, even if the client that locked a resource crashed or gets partitioned.
- Liveness property B: Fault tolerance. As long as the majority of Redis nodes are up, clients are able to acquire and release locks.
CFRedlock is my ColdFusion implementation of the Redlock algorithm. And, while CFRedlock can work with 1-N Redis servers as its source of truth, the concept of "key server" is abstracted away. So, while I have a key server implementation that wraps around a Jedis connection pool (a Redis client for Java), just about anything can serve as the source of truth for locking. In fact, I even have a key server that just stores keys in a ColdFusion struct. This "isolated key server" implementation is intended to provide the Redlock API without the distributed infrastructure. Essentially, it's a way to migrate your code to a distributed locking workflow before the dev-ops team has the distributed servers ready. And, when they are ready, changing the key server implementation is a seamless transition.
To start using CFRedlock all you need to do is create the distributed locking Client. This can be a client for multiple Jedis connection pools, an isolated ColdFusion instance, hard-coded test servers, or even a custom implementation of your own:
<cfscript>
// Create the locking client. The keyServers must adhere to the key server API.
// But, other than that, they can be implemented any way you want.
var locking = new lib.client.DistributedLockClient(
keyServers,
retryDelayInMilliseconds,
maxRetryCount
);
</cfscript>
Because I intended to use this with Jedis, I have a client factory that automatically wraps Jedis Pools in the key server API:
<cfscript>
// Create a Jedis-powered client - requires an array of JedisPool instances.
var locking = new lib.CFRedlock().createJedisClient(
jedisPools,
retryDelayInMilliseconds,
maxRetryCount
);
// Create an ISOLATED ColdFusion client (uses CFLock internally).
var locking = new lib.CFRedlock().createIsolatedClient(
retryDelayInMilliseconds,
maxRetryCount
);
// Create a test client with static GET / DELETE behavior.
var locking = new lib.CFRedlock().createTestClient(
[
[ true, true ],
[ false, false ],
[ true, true ]
],
retryDelayInMilliseconds,
maxRetryCount
);
</cfscript>
Unlike the native CFLock tag, there is no "timeout" property that dictates how long the calling code will wait to acquire a lock. The distributed lock client has a configurable number of retries that it will perform (including a provided and slightly randomized delay); but, you can't customize this on a per-lock basis. I happen to think this is a much easier mental model. And, as one who's had to deal with a lot of locking, I think this makes it much easier to reason about.
If the lock cannot be obtained after the number of retries is exhausted, an error will be thrown. Unlike the native CFLock tag, there is no option to skip the lock without error because, unlike the CFLock tag, this doesn't represent native control flow functionality. However, if you like that general workflow (which is great for things like scheduled task execution), you could write a ColdFusion custom tag and then use it in your CFScript / CFComponent context in ColdFusion 11.
Once you have acquired a lock, you just need to remember to release it. In order to ensure that this always happens, the release code is usually placed inside of a try-finally block so that it gets released no matter what:
<cfscript>
var myLock = locking.getLock( "my-lock-name", expirationInMilliseconds );
try {
someSynchronizedAction();
// No matter what, release the lock when you are done with it.
} finally {
myLock.releaseLock();
}
</cfscript>
NOTE: If you forget to release a lock, the key will expire eventually - this is part of the "Liveness" property guarantee of the Redlock algorithm.
Now, keep in mind, this algorithm works with a single key server instance behind it. This means that you can start with just one server and then add more servers as your infrastructure becomes available. Heck, you can even start without Redis using a single ColdFusion instance. Once the ground-work is laid, the abstraction layer for distributed locking is pretty flexible.
Want to use code from this post? Check out the license.
Reader Comments
Nice work Ben. I'll be sure to check it out. Good to see we haven't completely lost you to that Angular witchcraft.
@Jim,
Ha ha, I still use ColdFusion every single day, I just write about it much less because I haven't necessarily done anything new with it lately. I spend most of my time in the JavaScript / AngularJS side. Right now, my ColdFusion stuff is primarily concerned with writing SQL queries :D So, it's nice to be able to step outside my box with things like this and with things like the JSON Web Tokens project I released a few weeks ago.
ColdFusion for the win!
Funny timing Ben, we just ported the redlock code to CF as part of a move to using Redis to store session data. Too bad we didn't know we were working on the same thing. :) Nice touch on offering a CF-based hack for infrastructure move.
@Brian,
Ha ha, small world sometimes, right?! To be fair, though, I mostly added the ColdFusion-based key-server implementation for testing. I honestly don't know much about testing. But, with the little that I do, every time that I might want to "mock something out," I think about how I could turn it into an implementation of of an abstract concept. Since my testing framework (TinyTest) doesn't have fancy mocking and stubbing features, it actually forces me to do this in order to test things.
But, the nice side-effect of it, in this case, is that it can definitely ease the transition :D
Hi Ben,
Nice implementation... One of the biggest drawbacks to cflock is its server specificity... and having an alternative (well having an alternative that I don't have to build myself) is great.
Cheers
@Gary,
Thank you, kind sir. It was a lot of fun to write :D