Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
Ben Nadel at RIA Unleashed (Nov. 2010) with: Carol Loffelmann and Vicky Ryder and Simon Free
Ben Nadel at RIA Unleashed (Nov. 2010) with: Carol Loffelmann ( @Mommy_md ) , Vicky Ryder@fuzie ) , and Simon Free@simonfree )

Playing With Lists And Blocking Pop Operations In Redis And Lucee 5.2.9.40

By Ben Nadel on
Tags: ColdFusion, Redis

I have a really embarrassing confession to make: until just recently, I thought that the Redis List data-type worked like the ColdFusion List data-type. Which is to say, I assumed that it was just an abstraction over simple, delimited string values. This wasn't based on anything that I read - it was just a really, really poor assumption that my brain made. As anyone who uses Redis Lists would tell you, Lists in Redis are really more like (bi-directionally linked) Arrays, where each index is a completely isolated string value. Once I realized how very wrong I was about Redis Lists, I wanted to put together a little learning experiment, where I Push and Pop JSON (JavaScript Object Notation) values on to and off of a Redis List using a Blocking Read operation (BLPOP) in Lucee 5.2.9.40.

ASIDE: I'm using Lucee 5.2.9.40, not the latest Lucee, simply because I have a running instance for work that already has a Redis server. I didn't want to have set up a new Redis instance just for this experiment.

To experiment with the Redis List data-type, I've created a demo with two ColdFusion pages: a Write Message page and Read Message page:

  • The Write Message page is an HTML Form that allows me to submit a Message value to the Lucee server. This value is composed in a ColdFusion Struct, which is then serialized, as JSON, and pushed onto the tail (Right) of the List.

  • The Read Message page uses the the blocking read operation - BLPOP (Blocking Left Pop) - to sit and wait for list items to be available on the head (Left) of the list. As items become available, they are deserialized and rendered to the page. The Read Message page will continue blocking, reading, and dumping values for about 30-seconds before it gives up.

For this experiment, I'm using the JavaLoader and JavaLoaderFactory projects to load the Redis JAR file, along with its compile dependencies. This JavaLoader is initialized on ColdFusion Application start-up. Then, with each page request, I expose a .withRedis() method on the request scope that can be used to access an instance of the Jedis Connection object.

Here's my Application.cfc ColdFusion framework file:

component
	output = false
	hint = "I define the application settings and event handlers."
	{

	// Configure the application runtime.
	this.name = "RedisListWithJsonExploration";
	this.applicationTimeout = createTimeSpan( 0, 1, 0, 0 );
	this.sessionManagement = false;

	// Setup the mappings for our path evaluation.
	this.webrootDir = getDirectoryFromPath( getCurrentTemplatePath() );
	this.mappings = {
		"/": this.webrootDir,
		"/javaloader": "#this.webrootDir#vendor/JavaLoader/javaloader/",
		"/JavaLoaderFactory": "#this.webrootDir#vendor/JavaLoaderFactory/",
		"/jedis": "#this.webrootDir#vendor/jedis-2.9.3/"
	};

	// ---
	// EVENT METHODS.
	// ---

	/**
	* I get called once when the application is being initialized.
	*/
	public boolean function onApplicationStart() {

		var javaLoaderFactory = new JavaLoaderFactory.JavaLoaderFactory();

		application.javaLoaderForJedis = javaLoaderFactory.getJavaLoader([
			expandPath( "/jedis/commons-pool2-2.4.3.jar" ),
			expandPath( "/jedis/jedis-2.9.3.jar" ),
			expandPath( "/jedis/slf4j-api-1.7.22.jar" )
		]);

		var jedisConfig = application.javaLoaderForJedis
			.create( "redis.clients.jedis.JedisPoolConfig" )
			.init()
		;

		application.jedisPool = application.javaLoaderForJedis
			.create( "redis.clients.jedis.JedisPool" )
			.init( jedisConfig, "127.0.0.1" )
		;

		return( true );

	}


	/**
	* I get called once at the start of each request.
	*/
	public boolean function onRequestStart() {

		request.withRedis = this.withRedis;

		return( true );

	}

	// ---
	// PUBLIC METHODS.
	// ---

	/**
	* I invoke the given Callback with an instance of a Redis connection from the Jedis
	* connection pool. The value returned by the Callback is passed-back up to the
	* calling context. This removes the need to manage the connection in the calling
	* context.
	* 
	* @callback I am a Function that is invoked with an instance of a Redis connection.
	*/
	public any function withRedis( required function callback ) {

		try {

			var redis = application.jedisPool.getResource();

			return( callback( redis ) );

		} finally {

			redis?.close();

		}

	}

}

As you can see, the .withRedis() method accepts a Callback function; then, it gets a Jedis / Redis connection from the connection pool and invokes the Callback, passing-in the connection object. This allows the calling context to consume the Redis connection without having to worry about the complexities of interacting directly with the connection pool.

And now that we have a way to access Redis, let's get on with the experiment. First, let's look at the Write Message page. Again, this is just a simple ColdFusion page that uses a post-back model to read a Message in from the user. This message is then wrapped in a ColdFusion Struct, serialized, and pushed onto the tail of the Redis List:

<cfscript>

	// If the form has been submitted, push a Message onto the List.
	// --
	// NOTE: Each list item is a String that represents a serialized object.
	if ( form.keyExists( "message" ) && form.message.len() ) {

		listItem = {
			"id": createUUID().lcase(),
			"message": form.message,
			"createdAt": getTickCount()
		};

		request.withRedis(
			( redis ) => {

				redis.rpush( "list:messages", [ serializeJson( listItem ) ] );

			}
		);

	}

</cfscript>
<cfoutput>

	<!doctype html>
	<html lang="en">
	<head>
		<meta charset="utf-8" />

		<title>
			Post a Message
		</title>

		<link rel="stylesheet" type="text/css" href="./styles.css" />
	</head>
	<body>

		<h1>
			Post a Message
		</h1>

		<form method="post" action="#cgi.script_name#">

			<strong>Message:</strong><br />
			<input type="text" name="message" autofocus size="30" />
			<button type="submit">
				Post
			</button>

		</form>

	</body>
	</html>

</cfoutput>

As you can see, I'm using the RPUSH operation to push the serialized JSON payload onto the Redis List at key, list:messages. If this is the first time a list item is being pushed, Redis will automatically allocate a new List data-type at the given key. Once the list item is pushed, the ColdFusion page re-renders the form, allowing the user to continue pushing new messages.

On the Read side of the experiment, I created a ColdFusion page that sits and waits for messages using the blocking read operation, BLPOP. As messages become available, I'm dumping them to the output buffer and then telling the Lucee Server to flush (cfflush) the content to the browser. This allows the messages to show up in "realtime" (for about 30-seconds).

<cfoutput>

	<!doctype html>
	<html lang="en">
	<head>
		<meta charset="utf-8" />
		<title>
			Read a Message
		</title>
	</head>
	<body>

		<h1>
			Read a Message
		</h1>

		<cfscript>

			// For the purposes of the demo, we're going to block and listen for about
			// 30-seconds until the page has to be refreshed (for more messages).
			startedAt = getTickCount();
			cutoffAt = ( startedAt + ( 30 * 1000 ) );

			while ( getTickCount() < cutoffAt ) {

				// Since the BLPOP operation is going to BLOCK the page request, let's
				// flush data to the Browser incrementally so that we can consume and
				// display the messages in "realtime" as they are pushed onto the list.
				cfflush( interval = 1 );

				results = request.withRedis(
					( redis ) => {

						// NOTE: If the operation doesn't receive any message in 5
						// seconds, it will stop blocking and return NULL.
						return( redis.blpop( 5, [ "list:messages" ] ) );

					}
				);

				// The result of the BLPOP operation is 2-tuple where the first index is
				// the name of the key (since we can block on multiple keys at the same
				// time); and, the second index is the value of the list item.
				if ( ! isNull( results ) ) {

					dump( deserializeJson( results[ 2 ] ) );
					echo( "<script>scrollTo( 0, 999999 );</script>" );
					echo( "<br />" );

				}

			}

		</cfscript>

		<a href="#cgi.script_name#">Refresh listener</a>

	</body>
	</html>

</cfoutput>

The Read page is a bit more complicated than the Write page; but, the logic is still fairly simple. We're looping until the current time passed the cut-off time (about 30-seconds in the future). Then, for each loop iteration, we're blocking and waiting (up to 5-seconds) for a Redis List item to become available. If the blocking operation, BLPOP, times-out, it returns null. If it receives a list item within those 5-seconds, the result is a 2-tuple containing the Redis key and the List Item value.

So, essentially what we have is a page that pushes values onto a Redis list; and then, a page that reads items from that list and renders them in real time. Basically, we've created a super simple Message Queue.

Now, if we run these two ColdFusion pages side-by-side, we get the following output:

Messages being pushed and popped from a Redis List using Lucee 5.2.9.40.

As you can see, as I write / push messages on to the Redis List from one ColdFusion page, they are immediately read / popped off of the Redis List by the other ColdFusion page. As the values are read, each list item JSON payload is deserialized as a ColdFusion Struct and then dumped to the page response output buffer.

There's something kind of magical about Redis. Every time I interact with it, I get a warm feeling in my tum-tum. I believe that Redis holds a tremendous amount of power; and, that I'm only just scratching the surface in terms of the ways in which I can use it in my Lucee / ColdFusion applications. Of course, the first step in better consumption is actually understanding how the Redis data-types work. My mental model for Redis Lists was laughably wrong; but, now I'm moving in the direction.



Reader Comments

Ben. This is very interesting. Is this similar to an XHR request? But I can see the browser refresh button flickering?

And, secondly is Redis like MongoDB? A key value storage mechanism using JSON?

I really like your Blogs on Redis. I keep wanting to try it out, but then the time starts to fly again...

Reply to this Comment

@Charles,

In this post, you can think about my technique (for the demo) as being akin to "Long Polling". I've never actually used long-polling before; but, my understanding is that the browser would make a request (via something like AJAX) to the server and the server would hold the request open. Then, it would start flushing data to the client. On the client-side, the code would be buffering the flushed data; and, would be looking for certain delimiters (like a newline character), at which point it would take the buffered data and treat it like a cohesive value.

It's not an easy pattern to explain. It's kind of like Morris Code.

So, in my demo, the "Read a Message" pages are just making a single request. Only, the Lucee server isn't sending all the data right-away. It's using a combination of sleep() and blpop() to "hold the request open". And, as more data becomes available, the cfflush() command flushes that data from the server down the browser on the connection that's being held-open.

Basically, the "Read a Message" page is really just a very slow loading page, that takes like 30-seconds to load :D

This makes sense for a demo; but, in a "real" app, you'd probably just use an AJAX request or something like a WebSocket.

Regarding Redis, it's really cool and really fast. I don't think I use it to even a fraction of its potential - we just use it for session management / caching. But, it seems like it can do so much more, including Publish/Subscribe and interesting event-based stuff. But, as you say, the time just flies!

Reply to this Comment

@Charles,

Oh, but the "Post a Message" page is doing a full page-POST. Which is why you see the refresh button refreshing. The "Post a Message" is like:

Browser => FORM Post => Lucee CFML => Redis

Then, the "Read a Message" is just reading out of Redis. So, to be clear, the demo has two different pages in it.

Reply to this Comment

Cheers for the explanation.

I have just done the interactive tutorial on the Redis website, and I really like its simplicity.

If the application scope is refreshed, does the data, in the Redis cache, persist, or does it get wiped, as well?

I am just trying to figure out what the advantages are of using a Redis object over an Application scope Struct? I would like to start using it, but I need to know why I should use it?

Reply to this Comment

@Charles,

Great question - Redis is entirely external to the ColdFusion application. Think of Redis just like you would MySQL or MongoDB. So, if ColdFusion restarts, it just needs to reconnect to Redis and it will get all the same data. So, if you have multiple instances of ColdFusion running, for example, they can all connect to the same Redis and you can use Redis as the session store so that your user will have the same session regardless of which ColdFusion instance they hit (assuming they are behind some load-balancer that is distributing requests across the ColdFusion instances).

The biggest difference between Redis and other DBs i that, by default, it's just in-memory data; so, if Redis crashes or restarts, the data is wiped. However, I believe you can configure Redis to write to disk just like other DBs. But, that's going a bit beyond my rather shallow understanding of Redis.

Reply to this Comment

That's really helpful information.

I will tell my boss about this, because he is looking for an alternative session storage mechanism and this sounds like a good option. I presume you can store complex objects by converting to JSON, before storing in Redis and then deserialising after reading from Redis?

I guess I could hold the Redis DB on it's own server that sits behind the application server cluster. So, this will serve as a single source of truth and prevent sticky sessions?

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
NEW: Some basic markdown formatting is now supported: bold, italic, blockquotes, lists, fenced code-blocks. Read more about markdown syntax »
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.