Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
Ben Nadel at CFUNITED 2010 (Landsdown, VA) with: Sandeep Paliwal and Kiran Sakhare and Rupesh Kumar and Jayesh Viradiya
Ben Nadel at CFUNITED 2010 (Landsdown, VA) with: Sandeep Paliwal , Kiran Sakhare , Rupesh Kumar@rukumar ) , and Jayesh Viradiya

Building A Simple Redis Key Scanner Using Lucee CFML 5.2.8.50 And Jedis

By Ben Nadel on
Tags: ColdFusion, Redis

The other day at work, I noticed that our production Redis instance was holding steady, on balance, at about 7GB of memory usage and 14M keys. Anything "holding steady" in Redis makes me suspicious, especially over the course of several months, as I would expect the Redis resource consumption to ebb-and-flow with the natural rhythm of user engagement. As such, I wanted to start digging into the Redis data to see if anything looked wonky. To aid in this discovery process (and to have some fun), I created a super light-weight Redis Key Scanner using Lucee CFML 5.2.8.50 and Jedis, which I can run locally using CommandBox.

View this code in my Redis Key Scanner project on GitHub.

Because I don't know what I'm looking for, inspecting the Redis instance requires a brute-force approach. My suspicion is that many Redis keys don't have a specified TTL (Time To Live); which means, they will persist in Redis indefinitely. But, in order to figure out what those keys are (so that I can then figure out where in the application they are being generated), I need to traverse the Redis key space and start spot-checking value payloads.

To this end, my Redis Key Scanner uses the SCAN command to retrieve 100 keys at a time. This collection of keys is then listed on the screen along with pagination that allows me to move forward to the next cursor offset.

For each key listed, I can click-through to a detail page that includes the following data:

  • Key
  • Type - String, List, Hash, Set, ZSet, etc.
  • TTL - Time To Live
  • Value - The payload stored in Redis.

To render the Redis Key Scanner, I'm using an old-school frameset. Even though the frameset is a deprecated HTML element, it seemed like exactly the right tool for job. In fact, getting this whole project up and running in about two hours was just a reminder of how powerful ColdFusion is, especially when you want to get things up and running fast.

Interacting with the Redis Key Scanner looks like this:

Redis Key Scanner demo using Lucee CFML and Jedis in a local Redis instance.

As you can see, it's super simple. I'm just iterating over the Redis key-space and then inspecting individual keys. The bulk of the logic is handled by a single ColdFusion component, Scanner.cfc. This component gets configured to look at a Redis host; and then, provides two methods, .scan() and .inspect(). The .scan() method provides the data for the "list" page; and, the .inspect() method provides the data for the "detail" page:

component
	output = false
	hint = "I provide a simple API for iterating and viewing keys in a given Redis database."
	{

	/**
	* I initialize the Scanner with the given JavaLoader.
	* 
	* @javaLoaderForJedis I am the JavaLoader for the Jedis library.
	*/
	public any function init( required any javaLoaderForJedis ) {

		variables.loader = javaLoaderForJedis;

		// These properties will only be made available when the Scanner is configured to
		// inspect a given Redis host.
		variables.host = "";
		variables.pool = "";

	}

	// ---
	// PUBLIC METHODS.
	// ---

	/**
	* I configure the Scanner to inspect the given Redis host.
	* 
	* @newHost I am the Redis host to connect to.
	*/
	public void function configure( required string newHost ) {

		// If there is an existing connection pool, close it.
		if ( isConfigured() ) {

			variables.pool.close();
			variables.pool = "";
			variables.host = "";

		}

		var config = loader
			.create( "redis.clients.jedis.JedisPoolConfig" )
			.init()
		;
		var pool = loader
			.create( "redis.clients.jedis.JedisPool" )
			.init( config, newHost )
		;

		variables.pool = pool;
		variables.host = newHost;

	}


	/**
	* I get the data and meta-data stored at the given key.
	* 
	* @key I am the Redis key being inspected.
	*/
	public struct function inspect( required string key ) {

		assertIsConfigured();

		var results = {
			key: key,
			type: "none",
			ttl: "none",
			value: ""
		};

		results.type = withRedis(
			( redis ) => {

				return( redis.type( key ) );

			}
		);

		// If the key doesn't exist, there's no point in trying to access the rest of
		// key meta-data.
		if ( results.type == "none" ) {

			return( results );

		}

		results.ttl = formatTTL(
			withRedis(
				( redis ) => {

					return( redis.ttl( key ) );

				}
			)
		);

		results.value = getValueByType( key, results.type );

		return( results );

	}


	/**
	* I determine if the Scanner has been configured for a Redis host.
	*/
	public boolean function isConfigured() {

		return( ! isSimpleValue( pool ) );

	}


	/**
	* I scan over the Redis keys, using the given cursor and pattern.
	* 
	* NOTE: The pattern is applied to the keys AFTER they have been scanned. As such,
	* it's possible to use a pattern that returns zero results prior to the end of a
	* full iteration of the Redis database.
	* 
	* @scanCursor I am the cursor performing the iteration.
	* @scanPattern I am the post-scan filter to apply to the result-set.
	* @scanCount I am the number of keys to scan in one operation.
	*/
	public struct function scan(
		required numeric scanCursor,
		required string scanPattern,
		numeric scanCount = 100
		) {

		assertIsConfigured();

		var scanParams = loader
			.create( "redis.clients.jedis.ScanParams" )
			.init()
			.match( scanPattern )
			.count( scanCount )
		;

		var results = withRedis(
			( redis ) => {

				return( redis.scan( scanCursor, scanParams ) );

			}
		);

		return({
			cursor: results.getCursor(),
			keys: results.getResult()
		});

	}

	// ---
	// PRIVATE METHODS.
	// ---

	/**
	* I assert that the Scanner is configured; and, throw an error if not.
	*/
	private void function assertIsConfigured() {

		if ( ! isConfigured() ) {

			throwNotConfiguredError();

		}

	}


	/**
	* I format the given TTL value to make it more human-readable.
	* 
	* @ttl I am the TTL in seconds being formatted.
	*/
	private string function formatTTL( required numeric ttl ) {

		if ( ttl < 0 ) {

			return( "none" );

		}

		if ( ttl < 60 ) {

			return( ttl & " seconds" );

		}

		var ttlInMinutes = ( ttl / 60 );

		if ( ttlInMinutes < 60 ) {

			return( numberFormat( ttlInMinutes, "0.0" ) & " minutes" );

		}

		var ttlInHours = ( ttlInMinutes / 60 );

		if ( ttlInHours < 24 ) {

			return( numberFormat( ttlInMinutes, "0.0" ) & " hours" );

		}

		var ttlInDays = ( ttlInHours / 24 );

		if ( ttlInDays < 28 ) {

			return( numberFormat( ttlInDays, "0.0" ) & " days" );

		}

		var ttlInWeeks = ( ttlInDays / 7 );

		return( numberFormat( ttlInWeeks, "0.0" ) & " weeks" );

	}


	/**
	* I get the Redis key value for a key of the given type.
	* 
	* @key I am the key being read.
	* @type I am the data-type stored at the given key.
	*/
	private any function getValueByType(
		required string key,
		required string type
		) {

		var value = withRedis(
			( redis ) => {

				switch ( type ) {
					case "hash":
						return( redis.hgetAll( key ) );
					break;
					case "list":
						return( redis.lrange( key, 0, -1 ) );
					break;
					case "set":
						return( redis.smembers( key ) );
					break;
					case "string":
						return( redis.get( key ) );
					break;
					case "zset":
						return( redis.zrange( key, 0, -1 ) );
					break;
					default:
						return( "Redis type [#type#] not supported by Scanner." );
					break;
				}

			}
		);

		return( isNull( value ) ? "" : value );

	}


	/**
	* I throw a Not Configured error.
	*/
	private void function throwNotConfiguredError() {

		throw(
			type = "RedisScannerNotConfigured",
			message = "Redis Scanner not yet configured.",
			detail = "Before the Redis Scanner can be used, it must be configured using the .configure() method."
		);

	}


	/**
	* I invoke the given Callback with an instance of a Redis connection from the Jedis
	* connection pool. The value returned by the Callback is passed-back up to the
	* calling context. This removes the need to manage the connection in the calling
	* context.
	* 
	* @callback I am a Function that is invoked with an instance of a Redis connection.
	*/
	private any function withRedis( required function callback ) {

		try {

			var redis = pool.getResource();

			return( callback( redis ) );

		} finally {

			redis?.close();

		}

	}

}

The rest of the code in this ColdFusion application consists of a few no-frills CFML pages that work using simple request-response workflows. No AJAX; barely any JavaScript; no compile step; just raw CFML horsepower! It's not even worth sharing the code here - you can view it in the GitHub repo if you want.

I am sure there are plenty of professional GUI (Graphical User Interface) applications for Redis. But, what's the fun in using one of those when I can spend an hour or two building one for myself. Plus, when I build it, I can tailor the UI (User Interface) for my own needs. If nothing else, just another opportunity to practice coding; and, a nice vacation from all the thick-client, single-page application development that I'm usually doing.



Reader Comments

Ben. This is interesting stuff!

The part I get a bit lost at:

redis.scan( scanCursor, scanParams )

Is this a call to the java library or a call to the CFC:

scan()

Method

I had a look at:

http://xetorthio.github.io/jedis/

But, I couldn't see:

redis.scan()

And I presume the cursor, acts like pagination for Redis. Are you collecting 100 keys at a time?

Reply to this Comment

@Charles,

Yes, that is all correct. I have a ColdFusion Component Scanner.cfc, which has a .scan() method. But, turns around and calls the .scan() method on the Java Driver for Redis (Jedis). And then, the actual scan is powered by a Cursor which takes the current Cursor value and the number of keys (100 in this case).

The Cursors in Redis are interesting. They maintain no state; so, you can have as many Cursors as you want to at any one time. And, you can change the key-count at any time during the iteration to get a larger collection of keys. The benefit of the Cursor is that you can safely iterate over the database without locking up access. Trying to do something like KEYS * would be a huge mistake as it would (among other issues) lock access to all other requests while the keys were being gathered. SCAN on the other hand, can safely and efficiently look at a small slice at a time without a negative impact.

Reply to this Comment

Wow. This cursor feature is amazing.

I'm really excited about trying some of this Redis stuff out. I am going to create a POC session storage application, to try and persuade my boss that this would be a good alternative to Coldfusion's native session scope.

Reply to this Comment

@All,

I updated the Redis Key Scanner project to include some RegEx filtering for both include and exclude functionality:

https://www.bennadel.com/blog/3708-using-regex-to-filter-keys-with-redis-key-scanner-in-lucee-cfml-5-2-8-50-and-jedis.htm

By default, the SCAN operation only has a MATCH which uses glob-style includes. In my update, I am pulling the key-set back into memory (just the keys in the given cursor iteration); then, I am using .filter() Lucee CFML methods to perform the include / exclude on the ColdFusion side.

It will pull back more keys than using MATCH; but, with a much-improved user experience (UX).

Reply to this Comment

@Charles,

Yeah, Redis is just a lot of fun to play with for some reason. At least, for me. I'm actually working on a fun little algorithm to scan over the Redis database and set a TTL on any key that is persistent (ie, never going to expire). That's why I created this Redis Scanner -- to get a sense of what was going on. I should have more to post about that tomorrow.

Reply to this Comment

@All,

As a follow-up, I created this Redis Key Scanner to help me better understand the contents of the Redis database such I could create a subsequent task that scans the key-space and adds TTL values:

https://www.bennadel.com/blog/3712-adding-a-ttl-to-all-persistent-keys-in-redis-using-launchdarkly-feature-flags-and-lucee-cfml-5-2-9-40.htm

This code was a lot of fun to write! And, what's really cool is that I am use LaunchDarkly feature flags to both enable/disable the clean-up task as well as ramp up the aggressiveness of the task's SCAN operation.

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
NEW: Some basic markdown formatting is now supported: bold, italic, blockquotes, lists, fenced code-blocks. Read more about markdown syntax »
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.