Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at cf.Objective() 2013 (Bloomington, MN) with:

Serializing ColdFusion Variables Into ColdFusion Code Using WDDX

By Ben Nadel on
Tags: ColdFusion

ColdFusion provides us with a number of ways in which to serialized our ColdFusion values - JSON, WDDX, and Binary (as of ColdFusion 9) are all formats that can be generated on the fly. But these formats all require a file read in order to be consumed. What if, rather than serializing a value into a simple data type, we could serialize it into the ColdFusion code that could be used to generate it? In doing so, we could essentially cache values to the template cache rather than to the "file system."

Now, of course, CFM files are stored on the file system, just like every other file; but, with ColdFusion features like the template cache and the trusted cache, file access for CFM files is highly optimized. Furthermore, the density of the cache gets managed by ColdFusion in such a way that you can't "overflow" your template cache (it will purge cached templates as needed). Now, I'm not saying that ColdFusion manages the template cache in a very intelligent way - I think it's a time-based queue; but, I am saying that you won't crash your server by filling up your JVM with cached templates.

Cache-talk aside, I put together a little proof-of-concept using WDDX. When I first started this, I was going to try and build the code recursively based on the data type; but, ColdFusion's WDDX support is pretty awesome and XML parsing is pretty freaking fast. As such, I didn't see a need to reinvent the wheel (at least not for this exploration).

  • <cffunction
  • name="toCode"
  • access="public"
  • returntype="string"
  • output="false"
  • hint="I take a given variable and return the ColdFusion code (via WDDX) that would be required in order to create it.">
  •  
  • <!--- Define arguments. --->
  • <cfargument
  • name="value"
  • type="any"
  • required="true"
  • hint="I am the ColdFusion variable that is being serialized to ColdFusion code (via inline WDDX)."
  • />
  •  
  • <cfargument
  • name="prefix"
  • type="string"
  • required="false"
  • default="data"
  • hint="I am the variable name prefix to use when generating the ColdFusion code."
  • />
  •  
  • <!--- Define the local scope. --->
  • <cfset var local = {} />
  •  
  • <!---
  • When converting a value using WDDX, we need two
  • intermediary values - the content buffer that holds the
  • actual WDDX data and the intermediary value that holds the
  • deserialization. Let's create a "Safe" suffix for use in
  • these values.
  • --->
  • <cfset local.prefixHash = ("toCode" & hash( arguments.prefix )) />
  •  
  • <!--- Convert the value to WDDX. --->
  • <cfwddx
  • output="local.wddxValue"
  • action="cfml2wddx"
  • input="#arguments.value#"
  • />
  •  
  • <!---
  • Return the value going through the WDDX conversion
  • life-cycle. We'll need to use an intermediary
  • CFSaveContent tag to hold the XML.
  • --->
  • <cfreturn (
  • "<cfsavecontent variable=""in_#local.prefixHash#"">" &
  • local.wddxValue &
  • "</cfsavecontent>" &
  • (chr( 13 ) & chr( 10 )) &
  • "<cfwddx " &
  • "action=""wddx2cfml"" " &
  • "input=""##in_#local.prefixHash###"" " &
  • "output=""out_#local.prefixHash#"" " &
  • "/>" &
  • (chr( 13 ) & chr( 10 )) &
  • "<cfset #arguments.prefix# = out_#local.prefixHash# />"
  • ) />
  • </cffunction>
  •  
  •  
  • <!--- ----------------------------------------------------- --->
  • <!--- ----------------------------------------------------- --->
  • <!--- ----------------------------------------------------- --->
  • <!--- ----------------------------------------------------- --->
  •  
  •  
  • <!--- Create a value that we want to "cache" to code. --->
  • <cfset data = {} />
  • <cfset data.label = "Girls" />
  • <cfset data.tags = [ "Cute", "Adorable" ] />
  • <cfset data.collection = queryNew( "" ) />
  • <cfset queryAddColumn(
  • data.collection,
  • "id",
  • "cf_sql_integer",
  • listToArray( "1,2,3" )
  • ) />
  • <cfset queryAddColumn(
  • data.collection,
  • "name",
  • "cf_sql_varchar",
  • listToArray( "Jill,Tricia,Katie" )
  • ) />
  •  
  •  
  • <!--- Serialize the data value to ColdFusion code. --->
  • <cfset code = toCode( data, "newData" ) />
  •  
  • <!--- Write the ColdFusion code to a CFM file. --->
  • <cffile
  • action="write"
  • file="#expandPath( './code.cfm' )#"
  • output="#code#"
  • />
  •  
  •  
  • <!--- ----------------------------------------------------- --->
  • <!--- ----------------------------------------------------- --->
  •  
  •  
  • <!--- Include the generated code file. --->
  • <cfinclude template="code.cfm" />
  •  
  • <!--- Output the resultant variable. --->
  • <cfdump var="#newData#" />

As you can see, the toCode() function takes a variable and returns the ColdFusion code needed to generate it (as close as possible). In the end, this isn't doing much more than creating an inline XML buffer, parsing it, and assigning the result. The value that gets returned is the ColdFusion code needed to generate the original variable. This code can then be written to disk and used as a CFM template.

When we run the above code, we get the following CFDump output:

 
 
 
 
 
 
ToCode() can serialize ColdFusion variables into the ColdFusion code needed to generate them. 
 
 
 

As you can see, the ColdFusion variable was serialized into ColdFusion code, which was then used to create a mirrored ColdFusion variable.

So, to be honest, I really have no idea if this is a good idea or not. I wouldn't be surprised if the first person to comment told me that this was insane and that if I needed a better cache, I should be using something like CouchDB.... if only my VPS had more memory. This idea just popped into my head the other day and I wanted to try and think it through a bit before I completely discarded it.



Reader Comments

Okay . . . so you can scrape serialized data and create CF code from these, but why would you want to do so?

Reply to this Comment

This is insane and you need a better cache (there, I said it!). This reminds me of a time when someone I knew wanted a way to put CF code in the database and have it execute by writing a file to disk and then including it (this was way back in the CF5 days).

Serializing objects/data to disk and reading them back in is certainly nothing new, but generally it's a good practice to separate the contexts of "data" and "executable code". Turning the data into code is generally not a good idea, and is the reason people harp on the use of CFQUERYPARAM all the time (takes the "data" out of the "code" of the SQL query so it cannot be executed).

Granted this is a little more controlled since it's being fully encapsulated into WDDX, so in theory it should work/act exactly the same as if you were putting the WDDX package into the database and reading it from there. I would certainly be exploring options other than generating executable code on-the-fly.

It's an interesting proof of concept though.

Reply to this Comment

@Lola,

I could see this being used for long-term storage for something like basic application settings. For example, the setup process asks for your datasource information and then generates some executable code to set those variables in an included file so they're available for the life of the application on that server.

Now that I think back, I recall doing something similar to cache generated page content in places where speed was important and the generated content took a while to produce but didn't change often. This was back in CFMX 6 before the CFCACHE tag could cache parts of a page (it was all or nothing back then). I wrote a CFML custom tag which you could wrap around a piece of content and set a unique name and a time-to-live in seconds. It would execute the code on the first run, then write it to disk as a CFM file in a cache folder, then on subsequent runs if the TTL hadn't expired, it would CFINCLUDE the generated file. The first page load would be slow since it had to execute the code, then the second page load would be faster but still slow because the JVM would have to compile the new CFM file into byte code and cache it before it could be included, but then subsequent page loads would be really quick. It was originally written to speed up the home page on a bunch of floral e-commerce sites I worked on for a while. CF's internal cache mechanisms provide the same functionality now with a lot more flexibility though, so I wouldn't do it again (so I still think it's insane with modern versions of CF. :)

Reply to this Comment

@Justin,

Ha ha, yeah, maybe it is a bit insane.

Here's my situation - I'm on a small VPS with 1 Gig of RAM, a sub-portion of which actually goes to ColdFusion and the JVM. I currently cache things like queries into an object (which is, itself, cached in the Application scope). But, I get nervous that on such a small machine, caching large amounts of database-driven data will quickly soak up the valuable JVM heap space.

Disk space, on the other hand, I am dirty with - gigs of it. So, I figured I'd do a little exploration into how one might cache data to disk rather than memory. Now, of course, disk access is really slow. But, I figured the Template Cache would help alleviate that by keeping popular data in the cache and then re-compiling non-popular data as needed. But, cached or in-need of compiling, it's still skipping the database (which I believe is really becoming a bottle-neck on my site).

So, anyway, just an exploration. Trying various ways of getting rid of my thousand nightly Error emails and lock-timeouts :D

Reply to this Comment

@Lola,

The CFLocks are around database calls that populate my application cache. As such, I am pretty sure it's my database that is causing the bottleneck. That's why I'm trying to move away from the database and a source of data (at least, to limit the amount that I have to go to it).

And, I don't want to cache rendered content because there is still a good amount of logic that I do inside of that content - it is not static.

Reply to this Comment

@Ben,

Ah, the classic computer science exercise of generating code and then executing it.

I get what you're saying about CF template cache management not running the JVM out of memory. An alternative (caching already-compiled data structures in the Server scope) can definitely run you out of memory. What you've demonstrated is safer, in terms of memory management.

In the early days of CFMX, we had to fight the JDBC problem of cfstoredproc not supporting dbvarname (call-by-name). We had LOTS of cfstoredproc calls that didn't have cfprocparams in the correct order for CFMX's call-by-position. It threatened to keep us from upgrading to CFMX.

So I wrote a utility to read the script that defines a stored procedure, parse it and generate a "stored procedure call file", in which the cfprocparams were guaranteed to be in definition order (with the correct datatypes). That way, the call-by-position limitation of CFMX would no longer be a problem. It's what allowed us to go to CFMX.

Eventually our stored procedure call files incorporated tons of standardized error recovery, null parameter management, performance logging and other non-CFMX-conversion benefits. And it also allowed us to easily switch between DBMSs (Microsoft SQL Server, Oracle, Sybase), simply by calling a different stored procedure call file targeted to the other DBMS.

Recently I've started extracting table column properties as JSON, for use in a general purpose utility that validates whether data can be stored into a database column, based on the table's definition script. Sure, you could call cfdbinfo, but the files to generate data structures from JSON are CFM files, so they can generate a more useful data structure than cfdbinfo and be cached in the template cache (essentially no I/O).

So yes, generating code can be a very good idea indeed.

Reply to this Comment

@WebManWalking,

I've never had to deal with stored procedures before, but that definitely sounds like a great deal of work :) Ultimately, I wish I had some retarded awesome dedicated box where I could just play with different ideas like CouchDB and MongoDB and all that jazz.

Reply to this Comment

@Ben,

It was a great deal of work back in May, 2003. But ever since, it's been an enormous amount of work saved.

Steve Drucker of Fig Leaf Software wrote something he called his "SQL Tool of Justice!" It also generates code from the database, all the way down to input/edit screens. It even honors foreign key constraints and many-to-many relationships. It's Microsoft SQL Server only, because it uses built-in stored procedures such as sp_databases, sp_tables, sp_columns, etc, to find out database information.

For a while there, I thought Steve Drucker was trying to put all of us web-to-database types out of our jobs with that tool!

Reply to this Comment

@WebManWalking Is this SQL Tool of Justice available anywhere? I suppose there have been more improved code generators since then.

Reply to this Comment

@Lola,

I just Googled it and found some posts on www.houseoffusion.com. Steve Drucker posted a Flash Remoting piece of it and others wanted to see more. But I don't see the tool there. I would've thought there would be something on figleaf.com about it, but I didn't find anything. Sorry.

You might be able to contact him directly. He's at their DC office

http://www.figleaf.com/Company/Locations.cfm#CP_JUMP_854

Reply to this Comment

Ben,

One idea you might consider to deal with your caching issue would be to use ColdFusion 9's ehcache implementation. You could set the maxElementsInMemory to a relatively low value (to account for your low amount of RAM) and set overflowToDisk="true" so that when the memory cache is filled, ehcache will automatically start caching additional objects/data to disk instead.

In general, I would not recommend using a NOSQL database as a cache - mainly because the majority of them cache to disk which is at least an order of magnitude than caching to memory.

Reply to this Comment

@Rob,

Unfortunately, my VPS is on CF8. And, the amount of memory on it is so low, that I am not sure how easy the upgrade process would be. I have a new sever in the works, using CF9; hopefully, moving to that will help me out a lot!

Reply to this Comment

Hi Ben - How to create a JSON object from a query in coldfusion 5 ? The SerializeJSON is not available there.

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.