Compiling Several Linked Files Into One File
Posted April 8, 2008 at 8:53 AM by Ben Nadel
When I create my Javascript files and Style sheets, I like to break them up into logically cohesive files so that I don't end up editing a CSS files that is four thousand lines long. I will do things like break my CSS up into Content, Meta Content, and Structural files; not only does this allow me to find things quicker, it allows me to easily reuse subsections of my style sheets, for example, including just the content.css file into my XStandard rich text editor. The problem with this technique is that it requires the browser to make an excessive number of subsequent requests to the server to gather all the Javascript and style information.
I know there are build scripts out there that will take a bunch of Javascript or CSS files and compile them down into one file, but, from what I have seen, you have to explicitly link to the final file in your code. I don't like this concept; I really like seeing my linked files in the HEAD tag of the XHTML file - it feels so explicit, so cozy and comforting. Not to mention easy to modify and understand, even for someone who might not be familiar with the code.
I wanted to try and come up with something that was a "best of both worlds" kind of solution. And so, I called once again upon my friend, the ColdFusion custom tag. I am not 100% satisfied with this solution, but I have built a small set of ColdFusion custom tags that allow you to list the files in the HEAD tag, as usual, but will compile the files and then write the HTML to link to the final file. Take a look at this example:
- <!--- Import tag library. --->
- <cfimport taglib="./" prefix="linked" />
-
- <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
- <html>
- <head>
- <title>Linked File ColdFusion Custom Tags</title>
-
- <!--- Include javascript files. --->
- <linked:files
- type="javascript"
- rebuildparam="reset"
- file="#APPLICATION.Root#scripts.js"
- url="#REQUEST.WebRoot#scripts.js">
-
- <linked:file path="#APPLICATION.Root#forms.js" />
- <linked:file path="#APPLICATION.Root#util.js" />
- <linked:file path="#APPLICATION.Root#calendar.js" />
- </linked:files>
- </head>
- <body>
-
- <h1>
- Linked File ColdFusion Custom Tags
- </h1>
-
- <p>
- Hello World.
- </p>
-
- </body>
- </html>
Here, we are linking several Javascript files (but secretly, they are getting compiled down into one file). Running this page, we actually get this output:
- <head>
- <title>Linked File ColdFusion Custom Tags</title>
-
- <script type="text/javascript" src="./scripts.js"></script>
- </head>
Notice that all the scripts were compiled down to a single, subsequent HTTP request.
Let's take a look at the attributes of the Files ColdFusion custom tag. The Type attribute just determines what type of HTML tag we are going to have to write. Currently, it just supports Javascript and Style (but no fancy @media or anything like that). The RebuildParam attribute is the URL key that will trigger a rebuild of the compiled file (the target file will also get rebuilt if it does not exist); so, for example, if I put "?reset" in my URL query string, the scripts.js file would be rebuilt and all new changes would take place. The File attribute is the expanded path of the target file (remember, we are building this on the server). Then, the Url attribute is the web-based URL of the target file relative to the current request.
While I like this, the down side to a technique like this is that it has to process the ColdFusion custom tags for every single page request; the nice thing about a build script, like an ANT script, is that it only has to be done once. But, like I said, I don't like "compilation gestures"; I find them to be not intuitive to your average programmer. So, to get around this, I tried to make the ColdFusion custom tags as clean as possible and have them build the file only when necessary, taking super care to read from the file system as little as possible!
The best I could do was limit the file reads to a single FileExists() on the target file for each page request. I am not sure if I am satisfied with this. I really don't like the idea of reading from the file system for every single request - this could become a huge overhead if I am not careful. I think I might dump the idea of checking file existence and make the rebuild process something that can only be launched manually using the RebuildParam. After all, it's not like I have to rebuild the file for every request - once the file is built, my only real responsibly, which is the largest use-case, is to output the SCRIPT or LINK tags.
Anyway, here is what I have so far. This is the Files ColdFusion custom tag:
- <!--- Kill extra output. --->
- <cfsilent>
-
- <!--- Check to see which mode we are executing. --->
- <cfswitch expression="#THISTAG.ExecutionMode#">
-
- <cfcase value="Start">
-
- <!--- Param tag attributes. --->
-
- <!---
- The type of linked file. Currently, only
- Javascript and Style are supported.
- --->
- <cfparam
- name="ATTRIBUTES.Type"
- type="regex"
- pattern="(?i)javascript|style"
- />
-
- <!---
- This is the URL key that will trigger a rebuild
- of the merged file data.
- --->
- <cfparam
- name="ATTRIBUTES.RebuildParam"
- type="string"
- default=""
- />
-
- <!---
- This is the expanded path to the compiled file
- that we are going to create.
- --->
- <cfparam
- name="ATTRIBUTES.File"
- type="string"
- />
-
- <!---
- This is the page-relative URL to the compiled
- file that we are going to create. This will
- have to make sense from the client's (browser)
- perspective.
- --->
- <cfparam
- name="ATTRIBUTES.Url"
- type="string"
- />
-
-
- <!---
- This struct will hold the nested file path
- data. Due to logic in the nested File tag, we
- can be sure that this holds ONLY valid paths.
- --->
- <cfset VARIABLES.FilePaths = [] />
-
-
- <!---
- Before we execute the child tags, we want to
- set a flag to determine whether the local file
- system should even be checked. File reads are
- a relatively expensive process, so we want to
- limit them whenever we don't need them
- (including FileExists() checks).
-
- We will want to rebuild the file (and therefore
- need access to the local file system) if the
- rebuild param exists in the URL or the actual
- compiled file does not exist.
- --->
- <cfset VARIABLES.Rebuild = (
- StructKeyExists( URL, ATTRIBUTES.RebuildParam ) OR
- (NOT FileExists( ATTRIBUTES.File ))
- ) />
-
- </cfcase>
-
- <cfcase value="End">
-
- <!---
- At this point, we might need to build or re-build
- the compiled file. This will happen if the file
- either does not exist, or the rebuild parameter
- is present.
- --->
- <cfif VARIABLES.Rebuild>
-
- <!--- Try to delete the existing file. --->
- <cfif FileExists( ATTRIBUTES.File )>
-
- <cffile
- action="delete"
- file="#ATTRIBUTES.File#"
- />
-
- </cfif>
-
-
- <!--- Loop over the file paths. --->
- <cfloop
- index="VARIABLES.FilePath"
- array="#VARIABLES.FilePaths#">
-
- <!--- Read in the file data. --->
- <cffile
- action="read"
- file="#VARIABLES.FilePath#"
- variable="VARIABLES.FileData"
- />
-
- <!---
- Write the individual file data to the
- compiled file, appending the data and
- a new line (to make sure nothing breaks
- cross-file).
- --->
- <cffile
- action="append"
- file="#ATTRIBUTES.File#"
- output="#VARIABLES.FileData#"
- addnewline="true"
- />
-
- </cfloop>
-
- </cfif>
-
-
- <!--- Check to see which type of file we linking. --->
- <cfswitch expression="#ATTRIBUTES.Type#">
-
- <cfcase value="Javascript">
-
- <!--- Store the output. --->
- <cfsavecontent variable="THISTAG.GeneratedContent">
- <cfoutput>
- <script type="text/javascript" src="#ATTRIBUTES.Url#"></script>
- </cfoutput>
- </cfsavecontent>
-
- </cfcase>
-
- <cfcase value="Style">
-
- <!--- Store the output. --->
- <cfsavecontent variable="THISTAG.GeneratedContent">
- <cfoutput>
- <link rel="stylesheet" type="text/css" href="#ATTRIBUTES.Url#"></link>
- </cfoutput>
- </cfsavecontent>
-
- </cfcase>
-
- </cfswitch>
-
-
- <!---
- As one final cleaning, just trim the generated
- content to make sure there is no leading or
- trailing spaces.
- --->
- <cfset THISTAG.GeneratedContent = Trim(
- THISTAG.GeneratedContent
- ) />
-
- </cfcase>
-
- </cfswitch>
-
- </cfsilent>
The File tag doesn't have much to it. It basically just collects file paths and stores them in the base tag (Files):
- <!--- Check to see which mode we are executing. --->
- <cfswitch expression="#THISTAG.ExecutionMode#">
-
- <cfcase value="Start">
-
- <!--- Get base tag data. --->
- <cfset VARIABLES.FilesTag = GetBaseTagData( "cf_files" ) />
-
-
- <!--- Param tag attributes. --->
-
- <!--- This is the expanded path to the given file. --->
- <cfparam
- name="ATTRIBUTES.Path"
- type="string"
- />
-
-
- <!---
- Check to see if we even need to worry about this
- file. Only go further if the base tags contains
- the rebuild flag.
- --->
- <cfif NOT VARIABLES.FilesTag.Rebuild>
-
- <!---
- We have no need to use the child tag, so exit
- it without processing.
- --->
- <cfexit method="exittag" />
-
- </cfif>
-
-
- <!---
- ASSERT: At this point, we know that the compiled
- file needs to be rebuilt and therefore we will need
- to check this child tag.
- --->
-
-
- <!---
- Store path in base tag. Check to see if the file
- exists so that we only have valid files passed
- up to the base tag.
- --->
- <cfif FileExists( ATTRIBUTES.Path )>
-
- <cfset ArrayAppend(
- VARIABLES.FilesTag.FilePaths,
- ATTRIBUTES.Path
- ) />
-
- </cfif>
-
- </cfcase>
-
- <cfcase value="End">
-
- </cfcase>
-
- </cfswitch>
So, there you have it. Like I said, I'm not fully satisfied with it, but I think I am headed in the right direction. I didn't include any kind of file compression in this version. When reading in the individual files, I could have done things like strip out white space and comments, but this was more a proof of concept than a final product.
Reader Comments
@Ben,
One concern I'd have with this implementation would be that you can run into caching issues. For example, you're combining 3 JavaScript files into one file without a "build number" in the file name. If you change one of your source JavaScript files, and rebuild the combined file, then that file has the same file name and may not be re-requested by the browser when refreshed on the server.
One potential way around this would be to add a "build" number to the file name, perhaps from your source control, or the date of the last modified file from the set of combined files, or maybe use a GUID as part of the file name.
Another potential issue is that you should account for the file list to change based upon the specific page that is being included into the page, to use an example from your post, not every page will have the rich text editor on it, so some pages will need to include the JavaScript for it and some won't.
So thinking on this as I'm writing it, it may be worthwhile to generate a hash based upon all the file names that are being included into the page, and perhaps include the last modification dates of the files (maybe stored in an application scoped variable to prevent unnecessary disk access).
Anyway, just a couple of thoughts on what you're working through.
@Danilo,
Brilliant idea! I like the hash. I think that takes care of all the issues that you mention. Thanks a lot.
if you have sessions enabled, you can just append session.sessionid to the url of the script:
# <script type="text/javascript" src="./scripts.js?#session.sessionid#"></script>
I do this to all the script and link tags so that it only downloads once for the session instead of downloading on every page refresh.
@Tony,
Also a good suggestion, thanks.
Rather than doing it at runtime, why not do it at deploy time? We use a Ant script to aggregate all our JS/CSS files at build time, and then run the JS through the YUI Compressor. The result is nice modular JS/CSS at develop time and a compressed single file in production.
It may be a good idea to include some type of browser detection (cgi, perhaps) to append or substitute IE6 specific CSS.
@Barney,
How does that work with the linking in the HTML? The one thing I want to get around is having to explicitly link to a single file, assuming that it will be created with deployment. I think that is too "release oriented" and not aligned with the less than well planned world of web development.
@David,
One could still use conditional logic to include the File tags in this way. That is not something that I would want to move into the logic of the tag internals.
How about stripping out the line-breaks, tabs and whitespace?
It would save me the trip to the CSS/JS compressor. :)
As AJAX use picks up more and more, this type of utility becomes increasingly beneficial.
Here's how we do it, complete with code samples: http://www.barneyb.com/barneyblog/2008/04/08/build-time-aggregation-of-jscss-assets/
Figured that'd be easier than trying to communicate that in a comments discussion, though it probably robs Ben of some traffic. ;)
Ben-
I'd hesitate to use "smart numbers" in your file name to force a browser to re-request. Instead, try using ETags. Shouldn't be difficult to use CF to insure that a new ETag is used.
http://en.wikipedia.org/wiki/HTTP_ETag
What about using the new CF8 ajax goodness? Even doing a simple <cflayout> adds a ton of js and a few css files to your <head> section. There probably isn't any easy way to go back and grab those to put into one file.
You could repackage the aggregator as a Servlet filter, I suppose. Or fix CF's AJAX stuff. The former would probably be easier. ;)
@Ben:
Strangely enough, this is a problem I was getting ready to address in the next few days.
One of the things I wanted to tackle is loading of scripts outside of the main template. I use custom tags to drive all my UI components. I like my tags to load all the appropriate external files (style sheets, JS files etc.) That way I don't worry about missing dependencies.
Since I drive all my views through a central <cf_layout/> tag, I was going to make my link tag be usable from any file and then accumulate all the link files when my <cf_layout /> tag process the "end" mode.
As mentioned already, I was thinking of using a hash value of all the filenames combined together. The problem with concatentating all the files together per page request, is you could end up with several large files that are almost identical.
Of course, I may just end up deciding it's easier to place all my dependent scripts that I want to "merge" into a build directory and then just rebuild the scripts using an ANT script. That may end up being the most efficient method.
@Matt,
While I don't quite understand ETags or how they would be set, since the actual linked file is a JS/CSS file, I am not sure that ColdFusion would even get a hand in the matter. Do you have any more information / links on where CF and ETags have played together (just for my own curiosity).
@Dan,
I used to hate it that you couldn't flush content inside of a custom tag, but as I have gotten more and more into ColdFusion custom tags, I have grown to really love the fact that you have crazy control over the generated content up until the last close tag is executed.
That aside, I wouldn't be too worried about having large JS files being almost identical. After all, it's not like those similar files are being downloaded by the same client, right? Really, it just comes down to have some more files on the server, not a real concern these days. Yes, there is something to be said about things being clean as possible, but I wouldn't worry about it.
