Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at CFUNITED 2009 (Lansdowne, VA) with: Sumit Verma and Jarrod Stiles

Streaming Secure Files Efficiently With ColdFusion And MOD XSendFile

By Ben Nadel on
Tags: ColdFusion

ColdFusion is truly the cat's pajamas. But, it only works on CFM and CFC pages. This typically isn't a problem; but, when it comes to adding access-permissions around non-ColdFusion files, we can find ourselves in a less-than-optimal situation. While it has been shown that ColdFusion can stream binary files without a significant memory hit, we still have the issue of threads - streaming a file using ColdFusion and the CFContent tag ties up a ColdFusion thread for the duration of the file transfer. Ideally, we'd like to be able to perform a security check in ColdFusion and then pass the file-transfer burden off to the underlying operating system. Luckily, MOD XSendFile does exactly this.

 
 
 
 
 
 
 
 
 
 

XSendFile is an Apache module that allows ColdFusion (and other application platforms) to delegate file transfer actions back to Apache. It does this through the use of an HTTP Header. After ColdFusion has processed an incoming request, the XSendFile module will examine the outgoing HTTP headers; if the "x-sendfile" header exists, Apache overwrites the outgoing request with the file defined in the "x-sendfile" header.

NOTE: While XSendFile is an Apache module, there are ways that you can run Apache configurations in IIS (ex. Helicon Ape).

To demonstrate how MOD XSendFile works, I've set up a super simple, mock store on my local Apache server. In this store, users can exchange credits in return for access to secure photos. These photos reside outside the web root; as such, we need to use ColdFusion to determine access rights on a per-photo basis and then to pass the non-public photo paths back to Apache for binary streaming.

To get this to work, I had to install the XSendFile module in Apache, load the module, and then enable it for one of my virtual hosts:

  • ## NOTE: The installation automatically added this module loader
  • ## into my main httpd.conf file (not shown in the demo).
  • ##
  • ## LoadModule xsendfile_module libexec/apache2/mod_xsendfile.so
  •  
  •  
  • <VirtualHost *:80>
  •  
  • # Define our domain name for local routing.
  • ServerName xsendfile.com
  •  
  • # Define the webroot for the site.
  • DocumentRoot "/Sites/bennadel.com/testing/mod_xsendfile/www"
  •  
  • # Include the ColdFusion connector.
  • Include "/private/etc/apache2/cf8-main.conf"
  •  
  • # Set permissions for web root directory.
  • <Directory "/Sites/bennadel.com/testing/mod_xsendfile/www">
  •  
  • # Enable the use of MOD_XSendFile.
  •  
  • XSendFile On
  •  
  • # Allow MOD_XSendFile to access files in the directory
  • # parallel to the web root (images). If we don't set this,
  • # the MOD_XSendFile will only be able to access files in
  • # or below the defined web root.
  • #
  • # NOTE: This used to be the property, XSendFileAllowAbove,
  • # but it has been deprecated in newer versions of the
  • # XSendFile module.
  •  
  • XSendFilePath "/Sites/bennadel.com/testing/mod_xsendfile/images"
  •  
  • </Directory>
  •  
  • </VirtualHost>

Once the XSendFile module is installed, it has to be explicitly enabled. In the above Apache configuration, the following command turns XSendFile on for a given directory:

  • XSendFile On

This will allow XSendFile to access files anywhere in or below the contextual directory. The problem with that is that I am defining this within my web root; and, for obvious security purposes, I am storing the photos outside of the web root. As such, I then have to specify that XSendFile can access files that reside outside of the web root; specifically, that it can access files in my private "images" directory:

  • XSendFilePath "/Sites/bennadel.com/testing/mod_xsendfile/images"

NOTE: In previous versions of XSendFile, this same action used to be accomplished with XSendFileAllowAbove. This directive no longer exists and has been replaced with XSendFilePath.

In the end, this is what my directory structure looked like:

  • ./site/www/
  • ./site/www/thumbs/
  • ./site/images/

As you can see, the "www" is our public website. Notice that the thumbnails are stored in a public, web-accessible directory - thumbs. There's no need to secure those. The large photos, on the other hand, are our primary assets and are stored outside the web root in the parallel, "images," directory.

Now that we have XSendFile installed and configured, let's take a look at the ColdFusion code. I'm going to move through this code rather quickly since 90% of it isn't really relative to the XSendFile Apache module - it's just there to provide context.

Application.cfc

  • <cfcomponent
  • output="false"
  • hint="I define the application settings and event handlers.">
  •  
  • <!--- Define the application settings. --->
  • <cfset this.name = hash( getCurrentTemplatePath() ) />
  • <cfset this.applicationTimeout = createTimeSpan( 0, 0, 10, 0 ) />
  • <cfset this.sessionManagement = true />
  • <cfset this.sessionTimeout = createTimeSpan( 0, 0, 10, 0 ) />
  •  
  •  
  • <cffunction
  • name="onApplicationStart"
  • access="public"
  • returntype="boolean"
  • output="false"
  • hint="I initialize the application.">
  •  
  • <!---
  • Get the path to the images directory. For this demo,
  • that is simply the directory paralle to the webroot
  • (ie. ../images/).
  •  
  • NOTE: For this demo, I'm going to use "../" notation for
  • path traversal rather than actually calculating the exact
  • path - just makes the demo easier to read.
  • --->
  • <cfset application.imagesDirectory = (
  • getDirectoryFromPath( getCurrentTemplatePath() ) &
  • "../images/"
  • ) />
  •  
  • <!--- Return true so the application can load. --->
  • <cfreturn true />
  • </cffunction>
  •  
  •  
  • <cffunction
  • name="onSessionStart"
  • access="public"
  • returntype="void"
  • output="false"
  • hint="I initialize the session.">
  •  
  • <!---
  • Start the user off with a balance (which can be traded
  • in for image access).
  • --->
  • <cfset session.balance = 5.0 />
  •  
  • <!---
  • Keep track of the user's purchases. When the user trades
  • in their balance for an image, we will keep a record of
  • it so they can access the given files.
  • --->
  • <cfset session.purchases = {} />
  •  
  • <!--- Return out. --->
  • <cfreturn />
  • </cffunction>
  •  
  •  
  • <cffunction
  • name="onRequestStart"
  • access="public"
  • returntype="boolean"
  • output="false"
  • hint="I initialize the request.">
  •  
  • <!--- Check for a re-init flag. --->
  • <cfif structKeyExists( url, "init" )>
  •  
  • <!--- Manually reset the application. --->
  • <cfset this.onApplicationStart() />
  • <cfset this.onSessionStart() />
  •  
  • </cfif>
  •  
  • <!--- Return true so the request can load. --->
  • <cfreturn true />
  • </cffunction>
  •  
  • </cfcomponent>

When a user's session gets initialized, they get two properties - balance and purchases. The balance is their store credit and the purchases is an ID-based collection of photos for which they have paid.

The main page of the application handles the photo-list and purchase-processing. The user selects a photo and, if they have enough credits, the photo is added to their list of purchases and the user is forwarded to the purchase page.

Index.cfm

  • <!--- Param the photo selection. --->
  • <cfparam name="url.selectedPhoto" type="numeric" default="0" />
  •  
  • <!--- Set an error message. --->
  • <cfset errorMessage = "" />
  •  
  • <!--- Set the cost of the photo purhcases. --->
  • <cfset photoCost = 2 />
  •  
  •  
  • <!--- ----------------------------------------------------- --->
  • <!--- ----------------------------------------------------- --->
  •  
  •  
  • <!--- Check to see if any photo selection was made. --->
  • <cfif url.selectedPhoto>
  •  
  • <!---
  • Check to see if the user has enough credits to make the
  • given photo purchase (or if they have already purchased
  • this photo - in which case, we'll just forward them to
  • their purchased photo).
  • --->
  • <cfif (
  • (session.balance gte photoCost) ||
  • structKeyExists( session.purchases, url.selectedPhoto )
  • )>
  •  
  • <!---
  • For this demo, we don't want to re-charge the user for
  • photos they've already purchased. As such, only charge
  • them if the photo is not already in the collection of
  • purchased items.
  • --->
  • <cfif !structKeyExists( session.purchases, url.selectedPhoto )>
  •  
  • <!--- Deduct the credit from the balance. --->
  • <cfset session.balance -= photoCost />
  •  
  • <!--- Add the purchase record. --->
  • <cfset session.purchases[ selectedPhoto ] = now() />
  •  
  • </cfif>
  •  
  • <!---
  • Forward the user to purchase page where they can view
  • and download the purchased photo.
  • --->
  • <cflocation
  • url="./purchase.cfm?photo=#url.selectedPhoto#"
  • addtoken="false"
  • />
  •  
  • <cfelse>
  •  
  • <!---
  • The use doesn't have enough balance to make the purchase
  • - set an error message to display.
  • --->
  • <cfset errorMessage = "You do not have enough credits." />
  •  
  • </cfif>
  •  
  • </cfif>
  •  
  •  
  • <!--- ----------------------------------------------------- --->
  • <!--- ----------------------------------------------------- --->
  •  
  •  
  • <cfoutput>
  •  
  • <!DOCTYPE html>
  • <html>
  • <head>
  • <title>Using MOD_XSendFile With ColdFusion To Stream Files</title>
  •  
  • <style type="text/css">
  •  
  • img {
  • border: 5px solid ##454545 ;
  • margin-right: 10px ;
  • vertical-align: middle ;
  • }
  •  
  • button {
  • height: 40px ;
  • }
  •  
  • p.error {
  • color: ##CC0000 ;
  • font-style: italic ;
  • }
  •  
  • </style>
  • </head>
  • <body>
  •  
  • <h1>
  • Using MOD_XSendFile With ColdFusion To Stream Files
  • </h1>
  •  
  • <p>
  • Credit Balance: <strong>#session.balance#</strong>
  • </p>
  •  
  • <!--- Check to see if there is any error to report. --->
  • <cfif len( errorMessage )>
  •  
  • <p class="error">
  • <strong>Oh Noes!</strong> #errorMessage#
  •  
  • <a href="#cgi.script_name#?init=1">Re-initialize!</a>
  • </p>
  •  
  • </cfif>
  •  
  • <form action="#cgi.script_name#" method="get">
  •  
  • <p>
  • Please select the image to purchase:
  • </p>
  •  
  • <!--- Output a purchase option for each image. --->
  • <cfloop
  • index="photoIndex"
  • from="1"
  • to="15"
  • step="1">
  •  
  • <!---
  • Notice that thumbnails are below the webroot
  • (can be publicly accessed) but that the original
  • image are NOT in the webroot and therefore cannot
  • be accessed directly.
  • --->
  • <p>
  • <img src="./thumbs/#photoIndex#.jpg" />
  •  
  • <button
  • type="submit"
  • name="selectedPhoto"
  • value="#photoIndex#">
  • Photo #photoIndex# : #photoCost# Credits
  • </button>
  • </p>
  •  
  • </cfloop>
  •  
  • </form>
  •  
  • </body>
  • </html>
  •  
  • </cfoutput>

Once the user had made the purchase of a photo, they are forwarded to the purchase page. The purchase page doesn't actually stream the binary file to the client; rather, it acts as an intermediary landing page that gives the user some informational feedback. It's one of those "your download will begin shortly" pages.

Purchase.cfm

  • <!--- Param the photo id. --->
  • <cfparam name="url.photo" type="numeric" />
  •  
  • <cfoutput>
  •  
  • <!DOCTYPE html>
  • <html>
  • <head>
  • <title>Your Download Will Begin Shortly</title>
  •  
  • <!---
  • Forward the user to the download page. This is the
  • page that will actually stream the file to client
  • (the user's browser).
  • --->
  • <meta
  • http-equiv="refresh"
  • content="1; url=./download.cfm?photo=#url.photo#"
  • />
  • </head>
  • <body>
  •  
  • <h1>
  • Thank You - Your Download Will Begin Shortly
  • </h1>
  •  
  • <p>
  • If your download does not begin within a few seconds,
  • <a href="./download.cfm?photo=#url.photo#">click here</a>
  • to proceed manually.
  • </p>
  •  
  • <p>
  • <a href="./index.cfm">Return to Photos</a>
  • </p>
  •  
  • </body>
  • </html>
  •  
  • </cfoutput>

This page then forwards the user onto the actual download page which is where we finally get to deal with code that relates to XSendFile. Notice that ColdFusion isn't actually reading in or streaming out any file.

Download.cfm

  • <!--- Param the photo id. --->
  • <cfparam name="url.photo" type="numeric" />
  •  
  •  
  • <!---
  • Check to make sure the photo is in the user's collection
  • of purchases.
  • --->
  • <cfif structKeyExists( session.purchases, url.photo )>
  •  
  •  
  • <!---
  • The user has made the purchase - add the full file path of
  • the purchased image to the outgoing headers. By using the
  • XSendFile header, the file streaming will be passed off to
  • Apache (ie. it won't tie up a ColdFusion thread).
  •  
  • NOTE: Apache will strip this header out - it will never
  • reach the client.
  • --->
  • <cfheader
  • name="x-sendfile"
  • value="#application.imagesDirectory##url.photo#.jpg"
  • />
  •  
  • <!---
  • Deliver this purchased photo as an attachment so
  • the "Purchase" screen will not be navigated away from
  • (visually speaking).
  • --->
  • <cfheader
  • name="content-disposition"
  • value="attachment; filename=purchase-#url.photo#.jpg"
  • />
  •  
  • <!--- Set the mime-type. --->
  • <cfcontent type="image/jpeg" />
  •  
  • <!---
  • Exit the request - we don't really need to do this --
  • this is here only to emphasize that ColdFusion is no
  • longer doing anything in this request that is realted to
  • the file delivery.
  • --->
  • <cfexit />
  •  
  •  
  • <cfelse>
  •  
  •  
  • <!---
  • The user has not purhcased this photo. Return an accessed
  • denied header response.
  • --->
  • <cfheader
  • statuscode="401"
  • statustext="Unauthorized"
  • />
  •  
  • <h1>
  • Access Denied
  • </h1>
  •  
  • <p>
  • You are not authorized to access this page.
  • </p>
  •  
  •  
  • </cfif>

Before it does anything else, this ColdFusion page checks to see if the user has actually purchased the photo. Remember, that's really the reason we are here - to figure out how to create a secure-access layer in ColdFusion in which we can perform user-specific security checks. But, once the permissions have been evaluated, ColdFusion simply defines a new HTTP header, "x-sendfile." The value stored in this header is the absolute file path of the secure file that we are streaming. Once ColdFusion finishes the request, the XSendFile Apache module then examines the outgoing HTTP headers and streams the given file back to the client.

NOTE: The X-SendFile header never reaches the client. Once the XSendFile module evaluates the outgoing HTTP headers, it strips the X-SendFile entry and streams the file at the defined path.

I have to say, MOD XSendFile is pretty darn awesome. Admittedly, I like to keep my application as modular as possible; that is, I like to keep as much as the application configuration inside the application itself. But, when you see the kind of power and functionality that can be provided by things like XSendFile and URL Rewriting, it definitely makes you a lot more open-minded. XSendFile makes it a breeze to provide ColdFusion-powered secure file access without putting an undue burden on the finite collection of ColdFusion threads.




Reader Comments

This may be the first time I actually understood what you are doing on my first read-through. Really great example of integrating web server features with CF. Also, great example on how you integrate CF into Apache.

Reply to this Comment

@Brandon,

Ha ha, awesome - hopefully that means I am getting better at explaining myself :) I think this stuff is really cool. I can definitely see a lot of opportunities to use this module. I've never been truly happy with secure file downloads in the past.

Reply to this Comment

Thanks, Ben! I could have used this at my previous company, but now have this page bookmarked and indexed in Evernote so that I can figure out how to do it for my startup, as I know that file streaming will be needed.

I found your explanation easy to follow - you have always done a great job and continue to improve - we thank you!

Reply to this Comment

Been doing this and similar for the past 4 or 5 years. :) FYI, writing an IIS module for IIS7+ that does this is not very difficult (c#).

Prior to IIS7 it's a... hassle (ISAPI filters and native code), so I've never bothered.

Reply to this Comment

@Mark,

I haven't touched C# in a long time :) But, it's good to know that it's not a dead end in IIS. Glad this kind of approach has been proven over the past few years. I really like it!

Reply to this Comment

Interesting technique. Only problem is that it requires apache. I am using nginx as my front end web server. A little googling turned up an nginx module http://wiki.nginx.org/XSendfile that uses a very similar concept... so the same idea may work for me.

Thanks for the idea!

Reply to this Comment

Great idea!

We used the combination of mod_auth_cookie and Apache authentication to manage secure downloads, but your approach seems much better. Especially since mod_auth_cookie seems dead.

Thanks for pointing XSendfile.

Reply to this Comment

My apologies if my question is a bit basic, but is it possible to have Apache and CF running in separate servers to get this to work?

BTW also wanted to thank you, Ben. Since I recently started working with CF, I must say that I always find great information in your blog!

Reply to this Comment

Ben,

Are you sure the X-SendFile header never reaches the client? I am using this and it all appears to be working great...except for the fact then when looking at the headers in Charles the full path of the file is included in the header

Reply to this Comment

@Jason,

That's really strange. I have not seen that. In my requests, the X-Send-File header comes through, but the value has been stripped-out. Are you sure XSendFile is actually processing the request?

Reply to this Comment

@All,

Also, I just came across a strange problem on IIS + XSendFile. We're using XSendFile on IIS7 and I *think* we're using Ionic to do the integration. Any how, on IIS7, XSendFile does not work unless you have output="false" in some part of the request.

This could be on the application:

  • <cfcomponent output="false> ... </cfcomponent>

... or on the request start:

  • <cffunction name="onRequestStart" output="false"> ... </cffunction>

It doesn't seem to matter *where* the output=false is, so long as it's part of the pre-request processing. Doing so seems to trip the some of internal flag that allows XSendFile to work.

If you don't have the above, the binary that comes down from IIS has a space character as the first byte (which makes the files unusable unless you manually strip it out).

Again, this is only for XSendFile running under IIS7 as far as I know. I have not seen this error under Apache.

Reply to this Comment

Ben,

Ok yeah that was the issue, it looked like the PDF was being downloaded but upon further investigation it was not. I changed a few config params and it is working as expected! Awesome stuff, thanks for the quick reply.

Reply to this Comment

Hi Ben, I faced an issue that I partially solved with the streaming process you described in http://www.bennadel.com/blog/1227-Using-ColdFusion-To-Stream-Files-To-The-Client-Without-Loading-The-Entire-File-Into-Memory.htm

I wrote a SO question that defines the problematic which you can see at http://stackoverflow.com/questions/19385223/how-to-transparently-stream-a-file-to-the-browser

The solution works, but it will not support video seeking, which I was expecting, but it is a very important feature that we need.

So far I was unable to find enough information on the protocol used by Windows Media Player and IIS (default) and because of the controlled environment I work in, I couldn't inspect the HTTP request details to see what headers the client is sending to IIS and how the server responds.

The only feature we are interested in is seeking, not implementing a full media server with adapative bitrate. Do you have an idea how we could tackle the problem?

Thanks!

*NOTE: I cannot install any plugins on the server.*

Reply to this Comment

I just thought about a different approach. Perhaps we could perform the HTTP request from ColdFusion to IIS by just cloning the current client request details and find a way to read from the http response stream write and flush to the client as we read from the stream.

Basically ColdFusion would just act as a proxy. I know that it's probably not possible with cfhttp however it probably is using Java classes.

Have you ever done something similar?

*NOTE: I have no idea if that would fix the video seeking feature however.*

Reply to this Comment

@Alexandre,

That's a really interesting approach! I can't say that I know much of anything about media streaming, but I love how you're basically piping one response into another. Very cool!

Reply to this Comment

Post A Comment

?
You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.