Skip to main content
Ben Nadel at the jQuery Conference 2010 (Boston, MA) with: Blain Smith
Ben Nadel at the jQuery Conference 2010 (Boston, MA) with: Blain Smith ( @blainsmith )

Streaming Secure Files Efficiently With ColdFusion And MOD XSendFile

By on
Tags:

ColdFusion is truly the cat's pajamas. But, it only works on CFM and CFC pages. This typically isn't a problem; but, when it comes to adding access-permissions around non-ColdFusion files, we can find ourselves in a less-than-optimal situation. While it has been shown that ColdFusion can stream binary files without a significant memory hit, we still have the issue of threads - streaming a file using ColdFusion and the CFContent tag ties up a ColdFusion thread for the duration of the file transfer. Ideally, we'd like to be able to perform a security check in ColdFusion and then pass the file-transfer burden off to the underlying operating system. Luckily, MOD XSendFile does exactly this.

XSendFile is an Apache module that allows ColdFusion (and other application platforms) to delegate file transfer actions back to Apache. It does this through the use of an HTTP Header. After ColdFusion has processed an incoming request, the XSendFile module will examine the outgoing HTTP headers; if the "x-sendfile" header exists, Apache overwrites the outgoing request with the file defined in the "x-sendfile" header.

NOTE: While XSendFile is an Apache module, there are ways that you can run Apache configurations in IIS (ex. Helicon Ape).

To demonstrate how MOD XSendFile works, I've set up a super simple, mock store on my local Apache server. In this store, users can exchange credits in return for access to secure photos. These photos reside outside the web root; as such, we need to use ColdFusion to determine access rights on a per-photo basis and then to pass the non-public photo paths back to Apache for binary streaming.

To get this to work, I had to install the XSendFile module in Apache, load the module, and then enable it for one of my virtual hosts:

## NOTE: The installation automatically added this module loader
## into my main httpd.conf file (not shown in the demo).
##
## LoadModule xsendfile_module libexec/apache2/mod_xsendfile.so


<VirtualHost *:80>

	# Define our domain name for local routing.
	ServerName xsendfile.com

	# Define the webroot for the site.
	DocumentRoot "/Sites/bennadel.com/testing/mod_xsendfile/www"

	# Include the ColdFusion connector.
	Include "/private/etc/apache2/cf8-main.conf"

	# Set permissions for web root directory.
	<Directory "/Sites/bennadel.com/testing/mod_xsendfile/www">

		# Enable the use of MOD_XSendFile.

		XSendFile On

		# Allow MOD_XSendFile to access files in the directory
		# parallel to the web root (images). If we don't set this,
		# the MOD_XSendFile will only be able to access files in
		# or below the defined web root.
		#
		# NOTE: This used to be the property, XSendFileAllowAbove,
		# but it has been deprecated in newer versions of the
		# XSendFile module.

		XSendFilePath "/Sites/bennadel.com/testing/mod_xsendfile/images"

	</Directory>

</VirtualHost>

Once the XSendFile module is installed, it has to be explicitly enabled. In the above Apache configuration, the following command turns XSendFile on for a given directory:

XSendFile On

This will allow XSendFile to access files anywhere in or below the contextual directory. The problem with that is that I am defining this within my web root; and, for obvious security purposes, I am storing the photos outside of the web root. As such, I then have to specify that XSendFile can access files that reside outside of the web root; specifically, that it can access files in my private "images" directory:

XSendFilePath "/Sites/bennadel.com/testing/mod_xsendfile/images"

NOTE: In previous versions of XSendFile, this same action used to be accomplished with XSendFileAllowAbove. This directive no longer exists and has been replaced with XSendFilePath.

In the end, this is what my directory structure looked like:

  • ./site/www/
  • ./site/www/thumbs/
  • ./site/images/

As you can see, the "www" is our public website. Notice that the thumbnails are stored in a public, web-accessible directory - thumbs. There's no need to secure those. The large photos, on the other hand, are our primary assets and are stored outside the web root in the parallel, "images," directory.

Now that we have XSendFile installed and configured, let's take a look at the ColdFusion code. I'm going to move through this code rather quickly since 90% of it isn't really relative to the XSendFile Apache module - it's just there to provide context.

Application.cfc

<cfcomponent
	output="false"
	hint="I define the application settings and event handlers.">

	<!--- Define the application settings. --->
	<cfset this.name = hash( getCurrentTemplatePath() ) />
	<cfset this.applicationTimeout = createTimeSpan( 0, 0, 10, 0 ) />
	<cfset this.sessionManagement = true />
	<cfset this.sessionTimeout = createTimeSpan( 0, 0, 10, 0 ) />


	<cffunction
		name="onApplicationStart"
		access="public"
		returntype="boolean"
		output="false"
		hint="I initialize the application.">

		<!---
			Get the path to the images directory. For this demo,
			that is simply the directory paralle to the webroot
			(ie. ../images/).

			NOTE: For this demo, I'm going to use "../" notation for
			path traversal rather than actually calculating the exact
			path - just makes the demo easier to read.
		--->
		<cfset application.imagesDirectory = (
			getDirectoryFromPath( getCurrentTemplatePath() ) &
			"../images/"
			) />

		<!--- Return true so the application can load. --->
		<cfreturn true />
	</cffunction>


	<cffunction
		name="onSessionStart"
		access="public"
		returntype="void"
		output="false"
		hint="I initialize the session.">

		<!---
			Start the user off with a balance (which can be traded
			in for image access).
		--->
		<cfset session.balance = 5.0 />

		<!---
			Keep track of the user's purchases. When the user trades
			in their balance for an image, we will keep a record of
			it so they can access the given files.
		--->
		<cfset session.purchases = {} />

		<!--- Return out. --->
		<cfreturn />
	</cffunction>


	<cffunction
		name="onRequestStart"
		access="public"
		returntype="boolean"
		output="false"
		hint="I initialize the request.">

		<!--- Check for a re-init flag. --->
		<cfif structKeyExists( url, "init" )>

			<!--- Manually reset the application. --->
			<cfset this.onApplicationStart() />
			<cfset this.onSessionStart() />

		</cfif>

		<!--- Return true so the request can load. --->
		<cfreturn true />
	</cffunction>

</cfcomponent>

When a user's session gets initialized, they get two properties - balance and purchases. The balance is their store credit and the purchases is an ID-based collection of photos for which they have paid.

The main page of the application handles the photo-list and purchase-processing. The user selects a photo and, if they have enough credits, the photo is added to their list of purchases and the user is forwarded to the purchase page.

Index.cfm

<!--- Param the photo selection. --->
<cfparam name="url.selectedPhoto" type="numeric" default="0" />

<!--- Set an error message. --->
<cfset errorMessage = "" />

<!--- Set the cost of the photo purhcases. --->
<cfset photoCost = 2 />


<!--- ----------------------------------------------------- --->
<!--- ----------------------------------------------------- --->


<!--- Check to see if any photo selection was made. --->
<cfif url.selectedPhoto>

	<!---
		Check to see if the user has enough credits to make the
		given photo purchase (or if they have already purchased
		this photo - in which case, we'll just forward them to
		their purchased photo).
	--->
	<cfif (
		(session.balance gte photoCost) ||
		structKeyExists( session.purchases, url.selectedPhoto )
		)>

		<!---
			For this demo, we don't want to re-charge the user for
			photos they've already purchased. As such, only charge
			them if the photo is not already in the collection of
			purchased items.
		--->
		<cfif !structKeyExists( session.purchases, url.selectedPhoto )>

			<!--- Deduct the credit from the balance. --->
			<cfset session.balance -= photoCost />

			<!--- Add the purchase record. --->
			<cfset session.purchases[ selectedPhoto ] = now() />

		</cfif>

		<!---
			Forward the user to purchase page where they can view
			and download the purchased photo.
		--->
		<cflocation
			url="./purchase.cfm?photo=#url.selectedPhoto#"
			addtoken="false"
			/>

	<cfelse>

		<!---
			The use doesn't have enough balance to make the purchase
			- set an error message to display.
		--->
		<cfset errorMessage = "You do not have enough credits." />

	</cfif>

</cfif>


<!--- ----------------------------------------------------- --->
<!--- ----------------------------------------------------- --->


<cfoutput>

	<!DOCTYPE html>
	<html>
	<head>
		<title>Using MOD_XSendFile With ColdFusion To Stream Files</title>

		<style type="text/css">

			img {
				border: 5px solid ##454545 ;
				margin-right: 10px ;
				vertical-align: middle ;
				}

			button {
				height: 40px ;
				}

			p.error {
				color: ##CC0000 ;
				font-style: italic ;
				}

		</style>
	</head>
	<body>

		<h1>
			Using MOD_XSendFile With ColdFusion To Stream Files
		</h1>

		<p>
			Credit Balance: <strong>#session.balance#</strong>
		</p>

		<!--- Check to see if there is any error to report. --->
		<cfif len( errorMessage )>

			<p class="error">
				<strong>Oh Noes!</strong> #errorMessage#

				<a href="#cgi.script_name#?init=1">Re-initialize!</a>
			</p>

		</cfif>

		<form action="#cgi.script_name#" method="get">

			<p>
				Please select the image to purchase:
			</p>

			<!--- Output a purchase option for each image. --->
			<cfloop
				index="photoIndex"
				from="1"
				to="15"
				step="1">

				<!---
					Notice that thumbnails are below the webroot
					(can be publicly accessed) but that the original
					image are NOT in the webroot and therefore cannot
					be accessed directly.
				--->
				<p>
					<img src="./thumbs/#photoIndex#.jpg" />

					<button
						type="submit"
						name="selectedPhoto"
						value="#photoIndex#">
						Photo #photoIndex# : #photoCost# Credits
					</button>
				</p>

			</cfloop>

		</form>

	</body>
	</html>

</cfoutput>

Once the user had made the purchase of a photo, they are forwarded to the purchase page. The purchase page doesn't actually stream the binary file to the client; rather, it acts as an intermediary landing page that gives the user some informational feedback. It's one of those "your download will begin shortly" pages.

Purchase.cfm

<!--- Param the photo id. --->
<cfparam name="url.photo" type="numeric" />

<cfoutput>

	<!DOCTYPE html>
	<html>
	<head>
		<title>Your Download Will Begin Shortly</title>

		<!---
			Forward the user to the download page. This is the
			page that will actually stream the file to client
			(the user's browser).
		--->
		<meta
			http-equiv="refresh"
			content="1; url=./download.cfm?photo=#url.photo#"
			/>
	</head>
	<body>

		<h1>
			Thank You - Your Download Will Begin Shortly
		</h1>

		<p>
			If your download does not begin within a few seconds,
			<a href="./download.cfm?photo=#url.photo#">click here</a>
			to proceed manually.
		</p>

		<p>
			<a href="./index.cfm">Return to Photos</a>
		</p>

	</body>
	</html>

</cfoutput>

This page then forwards the user onto the actual download page which is where we finally get to deal with code that relates to XSendFile. Notice that ColdFusion isn't actually reading in or streaming out any file.

Download.cfm

<!--- Param the photo id. --->
<cfparam name="url.photo" type="numeric" />


<!---
	Check to make sure the photo is in the user's collection
	of purchases.
--->
<cfif structKeyExists( session.purchases, url.photo )>


	<!---
		The user has made the purchase - add the full file path of
		the purchased image to the outgoing headers. By using the
		XSendFile header, the file streaming will be passed off to
		Apache (ie. it won't tie up a ColdFusion thread).

		NOTE: Apache will strip this header out - it will never
		reach the client.
	--->
	<cfheader
		name="x-sendfile"
		value="#application.imagesDirectory##url.photo#.jpg"
		/>

	<!---
		Deliver this purchased photo as an attachment so
		the "Purchase" screen will not be navigated away from
		(visually speaking).
	--->
	<cfheader
		name="content-disposition"
		value="attachment; filename=purchase-#url.photo#.jpg"
		/>

	<!--- Set the mime-type. --->
	<cfcontent type="image/jpeg" />

	<!---
		Exit the request - we don't really need to do this --
		this is here only to emphasize that ColdFusion is no
		longer doing anything in this request that is realted to
		the file delivery.
	--->
	<cfexit />


<cfelse>


	<!---
		The user has not purhcased this photo. Return an accessed
		denied header response.
	--->
	<cfheader
		statuscode="401"
		statustext="Unauthorized"
		/>

	<h1>
		Access Denied
	</h1>

	<p>
		You are not authorized to access this page.
	</p>


</cfif>

Before it does anything else, this ColdFusion page checks to see if the user has actually purchased the photo. Remember, that's really the reason we are here - to figure out how to create a secure-access layer in ColdFusion in which we can perform user-specific security checks. But, once the permissions have been evaluated, ColdFusion simply defines a new HTTP header, "x-sendfile." The value stored in this header is the absolute file path of the secure file that we are streaming. Once ColdFusion finishes the request, the XSendFile Apache module then examines the outgoing HTTP headers and streams the given file back to the client.

NOTE: The X-SendFile header never reaches the client. Once the XSendFile module evaluates the outgoing HTTP headers, it strips the X-SendFile entry and streams the file at the defined path.

I have to say, MOD XSendFile is pretty darn awesome. Admittedly, I like to keep my application as modular as possible; that is, I like to keep as much as the application configuration inside the application itself. But, when you see the kind of power and functionality that can be provided by things like XSendFile and URL Rewriting, it definitely makes you a lot more open-minded. XSendFile makes it a breeze to provide ColdFusion-powered secure file access without putting an undue burden on the finite collection of ColdFusion threads.

Want to use code from this post? Check out the license.

Reader Comments

3 Comments

This may be the first time I actually understood what you are doing on my first read-through. Really great example of integrating web server features with CF. Also, great example on how you integrate CF into Apache.

15,663 Comments

@Brandon,

Ha ha, awesome - hopefully that means I am getting better at explaining myself :) I think this stuff is really cool. I can definitely see a lot of opportunities to use this module. I've never been truly happy with secure file downloads in the past.

7 Comments

Thanks, Ben! I could have used this at my previous company, but now have this page bookmarked and indexed in Evernote so that I can figure out how to do it for my startup, as I know that file streaming will be needed.

I found your explanation easy to follow - you have always done a great job and continue to improve - we thank you!

10 Comments

Been doing this and similar for the past 4 or 5 years. :) FYI, writing an IIS module for IIS7+ that does this is not very difficult (c#).

Prior to IIS7 it's a... hassle (ISAPI filters and native code), so I've never bothered.

15,663 Comments

@Mark,

I haven't touched C# in a long time :) But, it's good to know that it's not a dead end in IIS. Glad this kind of approach has been proven over the past few years. I really like it!

27 Comments

Great idea!

We used the combination of mod_auth_cookie and Apache authentication to manage secure downloads, but your approach seems much better. Especially since mod_auth_cookie seems dead.

Thanks for pointing XSendfile.

1 Comments

My apologies if my question is a bit basic, but is it possible to have Apache and CF running in separate servers to get this to work?

BTW also wanted to thank you, Ben. Since I recently started working with CF, I must say that I always find great information in your blog!

3 Comments

Ben,

Are you sure the X-SendFile header never reaches the client? I am using this and it all appears to be working great...except for the fact then when looking at the headers in Charles the full path of the file is included in the header

15,663 Comments

@Jason,

That's really strange. I have not seen that. In my requests, the X-Send-File header comes through, but the value has been stripped-out. Are you sure XSendFile is actually processing the request?

15,663 Comments

@All,

Also, I just came across a strange problem on IIS + XSendFile. We're using XSendFile on IIS7 and I *think* we're using Ionic to do the integration. Any how, on IIS7, XSendFile does not work unless you have output="false" in some part of the request.

This could be on the application:

<cfcomponent output="false> ... </cfcomponent>

... or on the request start:

<cffunction name="onRequestStart" output="false"> ... </cffunction>

It doesn't seem to matter *where* the output=false is, so long as it's part of the pre-request processing. Doing so seems to trip the some of internal flag that allows XSendFile to work.

If you don't have the above, the binary that comes down from IIS has a space character as the first byte (which makes the files unusable unless you manually strip it out).

Again, this is only for XSendFile running under IIS7 as far as I know. I have not seen this error under Apache.

3 Comments

Ben,

Ok yeah that was the issue, it looked like the PDF was being downloaded but upon further investigation it was not. I changed a few config params and it is working as expected! Awesome stuff, thanks for the quick reply.

4 Comments

Hi Ben, I faced an issue that I partially solved with the streaming process you described in www.bennadel.com/blog/1227-Using-ColdFusion-To-Stream-Files-To-The-Client-Without-Loading-The-Entire-File-Into-Memory.htm

I wrote a SO question that defines the problematic which you can see at http://stackoverflow.com/questions/19385223/how-to-transparently-stream-a-file-to-the-browser

The solution works, but it will not support video seeking, which I was expecting, but it is a very important feature that we need.

So far I was unable to find enough information on the protocol used by Windows Media Player and IIS (default) and because of the controlled environment I work in, I couldn't inspect the HTTP request details to see what headers the client is sending to IIS and how the server responds.

The only feature we are interested in is seeking, not implementing a full media server with adapative bitrate. Do you have an idea how we could tackle the problem?

Thanks!

*NOTE: I cannot install any plugins on the server.*

4 Comments

I just thought about a different approach. Perhaps we could perform the HTTP request from ColdFusion to IIS by just cloning the current client request details and find a way to read from the http response stream write and flush to the client as we read from the stream.

Basically ColdFusion would just act as a proxy. I know that it's probably not possible with cfhttp however it probably is using Java classes.

Have you ever done something similar?

*NOTE: I have no idea if that would fix the video seeking feature however.*

15,663 Comments

@Alexandre,

That's a really interesting approach! I can't say that I know much of anything about media streaming, but I love how you're basically piping one response into another. Very cool!

I believe in love. I believe in compassion. I believe in human rights. I believe that we can afford to give more of these gifts to the world around us because it costs us nothing to be decent and kind and understanding. And, I want you to know that when you land on this site, you are accepted for who you are, no matter how you identify, what truths you live, or whatever kind of goofy shit makes you feel alive! Rock on with your bad self!
Ben Nadel