Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at the New York ColdFusion User Group (Mar. 2009) with:

Creating Semi-Secure File Downloads Without Using CFContent

By Ben Nadel on
Tags: ColdFusion

In the past, I have been asked several times how to create secure file downloads without tying up resources in the ColdFusion thread pool. Really, when you want to create a good, secure download, you would want to funnel the file requests through some sort of ColdFusion control logic and then stream a non-web-accessible file to the user via ColdFusion's CFContent tag. The problem with this is that the CFContent tag ties up a ColdFusion thread for the entire duration of the download. This works nicely, but ColdFusion shouldn't be responsible for streaming files - let IIS do that; ColdFusion should be freed up to handle the application logic and work flow.

One thing that I have suggested in the past was to copy files to a new, web-accessible location before forwarding the user directly to the file. This would put the onus of the download on the web server, not ColdFusion. It would create a somewhat non-secure download since the file is now in a public location, but we can take steps to make sure that that is a "good enough" solution.

I told people that this was a theoretical suggestion since I had never tried it out myself. But, curiosity got the best of me and I wanted to see if I could come up with a really simple solution for this problem. I think what I came up with is pretty decent. You start off with a directory of secure files that are above the web root. Then, you have a list of files that a user can download. When they click on a file link, we funnel this request through a ColdFusion action page that checks to see if the requested file is already accessible somewhere. If it isn't, it copies the file to a public folder which has a UUID directory name. Once the copy is done, the ColdFusion page forwards the user directly to the new file location so that IIS can handle the file download.

There's a bit more too it (not much more), but let's take a look at the set up first. Here is my Application.cfc file:

  • <cfcomponent
  • output="false"
  • hint="I set up the application and define the application-level events.">
  •  
  • <!--- Define application settings. --->
  • <cfset THIS.Name = "FileDownloadDemo" />
  • <cfset THIS.ApplicationTimeout = CreateTimeSpan( 0, 0, 10, 0 ) />
  •  
  • <!--- Define page request settings. --->
  • <cfsetting
  • showdebugoutput="false"
  • requesttimeout="10"
  • />
  •  
  •  
  • <cffunction
  • name="OnApplicationStart"
  • access="public"
  • returntype="boolean"
  • output="false"
  • hint="I initialize the application.">
  •  
  • <!--- Clear the APPLICATION scope. --->
  • <cfset StructClear( APPLICATION ) />
  •  
  • <!---
  • Get the file directory. This is where we permanently
  • store files on the server. This could be a non-web-
  • accessible directory.
  • --->
  • <cfset APPLICATION.FileDirectory = (
  • GetDirectoryFromPath( GetCurrentTemplatePath() ) &
  • "files\"
  • ) />
  •  
  • <!---
  • Store the path to the temp directory. This is the
  • directory to which we will be copying files when
  • users request a download.
  • --->
  • <cfset APPLICATION.TempDirectory = (
  • GetDirectoryFromPath( GetCurrentTemplatePath() ) &
  • "temp\"
  • ) />
  •  
  • <!---
  • When we re-initialize the directory structure, be
  • sure to wrap this in a CFTry / CFCatch since we
  • are dealing with a third party system that is out
  • of our control. If any of those files is currently
  • being downloaded, then it will be locked and the
  • delete will throw an error.
  • --->
  • <cftry>
  •  
  • <!---
  • Delete all the files in the temp directory if the
  • temp directory already exists (reinitialization).
  • --->
  • <cfif DirectoryExists( APPLICATION.TempDirectory )>
  •  
  • <cfdirectory
  • action="delete"
  • directory="#APPLICATION.TempDirectory#"
  • recurse="true"
  • />
  •  
  • </cfif>
  •  
  • <!--- Re-create the temp directory. --->
  • <cfdirectory
  • action="create"
  • directory="#APPLICATION.TempDirectory#"
  • />
  •  
  • <!--- Catch any errors. --->
  • <cfcatch>
  •  
  • <!---
  • Something went wrong with the file directory.
  • Mostly likely, this was not a "creation"
  • issue, but rather a deletion issue, so we can
  • be pretty sure that the Temp directory is
  • still in place.
  • --->
  •  
  • </cfcatch>
  • </cftry>
  •  
  • <!---
  • Create a struct to hold the file names of our
  • temporary files. We are doing this so that we can
  • resuse file without having to copy them over and
  • over again.
  • --->
  • <cfset APPLICATION.TempFiles = StructNew() />
  •  
  • <!--- Return out. --->
  • <cfreturn true />
  • </cffunction>
  •  
  •  
  • <cffunction
  • name="OnRequestStart"
  • access="public"
  • returntype="boolean"
  • output="false"
  • hint="I initialize the page request.">
  •  
  • <!--- Define arguments. --->
  • <cfargument
  • name="Page"
  • type="string"
  • required="true"
  • hint="I am the page template that the user requested."
  • />
  •  
  • <!---
  • Check to see if we need to manually re-initialize
  • the application.
  • --->
  • <cfif StructKeyExists( URL, "reset" )>
  •  
  • <!--- Manually execute initialization. --->
  • <cfset THIS.OnApplicationStart() />
  •  
  • </cfif>
  •  
  • <!--- Return out. --->
  • <cfreturn true />
  • </cffunction>
  •  
  • </cfcomponent>

When the application is initialized or re-initialized, we start off by clearing out all of our temp (publicly-accessible) files. Then we create a cache of file names. This cache is keyed using the requested file name and each item in the cache has the name of the randomly-named directory that houses each publicly accessible file and the date/time stamp on which that directory was created.

Then we list out each of our secure files (these files are not web accessible at this point):

  • <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  • <html>
  • <head>
  • <title>Copy Files For Download Demo</title>
  • </head>
  • <body>
  •  
  • <h1>
  • Copy Files For Download Demo
  • </h1>
  •  
  • <p>
  • To see Olympic training videos and pictures of
  • the beautiful and sexy Laure Manaudou, the French
  • Olympic gold metalist, please select from the file
  • list below.
  • </p>
  •  
  • <ul>
  • <li>
  • <a
  • href="get_file.cfm?file=large_1.avi"
  • target="Laure Manaudou Video"
  • >Laure Manaudou Video</a>
  • </li>
  • <li>
  • <a
  • href="get_file.cfm?file=large_2.avi"
  • target="Laure Manaudou Video II"
  • >Laure Manaudou Video II</a>
  • </li>
  • <li>
  • <a
  • href="get_file.cfm?file=small_1.jpg"
  • title="Laure Manaudou Swimming Picture"
  • >Laure Manaudou Swimming Picture</a>
  • </li>
  • <li>
  • <a
  • href="get_file.cfm?file=small_2.jpg"
  • title="Laure Manaudou Swimming Picture 2"
  • >Laure Manaudou Swimming Picture 2</a>
  • </li>
  • </ul>
  •  
  • </body>
  • </html>

Notice that each file link goes through the ColdFusion page, get_file.cfm. The get_file.cfm page checks to see if the requested file has been made publicly available yet. It does this be checking to see if the file is in the application cache. If it is not, then a public directory is created using a UUID name and the secure file is copied to the public directory and the name of the directory and creation date are cached in the application. If, however, the file is already publicly accessible, it checks the cache to see when it was created. In our demo (for testing purposes), if the directory was created more than two minutes ago, the public directory [that contains the file in question] is renamed to a new UUID and its new name and creation date are cached in the application. This renaming of the directory helps to keep the direct-file-access availability to a minimum while at the same time not having to re-copy the file every time it is requested.

Let's take a look at get_file.cfm:

  • <!--- Param the URL variables. --->
  • <cfparam name="URL.file" type="string" default="" />
  •  
  •  
  • <!---
  • HERE: You could implement any kind of security or logging
  • that you want. It won't be 100% fool-proof, but it should
  • be good enough if your files are not 100% mission critical.
  • --->
  •  
  •  
  • <!---
  • Check to see if the given file exists. If it doesn't, then
  • we just want to display an error message and exit out of
  • this page.
  • --->
  • <cfif NOT FileExists( APPLICATION.FileDirectory & URL.file )>
  •  
  • <!--- Redirect to 404 page. --->
  • <cflocation
  • url="./file_not_found.cfm"
  • addtoken="false"
  • />
  •  
  • </cfif>
  •  
  •  
  • <!---
  • Let's put a names lock on this file since we are reading
  • and writing from shared memory that might affect other
  • people's page flow.
  • --->
  • <cflock
  • name="#URL.file#"
  • type="exclusive"
  • timeout="60">
  •  
  • <!---
  • Check to see if this file has already been moved
  • over to the temp directory. If it has, we need to
  • check the date stamp on that as well. We only want
  • files to be available for a certain amount of time
  • before we create a new copy / name.
  •  
  • After a certain amount of time, we don't want this
  • file to be available. After a certain amount of
  • time, we want to make sure this file can no longer
  • be found at a directly accessible URL. However, we
  • don't want to have to re-copy the file since that
  • is wasted processing. So, if the file is no longer
  • available, we simply want to re-cache it under
  • another directory.
  •  
  • For DEMO purposes, our files we be available at
  • the previous URL for only 2 minutes (so I could
  • actually test that this was working).
  • --->
  • <cfif (
  • StructKeyExists( APPLICATION.TempFiles, URL.file ) AND
  • (APPLICATION.TempFiles[ URL.file ].DateCreated LT DateAdd( "n", -2, Now() ))
  • )>
  •  
  • <!--- Get a reference to the temp file. --->
  • <cfset objTempFile = APPLICATION.TempFiles[ URL.file ] />
  •  
  • <!--- Create a new directory name. --->
  • <cfset strNewDirectory = CreateUUID() />
  •  
  •  
  • <!---
  • Now, we need to rename the directory to prevent
  • further DIRECT access to the file. However, it is
  • possible that the file is in use and the system will
  • not let us rename it. Thefefore, we have to TRY to
  • rename the directory.
  • --->
  • <cftry>
  •  
  • <!--- Rename the directory. --->
  • <cfdirectory
  • action="rename"
  • directory="#APPLICATION.TempDirectory##objTempFile.Directory#"
  • newdirectory="#APPLICATION.TempDirectory##strNewDirectory#"
  • />
  •  
  • <!---
  • Now that we have renamed directory, store the
  • new name in the file cache.
  • --->
  • <cfset objTempFile.Directory = strNewDirectory />
  •  
  • <!--- Create the new timestamp. --->
  • <cfset objTempFile.DateCreated = Now() />
  •  
  •  
  • <!--- Catch any errors. --->
  • <cfcatch>
  •  
  • <!---
  • There was some problem with the directory
  • renaming. Therefore, just let the page
  • process and allow the user to access the
  • file. We'll have to try this the next
  • go-round.
  • --->
  •  
  • </cfcatch>
  • </cftry>
  •  
  • <cfelseif NOT StructKeyExists( APPLICATION.TempFiles, URL.file )>
  •  
  • <!---
  • This file has not been moved over yet. Let's create
  • an object to hold the temp file information. For
  • the temp directory, we are going to create a UUID
  • to make it very hard to guess the location.
  • --->
  • <cfset objTempFile = {
  • Directory = CreateUUID(),
  • DateCreated = Now()
  • } />
  •  
  • <!---
  • Create a short hand for our new temp directory
  • for the new file.
  • --->
  • <cfset strTempDirectoryPath = (
  • APPLICATION.TempDirectory &
  • objTempFile.Directory
  • ) />
  •  
  • <!--- Create the temp directory. --->
  • <cfdirectory
  • action="create"
  • directory="#strTempDirectoryPath#"
  • />
  •  
  • <!---
  • Copy the file from the file storage to our
  • new temp directory.
  • --->
  • <cffile
  • action="copy"
  • source="#APPLICATION.FileDirectory##URL.file#"
  • destination="#strTempDirectoryPath#\#URL.file#"
  • />
  •  
  • <!---
  • Create an entry in the APPLICATION scope so
  • that we know that this file has already been
  • copied over.
  • --->
  • <cfset APPLICATION.TempFiles[ URL.file ] = objTempFile />
  •  
  • </cfif>
  •  
  •  
  • <!---
  • ASSERT: At this point, we have either dealt with an
  • existing file or a new file. Either way, the file
  • now has an entry in the TempFiles cache.
  • --->
  •  
  •  
  • <!--- Get a reference to the temp file. --->
  • <cfset objTempFile = APPLICATION.TempFiles[ URL.file ] />
  •  
  • <!---
  • Redirect use to actual file so that we can let IIS
  • do the streaming rather than tying up a ColdFusion
  • thread.
  • --->
  • <cflocation
  • url="./temp/#objTempFile.Directory#/#URL.file#"
  • addtoken="false"
  • />
  •  
  • </cflock>

The concept is fairly straight forward - we are periodically moving accessible files to new directories so that if anyone has the direct-file link bookmarked, it will only be available for a certain amount of time. Now, in the above file, the file-move is triggered by user actions; you could, however, have some sort of a scheduled task that runs every 20 minutes or something and renames old directories. I try to avoid using scheduled tasks whenever possible and I felt that since this kind of technique would really only make sense on a high-traffic site, the user-triggered event would be more than sufficient.

At first, I was concerned that it would take too long to copy files from the private directory to a public directory, but it was surprisingly quick. In my demo, the video files were about 730 Mega bytes and they copied over, on my slow local development server, in about 24 seconds. This seems like a long time for a SAVE AS prompt to come up, but realize also that we are only incurring that time cost on the first file request (for that file). All subsequent file requests will be made directly to the publicly accessible, but randomly located file. Smaller files, like the JPGs were copied over instantly.

Again, this is a "good enough" solution. It is not mean to be ultra, super secure. If someone wanted to, they could get a publicly accessible file link and email that to a million people who could all start downloading. But really, that's not what we're worried about - those people could just as easily download the file and send the file to a ton of people. Really, what this solution is for is to keep honest people honest and do as much as possible with as little effort to get people to go through standard procedures to download files.

Now, as much as this is less theoretical, since I have tried it out myself, I am still not sure that I would recommend this. I know very little about high-traffic download sites. I am sure that there is a lot more to consider and many patterns, techniques, and best practices have already been discovered for this type of problem. This, however, is what I could come up with this morning.




Reader Comments

My only complaint is that it's not 100% secure. There is a way to secure downloads but Adobe has to make a chance to cfcontent to do so. Right now, in order to cfcontent something down to the end user's client, they read the entire file into memory. There's no such thing as a buffer or anything? :P Anyway, I did put as an enhancement request for CF9 and if you want the code that I researched / demo'ed, I'd be happy to share it.

Reply to this Comment

@Todd,

Are you saying that you don't like CFContent because is ties up a ColdFusion thread? Or because it reads the whole file into memory? I am curious to which of these issues your enhancement addresses? Also, did you come up with a way to do it in CF8? Or was your demo theoretical?

Reply to this Comment

Well, I hate both aspects of CFContent. :) The fact that it has to load the entire file into memory instead of just buffering and streaming the buffer is just silly. Then again, that would make CF a streaming server if they changed it, wouldn't it? Probably why they didn't do it. As for cfcontent taking up a thread, for the sake of security, I'll deal with it, just buffer/stream the file appropriately to make the memory aspect of it at little more friendly. They did it for cffile's upload, they can certainly do it for downloading.

The code I have does work and does buffer/stream. I have not load tested it at all to see if it was on the right path. Said code is at home at the moment (for some reason), so I'll have to get back to you on it. If you're impatient, you can go here ( http://www.realitystorm.com/experiments/flash/streamingFLV/index.cfm ) and check this code out yourself and rewrite it to make it a little more generic (which is all I did).

Reply to this Comment

@Todd,

I actually have used that Christian Cantrell post before, especially in pre-MX7 days before the CFContent tag had a Variable attribute for binary content. When the CFContent added the Variable attribute, I stopped using this. But, this didn't stream anyway, this was just a way to send binary data to the client.

The Reality Storm post does a similar thing, but rather than writing the whole binary content at one time, it is periodically reading in the flushing the content.

So, I guess the benefit here is not that it frees up a thread, but rather that that thread in question is not hogging memory?

Reply to this Comment

Correct. CFContent as it stands reads the entire file into memory to send it down to the user. With the code that I had written, I was able to create a buffer of 10 megs and stream that. So, only 10 megs of continuous memory was being used for that one request. So, if my client has 200 meg PDF files (and, I could show you a few), then it's not going to shove everything into memory, it's only going to buffer 10 megs at a time. That 10 megs was arbitrary so, I could set it to anything I want.

What I would be curious about and I doubt if Adobe is going to answer this. If the 1st request requested my 200 meg 'abcdef.pdf' file and a 2nd request came in for the same pdf, does it check the memory for it first (since it's already there?) or is it going to get a new copy of the file? If it's going to the memory first, then cfcontent's as it exists today is probably the more efficient way to do it. Since we don't want 400 megs lingering in memory, we want 200 megs being served out to 2+ requests.

This whole cfcontent thing is a mess because I have written a document library for clients. I need absolute tracking / security and I need cfcontent to serve out that file, but not at the expense of killing the server.

Reply to this Comment

An off topic nit pick - but important. Your onRequestStart has:

<cfset THIS.OnApplicationStart() />

Don't forget that when you call CFC methods using this.X, it acts like an "outside" call. If X were private, you would get an error. You should do

<cfset OnApplicationStart() />

instead.

Reply to this Comment

@Todd,

I would doubt that it is checking the memory for existing buffers. I think the use case of two users downloading the same file concurrently is probably fairly small, unless its something like a Product download on a hugely popular site?

I hear what you are saying about the document library. I think that is where we all ultimately want to go with a scenario like this. I'd like to give that code-write a go. I know you have written it already, and that other site has a version, but I love to flex my brain muscles :)

Reply to this Comment

@Ray,

Yeah, that is weird habit of mine. I love to see things scoped. What's ironic is that it works against me sometimes. If the method were private, you could call it like this:

<cfset VARIABLES.Method() />

Sure, if the access changes, then you have to change where it is accessed, but I am not so worried about that. What really bothers me, though, is that you can't use Named arguments with private methods:

<cfset VARIABLES.Method( FirstName = "Ben", LastName = "Nadel" ) />

This throws an error for some reason saying that it can't be called using named arguments and to try calling it using ordered arguments. Of course, if it were THIS instead of VARIABLES, this works just fine.

So, that irks me. But really, you are right, there is no need to scope these methods at all. I think I did it to demonstrate that they were not built-in ColdFusion methods, but it's probably something that I will drop eventually.

Reply to this Comment

The other option is to integrate the web tier with your application's security. It can be a pain, but I've written an ISAPI filter that, before sending a file, delegates a security check back to the CF app via. a separate http request (you could also communicate via. db or just about any other mechanism - both processes are on the same server). Keep in mind that ISAPI (or the Apache equivalent) has access to session cookies set by your CF app.

Also keep in mind that app server threads aren't as expensive as they used to be, especially if all they are doing is chunking bytes to an output stream.

Reply to this Comment

@Dave Ross: Also, majority of my files have some kind of security restriction so they're not web accessible at all which is why cfcontent is cool, because you can put it wherever and read it from that location as long as the permissions are in place. How would Apache/IIS even go after those if it's not defined globally in some settings or setup as an alias (which defeats security)?

Reply to this Comment

@Todd - you would use a virtual directory within the webserver that has the filter applied to it.

1) Make a virtual directory that maps to where you keep your files
2) Apply the ISAPI filter to that virtual directory

The filter would redirect someone to whatever your login page is (or just return a 403) if someone wasn't logged into the application (or your application tells the filter that they shouldn't access that particular file.

@Ray - I'll do my best. After all was said and done we ended up using cfcontent, and adding threads and hardware.

Reply to this Comment

This is a very interesting topic to me. I'm working on a document library as well. I need full security to make sure person in department a isn't looking at files they shouldn't be seeing in department b. I'm using cfcontent at this time, but I'm concerned about the memory usage too.

I like Dave Ross' idea about an ISAPI filter, but even a CF solution that keeps the memory footprint small per download would be workable. (Even at the expense of being a slightly longer download time)

I like Ben's original solution on this post, but only if a cleanup is done afterwards to remove the temp file to retain security.

Dan

Reply to this Comment

@Dan,

The problem with clean up is that from a Server standpoint, you have no idea how long the download will take. You'd have to keep checking the file to see if its locked.

Reply to this Comment

I'll chime in and add that we actually handle secure file downloads on another server (although not necessarily so) with a PHP script. My ColdFusion download page creates a encrypted reference to a primary key in the DB and passes it to the browser with a redirect to a PHP script with the encrypted reference in the URL.

The PHP script then parses the URL, makes a call back to the CF-powered app tier via web services which will return the authorization to download (by giving a file path outside of the webroot). The PHP script at that point reads and streams the file to the client with the appropriate HTTP headers.

The reason for this approach had to do with scaling existing infrastructure (our problem was that downloads were thread-bound in the app-tier). Move the download process to PHP (on a different server in our case) and your shortage of ColdFusion threads clears right up!

Reply to this Comment

If you are running on a Unix box, you can use <cfexecute> to create a Symlink (ln -s /path/to/sourcefile /path/to/tempfile), redirect the user to this file, wait a few seconds (<cfthread action="sleep" duration="5000">), then remove /path/to/tempfile.

As long as the user has started downloading the file, they will receive the full file, and in addition you don't have all the disk IO involved in copying the large file, and you also don't have to worry about other users being able to snag the file even 10 seconds later.

The way that most Unix filesystems work, files can be deleted even while there are open read handles on the file, the file is not cleaned up from the disk until the last read handle is closed - though no new read handles can access the file, for users not already reading the file it's effectively gone.

Reply to this Comment

@Eric : Yeah, I remember looking at that when I was researching all this. I even looked into something for windows as well (it's a mess on windows, not even worth it).

Reply to this Comment

We've had this problem for a while. To enable secure downloads without having to have temporary files I wrote a c# asp.net page. The page communicates via webservice with the coldfusion application and then returns the file.

The asp.net page needs to run under IIS.

1. Create a webservice in coldfusion. The service takes information about the download request and determines if the current user in the session has access to the specified file.

2. create webservice stub in C# and compile it to a dll

3. create a .aspx page in the same directory as the webroot of the coldfusion application.

Something like

cfapp\wwwfiles\index.cfm
cfapp\wwwfiles\download.aspx

4. using the webservice stub, the aspx page adds the cfid/cftoken cookies to the request to the webservice and asks the cf app if the user is authorized.

5. The .aspx page either shows an error or returns the file.

Reply to this Comment

@JohnEric - Doesn't this just shift the tied up request from CF to ASP? You still end up with a thread somewhere doing nothing but reading the file and passing it to IIS which does nothing but read from your middleman and send it to the end user. The idea is to get ourselves out from the middle of the transfer work to free up server resources.

Reply to this Comment

A thread is a thread - there will always be one per ongoing download. The problem is that with CF it's tough to differentiate between threads and prioritize them effectively.

So, another option is to run CF enterprise, and have a second "content hosting" instance which has a high number of request handling threads (hundreds if not thousands). You would stream using standard java io (example was posted above), and communicate with the "application" instance via. webservices to establish security.

Reply to this Comment

@Eric : 1 questions about using cfthread to delete the file after redirecting the user:

1. how are you making sure that the cfthread continues after the user has been relocated - are you starting and sleeping the cfthread before the cflocation occurs?

2. using this technique, is the server-client connection completely cut off by the relocation? I ask as I wonder what happens to any remaining output after the cflocation? (not that there should be any, but I'm curious)

thanks!

Reply to this Comment

@jdbo: You don't have to maintain a read handle on the file from CF, once the user follows the redirect to the file, Apache or whatever your web server is will obtain the read handle in order to deliver the content to the user.

You just have to make CF sleep for long enough that you're certain the user will have started their request for the file before you delete the symlink. My example was 5 seconds - even a very slow connection should have been able to make that round trip in this time.

Here's my lots of caveats: You might not be able to use <cflocation> since CF might not send the Location header until after page processing is done (ie, after you deleted the file), and this might even depend on what web server you're using. You might have to resort to a meta redirect or javascript redirect, <cfflush>, then sleep 5 seconds and delete. Further, this might be interfered with if you use mod_gzip or the like too (which might wait for more response data in order to complete a compression unit to send to the browser), in which case you might have to send a few K of random data in a HTML comment after your redirect commands.

The difficulty with this method is convincing the browser that it needs to start reading the new file while you're still inside CF's thread sleep. You can rest easy that once the browser starts to download the file, it's safe to delete the symlink.

Reply to this Comment

@Eric: that helped clear things up immensely - I was make exactly the wrong assumptions (thinking that cflocation and cfthread would work together happily), and you've just saved me a lot of time, so thanks much!

Reply to this Comment

Threads/Memory/CFContent - yes memory is a big issue but I actually ran into the threads problem with weekly download of an updated schedule published as PDF. The PDF was very small but since the update was always published at the same time and everyone went to get it at that time the threads were consumed quickly. Even with a small file it just takes a few people with very slow connections to bog things down.

My solution is similar to yours - if file is public then I just copy it to a simple named download dir and redirect. If private then I create a temp download dir with UUID style name and copy file there and redirect. Dirs are flushed based on comparing create time stamp and a settable value. Although not completely secure is does also allow people to use file download utilities.

A really nice solution (especially with large files) is to use Amazon S3 to store the files and set an expiring link. S3 gives you unlimited and cheap storage, delivers the files fast (scales on demand) and of course handles the link expiry.

Reply to this Comment

@Ben

Interesting approach. We needed something similar to deal with presentations, video and audio for the conferences and I found AdmitOne which seems to work pretty well.

http://www.qwerksoft.com/products/AdmitOne/

I really wish CF was smarter about how to handle these though. I'm reasonably sure that while it doesn't load the entire file into memory, it does block a CF template request while it's running.

Rails (on mongrel) does this much smarter. There's a function that returns a file handle directly back to the web server and the web server streams the file completely separate of the rails application and your code.

To my knowledge CF still ties up a CF thread with requests... or does it do what mongrel/rails does? Can someone confirm this?

Reply to this Comment

I just read up on the Rails feature - and it's actually not Rails but Apache and Lighttpd that look in the response for a special header - very slick.

http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/

This should work w/ CF on Apache too... quick someone try!

Reply to this Comment

Wow, that's awesome!

I was actually thinking of another feature where rails passes an open file descriptor back to mongrel and it streams to the client. Rails isn't thread safe so it blocks the mongrel process I think, but a different framework can technically have more than one client streaming files and handling requests at the same time.

X-SendFile definitely appears to the best way to handle this though. I wonder if IIS has a feature like that...

Reply to this Comment

...snip...

"Rails (on mongrel) does this much smarter. There's a function that returns a file handle directly back to the web server and the web server streams the file completely separate of the rails application and your code."

...snip...

Sounds like this should be a new feature in CF9

Reply to this Comment

@Kurt: The web server it's running on has to support it. I believe that web servers which already support it can be interacted this way by CF - it only requires crafting a specific response header, so can probably be done in one or two lines of code.

Reply to this Comment

@Dave - I just tried mod_xsendfile for Apache and it works perfectly. I'm so glad you found this!!! My CF script hands things off to apache for serving so fast it's sick!!! And now my CF threads are free to do real work instead of being tied up serving files, woo hoo!!! I have been looking for a solution like this for a while, I knew it had to be out there! Thanks again!

Reply to this Comment

I looked for a while and didn't find anything. I'd really like to roll an ISAPI filter for this - it wouldn't be that hard but I'd be concerned about streaming the file properly with my (poor) C++ coding abilities.

Reply to this Comment

@Dave Ross,
I had the same idea, but never knew if it could be accomplished.

@all,
Thanks for those great ideas!

@Ben,
So many ideas on a single page because of your post, Thanks much!

Reply to this Comment

Hey Ben,

I realize this is an old posting but... I am using this technique to secure documents. We have two PDFs that just so happen to have '<cfc' in the code, when viewed in a text editor. This is causing CF to choke giving me the following error. Any ideas on how to get around this?

Invalid token  found on line 5148 at column 682. <p>The CFML compiler was processing:<ul><li>A cFC tag beginning on line 5148, column 679.</ul> <br>The error occurred on line -1.

K?#<d =1|(<cFC

Thank you

Reply to this Comment

@Raymond,

I am not exactly sure what you mean, I am just trying to open the PDF. I click on a link that goes to 'someFile.pdf.cfm' and I get that error.

Also, 'onRequestStart' does not seem to be firing before this error happens, so I am not sure how I can intercept it.

Reply to this Comment

Um, I don't know then. If you look at Ben's code, it copies a file and uses cflocation to send you to it. Can you confirm that the file copy worked fine? Also, if you are being sent to a PDF, the onRequestSart would obviously not fire for that request since it isn't a CF one.

Reply to this Comment

@Raymond,

It turns out that there was an issue with the PDF itself. We recreated it and all seems to be working now.

Thank you for the help Ray.

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.