My First ColdFusion 8 CFFTP Experience - Rocky But Triumphant

Posted August 29, 2008 at 9:46 AM

Tags: ColdFusion

Yesterday, I performed my first ever ColdFusion CFFTP task. I've needed to perform FTP tasks from ColdFusion before but required sFTP functionality, which was only added in ColdFusion 8. As such, until now, I have only ever used third-party utilities. I got the ColdFusion 8 secure FTP to work, but it took me a while to figure it out all. I am sure others will run into similar road blocks, so I thought I'd share my learnings.

The first thing we want to do is create a large test file to upload (PUT) to our FTP account:

 Launch code in new window » Download code as text file »

Here, we are just creating a file path (to be used throughout this example) and writing about 6 MB of data to it. It is important that the file be large because that's when things get a little bit more exciting.

The next thing we need to do is define our connection properties. We could do this inline with the ColdFusion 8 CFFTP tag, but using the AttributeCollection gives us the ability to cache the connection information in our site's configuration mechanism:

 Launch code in new window » Download code as text file »

Ok, now that we have our test file and our sFTP configuration object in place (for use with the tag's AttributeCollection), let's go ahead and try to upload (PUT) the file to the remote FTP server:

 Launch code in new window » Download code as text file »

There's a couple of things going on here. First, we are naming our sFTP connection, "objConnection." By doing this, it gets ColdFusion 8 to cache to the connection to the FTP server. This allows us to execute additional FTP commands on the cached connection without using our configuration attributes. That is why only the OPEN command uses the AttributeCollection; the PUT and CLOSE commands simply refer to the name of the cached connection.

Simple enough, right? Unfortunately, this does not work. When running the above code, ColdFusion throws the following exception:

An error occurred during the sFTP putfile operation. Error: putFile operation exceeded TIMEOUT. java.io.IOException: putFile operation exceeded TIMEOUT.

Ok, that make sense; I'm uploading a 6 MB file and the default timeout for FTP commands is 30 seconds.

To fix this, I went to increase the timeout of the PUT command to 300 seconds (5 minutes):

 Launch code in new window » Download code as text file »

The connection is really fast so 5 minutes should be no problem for 6 MB. However, when I integrated this PUT file with the above example, I got the same exact error:

An error occurred during the sFTP putfile operation. Error: putFile operation exceeded TIMEOUT. java.io.IOException: putFile operation exceeded TIMEOUT.

Maybe 5 minutes wasn't enough. I tried pumping it up to 8 minutes. Still no luck.

After pouring over the ColdFusion live docs for a while, it suddenly occurred to me: the timeout attribute doesn't go on the PUT command. I don't know if this is true for one-off commands, but when we are using a ColdFusion cached connection (named connection), the Timeout attribute must be included in the OPEN command. This way, the Timeout will define the timeout of all commands executed on that cached connection.

So, no problem, I went ahead and put the Timeout in the OPEN CFFTP command:

 Launch code in new window » Download code as text file »

I was feeling confident when I refreshed the page. As it was running, I monitored the progress of the FTP upload using FileZilla. Every few seconds, I would refresh the remote directory listing in FileZilla to see how big the file size was. When it got to 6 MB, I was excited - it has worked! But then, I was shocked to switch back to the web page and find the following error:

The request has exceeded the allowable time limit Tag: cfftp. The error occurred on line 210. An error occurred during the sFTP putfile operation. Error: putFile operation exceeded TIMEOUT. java.io.IOException: putFile operation exceeded TIMEOUT

When I saw the line:

putFile operation exceeded TIMEOUT

... I assumed the FTP just timed out again. So I increased the Timeout value and ran it again. And, once again, the same problem - the file finished uploading according to FileZilla, but the page kept reporting that the PUT command timed out. What was going on? I started looking at the log files and messing with the timeout value and going back over the documentation.

What had I missed?!? It seemed to be both working and failing at the same time?

Then it occurred to me! The above error is actually telling us two things: the PUT file exceeded the timeout and the request had exceeded the allowable time limit. Of course! The request! This time, it wasn't the FTP command that was timing out, it was the page itself. What was happening was that the ColdFusion sFTP PUT command was finishing properly, but when it returned, the executing page timed out.

The fix for this was simple:

 Launch code in new window » Download code as text file »

By increasing both the timeout of the cached CFFTP connection as well as the timeout of the page, everything went smoothly.

So, it was a bit of a rocky start with ColdFusion 8's CFFTP tag, but once I got all the kinks ironed out, I had some pretty powerful functionality.

I won't bother putting this in the example, but after I was done testing, I moved this into a CFThread tag so that it could run asynchronously as part of a list of scheduled tasks. By putting the CFFTP command in a CFThread, it eliminated the need for the CFSetting tag. ColdFusion threads launched using CFThread do not have a timeout; therefore, only the timeout of the CFFTP connection needs to be considered.

Download Code Snippet ZIP File

Post Comment  |  Ask Ben  |  Permalink  |  Other Searches  |  Print Page

• Add To Del.icio.us
• Add To Digg
• Add to Spurl.net
• Add to Wists
• Add to Simpy
• Add to Newsvine
• Add to BlinkList
• Add to Furl
• Add to reddit
• Add to FARK
• Add to BlogMarks
• Add to Yahoo! My Web
• Add to smarking
• Add to Ma.gnolia
• Add to Segnalo

Reader Comments

I have a small nitpick:
"There's a couple of things going on here. First, we are naming our sFTP connection, "objConnection." By doing this, it gets ColdFusion 8 to cache to the connection to the FTP server."

The way you wrote this makes it seem like naming your connection "objConnection" is doing something magical. Maybe it's the use of the quotes. I also don't think it's a 'caching' mechanism as much as it is a naming of a thread. Connection is an optional thing, but if you don't supply a connection name, CF will make one for itself.

Quote from the CFDocs: Name of the FTP connection. If you specify the username, password, and server attributes, and if no connection exists for them, ColdFusion creates one. Calls to cfftp with the same connection name reuse the connection.

The biggest thing I ran into when I upgraded to CF 8 was that CF 8 required the connection attribute but the error you get without it "Null, Null" doesn't give you a clue that that is the problem.

@Todd,

I was just referencing some of the documentation:

When you establish a connection with cfftp action="open" and specify a name in the connection attribute, ColdFusion caches the connection so that you can reuse it to perform additional FTP operations. When you use a cached connection for subsequent FTP operations, you do not have to specify the username, password, or server connection attributes. The FTP operations that use the same connection name automatically use the information stored in the cached connection. Using a cached connection helps save connection time and improves file transfer performance..... Changes to a cached connection, such as changing retryCount or timeout values, might require reestablishing the connection.

Maybe I am using weird wording, but I was trying to get the above sentiment across. And, as I am new to the CFFTP world, I am not aware of all the ins-and-outs.

@Ian,

Yeah, the Null Null error on anything is frustrating :)

Thanks for writing up your experience with this. I just migrated to CF 8 and had this exact problem. Your instructions on solving it saved me a lot of time figuring it out.

@Chris,

Glad to help man!

To Todd and other readers, the purpose of naming your connection is so that you actually know which connection to refer to when making calls (e.g. lets say you have multiple FTP calls happening simutaneously). If you allow CF to create/name the call, you wont know which one to refer to during multiple FTP calls.

HTH,
Tom

As always, thanks for figuring this out for us!

Hi Ben

I was reading this post now and I saw a reference to the "retryCount" property of CFFTP.

I did some research into this and I must admit the documentation relating to this property is very minimal.

What I am wondering is:

1) Does this ONLY apply to the "open" action of CFFTP or other tags, as I've only seen it referenced in the "open" action.

2) If so, does this simply ensure that if a connection can't be established, that it tries [retrycount] number of times to establish the connection
OR
Does this mean that ANY CFFTP action attempt that fails (e.g. action="putfile") will retry [retrycount] number of times??

Any idea?

I was just laying in bed thinking about how to handle FTP file resumes if a file upload should get disrupted somehow and then I saw this post and reference to the "retrycount" property and was wondering if this is what I need? :)

Thanks

Thanks for this post, I was about to shoot myself in the face.

Thanks Ian for your post. you saved me a lot of hair,
ie. The biggest thing I ran into when I upgraded to CF 8 was that CF 8 required the connection attribute but the error you get without it "Null, Null" doesn't give you a clue that that is the problem

Hi Ben,

Here's a great post on CF8 and a file size issue here: http://coldfused.blogspot.com/2007/08/coldfusion-8-changes-with-file-upload.html

I wasn't sure how big a file CF8 would manage to transfer without timing out as i've had troubles with reading large 400meg xml files.

Any ideas on how to receive a response on the progress of the file?

Cheers

@Leigh,

I am confused as to what you are asking. This post was about FTP, but the link you listed was about HTTP. Are you asking about timing out in FTP or HTTP?

@Leigh

Dude that's insane to have such a huge XML file, especially since you are just reading it. Why don't you just generate your XML file from a database? It would save you so much processing time as 400 megs of XML is really insane I have seen some huge XML files but 400? Dude you need to rethink your structuring a little bit better I know that is killing your server when you request that file.

Hi All,

I should be a bit more specific.

I am looking for a file transfer solution through CF to upload a single file at a time (20-100megs). Looks like i'll use CFFTP to do after reading some posts.

The article i posted was mentioning issues with IIS and Windows server 2003 which i thought might effect the CFFTP but i could be wrong?

I've had trouble in the past with CF timing out when reading large xml files, but haven't attempted to copy large files at all.

I have a current client system that generates a massive XML file of property data (full push) which came to 400meg! This is a seperate issue i face in CF as i believe a COM object on a windows server is a solution after doing some research on how to read large files. *The xml is also encased inside a zip file full of images.

I've found i can only read into CF around 3.7 meg of XML data in 10mins before it times out.

Leigh

Is there anything else that must be set-up/configured to allow ColdFusion 8 to establish a secure FTP connection? I have set-up multiple tests with various code configurations, each test results in a "User Authentication failed" error. I can connect to the secure FTP server using CoreFTP or psFTP which confirms my user parameters, server, and port are correct.

@JFC,

From the docs, it looks like all you need to do is set secure="true" in the CFFTP tag. If you need to get more complex, it looks like you might need to generate some secure private key stuff:

http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=Tags_f_18.html

I have not worked with SFTP very much.

Handy article..Ben

you may also want to set the fingerprint parameter to the cfftp tag for SFTP..

<cfsetting requesttimeout="600" />

<cfftp action = "open"
username = "***"
connection = "**********"
password = "*********"
fingerprint = "40:12:77:67:43:bb:2f:71:06:b8:3b:b7:ee:v5:ee:31"
server = "111.11.111.111"
secure = "yes"
timeout="300">

use Bitvise Tunnelier to get the fingerprint for your SFTP login

Ben,

Not sure if you're familiar with mainframe files or not, but here goes. One of our processes FTP's a file to the mainframe, but one caveat is that we have to pre-allocate some of the parameters for the file, with CF8 they introduced a handy new action called "site" which would allow you to put in syntax to be run on the ftp server, so in this instance I'm running the following:

<cfftp connection="Myftp"
action="site"
actionparam="LITERAL SITE LRECL=2088 RECFM=FB CYLINDERS PRIMARY=3 SECONDARY=1">

Works great in CF8, the problem is that our production environment is still running CF7 and, unfortunately, we do not have plans to upgrade in the near future. Do you know of any similar commands in CF7 that I could use to accomplish this task?

Any guidance would be much appreciated.

Chuck

@Chuck,

Unfortunately, that is beyond my understanding. I've never even seen the Site feature before. If I come across anything, I'll let you know.

Hi Ben! I do not if i should ask this question or not, but i am bit in complexity.

Can we use cfftp to upload multiple files and insert the same in the database too at the same time.

is This possible.

Ben,
I've been able to make the ftp connection to the server ok. but I'm having trouble uploading a file to a remote server. I have the correct path on the remote server and that looks ok. what I'm having trouble with is I use a form to look up the file I want to upload, which give a honking long filename, nothing like the real filename. I always get an error that says it can't make the file. any suggestions on what's going on here?

@Terri,

The most common problem people have with uploading files in CF is they fail to account for the cffile variable. This article sums it up pretty nicely: http://www.quackit.com/coldfusion/tutorial/cffile_parameters.cfm. Check your code and see if this might be the issue. The tip was the "honking long filename." :)

Terri,

Just double check you have the correct permissions on the folder..sometimes easy to overlook

You could test with a normal upload function to see if you can write files ok

@Terri,

If it is giving you a really long file name, it's probably giving you the .TMP file path (extracted from the FORM). What you need to do is actually save the file to disk first (using CFFILE) and then upload the destination file.

Chris and Ben,

Have you guys tried to secure ftp into a unix box using just the username and pub/pivate key access methods?

@Tony,

I have not tried that.

Tony

I use the secure ftp to transfer video files from a windows server to a linux server and this works like a dream..

I do pass in the fingerprint parameter...but I guess the determining factor is the sftp service you have setup on your unix box as to how you can get this to work with the cfftp tag.

@Ben, ok thanks....

@Chris, are you using the "username/key" instead of "username/password"???

Hi Tony,

we are using username/password/fingerprint.

Sorry if the previous post was unhelpful.

@Chris, any input on this is helpful. I appreciate the response!

Thanks for the post. I spent like 3 hours trying to figure out why it kept on timing out. But it was worse for me, on our Dev server it worked right away, but on the test I kept on getting a read time out error. But when I put the longer timeouts in the open connection and on the calling page seems to have solved the problem.

@Roman,

Glad you got it working; ColdFusion is sometimes funny that way, in that it won't timeout while a 3rd party interaction is executing; it will only timeout once that process has returned.

Post Comment  |  Ask Ben

Post A Comment

Please review the following issues:

Author Name:
Author Email:
Author Website:
Comment:	Formatting: <strong>bold</strong> <em>italic<em>
	Remember my information
	Subscribe to entry *
	Send me a copy of this comment (as part of the comment thread)

 

* If you subscribe, any new posts to this thread will be sent to your email address.