ColdFusion CGI Variables, Hot Linking, And Poor Documentation
Posted October 4, 2006 at 8:46 AM by Ben Nadel
A HUGE part of what project Skin Spider does is use the ColdFusion CFHttp tag to grab image and video files off of public sites for local storage. For anyone who has ever tried to use CFHttp to grab an image, a good amount of time it returns with a 403 : Forbidden Access error. This is done by the remote server as a "security" feature that stops users from hot linking their photos (linking to them across domains). To over come this, you can trick the remote server into thinking that you are using it locally by supplying a fake referer CGI variable:
- <!---
- Grab the remote image to save locally. In this example, we
- are grabbing the image as a binary and then writing it to
- disk.
- --->
- <cfhttp
- url="http://matureladies.adultbouncerhost.com/0007/mat1.jpg"
- method="GET"
- result="objSpider"
- resolveurl="yes"
- useragent="#CGI.http_user_agent#"
- getasbinary="yes"
- throwonerror="yes">
-
- <!---
- Trick the remote server into thinking you are browsing
- it's pages. Do this by sending a LOCAL url (local
- relative to the remote machine) as the referer.
- --->
- <cfhttpparam
- type="CGI"
- name="http_referer"
- value="http://matureladies.adultbouncerhost.com/0007/index.html"
- encoded="false"
- />
-
- </cfhttp>
However, if you run that code above, you will see that it actually fails. It returns a 403 : Forbidden Access error. The generated content of the returned page is:
You must supply a local referer to get URL '/0007/mat1.jpg' from this server.
What gives? We supplied the proper HTTP_REFERER value? After all, we gave it the video gallery that the image came from. I have dealt with this a number of times and have never gotten it to work. LUCKILY, just this morning, I stumbled across an old old comment from 2002 on the Macromedia web site about CGI variables and CFHttParam tags.
The comment explains that the CGI variables sent in the CFHttpParam tag are NOT verbatim the CGI variables that are exposed by the CGI scope. Now, as someone who first learned about CGI variables from the CGI scope, this was totally news to me. Here is the comment he placed:
This page fails to mention that when passing a value for the name attribute when trying to use type="cgi", the name is NOT the same as the CGI variable name as CF exposes it. For instance, if you want to pass in a "referer", you wouldn't say CFHTTPPARAM Type="cgi" Name="http_referer" Value="somevalue".
Instead, you need to drop the "http_" and just say Name="referer". And for headers/cgi variables that have multiple parts to their name, such as HTTP_Accept_Encoding, note too that the value to use isn't just "Accept_Encoding".
Instead, you have to use the formal name for the header, as speced at the w3c site among others, http://www.w3.org/Protocols/HTTP/HTRQ_Headers.html. In this case, you'd use Name="accept-encoding", which is typical for their multi-word values.
It would be nice for the CFMX docs to clarify this, as I'm sure many pull their hair out trying to get this to work, relying on their knowledge of these "CGI" variables solely from how they appear in the CF debugging info or a dump of the CGI scope.
Wait, what? Yeah, when you pass in a CGI value for HTTP_REFERER, all you pass in is REFERER as the CGI name. That's crazy. I checked the new MX 7 documentation and I didn't see anything about this. Thank God I found it on a page back from 2002!
So, updating the code above, we get:
- <!---
- Grab the remote image to save locally. In this example, we
- are grabbing the image as a binary and then writing it to
- disk.
- --->
- <cfhttp
- url="http://matureladies.adultbouncerhost.com/0007/mat1.jpg"
- method="GET"
- result="objSpider"
- resolveurl="yes"
- useragent="#CGI.http_user_agent#"
- getasbinary="yes"
- throwonerror="yes">
-
- <!---
- Trick the remote server into thinking you are browsing
- it's pages. Do this by sending a LOCAL url (local
- relative to the remote machine) as the referer.
-
- NOTE: I am now just passing "referer", NOT the original
- "http_referer".
- --->
- <cfhttpparam
- type="CGI"
-
- name="referer"
-
- value="http://matureladies.adultbouncerhost.com/0007/index.html"
- encoded="false"
- />
-
- </cfhttp>
I run that an Blam! It works like a charm. Hot linking has been circumvented and the image is now in the objSpider.FileContent as a binary object. I agree with the comment the dude make above; there are a lot of people out there, including myself, who learned about CGI from ColdFusion. I don't think it is intuitive that what ColdFusion exposes as a CGI variable is the actually CGI variable name. Hmmmph!
Also take note about the "encoded" attribute of the CFHttpParam tag. By default the CFHttpParam tag Url Encodes the passed in value. This puts in all those % characters in the value. We do NOT want this. In fact, if we allow the value in the above example to be encoded then the CFHttp will get denied via 403 again.
Reader Comments
Thanks a ton, this post just saved me after banging my head against the wall for a good hour!!
Rich,
Glad to help. Yeah, I wish this stuff was more apparent.
Great work...you are only one tnat wrote something on subject....
Thats OK...but how can I manage "refer" in .NET using WebClient assembly
Thank a lot
@zmrcic,
Unfortunately, I do not know enough about .NET to give you that kind of advice. Sorry.
It al worked for few days...naw I get 403 forbiden again....I checked other PC...it is working on them....I cant browsw page from IE.....
any suggestions?
You can't browse any pages using IE? Are you talking about using CFHTTP or just plain old browsing the internet?
I have new problem, and I guess you are some kind of authority for web questions...
I'm trying to get content of webpage containing frames...url in addressbar never cahnge and I can not see full url(with query string) I have few combo boxes on that page, and when I change its values, page is reloaded, but with no query string that I can read....I guess it should be www.somepage.com?date=2008/2/2
instead I always get www.somepage.com
...How can I get full url, with query string
@Zmrcic,
If you look page source of the top page, the frameset, it should show you frames and the URLs. But this will only be for the first loaded screen. If the frame has already been reloaded, try right-clicking in the frame. There might be an option somewhere for "open frame in new window" or something like that. Do that and you will be able to get the URL.
You could probably also use something like FireBug to inspect the live code.
I have run into an odd situation. In reviewing some old CF code I found a variable referenced as just "query" as in #query# in its usage. I at first assumed this was a local Variable scoped variable but I couldn't find anywhere when it had it's value defined.
I then did some testing and find that <cfoutput>#query#</cfoutput> displays the same value as #cgi.query_string#. Hmmm.
I then tried #cgi.query# and again it displays the same thing. I then tried this:
<cfset query = "nothing">
cgi.Query: <cfoutput>#cgi.query#</cfoutput><br>
and again it displays the URL query string.
Now maybe I just haven't found the script that is secretyly playing this game with me but this appears to be a non-documented "feature" built into ColdFusion. We are running v.8 but it clearly did this prior to our April 2009 switch over from v.6.x at least.
Can you please explain what is going on here?
@CFCoder,
When I try to dump out #query# I get an undefined value error. There must be something else in your code that is monkeying around.
