Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at Scotch On The Rock (SOTR) 2010 (London) with:

Ask Ben: Cross-Site AJAX Requests Using jQuery And ColdFusion

Posted by Ben Nadel

Using JQuery is it possible to use .load() or .ajax() to bring content from another site into your locally running page (Screen scraping)? I am basically trying to display the weather from the URL: http://weather.yahooapis.com/forecastrss?p=11385 into my div. Is this even possible with JQuery?

jQuery is the most awesome Javascript library ever created; but, as with all Javascript code, it exists in the security sandbox of the browser. Unfortunately (or fortunately, depending on how you look at it), the security of the browser prevents the client from making cross-site AJAX requests - that is, making an AJAX request to a page located under a different domain. If you try to make such a request, you will find that Javascript throws the following error:

Access to restricted URI denied

There are a number of ways that I have read about to allow Javascript to make cross-site AJAX requests; but, these all seem a bit hacky and are not always cross-browser compatible (not tested personally, but from what I have read - I may be ignorant on this matter). Personally, the method that I like the best is to create a proxy page under your own domain that acts as a server-side intermediary in your cross-domain AJAX. The purpose of this page is that it performs your AJAX request using server-side technologies that are not bound by the security of the client:

 
 
 
 
 
 
Cross-Site AJAX Requests Using A Server-Side Proxy Page To Get Around Client Security Model. 
 
 
 

As you can see from the diagram, your client code makes the AJAX request to this proxy page rather than the target page. Naturally, we have to tell the proxy page where to make its request so, in addition to any url or form values, we have to pass along the proxyURL (target URL):

  • <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  • <html>
  • <head>
  • <title>Cross-Site jQuery AJAX Demo With Proxy</title>
  • <script type="text/javascript" src="jquery-1.3.1.pack.js"></script>
  • <script type="text/javascript">
  •  
  • // Run when DOM is ready.
  • $(
  • function(){
  • var jForm = $( "form" );
  •  
  • // Hook up form submit.
  • jForm.submit(
  • function( objEvent ){
  • // Get weather from Yahoo.
  • GetWeather( jForm );
  •  
  • // Prevent default.
  • return( false );
  • }
  • );
  • }
  • );
  •  
  •  
  • // This method actually performs the cross-site AJAX
  • // using a proxy ColdFusion page.
  • function GetWeather( jForm ){
  • // Hook up form submit to pull down weather from
  • // Yahoo API using proxy AJAX request.
  • $.ajax(
  • {
  • url: "ajax_proxy.cfm",
  • type: "get",
  • dataType: "xml",
  • cache: false,
  •  
  • // As part of the data, we have to pass in the
  • // the target url for our server-side AJAX request.
  • data: {
  • proxyURL: "http://weather.yahooapis.com/forecastrss",
  • p: jForm.find( ":text" ).val()
  • },
  •  
  • // Alert when content has been loaded.
  • success: function( xmlData ){
  • // Get the content from the response XML.
  • var strData = $( xmlData )
  • .find( "description" )
  • .text()
  • ;
  •  
  • // Load content into DOM.
  • $( "#content" ).html( strData );
  • }
  • }
  • );
  • }
  •  
  • </script>
  • </head>
  • <body>
  •  
  • <h1>
  • Cross-Site jQuery AJAX Demo With Proxy
  • </h1>
  •  
  • <form>
  •  
  • <p>
  • Zip Code:<br />
  • <input type="text" name="zip" />
  • <input type="submit" value="Get Weather" />
  • </p>
  •  
  • </form>
  •  
  • <div id="content">
  • <!-- Here, we will store the AJAX response. -->
  • </div>
  •  
  • </body>
  • </html>

As you can see in the demo, when the DOM loads, we are binding the form submit to fire the Javascript method, GetWeather(). This method then initiates an AJAX request to our proxy page, "ajax_proxy.cfm". Because this proxy page is on the same domain as the current page request, there are no security concerns. Notice, though, that in the data for our request, we are passing along the actual target URL (proxyURL), the Yahoo Weather API url: http://weather.yahooapis.com/forecastrss.

When the form is submitted, our proxy page makes the request and returns the resultant XML response to the client which uses jQuery to grab the content and inject it into the client's DOM.

Ok, so now that we see how this proxy page is being used, let's take a look at the code. This proxy page code can get more complicated than the following demo if you need to deal with cookies and authentication; but, I have found this ColdFusion code to be quite dependable for most generic situations:

  • <!---
  • Check to see if the page request is a POST or a GET.
  • Based on this, we can figure out our target URL.
  • --->
  • <cfif (CGI.request_method EQ "get")>
  •  
  • <!--- Get URL-based target url. --->
  • <cfset strTargetURL = URL.ProxyURL />
  •  
  • <!--- Delete target URL. --->
  • <cfset StructDelete( URL, "ProxyURL" ) />
  •  
  • <cfelse>
  •  
  • <!--- Get FORM-based target url. --->
  • <cfset strTargetURL = FORM.ProxyURL />
  •  
  • <!--- Delete target URL. --->
  • <cfset StructDelete( FORM, "ProxyURL" ) />
  •  
  • </cfif>
  •  
  •  
  • <!---
  • Remove any AJAX anit-caching that was used by jQuery. This
  • is a random number meant to help ensure that GET URLs are
  • not cached.
  • --->
  • <cfset StructDelete( URL, "_" ) />
  •  
  •  
  •  
  • <!---
  • Make the proxy HTTP request using. When we do this, try to
  • pass along all of the CGI information that was made by the
  • original AJAX request.
  • --->
  • <cfhttp
  • result="objRequest"
  • url="#UrlDecode( strTargetURL )#"
  • method="#CGI.request_method#"
  • useragent="#CGI.http_user_agent#"
  • timeout="15">
  •  
  • <!--- Add the referer tht was passed-in. --->
  • <cfhttpparam
  • type="header"
  • name="referer"
  • value="#CGI.http_referer#"
  • />
  •  
  • <!--- Pass along any URL values. --->
  • <cfloop
  • item="strKey"
  • collection="#URL#">
  •  
  • <cfhttpparam
  • type="url"
  • name="#LCase( strKey )#"
  • value="#URL[ strKey ]#"
  • />
  •  
  • </cfloop>
  •  
  • <!--- Pass along any FORM values. --->
  • <cfloop
  • item="strKey"
  • collection="#FORM#">
  •  
  • <cfhttpparam
  • type="formfield"
  • name="#LCase( strKey )#"
  • value="#FORM[ strKey ]#"
  • />
  •  
  • </cfloop>
  •  
  • </cfhttp>
  •  
  •  
  • <!---
  • <!--- Debug most current request. --->
  • <cfset objDebug = {
  • CGI = Duplicate( CGI ),
  • URL = Duplicate( URL ),
  • FORM = Duplicate( FORM ),
  • Request = Duplicate( objRequest )
  • } />
  •  
  • <!--- Output debug to file. --->
  • <cfdump
  • var="#objDebug#"
  • output="#ExpandPath( './ajax_prox_debug.htm' )#"
  • format="HTML"
  • />
  • --->
  •  
  •  
  • <!---
  • Get the content as a byte array (by converting it to binary,
  • we can echo back the appropriate length as well as use it in
  • the binary response stream.
  • --->
  • <cfset binResponse = ToBinary(
  • ToBase64( objRequest.FileContent )
  • ) />
  •  
  •  
  • <!--- Echo back the response code. --->
  • <cfheader
  • statuscode="#Val( objRequest.StatusCode )#"
  • statustext="#ListRest( objRequest.StatusCode, ' ' )#"
  • />
  •  
  • <!--- Echo back response legnth. --->
  • <cfheader
  • name="content-length"
  • value="#ArrayLen( binResponse )#"
  • />
  •  
  • <!--- Echo back all response heaers. --->
  • <cfloop
  • item="strKey"
  • collection="#objRequest.ResponseHeader#">
  •  
  • <!--- Check to see if this header is a simple value. --->
  • <cfif IsSimpleValue( objRequest.ResponseHeader[ strKey ] )>
  •  
  • <!--- Echo back header value. --->
  • <cfheader
  • name="#strKey#"
  • value="#objRequest.ResponseHeader[ strKey ]#"
  • />
  •  
  • </cfif>
  •  
  • </cfloop>
  •  
  •  
  • <!---
  • Echo back content with the appropriate mime type. By using
  • the Variable attribute, we will make sure that the content
  • stream is reset and ONLY the given response will be returned.
  • --->
  • <cfcontent
  • type="#objRequest.MimeType#"
  • variable="#binResponse#"
  • />

Note: I have left in some debugging code, but commented it out. The downsite of making a proxy-page AJAX request is that debugging can be a real pain. The debug code allows you to see the post and request made by the proxy page.

Once the ColdFusion CFHTTP request comes back, we echo the status code, response headers, and file content in its response to the client. This way, the client receives almost the same exact response as if it were calling the target page directly. The whole thing works quite nicely.

I hope this helps in some way.




Reader Comments

Great post Ben! Note that you can make cross-site calls to APIs that return JSON if you use the $.getJSON() method and the API allows for a callback method to be specified.

Reply to this Comment

Hi Ben, thanx 4 the insightful article, think I'm gonna start using it somewhere quite soon ;-) But how to get this dynamically, say on page load and in set intervals? How can the function then be called? And would these regular pings to the remote API not be seen as a spambot or something alike?

Reply to this Comment

@Rey,

That sounds pretty cool. Will this work on *any* JSON-based API? Or, does the API have to know about this callback concept?

@Sebastiaan,

You could access this the same way you access any API. So, you could do this with the page loads or on a setInterval() or setTimeout() call. As far as a spam flagging, each API is going to have its own behavior that dictates how often you can call its API before you get in trouble.

Reply to this Comment

@Bucky,

I saw that article,it's good stuff. The only down side to JSONP (JSON with Padding) is that it relies on the third party to implement such a convention. Of course, as that becomes more common place, it won't be such an issue.

Reply to this Comment

One of the things I've learned when it comes to ColdFusion development is that it almost always makes sense to go for simplicity. While not always possible, it doesn't hurt to stop and ask myself if the route I'm going down to solve some problem could be simplified in some way. Let me share the problem, and then I'll share the complex solutions I tried until I got to a much simpler fix.

Reply to this Comment

In the situation of an error like - page not found or an error code in the xml (<statusCode>-3001</statusCode>, how would you alert the user ? Any help would be apprecieated.

Thanks.

Reply to this Comment

@Nav,

That depends on you AJAX framework. For me, for instance, I always like to return an API response with the following format:

Success: boolean,
Errors: array,
Data: any

This way, if anything goes wrong on the server, I can simply return a non-Success message with the given error. The client code (on the browser) then just needs to know to check the Success flag in the response.

Reply to this Comment

Hey Ben!

I'm attempting to use your code in order to load an RSS feed from our main site onto our mobile site on a different domain.

It works great on many feeds but gets tripped up with the following XML/RSS url: "http://www.idfive.com/tasks/feed/?feedID=843AB539-C139-9E73-E2172D809FA78B99"

After dumping the httpRequest error I get this:

error in: crossdomain.cfm?_=1278476437853&proxyURL=http%3A%2F%2Fwww.idfive.com%2Ftasks%2Ffeed%2F%3FfeedID%3D843AB539-C139-9E73-E2172D809FA78B99 \nerror:\nTypeError: a is null

I'm assuming it's because the request URL is funky. I appreciate any advice you have. Thanks!

Reply to this Comment

@Jake,

Hmm, I'm not sure. I would try CFDump'ing out the FORM/URL values before you perform the CFHTTP request to see if there is anything in them that you can see, such as a Url-encoded value that shouldn't be encoded.

Reply to this Comment

@Ben:
Great Article man. I'd love to see a hybrid of this using the JSON Method, that Ray Camden wrote about.

This is a totally kick-ass article, and many many thanks on Posting it!

Reply to this Comment

Post A Comment

?
You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.