Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at cf.Objective() 2009 (Minneapolis, MN) with:

Ask Ben: Grabbing World Of Warcraft Data With ColdFusion And CFHTTP

By Ben Nadel on

You've been doing a lot with XML lately, so I thought maybe you could solve a problem I'm having. I'm a World of Warcraft junky and want to pull data from Blizzard's "armory" web site. It's all XML, so should be easy, but when I do

<cfhttp url="http://armory.worldofwarcraft.com/....." method="get" result="armoryXML">

It returns the full HTML, post style sheet transformation. Yet go to that page and view source and you see the lovely XML. I've seen PHP code that hits the data on this site by doing the PHP equivalent of a cfhttp.

I tried your ColdFusion CFHttp method and did, indeed, get the transformed page content. Then, I went directly to the page in my browser and viewed the source. The source, just as you said it would be, was an XML document with an attached XSLT processing instruction. This was confusing since ColdFusion's CFHttp tag works just like any browser request. So, what's different between the CFHttp request and the browser's request that would render different content?

There is nothing about this that is obvious at all, but if you have worked with CFHttp for a long time, you might know that CFHttp causes problems because the user agent it broadcasts is that of the ColdFusion server's HTTP agent. So, when you go to a site with your FireFox or Internet Explorer browser, the browser generally broadcasts its user agent as a Mozilla or MSIE compatible browser but ColdFusion, on the other hand, announces itself as the "ColdFusion" user agent.

Knowing this, I did a little experiment, making your ColdFusion CFHttp call in two ways: one with no explicit user agent and one with the FireFox user agent:

  • <!---
  • Store the user agent that I am using with my browser
  • (you're damn right I use FireFox!). I am breaking this
  • data into two lines for display purposes ONLY.
  • --->
  • <cfset strUserAgent = (
  • "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; " &
  • "rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3"
  • ) />
  •  
  •  
  • <!---
  • Get the target URL that we are grabbing. This page
  • provides an XML document with an XSL transformation
  • processing instruction. We want to grab the XML without
  • the data being transformed into HTML. I am breaking this
  • data into two lines for display purposes ONLY.
  • --->
  • <cfset strURL = (
  • "http://armory.worldofwarcraft.com/character-sheet.xml" &
  • "?r=Bloodhoof&n=Castlereagh"
  • ) />
  •  
  •  
  • <!---
  • Grab the target page without sending across any user agent
  • information. When we do this, ColdFusion automatically puts
  • in "ColdFusion" as the browser's user agent value.
  • --->
  • <cfhttp
  • url="#strURL#"
  • method="GET"
  • result="objHTTP"
  • />
  •  
  •  
  • <!---
  • Now, let's grab the same page, but this time, instead of
  • letting ColdFusion send over a default user agent, we are
  • going to explicitly define what user agent the HTTP reuqest
  • should announce.
  • --->
  • <cfhttp
  • url="#strURL#"
  • method="GET"
  • result="objHTTPWithUA"
  • useragent="#strUserAgent#"
  • />
  •  
  •  
  •  
  • <!---
  • Let's output the leading characters of the request without
  • the user agent value.
  • --->
  • <p>
  • <strong>Request With ColdFusion User Agent</strong>:
  • </p>
  •  
  • <p>
  • #HtmlEditFormat(
  • Left( objHTTP.FileContent, 500 )
  • )#
  • </p>
  •  
  •  
  • <!---
  • Let's output the leading characters of the request in which
  • we explicitly defined the FireFox user agent.
  • --->
  • <p>
  • <strong>Request With FireFox User Agent</strong>:
  • </p>
  •  
  • <p>
  • #HtmlEditFormat(
  • Left( objHTTPWithUA.FileContent, 500 )
  • )#
  • </p>

Running the above code, we get the following screen output:

Request With ColdFusion User Agent:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <?xml-stylesheet type="text/xsl" href="/layout/character-sheet.xsl"> <html> <head> <title>The Armory</title> <script src="/shared/global/third-party/detection.js" type="text/javascript"></script><script src="js/cookies.js" type="text/javascript"></script><script src="js/armory.js" type="text/javascript"></script><script src="js/functions.js" type="text/javascript"></script><s

Request With FireFox User Agent:

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/layout/character-sheet.xsl"?> <page globalSearch="1" lang="en_us" requestUrl="/character-sheet.xml"> <characterInfo> <character battleGroup="Ruin" charUrl="r=Bloodhoof&amp;n=Castlereagh" class="Hunter" classId="3" faction="Alliance" factionId="0" gender="Male" genderId="0" guildName="Veni Vidi Vici" guildUrl="r=Bloodhoof&amp;n=Veni+Vidi+Vici&amp;p=1" lastModified="May 18, 2007" level="70" name="Castlereagh" rac

As you can see, the first request in which we do not define a user agent (therefore letting "ColdFusion" be announced) sends us back the fully transformed HTML page complete with HTML, HEAD, and TITLE tags. But, in the second ColdFusion CFHttp request, where we announce the request as coming from a FireFox compatible browser, the true XML document is returned complete with XSLT, DocType, PAGE, and CHARACTERINFO XML nodes.

Now, just because I figured out what was going wrong, that does not mean that I can offer any insight into why this is happening. I assume this is just some functionality somewhere that is trying to be "clever" by servering up the content that it thinks the requester wants to see. And, for some reason, it assumes that FireFox wants the XML, but the ColdFusion request wants the transformed HTML page. Who knows... maybe someone reading this can offer more insight in that respect.




Reader Comments

All you have to do is pout World of Warcraft in the title, and you've got my attention! ;)

I'm going to play with this tomorrow if I have time, I wonder what kind of cool stuff you can do...

Reply to this Comment

I have only recently started blogging but my first couple of entries were on this very topic. Hopefully your other readers will find it useful. I should finish with an entry on how to breakout raiders by role using the armory data later this weekend:

http://blog.shooksweb.com/index.cfm/2007/4/20/Pulling-World-of-Warcraft-Armory-data-with-Coldfusion
http://blog.shooksweb.com/index.cfm/2007/4/25/World-of-Warcraft-Armory-Data
http://blog.shooksweb.com/index.cfm/2007/4/26/Burning-Crusade-Progress-Page
http://blog.shooksweb.com/index.cfm/2007/5/5/Burning-Crusade-Progress-Page-Part-2

Reply to this Comment

@Matt

Very nice! Wish I knew about your work months ago! I searched all over in February for other folks doing armory scraping with CF, only found ASP and PHP users. I had first wanted to pull armory data because I too was hosting my guild's web site. Then one evening the guild leader decided to go to a paid site, saying mine didn't have enough features. Yet he never requested I add anything that the other site had that he viewed as a must-have. I gave the guild the bird and switched servers. Now I just want the armory data to see what sorts of interesting things I can come up with and to practice some with XML manipulation.

If there's room for another developer on your guild's site, drop me a line.

Reply to this Comment

I actually thought you were playing World of Warcraft as well. However, I am going to take you up on the offer of posting this as a comment to this topic. I assume readers of this topic are WoW players, so this offer is more targeted at them:

We recently released a World of Warcraft Paladin Guide:
http://www.killerguides.com/guides/wow/guide/world-of-warcraft/paladin

Right now we are seeking a few reviewers who can provide us with some additional feedback. If you run a community site or blog we can also provide you with some copies as contest prizes (please use the contact form on the site to get in touch with me).

Reply to this Comment

I'm guessing the reason for this is that Firefox/IE are known to support all the different technologies used by the armory (xsl, ajax etc.), whereas any less-advanced browser might not, so to make sure the content is readable, they're sending a simplified version, which might work on something like a mobile phone! :-)

Reply to this Comment

lol, armory on your cell phone! "I wish I knew how I did in PVP last night, but I'm stuck in this all day meeting, I can't get on the armory to check! ... Oh wait, I can!"

Reply to this Comment

Thank you, thank you, thank you!

This annoying problem had me stumped for hours.

In my case, I was using stanard Java classes to make the requests.

Reply to this Comment

I think this entire thing might be a load of junk personally. Sites have been taking info for a long time now. Its not like something new has entered the filed. Just look at the mmorpg sites or the other database sites out there. Thats 100% what they are doing to help them get all the info from the game and advertise there things.

Reply to this Comment

Thanks for clearing this up! I've been writing a C# application to pull achievement information from the armory, and for the life of me couldn't figure out why my app was returning the entire page, and not just the XML bits.

Thanks again!

Reply to this Comment

Hi Ben,

Any idea of how to use CFHTTP method = "get" to populate the contents of a jQuery modal dialog?

We have a third party URL that we use for online support chat and it's presently in an ugly pop-up window.

Is this possible?

Mike

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.