Ask Ben: Grabbing World Of Warcraft Data With ColdFusion And CFHTTP
Posted May 18, 2007 at 8:32 AM by Ben Nadel
You've been doing a lot with XML lately, so I thought maybe you could solve a problem I'm having. I'm a World of Warcraft junky and want to pull data from Blizzard's "armory" web site. It's all XML, so should be easy, but when I do
<cfhttp url="http://armory.worldofwarcraft.com/....." method="get" result="armoryXML">
It returns the full HTML, post style sheet transformation. Yet go to that page and view source and you see the lovely XML. I've seen PHP code that hits the data on this site by doing the PHP equivalent of a cfhttp.
I tried your ColdFusion CFHttp method and did, indeed, get the transformed page content. Then, I went directly to the page in my browser and viewed the source. The source, just as you said it would be, was an XML document with an attached XSLT processing instruction. This was confusing since ColdFusion's CFHttp tag works just like any browser request. So, what's different between the CFHttp request and the browser's request that would render different content?
There is nothing about this that is obvious at all, but if you have worked with CFHttp for a long time, you might know that CFHttp causes problems because the user agent it broadcasts is that of the ColdFusion server's HTTP agent. So, when you go to a site with your FireFox or Internet Explorer browser, the browser generally broadcasts its user agent as a Mozilla or MSIE compatible browser but ColdFusion, on the other hand, announces itself as the "ColdFusion" user agent.
Knowing this, I did a little experiment, making your ColdFusion CFHttp call in two ways: one with no explicit user agent and one with the FireFox user agent:
- <!---
- Store the user agent that I am using with my browser
- (you're damn right I use FireFox!). I am breaking this
- data into two lines for display purposes ONLY.
- --->
- <cfset strUserAgent = (
- "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; " &
- "rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3"
- ) />
-
-
- <!---
- Get the target URL that we are grabbing. This page
- provides an XML document with an XSL transformation
- processing instruction. We want to grab the XML without
- the data being transformed into HTML. I am breaking this
- data into two lines for display purposes ONLY.
- --->
- <cfset strURL = (
- "http://armory.worldofwarcraft.com/character-sheet.xml" &
- "?r=Bloodhoof&n=Castlereagh"
- ) />
-
-
- <!---
- Grab the target page without sending across any user agent
- information. When we do this, ColdFusion automatically puts
- in "ColdFusion" as the browser's user agent value.
- --->
- <cfhttp
- url="#strURL#"
- method="GET"
- result="objHTTP"
- />
-
-
- <!---
- Now, let's grab the same page, but this time, instead of
- letting ColdFusion send over a default user agent, we are
- going to explicitly define what user agent the HTTP reuqest
- should announce.
- --->
- <cfhttp
- url="#strURL#"
- method="GET"
- result="objHTTPWithUA"
- useragent="#strUserAgent#"
- />
-
-
-
- <!---
- Let's output the leading characters of the request without
- the user agent value.
- --->
- <p>
- <strong>Request With ColdFusion User Agent</strong>:
- </p>
-
- <p>
- #HtmlEditFormat(
- Left( objHTTP.FileContent, 500 )
- )#
- </p>
-
-
- <!---
- Let's output the leading characters of the request in which
- we explicitly defined the FireFox user agent.
- --->
- <p>
- <strong>Request With FireFox User Agent</strong>:
- </p>
-
- <p>
- #HtmlEditFormat(
- Left( objHTTPWithUA.FileContent, 500 )
- )#
- </p>
Running the above code, we get the following screen output:
Request With ColdFusion User Agent:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <?xml-stylesheet type="text/xsl" href="/layout/character-sheet.xsl"> <html> <head> <title>The Armory</title> <script src="/shared/global/third-party/detection.js" type="text/javascript"></script><script src="js/cookies.js" type="text/javascript"></script><script src="js/armory.js" type="text/javascript"></script><script src="js/functions.js" type="text/javascript"></script><s
Request With FireFox User Agent:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/layout/character-sheet.xsl"?> <page globalSearch="1" lang="en_us" requestUrl="/character-sheet.xml"> <characterInfo> <character battleGroup="Ruin" charUrl="r=Bloodhoof&n=Castlereagh" class="Hunter" classId="3" faction="Alliance" factionId="0" gender="Male" genderId="0" guildName="Veni Vidi Vici" guildUrl="r=Bloodhoof&n=Veni+Vidi+Vici&p=1" lastModified="May 18, 2007" level="70" name="Castlereagh" rac
As you can see, the first request in which we do not define a user agent (therefore letting "ColdFusion" be announced) sends us back the fully transformed HTML page complete with HTML, HEAD, and TITLE tags. But, in the second ColdFusion CFHttp request, where we announce the request as coming from a FireFox compatible browser, the true XML document is returned complete with XSLT, DocType, PAGE, and CHARACTERINFO XML nodes.
Now, just because I figured out what was going wrong, that does not mean that I can offer any insight into why this is happening. I assume this is just some functionality somewhere that is trying to be "clever" by servering up the content that it thinks the requester wants to see. And, for some reason, it assumes that FireFox wants the XML, but the ColdFusion request wants the transformed HTML page. Who knows... maybe someone reading this can offer more insight in that respect.
Reader Comments
Thanks Ben! You're my hero!
@Jamie,
Always glad to help out :)
All you have to do is pout World of Warcraft in the title, and you've got my attention! ;)
I'm going to play with this tomorrow if I have time, I wonder what kind of cool stuff you can do...
I have only recently started blogging but my first couple of entries were on this very topic. Hopefully your other readers will find it useful. I should finish with an entry on how to breakout raiders by role using the armory data later this weekend:
http://blog.shooksweb.com/index.cfm/2007/4/20/Pulling-World-of-Warcraft-Armory-data-with-Coldfusion
http://blog.shooksweb.com/index.cfm/2007/4/25/World-of-Warcraft-Armory-Data
http://blog.shooksweb.com/index.cfm/2007/4/26/Burning-Crusade-Progress-Page
http://blog.shooksweb.com/index.cfm/2007/5/5/Burning-Crusade-Progress-Page-Part-2
@Matt
Very nice! Wish I knew about your work months ago! I searched all over in February for other folks doing armory scraping with CF, only found ASP and PHP users. I had first wanted to pull armory data because I too was hosting my guild's web site. Then one evening the guild leader decided to go to a paid site, saying mine didn't have enough features. Yet he never requested I add anything that the other site had that he viewed as a must-have. I gave the guild the bird and switched servers. Now I just want the armory data to see what sorts of interesting things I can come up with and to practice some with XML manipulation.
If there's room for another developer on your guild's site, drop me a line.
And here's the original PHP site I found with armory stuff http://wow.tachyonsix.com/armory/. This guy caches all data for 12 hours, I wonder how much disk space that uses. The code's open source, might be helpful for developing more tools.
like it
I actually thought you were playing World of Warcraft as well. However, I am going to take you up on the offer of posting this as a comment to this topic. I assume readers of this topic are WoW players, so this offer is more targeted at them:
We recently released a World of Warcraft Paladin Guide:
http://www.killerguides.com/guides/wow/guide/world-of-warcraft/paladin
Right now we are seeking a few reviewers who can provide us with some additional feedback. If you run a community site or blog we can also provide you with some copies as contest prizes (please use the contact form on the site to get in touch with me).
I'm guessing the reason for this is that Firefox/IE are known to support all the different technologies used by the armory (xsl, ajax etc.), whereas any less-advanced browser might not, so to make sure the content is readable, they're sending a simplified version, which might work on something like a mobile phone! :-)
lol, armory on your cell phone! "I wish I knew how I did in PVP last night, but I'm stuck in this all day meeting, I can't get on the armory to check! ... Oh wait, I can!"
If u like this you can go to
http://www.powerleveling-wow-powerleveling.com
to try best wow powerleveling service
Thank you, thank you, thank you!
This annoying problem had me stumped for hours.
In my case, I was using stanard Java classes to make the requests.
Thanks very much!
You lit a light in my particular tunnel!
I think this entire thing might be a load of junk personally. Sites have been taking info for a long time now. Its not like something new has entered the filed. Just look at the mmorpg sites or the other database sites out there. Thats 100% what they are doing to help them get all the info from the game and advertise there things.
Thanks for clearing this up! I've been writing a C# application to pull achievement information from the armory, and for the life of me couldn't figure out why my app was returning the entire page, and not just the XML bits.
Thanks again!
Hi Ben,
Any idea of how to use CFHTTP method = "get" to populate the contents of a jQuery modal dialog?
We have a third party URL that we use for online support chat and it's presently in an ugly pop-up window.
Is this possible?
Mike
