Ask Ben: Grabbing Google Results with CFHttp : Kinky Solutions : A Student's Perspective by Ben Nadel

Recent Entries Complete Entry List December 2008 November 2008 October 2008 September 2008 August 2008 July 2008 June 2008 May 2008 April 2008 March 2008 February 2008 January 2008 December 2007 November 2007 October 2007 September 2007 August 2007 July 2007 June 2007 May 2007 April 2007 March 2007 February 2007 January 2007 December 2006 November 2006 October 2006 September 2006 August 2006 July 2006 June 2006 May 2006 April 2006 AJAX (16) Ask Ben (121) Books (15) ColdFusion (972) Exercise List (17) Flash (3) Health / Fitness (35) HTML / CSS (31) Javascript / DHTML (101) Life (71) Movies (24) Project HUGE (30) Relationships (9) Search Engine Optimization (14) Skin Spider (30) SQL (85) Work (114) XStandard WYSIWYG (22) RSS 2.0 Custom RSS 2.0

Posted August 23, 2006 at 2:55 PM

Tags: ColdFusion, Ask Ben

I am getting errors when I try to grab google results with cfhttp. But, when I go to page with my browser, it works just fine. What am I doing wrong?

You are not doing anything wrong. Google wants to be used by regular web users. CFHttp does not announce itself as a regular user. When you do a CFHttp page grab, it passes along, as its User Agent a non-standard value. I am not sure offhand what it is, but I think it sends "ColdFusion" as its user agent. Doing a regular CFHttp will return this error:

Your client does not have permission to get URL /search?hl=en&lr=&q=Girls+Gone+Wild&btnG=Search from this server.

This is there for a reason: you might be violating the Google terms of service (I have not read them, nor do I condone working around this). If you want to avoid this, you can fake Google into thinking you ARE a web browser by sending a standard user agent in your CFHttp:

 Launch code in new window » Download code as text file »

Notice that I am sending the FireFox / Mozilla user agent. This should work just fine. But again, I am not aware of the legality of such an action - proceed with caution.

Download Code Snippet ZIP File

Comments (2)  |  Post Comment  |  Ask Ben  |  Permalink  |  Other Searches  |  Print Page

• Add To Del.icio.us
• Add To Digg
• Add to Spurl.net
• Add to Wists
• Add to Simpy
• Add to Newsvine
• Add to BlinkList
• Add to Furl
• Add to reddit
• Add to FARK
• Add to BlogMarks
• Add to Yahoo! My Web
• Add to smarking
• Add to Ma.gnolia
• Add to Segnalo

Reader Comments

Great post on pulling Google search results pages.
Now if I could only figure out how to get it to only pull
the result for a particular site's listing.
Trying to analyze the different result text for one domain
for different keyword searches.

Posted by Dr Adam on Dec 9, 2007 at 9:57 PM

@Dr. Adam,

Just put "site:youdomain.com" in the google query and it should only pull for a particular site.

Posted by Ben Nadel on Dec 10, 2007 at 7:06 AM

Post Comment  |  Ask Ben