Skip to main content
Ben Nadel at the jQuery Conference 2009 (Cambridge, MA) with: Rebecca Murphey
Ben Nadel at the jQuery Conference 2009 (Cambridge, MA) with: Rebecca Murphey ( @rmurphey )

Turning Off Session Management for Web Spiders

By on
Tags:

Michael Dinowitz came to my office to do some code review since our server was having some trouble, slowing down and what not. We ended up having to restart the server several times to kill the slowly growning memory usage of the ColdFusion service. One suggestion that Dinowitz made was to turn off session management for web spiders so that they do not get any server memroy usage. Since spiders do not use cookie, their sessions never hold. And so, for every single page the spider hits, a new session is created. And with that new session goes every bit of user-related memeory that you have created in the application.

As a solution to this problem, he did something that I have never seen before. He put logic in the Application.cfc around the application definition. It never even really occurred to me to put ColdFusion code before the CFAPPLICATION tag or the application definition in the CFC.

Here is my adaptation of his solution. I use the User Agent to set the Application session management. Additionally, browsers that I determine are "spiders," I set HasSessionScope variable in the request. Then, at any time during the request processing, I can determine if I need to alter my code to allow for non-session based usage.

// Define the application. To stop unnescessary memory usage, we are going to give
// web crawler no session management. This way, they don't have to worry about cookie
// acceptance and object persistance (except for APPLICATION scope).
if ((NOT Len(CGI.http_user_agent)) OR REFindNoCase("Slurp|Googlebot|BecomeBot|msnbot|Mediapartners-Google|ZyBorg|RufusBot|EMonitor|researchbot|IP2MapBot|GigaBot|Jeeves|Exabot|SBIder|findlinks|YahooSeeker|MMCrawler|MJ12bot|OutfoxBot|jBrowser|ZiggsBot|Java|PMAFind|Blogbeat|TurnitinBot|ConveraCrawler|Ocelli|ColdFusion", CGI.http_user_agent)){
 
	// This application definition is for robots that do NOT need sessions.
	THIS.Name = "KinkySolutions v.1 {dev}";
	THIS.SessionManagement = false;
	THIS.SetClientCookies = false;
	THIS.ClientManagement = false;
	THIS.SetDomainCookies = false;
	// Set the flag for session use.
	REQUEST.HasSessionScope = false;

} else {

	// This application is for the standard user.
	THIS.Name = "KinkySolutions v.1 {dev}";
	THIS.SessionManagement = true;
	THIS.SetClientCookies = true;
	THIS.SessionTimeout = CreateTimeSpan(0, 0, 20, 0);
	THIS.LoginStorage = "SESSION";
 
	// Set the flag for session use.
	REQUEST.HasSessionScope = true;

}

Want to use code from this post? Check out the license.

Reader Comments

I believe in love. I believe in compassion. I believe in human rights. I believe that we can afford to give more of these gifts to the world around us because it costs us nothing to be decent and kind and understanding. And, I want you to know that when you land on this site, you are accepted for who you are, no matter how you identify, what truths you live, or whatever kind of goofy shit makes you feel alive! Rock on with your bad self!
Ben Nadel