Skip to main content
Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.

Updated Session Management And Web Spiders & Bots

By Ben Nadel on
Tags: ColdFusion

As the list of spiders that hits my site grows, I am trying to keep the session management under control. I don't want to offer spidres sessions since I don't want to have unused session variables taking up RAM on the server. But at the same time, I don't want to add a lot of processing overhead to each page just checking to see if a user is really a user or really a spider/bot. Afterall, the server does have 4 gigs or RAM. Processing speed and the user experience is more important to me than RAM usage.

But, in the interest of optimization I have combined several of my user agent checks into one Regular Expression (RegEx) search for the string "bot" on a word boundry: "bot\b". As you can see below, this takes care of 18 user agent types. I doubt that this will give me any false positives on standard browsers, but if it does, the only difference is that they will not get sessions.

In previous posts I talked about how Short-Circuit evaluation was faster than large regular expressions. I have not gone back on this. While this is a regular expression, it is not a variable-length regular expression. It is meerly a qualified standard string search (qualified by the word bountry) and is therefore very fast.

// Create a lowercase version of the user agent so we can run without
// NoCase checks.
strTempUserAgent = LCase( CGI.http_user_agent );

// Check user agent.
if (
	(NOT Len(strTempUserAgent)) OR

	// We are gonna try to optimize even a little bit more. A good number
	// of the spider names end in "bot". If we check for names that have
	// BOT ending on a word bountry, we can eliminate severl of the other
	// spider checkes. The bot\b search here takes care of the spiders
	// that are now commented out below. As you can see, it takes the
	// place of 18 different spider Find()'s.
	REFind( "bot\b", strTempUserAgent ) OR

	Find( "slurp", strTempUserAgent ) OR
	// Find( "googlebot", strTempUserAgent ) OR
	// Find( "becomebot", strTempUserAgent ) OR
	// Find( "msnbot", strTempUserAgent ) OR
	Find( "mediapartners-google", strTempUserAgent ) OR
	Find( "zyborg", strTempUserAgent ) OR
	// Find( "rufusbot", strTempUserAgent ) OR
	Find( "emonitor", strTempUserAgent ) OR
	// Find( "researchbot", strTempUserAgent ) OR
	// Find( "ip2mapbot", strTempUserAgent ) OR
	// Find( "gigabot", strTempUserAgent ) OR
	Find( "jeeves", strTempUserAgent ) OR
	// Find( "exabot", strTempUserAgent ) OR
	Find( "sbider", strTempUserAgent ) OR
	Find( "findlinks", strTempUserAgent ) OR
	Find( "yahooseeker", strTempUserAgent ) OR
	Find( "mmcrawler", strTempUserAgent ) OR
	// Find( "mj12bot", strTempUserAgent ) OR
	// Find( "outfoxbot", strTempUserAgent ) OR
	Find( "jbrowser", strTempUserAgent ) OR
	// Find( "ziggsbot", strTempUserAgent ) OR
	Find( "java", strTempUserAgent ) OR
	Find( "pmafind", strTempUserAgent ) OR
	Find( "blogbeat", strTempUserAgent ) OR
	// Find( "turnitinbot", strTempUserAgent ) OR
	Find( "converacrawler", strTempUserAgent ) OR
	Find( "ocelli", strTempUserAgent ) OR
	Find( "labhoo", strTempUserAgent ) OR
	Find( "validator", strTempUserAgent ) OR
	Find( "sproose", strTempUserAgent ) OR
	// Find( "obot", strTempUserAgent ) OR
	// Find( "myfamilybot", strTempUserAgent ) OR
	// Find( "girafabot", strTempUserAgent ) OR
	// Find( "aipbot", strTempUserAgent ) OR
	Find( "ia_archiver", strTempUserAgent ) OR
	// Find( "snapbot", strTempUserAgent ) OR
	Find( "larbin", strTempUserAgent ) OR
	Find( "psycheclone", strTempUserAgent )
	// Find( "IRLbot", strTempUserAgent )
	){

	// This application definition is for robots that do NOT need sessions.
	THIS.Name = "KinkySolutions v.1 {dev}";
	THIS.SessionManagement = false;
	THIS.SetClientCookies = false;
	THIS.ClientManagement = false;
	THIS.SetDomainCookies = false;

	// Set the flag for session use.
	REQUEST.HasSessionScope = false;

} else {

	// This application is for the standard user.
	THIS.Name = "KinkySolutions v.1 {dev}";
	THIS.SessionManagement = true;
	THIS.SetClientCookies = true;
	THIS.SessionTimeout = CreateTimeSpan(0, 0, 20, 0);
	THIS.LoginStorage = "SESSION";

	// Set the flag for session use.
	REQUEST.HasSessionScope = true;

}


Reader Comments