Spiders Are Tricking My Session Management
Posted August 18, 2006 at 8:51 AM
In order to cut down on variables that are set on the server, I attempt to turn off session management for spiders so that no session variables need to be created. I do this based on user agents and black-listed IP addresses. However, recently, I have been getting a slew of hits from what I assume are spiders that have regular user agents:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Since I can't use that, I thought I would black list the IP addresses, but it seems that the spider is sending a randomized remote address for each page request. The following IP addresses all came from some sort of crawler within two minutes:
24.34.69.164 (3 hits)
62.252.224.18
65.35.198.39 (2 hits)
65.188.255.128
66.180.121.138
67.181.55.113
67.81.170.135
68.252.44.67 (2 hits)
68.52.163.179 (2 hits)
68.80.35.248
69.112.175.221
69.206.239.189
70.123.212.52
71.72.88.167
72.227.146.87 (2 hits)
74.134.101.18
74.136.92.229
80.183.99.222
80.218.123.202
80.38.98.243 (3 hits)
80.99.78.112
82.156.133.68
84.29.108.242
89.156.52.247
89.98.20.160
I know that it was a crawler because they all had the same http referer, which was my home page and not all of the requested pages are available from the home page, which means the referer was being set manually. This is so irritating! Now, I have dozens upon dozens of sessions being created on the server that will last 20 minutes without being used twice. That is poor memory management.
Why is the spider doing this? I suppose this is to stop people from serving up different content based on spiders, but that is not my purpose. Having no session management does not server different content. It just turns off certain server-side tracking. Uggg.
Comments (3) | Post Comment | Ask Ben | Permalink | Other Searches | Print Page
QuerySetCell(), StructInsert(), And Other Silly ColdFusion Functions
Ask Ben: Extracting Parts Of An Email Address In Javascript
Reader Comments
Can you tell if any of those "sneaky" spiders over look for robots.txt?
I've always wanted to mess around with mod rewrite or something to funnel robots.txt through CF so I can better pin down requests coming from spiders.
Of course, that ASSUMES they even bother looking for robots.txt. If they change their IP address with every request, that would make it difficult too.
Posted by Brad Wood on Oct 24, 2007 at 2:40 PM
I have no idea. I assume they don't even bother looking at it??
Posted by Ben Nadel on Oct 24, 2007 at 3:31 PM
80.38.98.243 thats my web
Posted by DieguezZ on Apr 8, 2008 at 9:28 PM
