Skin Spider : Max Size Data Storage Rules
Posted October 7, 2006 at 1:29 PM by Ben Nadel
To learn all about Skin Spider, click here.
To view the most updated application, click here.
To view the current code base, click here.
As I have been working on this application and testing and letting it run in the background, I am quickly becoming aware that one mega byte files add up very quickly. On my local server, after letting the spider run for about 20 or 30 minutes, I had almost one GIG or videos stored. Granted, this was a large number of videos, but it made me realize that something had to be done; the spider could not be allowed to run unconditionally.
To remedy this, I have updated the application initialization process in the ColdFusion Application.cfm file to set a value for max storage size (APPLICATION.MaxStorageSize). This is the number of bytes that the application is allowed to store before spidering of new content is shut down. I have defined the size interms of math so that it can be easily manipulated:
- <!--- Set size limit to 800 mega bytes. --->
- <cfset APPLICATION.MaxStorageSize = (1024 * 1024 * 800) />
This will make editing the max file size easy for anyone who does not know byte sizes.
Then, if you look on the spider.cfm, spider_gallery_links.cfm, spider_gallery.cfm, and spider_video.cfm pages, I am putting in a CFDirectory tag and an ArraySum() size check on the video storage directory. If the current size of the directory is greater than or equal to the max size allowed by the application, spidering is prevented. Additionally, I have put a special note on the home page, home.cfm, but there is nothing to "shut down" there.
I am not checking the size of the XML documents or the thumbnails. I figure these are such an insignificant means of hard drive space usage when compared to the videos themselves. If there is any concern, the max size of the application can always be lowered to accommodate thinking about all resources.
You will also notice that on the spider.cfm page, I am dealing with the max file size inline. Meaning, if the user has exceeded the file size usage, I tell them directly on that page, inline to the existing content. On other pages, however, such as the spider_gallery_links.cfm, spider_gallery.cfm, and spider_video.cfm, you will see that I am just performing a check and potentially forwarding the user onto another page. I just don't want those pages to get any more complicated then they have to. To keep all that checking inline on those pages would require the use of more IF statements and would make the page unnecessarily long.
As far as the pop-up spidering pages go (those mentioned above), they just forward onto a new page, spider_size_warning.cfm, whose only job is to display the size warning message. This makes everything easy and keeps the page logic as small as possible.
Now, we are, of course, violating the DRY principle. DRY stands for Do not Repeat Yourself. In this case, we are repeating ourselves on several page. This is something that will mostly likely be addressed in Phase II when we do general refactoring of the application.
I have also made some minor CSS modifications for display on the search page. Nothing big.
Reader Comments
Just ran across this by accident and want you to know that I appreciate you posting all the code iteration by iteration.
I'm learning several things by studying your code as you improve/add to it.
Your code viewer is very cool.
Bruce
Bruce,
I am glad you are finding the code posting useful. That is the objective that I am trying to accomplish. If you can think of anyway that I can make the whole process easier to understand or follow, please let me know.
One think that I will start doing that I just have not gotten around to is providing a ZIP file of all the code for the project. I appreciate you like the code viewer, but I also know that things are just easier to view on your local system. So, that should be up and running sometime this week.
Always glad to help and am dying for feedback on the project.
Thanks.
