search engine spider tool updates

The search engine spider tool on this site was recently updated and while I was testing it I realized that the spider lacked accuracy. The script for that spider is very old (from 2005 I think). It made me wonder how I could make it better. I tried a few things and realized I would have to start from scratch. But I am glad I did.

The tool is now much faster and more accurate. To make it more accurate I stopped using regular expressions, and started using DOM methods which are more like a real search engine spider.

It is also simpler. I use the spider as a way to see if something obvious is wrong with a webpage, and it is now even easier to see.

I also added some new features that allow users to see potential problems with their titles and descriptions. If they are too short the tool will tell you. Short titles and descriptions are often part of a problem with ranking, especially if titles and descriptions are short throughout a website. So this is a great addition to the tool.

I also removed some features, or at least streamlined them. The previous spider showed keyword density and that is just not something people should be worrying about. Instead of a matrix of percentages the tool just highlights four of the most used words.

This means that at a glance you can see if those words make sense to the page content. I also made the overall results of the tool shorter by hiding the full text and the link list (you can unhide them by clicking a button). This allowed a more cohesive look to the tool and in the earlier versions of the tool the amount of info returned could be quite overwhelming.

I am making tools that are useful to normal people, not for SEOs or marketers. Most of my tools will be updated to reflect this and I will be making lots of new tools in the near future. Hope you are around to see them, thanks.

Hey, go check out the tool!