Are you validating your site’s code and checking your site for broken links? You should be and until now you have probably used the W3C Markup Validation Service and something like Xenu’s Link Sleuth.
Now there is a new online tool that performs batch (X)HTML validation, link checking and more. It is called Nikita The Spider and is the brainchild of Philip Semanchuk. Philip found himself managing a growing number of Web pages and wanted to keep those pages valid and low on link rot without having to check each individual page and link. He couldn’t find a tool that would do that and since he is a programmer decided to build one himself.
Currently it is in Alpha test so sometimes you may have to wait to use it but register your interest by sending Philip an email (address on Nikita’s home page) and you can be one of the first to give it a try. (At the time of writing you can start a crawl without waiting).
The link checker will find broken internal and external hyperlinks and references to missing documents such as a missing PDF or MP3 file and uses the same validation parser engine as the W3C Validator. You get well organized HTML reports that summarize what Nikita has found on each page of your site and if you want to analyze your site in a way that these reports don’t you can have XML versions and reorganize the data yourself.
Here are a couple of screen shots:


Nikita can spider your whole site or just a portion of it and it has an online interface for ad hoc queries. For example you can show a list of pages that are delivered as text/html, pages that are larger than 100k or pages that use a certain doctype.
There are other features too. It allows you to control the speed at which your site is spidered and you can define a custom user agent so that you can filter Nikita’s visits from your Web logs. Also you receive statistics about your site such as a list of the doctypes you use, average page size and URL length.
Philip is very open to suggestions as to future features and even has a page where you can vote for the ones you like. After beta obviously power users will have to pay however he intends to always offer a free version with a limit to the number of pages per crawl.
This tool is from the top drawer and has a very bright future indeed.
Philip has also written an excellent article about protecting email addresses from spam with the results of a most interesting experiment.


Nikita The Spider - Read More...