Home > Developer Tools

The Website Utility

Based on the web crawling technology used in the Index Server Companion, The Website Utility navigates a website and creates reports based on what it finds. The reports contain a number of features that can be used to help optimise a website for search engines and check the website for errors. A free trial version is available.

• Broken hyperlinks in the website.
• Hyperlinks pointing to content that has moved to another location.
• Missing or duplicated page titles.
• Missing description and keywords meta tags.
• Underutilised FrameSets.
• Pages with too little or too much content.
• Generates a basic table of contents (site map) for a website.
• Saves the HTML source code from each page on the website.
• Optionally compiles the reports into a single, searchable HTML Help file (requires Microsoft's HTML Help Workshop - a free download from Microsoft.com).
• Shows a web robot's (crawlers) view of a website.
• Creates website search facilities based on client-side JavaScript [read more] or server-side using either Active Server Pages (ASP) [read more] or ASP.NET.
• Creates a Google^TM SiteMap XML file for the website it analyses.

Because The Website Utility uses web crawling technology, it views websites in much the same way as human users and search engine robots view websites.

Search Engine Optimization Features

Web Robot View of Website

The Website Utility creates a simulated web robot (crawler) view of a website. As such it shows what a page will look without images, JavaScript or interactive forms. This can prove very helpful in ensuring that each page in the website contains enough search engine indexable text - something that is not always easy to check when viewing the actual website in a web browser.

Missing Page Titles

The Website Utility can identify pages with missing page titles. Since many search engines place significant weighting on a page's title, it is essential to ensure all pages have a title.

Duplicated Page Titles

The Website Utility identifies pages with duplicate page titles. The report will show each duplicated title and which pages share the specific title. A surprising number of websites use the same page title for each page in the site, which is in most instances an obvious waste of potential site optimisation for search engines.

Creation of Table of Contents/Site Maps

A table of contents page (often referred to as a Site Map) contains a list of links to all of the other pages in a website. There are several good reasons for adding a site map to a website:

• It is another page of content for your website
• It allows search engine spiders and robots easy access to all of the pages in a website. By submitting the site map page to the search engines it allows their spiders and robots to visit most of your website by following the links from just that one page.
• It allows human visitors another method of finding what they're looking for on a website.

The Website Utility creates a very basic table of contents, sorted alphabetically by page title.

Missing/Blank Description and Keywords Meta Tags

Although most search engines now give much less weighting to the content of meta tags than they once did, it is still a good idea to include them. In particular, the description meta tag is often used as a page's summary in search engine results pages.

Extraction of Description and Keyword Meta Tags

The Website Utility is able to extract the content from Description and Keyword meta tags in all of the pages in a website. This saves substantial time compared to having to extract the information manually.

• The Site Descriptions Report [view sample] created by The Website Utility displays the Description meta tags by page.

• The Site Keywords Report [view sample] displays keywords by page. It also displays a list of all of the keywords on a page, ordered by their frequency of occurence.

Don't forget that The Website Utility can be used to crawl your competitors' websites to determine what keywords they are using!

Page Size Report

The Website Utility shows the site's pages ordered by page size. Search engines normally prefer pages to be of a certain size, so The Website Utility makes it straightforward to identify pages that may be too small or too large for efficient search engine indexing and placement. Needless to say, The Website Utility will also identify pages that may be too large for human visitors as well! Consider splitting up large pages - increasing the number of pages in your website will help with search engine placement as well.

Usage of Meta-Refresh Tags

The Website Utility identifies pages that use Meta Refresh Tags with a short refresh interval. Pages that make use of these may be indicative of websites designed to confuse and mislead human or web robot visitors, so their usage should be avoided where possible. Furthermore, Microsoft's Internet Explorer web browser allows the user to disable the action of this tag in its security settings. Handy tip: The refresh tag can be replaced with the Response.Write() statement in websites that use Active Server Pages (ASP) technology. PHP has the "location:" statement equivalent.

Splash Screen Detection

The Website Utility can detect the use of splash screens on a website. Typically these are home pages that display the organisation's logo for a few seconds before redirecting to the main content page, or they contain a Macromedia Flash animation and a click to continue link below. Whilst splash screens look attractive (and keep web designers in business!) they result in human website visitors being an extra click away from the content they may be looking for. Splash screens can also adversely affect a site's search engines rankings as they mean the first (and most important) page in the website has very little indexable content on it.

Error Checking Features

Broken Hyperlinks

The Website Utility has the option of checking hyperlinks found in a website to ensure they are still accessible on the Internet. Both internal (i.e links to other pages within the website being checked) and external 3rd party websites can be checked. The Site Errors Report will list problem hyperlinks by the page they are found in.

Moved Hyperlinks

As well as finding broken hyperlinks, The Website Utility will also identify links to pages that have moved. Identification of these can allow you to update the links, which may prevent them becoming broken in future.

Spaces in Page Titles

Websites frequently contain URLs with spaces in them. These are not supported by older web browsers (e.g. Netscape 4). They are also confusing for people to write down or remember, so the URLs should be renamed where possible.

Identification of file:// Prefixed Links

The Website Utility can identify links prefixed with file://. These links can often be accidentally inserted into HTML when using a HTML Editor such as Microsoft's FrontPage. Furthermore, these links will usually work when the website is viewed on the same machine where the HTML pages were created, so it can be a difficult error to spot and to resolve. In the worst case scenario, it is an error that will be spotted by the website's customers, or even the client who commissioned it.

Creation of JavaScript Search Facilities

The Website Utility is able to create a client-side JavaScript search facility for a website it crawls using the internal web robot. Adding a search facility to a website brings major benefits to a website, making it easier to find information as well as adding an additional method of navigating a website. Search facilities are generally well used, and will frequently appear within the top ten most requested pages on a website.

Although there are a number of server-side search solutions available such as Index Server or ht://Dig, they can be complicated to install and configure. To create a search page with these solutions, programming knowledge of server-side scripting languages such as Active Server Pages (ASP) or PHP is also usually required, or you will need to pay somebody to create the code for you.

You can also link to search results for your website from Google or other major search engines, but it can be difficult to integrate these with the design of your website. They also carry the obvious risk of a website visitor leaving your website and not returning!

The JavaScript search facility created by The Website Utility is straightforward for anyone with a basic knowledge of HTML and JavaScript to add to their website. The search box can be added to any page on the website (so you could, for example, have a search facility in the corner of all pages - like this website!). The search results page can be easily integrated within the look and feel of an existing website.

• Try a sample search facility created by The Website Utility: Search Brettb.com.
• Further information on creating a JavaScript Search Engine for your Website with The Website Utility.

Creation of ASP and ASP.NET Search Facilities

For websites using either Microsoft's Active Server Pages (ASP) or ASP.NET server-side scripting technologies, The Website Utility also creates an ASP and ASP.NET search facility. This search facility is similar to the JavaScript search facility, but the search files are retained on the server, meaning there is less content for the client's web browser to download.

The ASP and ASP.NET versions of the search engine also allows page descriptions (from page Description meta tags) to be displayed in the search results.

• Try a sample search facility created by The Website Utility: Search Brettb.com.

The ASP.NET search facility is used on www.softwaredocumentation.info (the search box is in the top left hand corner of the website's pages). The ASP.NET search facility is created in C# and requires at least version 2.0 of the .NET Framework.

Google SiteMap Generator

Creating a Google SiteMap for your site can help determine how and where Google is or isn't indexing your website. Once registered with Google, you gain access to valuable statistics about your website. These can often give your websites the edge over your competitors when it comes to getting your website noticed online!

The Website Utility is a simple way to create Google SiteMaps for your websites. Read more about creating Google SiteMaps using The Website Utility.

Download the Evaluation Version

The evaluation version is fully function except for the following limitations:

• Only the first 5 errors/issues found in a website are shown in the RTF format reports.
• Annoying JavaScript popups on some pages in the report.
• HTML source code pages only show the first 50 lines of the page's HTML.
• Various restrictions on other reports.
• JavaScript/ASP/ASP.NET search facilities created contain a small subset of the website's indexable content.

• Download The Website Utility Evaluation Version (3Mb ZIP file).

Purchase The Website Utility

• The Website Utility is available for immediate online purchase - visit The Website Utility's website for more information.

Useful Development Tools

ASP Documentation Tool™

Automatically creates developer documentation for ASP 2.0 and 3.0 web applications written in VBScript and JScript. Documentation for Microsoft Access, SQL Server 7/2000 databases and Visual Basic 6.0 components associated with the web application can also be incorporated into the reports. Documentation is created in HTML, HTML Help and plain text formats.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (5.2Mb ZIP file).

.NET Documentation Tool

Automatically creates technical documentation for .NET Framework Windows and ASP.NET applications written in C# or VB.NET and SQL Server 7/2000/2005 or Microsoft Access databases associated with the application. Documentation is created in HTML, HTML Help and plain text formats.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (5Mb ZIP file).

SQL Documentation Tool

The SQL Documentation Tool creates technical documentation for Microsoft SQL Server 7.0 and 2000 databases. Technical documentation is created in HTML and HTML Help formats. The HTML Help format documentation is fully searchable and cross referenced. The SQL Documentation Tool documents SQL Server Tables, Views, Stored Procedures, Triggers and Table Relationships.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (10.3Mb ZIP file).

VB Documentation Tool

The VB Documentation Tool creates technical documentation for Microsoft Visual Basic 6.0 projects. Technical documentation is created in HTML and HTML Help formats. The HTML Help format documentation is fully searchable and cross referenced.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (1Mb ZIP file).

Indexing Service Companion

The Indexing Service Companion is a Windows application that extends the functionality of the Microsoft Windows Indexing Service so that it is able to index content from remote websites and also from ODBC databases. As such it can be used as a low cost alternative to Sharepoint Portal Search Services.

Try Sample Search Facility.     Download Trial Version (1.7Mb ZIP file).

The Website Utility

The Website Utility examines websites for errors and areas that need to be optimised for search engines by using a built in web crawling engine. Errors checked for include broken or moved hyperlinks, missing page titles and missing meta tags. It also generates HTML for use in creating website site maps (table of contents pages - like this one), and is able to create both client-side JavaScript Search Engines and server-side ASP Search Engines for a website.

View Sample Output (HTML Format).     Download Trial Version (3Mb ZIP file).

PHP Documentation Tool™

Automatically creates developer documentation for PHP web applications. Documentation is created in HTML, HTML Help and plain text formats.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (1.0Mb ZIP file).