Transact SQL Other Articles Software Reviews
Building a Search Engine Friendly Website
The Internet is full of useful articles about how to build your website using technologies such as ASP and ASP.NET. But even though you may have the most technically brilliant website around, it's going to be of limited use if the website isn't able to be crawled by the web robots that Google, Yahoo and the other search engines use to find content to add to their search catalogs.
This article highlights some of the surprisingly common issues that affect search engine robots. The problems can be divided into two groups:
If you want to test your website for many issues, then an application I have created called The Website Utility can be used. It contains a built-in web crawler which can be used to test whether web robots can successfully navigate a website. It also reveals common website search engine optimization issues.
These are serious issues that will prevent a web crawling robot from indexing some or in a worst case scenario, all of a website's pages. Here are a few commonly encountered problems:
Web crawling robots may have difficulties navigating websites. These issues can be hard to spot, since most web browsers are tolerant of sloppy HTML. To resolve possible errors, try using an HTML validator such as the W3C's HTML validation service.
You should pay particular care to ensuring that the website's internal hyperlinks are using valid HTML. For example, this is a valid link:
<a href="default.asp" title="Home">Home Page</a>
Whereas this link may not be recognised by a web crawling robot:
<a href=default.asp title=Home>Home Page</a>
Robots.txt or Robots Meta Tag Issues
Most web robots, will respect both a web server's robots.txt file as well as the 'robots' meta tag that may be present in individual pages. These specify which pages a robot is allowed to crawl and retrieve content from.
If a website, or portions of that website cannot appear to be submitted successfully to search engines it is possible that the search engine is not accepting that website due to it being marked as non-retrievable in either the robots.txt file or in individual files through the use of the 'robots' meta tag.
Some web crawling robots are also able to support session cookies, such as those used by ASP and ASP.NET.
Missing Page Titles
Missing page titles is a common search engine optimisation deficiency. Since many search engines place significant weighting on a page's title, it is essential to ensure all pages have a title.
Duplicated Page Titles
Alongside missing page titles, duplicate page titles is another common issue. The report will show each duplicated title and which pages share the specific title. A surprising number of websites use the same page title every single page in the site, which is in most instances an obvious waste of potential site optimisation for search engines.
Missing/Blank Description and Keywords Meta Tags
Insufficient Indexable Content
Use of Splash Pages/Screens
Splash screens are home pages that display the organisation's logo for a few seconds before redirecting to the main content page, or they contain a Macromedia Flash animation and a click to continue link below.
Whilst splash screens look attractive (and keep web designers in business!) they result in human website visitors being an extra click away from the content they may be looking for. From a search engine placement point of view, splash screens can also adversely affect a site's search engines rankings as they mean the first (and most important) page in the website has very little indexable content on it, few links to other pages in the website and often use redirection to other pages.
Testing Your Website
A good way of testing to ensure your website is search engine friendly is to crawl it using a web crawler.
The Website Utility is a Windows application that can be used to test the crawlability of websites. It contains a built-in web crawler which can be used to test whether web robots can successfully navigate a website. It also reveals common website search engine optimization issues and errors such as broken links.
Finally, remember that testing a website for errors and search engine optimisation issues will almost certainly result in higher levels of satisfaction from the owner - useful if your livelihood depends on happy customers and repeat business!