Home > Articles

Creating an ASP Search Engine for your Website

A guide to using The Website Utility to create an ASP powered server-side Search Engine for your website.

Why Websites need Search Facilities

Once a website grows beyond a couple of dozen pages then it can sometimes be difficult to create a site navigation scheme that allows users to quickly find exactly what they're looking for. One way to improve site navigation is to add a search facility to the website. Adding a search facility brings major benefits to a website, making it easier for users to find information as well as adding an additional method of navigating a website. Search facilities are generally well used, and will often appear within the top ten most requested pages on a website.

Search Engines Allow Visitors to Search your Content

One of the easiest ways to add a facility for searching the pages in your website is to link to search results for your website from one of the major search engines. Google and other major search engines allow you to do this. However, using this method it can be difficult to integrate the search results with the design of your website. It also carries the obvious risk of a website visitor leaving your website and not returning! Even worse, your website visitors may see an advert for a competitor on the search results page, and so go and do their business elsewhere!

Building Your Own Search Engine

There are a number of software solutions that allow you to put your own search engine on your website. These include server-side search solutions available such as Microsoft's Index Server or ht://Dig. Although they allow sophisticated search facilities to be created, they generally require a high level of technical knowledge to install and configure. Another complication is that your chosen Internet Service Provider or hosting company may not support the search software required, or you may not be able to install additional software onto the server hosting the website.

A useful alternative is to use a server-side solution that only makes use of ASP, and does not require any other components to be installed onto a web server.

The Website Utility Builds ASP Search Engines

The Website Utility is a Microsoft Windows application that is able to create a server-side ASP search facility for a website. The resulting search facility requires no 3rd party components, databases or search software. It is also able to create client-side JavaScript search facilities for smaller websites.

The walkthrough below shows the steps involved in creating an ASP based search facility using The Website Utility:

Configuring The Website Utility to Produce ASP Search Engines

The Website Utility is configured using a small Windows application. There is a Create ASP/JavaScript Search Facility checkbox in the Report Settings part of the window that needs to be ticked in order for the ASP search facility to be created:

Note that if your website uses query strings then it is a good idea to tick the checkbox under Web Robot Settings called Use URL Query Strings. This will ensure that in the search results pages with different query strings will be treated as different search results pages. So for example www.mywebsite.net/news.php?ID=12 will link to a different news article from www.mywebsite.net/news.php?ID=21 and so The Website Utility will ensure they are indexed separately.

Running The Website Utility

Clicking on the Run button will start The Website Utility's web robot. This web robot start at a user specified page in the website and will automatically crawl all of the pages in that website. The Website Utility extracts all of the words from these pages, and finds the most relevant pages in the website for each word. Common English words (e.g. got, like, then) are removed, as are words of one or two characters. Word rankings depend on many factors, including their distribution through the entire website and their distribution in the content of a specific page.

Incorporating the ASP Search Facility into any Website

The Website Utility creates two ASP include files that can be used on the website's search results page:

• A Search Data ASP include file (TWUSearchData_JScript.asp) contains the rankings for each word and the most relevant pages for that word.
• A Search Code ASP include file (TWUSearchCode_JScript.asp) contains the code required to parse the user's search query and finds the most relevant pages for that query.

Note that the data and code files are created in JScript, but they can easily be used from within ASP pages that use VBScript.

Pages are sorted in search results according to their ranking for the particular word or words being searched for. The ranking scale goes from 0 to 99. Rank is higher for pages that most closely match the search term. In general, searching for words that are common on the site will produce search results with a lower rank than very specific words that occur on only one or two pages.

The search facility also requires a search form and a search results page. The search form can either be put on a separate search page on the site, or the search form could be added to all of the pages in a website (e.g. in the top right hand corner of the website's navigation). The HTML code for a typical search form is shown below. The search form needs a text box called TWUQuery. The form should use either the GET or POST form submission method to submit to the search results page.

<html>
<head>
<title>ASP JScript Search for http://www.brettb.com/</title>
</head>
<body>
<h1>Search http://www.brettb.com/</h1>
<form name="frmSearch" method="GET" action="searchresults.asp">

Search for: <input type="text" name="TWUQuery" maxlength="50">
<input type="submit" name="submitbutton" value="Submit">
</form>

</body>
</html>

The search results page (i.e. searchresults.asp) needs to include references to the two ASP include files created by The Website Utility (TWUSearchData_JScript.asp and TWUSearchCode_JScript.asp):

<%@Language="VBScript"%>
<script language="JScript" runat="Server">
var TWU_MaximumSearchResults = 20;
var TWU_DisplayPageTitles = true;
var TWU_DisplayPageDescriptions = true;
var TWU_DebugMode = false;
</script>
<!--#include file="TWUSearchData_JScript.asp"-->
<!--#include file="TWUSearchCode_JScript.asp"-->
<html>
<head>
<title>Search Results</title>
</head>

<body>
<h1>Search Results for "<%Response.Write(TWU_GetOriginalSearchQuery())%>"</h1>
<%Response.Write(TWU_DisplaySearchResults(TWU_GetModifiedSearchQuery()))%>

</body>
</html>   

The search results page can of course be customised to fit in with the existing design of your website. If you want to display the search terms the user was searching for, then use this ASP code (assuming VBScript is the default scripting language):

• <%Response.Write(TWU_GetOriginalSearchQuery())%>

To display the search results, place this ASP code where you want the search results to appear:

• <%Response.Write(TWU_DisplaySearchResults(TWU_GetModifiedSearchQuery()))%>

The search results page also defines four ASP JScript variables that can be used to change the output:

• TWU_MaximumSearchResults Controls the maximum number of pages that will be listed in the search results. This stops users getting confused by seeing large numbers of pages in the search results.
• TWU_DisplayPageTitles If set to true then the pages displayed in the search results will show their HTML titles as clickable links. If set to false then the URL is displayed instead. URLs are also shown if a page does not have a title. If the website does not contain accurate page titles you might have to turn this feature off.
• TWU_DisplayPageDescriptions If set to true then the pages displayed in the search results will show their HTML Description Meta Tags as page descriptions. If set to false then page descriptions are not included in the search results. If the website does not contain accurate page descriptions you might want to turn this feature off. Turning off descriptions will also increase search performance.
• TWU_DebugMode If set to true then debugging information is displayed (you should not need to use this).

Performance Issues

A server-side ASP search engine is obviously going to have a larger performance overhead compared to using a dedicated search solution such as Microsoft's Index Server or SQL Server's full text indexing capabilities. The size of the TWUSearchData_JScript.asp include file will depend on the number of pages in the website indexed, and also the amount of content on each page in the website. It is also dependent upon the nature of the website itself - websites with pages about similar subjects will tend to require a smaller file than a website with pages about different subjects.

To create a search facility for medium to large websites of 1000+ pages, or to allow the simultaneous searching of several websites at once, then Microsoft's Indexing Service is recommended. The Indexing Service Companion may also be used to extend the functionality of Indexing Services.

Download the Evaluation Version

The evaluation version of The Website Utility will allow you to determine whether the ASP Search Engines it creates are suitable for use on your own websites:

• Download The Website Utility Evaluation Version (3Mb ZIP file).

Sample Search Facilities

• Search Brettb.Com using an ASP search engine created by The Website Utility.
• Search part of ASPAlliance.Com using an ASP search engine created by The Website Utility.
• If you don't want to use ASP, it is also possible to build a client-side JavaScript search facility for your website.

Purchase The Website Utility

• The full version of The Website Utility is available for online purchase - visit The Website Utility's website for more information.

Useful Development Tools

ASP Documentation Tool™

Automatically creates developer documentation for ASP 2.0 and 3.0 web applications written in VBScript and JScript. Documentation for Microsoft Access, SQL Server 7/2000 databases and Visual Basic 6.0 components associated with the web application can also be incorporated into the reports. Documentation is created in HTML, HTML Help and plain text formats.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (5.2Mb ZIP file).

.NET Documentation Tool

Automatically creates technical documentation for .NET Framework Windows and ASP.NET applications written in C# or VB.NET and SQL Server 7/2000/2005 or Microsoft Access databases associated with the application. Documentation is created in HTML, HTML Help and plain text formats.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (5Mb ZIP file).

SQL Documentation Tool

The SQL Documentation Tool creates technical documentation for Microsoft SQL Server 7.0 and 2000 databases. Technical documentation is created in HTML and HTML Help formats. The HTML Help format documentation is fully searchable and cross referenced. The SQL Documentation Tool documents SQL Server Tables, Views, Stored Procedures, Triggers and Table Relationships.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (10.3Mb ZIP file).

VB Documentation Tool

The VB Documentation Tool creates technical documentation for Microsoft Visual Basic 6.0 projects. Technical documentation is created in HTML and HTML Help formats. The HTML Help format documentation is fully searchable and cross referenced.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (1Mb ZIP file).

Indexing Service Companion

The Indexing Service Companion is a Windows application that extends the functionality of the Microsoft Windows Indexing Service so that it is able to index content from remote websites and also from ODBC databases. As such it can be used as a low cost alternative to Sharepoint Portal Search Services.

Try Sample Search Facility.     Download Trial Version (1.7Mb ZIP file).

The Website Utility

The Website Utility examines websites for errors and areas that need to be optimised for search engines by using a built in web crawling engine. Errors checked for include broken or moved hyperlinks, missing page titles and missing meta tags. It also generates HTML for use in creating website site maps (table of contents pages - like this one), and is able to create both client-side JavaScript Search Engines and server-side ASP Search Engines for a website.

View Sample Output (HTML Format).     Download Trial Version (3Mb ZIP file).

PHP Documentation Tool™

Automatically creates developer documentation for PHP web applications. Documentation is created in HTML, HTML Help and plain text formats.

View Sample Output (HTML Help format).     View Sample Output (HTML Format).     Download Trial Version (1.0Mb ZIP file).