Dynamic Pages Are Easy To Maintain

The content of static pages doesn’t change unless you actually code the changes into your HTML file: open the file, edit the content, save the file, and upload it to the server. All search engine spiders can index static Web pages.

A dynamic Web page is a template that displays specific information in response to queries. Most of the page content comes from the database connected to the Web site. Visitors love them since they get quick access to the information they want. These sites are easy for webmasters to update: as product offerings or prices change, just edit your database instead of hundreds of individual Web pages.

Search engine spiders have a much tougher time with dynamic sites. Some get stuck because they can’t supply the information the site needs to generate the page. Other spiders deliberately stay away from dynamic pages to avoid getting trapped in the site.

What Was The Question Again?

Visitors find information in a dynamic site by using a search query. That query can either be typed into a search form by the visitor or already be coded into a link on the home page – making the link a pre-defined search of the site’s catalog. In that later case, the portion of the link containing the search parameters is called a ‘query string.’

But a search engine spider doesn’t know to use your search function – or what questions to ask. Dynamic scripts often need certain information before they can return the page content: cookie data, session id, or a query string are common requirements. Spiders usually stop indexing a dynamic site because they can’t answer the question.

If the spider does accidentally wander deeper into your site, it could inadvertently get caught in a “spider trap”: a badly written CGI script that requests information the spider can’t supply. Then the spider and your server navigate a never-ending loop where a request for a page is met with a request for information.

Getting a spider trapped inside your server is not just bad for the spider. The repeated requests for pages can crash the server. If you share server space with other Web sites and have a problem with site downtime, ask your Web host to check for CGI script problems on other sites.

It’s All In The Name

A page’s actual URL address often poses a problem too because most dynamic page URLs contain query strings. Here’s an example of the URL for a book search result page on Barnes and Noble’s Web site:

http://shop.barnesandnoble.com/booksearch/isbnInquiry.asp?
userid=2IMXLT5XN1&mscssid=QEUFGRFF5X2G9H2UCMJQLAKJ8
JV83FMD&isbn=0452269350

Look closely at the URL. See the question mark after /isbnInquiry.asp? Most search engine spiders get to the “?” in the query string and stop indexing because of the probability of getting caught in a spider trap.

Attract Spiders To Your Web

So, you’ve got all this invisible content – what do you do? Search engines know about the problem, but most have shown very little interest in addressing it. Infoseek and HotBot are the exception. Their search engine spiders can index dynamic page content, but don’t do it automatically. You have to invite them in.

HotBot recommends that you submit your dynamic page with all the arguments added onto the URL (www.website.com/products/search/product_query.asp?prod_id=22929). You can also submit a static page that contains links to the dynamic URLs you need indexed.

Infoseek’s spider, called Slurp, will index dynamic pages that you submit, but won’t crawl through your dynamic Web site by default.

You do have options to get indexed by the other search engines, but no matter which you select, you’ll have to spend some time and effort to make sure your dynamic content gets indexed.

Add Dynamic Links To Static Pages

Include links to important dynamic content on your static pages. The simplest way is a straightforward table of contents page that links to your most important dynamic pages. It gives spiders a way to index content without having to answer any questions. If you have a small site with few products, this is a quick way to get more of your content indexed.

However, the table of contents won’t help you with search engine spiders that stop at query strings. Increase your chances by including good, descriptive links to your major product categories on a static products page. Search engines that stop at query strings will still index the content of the products page – including your link titles. Other search engines that can follow dynamic links can visit the actual dynamic page content without a query.

Remove Query Strings From Dynamic URLs

Amazon.com uses this method to get its product selections indexed by search engines. For instance, a search on Google for Rachael Carson’s book, Silent Spring, returns a result that takes you directly to the appropriate dynamic page at Amazon: www.amazon.com/exec/obidos/ISBN=0395683297/thenatiowomensha/. Because the URL doesn’t contain any query strings, all search engines can index Amazon’s product line.

This method works, but it’s also the most technically demanding solution. If you decide to use this method, you can select from several different options, depending on the type of Web server you use and the software you’re using to integrate your database with your Web site:

Cold Fusion: Reconfigure your Cold Fusion setup to replace the “?” in a query string with a ‘/’ and pass the value to the URL. The browser interprets that as a static URL page.

Instead of http://www.mystore.com/products.cfm?prod_id=22343, you get a string like this: http://www.mystore.com/products.cfm/22343.

CGI Scripts: Path_Info (or Script_Name) is a variable in a dynamic application that contains the complete URL address (including the query string information). Write a script that strips out all the information before the query string and set the balance of the information equal to a variable. You can then use that variable in your URL address.

Apache: has a special rewrite module that allows you to translate URL’s containing query strings into URL addresses that search engine spiders can follow. The module, mod_rewrite, isn’t automatically installed with Apache software. Check with your Web host or administrator and see if it’s available on your server.

Visit the Apache Web site for more information on the mod_rewrite module.

Active Server Pages: Most search engines will index .asp pages if the “?” is removed from the URL. XQASP offers a product that will automatically remove the query strings from your .asp pages and replace them with “/” marks.

A note of caution: these four methods make your dynamic page appear to have its own sub-directory, so the browser will look for images and links there. You can completely avoid broken links and pages by using all absolute URL addresses on your page, but that will make maintenance more difficult later. Alternatively, use URL addresses that are relative to the root directory of your site, not the document. Use /homepage.htm instead of ../homepage.htm and you’ll be fine.

Remember The Rules

Don’t get so caught up in modifying your page design or URL addresses that you forget the basic rules for search engine optimization. Your pages need to have good content, META tags, a high link popularity score, appropriate keywords, and more before you can climb to the top of the search engine ranking.