search engines (General)

by BrianC, Thursday, April 19, 2007, 07:16 (6188 days ago) @ Alex
edited by Alfie, Thursday, April 19, 2007, 10:55

Thanks, Alex and done,

I've just tried some tests on our own forum with Google search and the results today are much better than a few months ago when I made a number of Google-recommended changes that are now causing our database to be indexed much more deeply. It's still a long way from comprehensive though so I'll read this further information you've provided, done, with much interest. We've got about 21 pages of posts each one about 40 strings deep. Confining the Google searches to short phrases that are likely to be quite particular to our discussion forum, Google searches about the first 4 pages exceedingly well. The performance falls off for older posts and for how deeply down a string one selects the particular search term from.

One of the tricks I learned from Google was placing a file labelled "robots.txt" in the forum folder with a list of files that need to be excluded if the robot is to search the database. In our case the list of exclussions is:

User-agent: *
Disallow: /img/
Disallow: /lang/
Disallow: /contact.php
Disallow: /email.php
Disallow: /register.php
Disallow: /user.php
Disallow: /admin.php
Disallow: /login.php
Disallow: /posting.php
Disallow: /board.php
Disallow: /board_entry.php
Disallow: /mix.php
Disallow: /mix_entry.php
Disallow: /inc.php
Disallow: /timedifference.php
Disallow: /db_settings.php
Disallow: /delete_cookie.php
Disallow: /info.php
Disallow: /more_smilies.php
Disallow: /upload.php
Disallow: /search.php
Disallow: /rss.php

I had to do a bit of tinkering with that list over a period of a month or so but it seems to have led to a significantly better, albeit still not perfect, indexing of our forum database by Google.

Cheers, Brian

locked
9288 views

Complete thread:

 RSS Feed of thread