The Robot (also called Spider, or Crawler) is a great traveler. Driving his car from one city to another, then to a different country, all over the world. Well, but in our terms countries are actually websites, cities are web pages, and all the country roads, tracks and highways are links between different pages.
He inspects all the websites, evaluates them, and he's always on the phone to report to the Search Engine. The Search Engine updates its index with the data the Robot reports about your website. And, as soon as the Robot's made a world trip, he's ready to start yet another one, to check the old spots and explore new ones — to keep the Search Engine's index up-to-date.
And what's a Search Engine's index? It's a huge database, where record is kept of everything that robots find on the Web: web pages and any possible information on them. In terms of SEO, it's crucial for a web page that all that's good on it is recorded in the Search Engine's index (= is indexed). If it's not there, the page won't be found through search, or will not bring you the results you wish.
This all means that you want the Robot to visit all your important pages, and look at every detail that can be found there.
Now, one by one, let's see the things that are important for the Robot, and even more important for you.
Fast and reliable hosting
The Robot loves high speed and hates driving slowly. And, I'll better not tell you what he thinks of traffic jams.
So let's make a good and smooth road for him. That is, let's get fast and reliable hosting. This will guarantee that your web server is never down when a Search Engine spider tries to index it, and it's always fast enough, both for the Robot and for users.
Yep, it's really worth it. Why? We all know that a site can be down sometimes. If this only happens on very rare occasions, it's not that bad. But if you've got problems with hosting, and your site doesn't respond quite often, the Robot may leave this kind of site not checked. And, it's even possible that the website gets removed from the Search Engine database. Not to say that you simply might be losing sales, because users can't reach you.
So here's what I advise: host your site on reliable servers that are very seldom down and that are fast. By the way, your users will like speed as much as the Robot does.
The faster your hosting, the sooner your site loads, the more visitors like it.
You know the 8 second rule? Here it is:
If your web page hasn't loaded within 8 seconds, your users won't wait longer.
Create a sitemap
A sitemap is like a giant crossroads for the Robot. Or, it's like a city that has direct roads to any other city in the country. It's a spot from which the Robot can get to any place easily.
In its simplest terms, a sitemap is a list of the pages on your website. Generally, there are two types of sitemaps.
An HTML sitemap is made both for human users and for Search Engines and helps them easily find the information they need.
An XML Sitemap (it's normally called a Sitemap, with a capital S) is for Search Engines only. Create and submit a Sitemap, and thus you'll make sure that Search Engines know about all the pages on your site, including URLs that can't be naturally discovered by Search Engines' crawlers.
Now what do you need, so that the Robot can visit all your pages, fast? Right, an accurate overall sitemap. Here you can download a tool to make Sitemaps:
http://goog-sitemapgen.sourceforge.net/
In fact, there're hundreds of tools of the kind. You can use this one, for example:
http://www.xml-sitemaps.com/
As soon as you've made your Sitemap, you have to submit it to Search Engines. The procedure of submission varies for different Search Engines. I'll now only help you with three biggest ones. Here you can find guidelines for Google.com (they have a webmaster tools section that helps you out).
Here's a link to Yahoo.com submission form:
https://siteexplorer.search.yahoo.com/submit
(at the bottom of this page, you see a form to submit a URL of your sitemap file).
Right now, MSN.com doesn't have a similar submission form. So to submit your sitemap, simply add the following line to your robots.txt file:
Sitemap: http://www.yoursite.com/sitemap.xml
Provide the complete URL for your Sitemap on this line, and MSN's crawler will pick it up.
If you need to submit to other search engines, look them up for instructions. Go to the Search Engine you need and type in submit sitemap and name_of_the_search_engine. You'll find a submission form, or just some instructions, like in MSN.com. Follow the guidelines you get, and — here we go: you not only invited Search Engines' Robots to visit your site, but ensured comfortable and easy travel!
Note! Sitemap has links to all pages that you've got on your site. So when you make a new page, don't forget to add it to your sitemap, too. You won't need to submit it to Search Engines again, just update the sitemap itself.
Rewrite dynamic URLs
A common problem for online stores, forums, blogs or other database-driven sites is: pages often have unclear URLs like this: www.website.com/?id=32554, and you cannot say which good or article it leads to. Though instead, they could have www.website.com/latest-news, or www.website.com/pots.html, where you can easily see what's on the page.
So the problem with such URLs like this one www.wbsite.com/?id=32554 is: no one (neither users, nor even the Robot) can tell what product can be found under the URL.
URLs like this, www.website.com/?id=32554, having parameters (here it's id=32554) are called Dynamic URLs, while URLs like www.website.com/latest-news are static. First of all, static URLs are much more user-friendly. For users, URLs with too much of "?", "&" and "=" are hard to understand and pretty inconvenient. Secondly, search engines like static URLs much better than dynamic ones.
I probably wouldn't believe this myself, but one of the biggest players in SEO industry confirmed that their search traffic jumped 20% due to static URL use instead of dynamic URLs.
It's possible that you also need static URLs but have dynamic ones instead. But, I wouldn't talk so much of this problem, if it couldn't be solved. There's a nice trick to make URLs look good to Search Engines.
A .htaccess file is a plain-text file, and using it, you can make amazing tricks with your web server. Just one example is rewriting dynamic URLs. And then when a user (or a robot) is trying to reach a page, this file gets a command to show a page URL that is user- and crawler-friendly.
This is, basically, hiding dynamic URLs behind the SE-friendly URLs. I'll give you an example for an online store.
As a rule, a page URL for some product looks like this:
http://www.mysite.com/showgood.php?category=34&good=146
where there are two parameters:
category — the group of goods
good — the good itself
You should rewrite pages, so the Robot will see and you'll get "speaking URLs" that are understood by the Robot and easy to check.
Writing an .htaccess file is an uneasy task that requires special knowledge. Moreover, it's your webmaster's business. I personally never do this myself. So if you have a database-driven site, search the web for a special SEO service that will write a .htaccess file for you.
Or, if you're using a fairly well-known 3rd-party engine, you can write the .htacess file yourself, using some scripts that you can find in the Internet. To do the search, you can type in the_name_of_your_site's_engine "URL Rewrite" htaccess or something like that.
Now, the idea is: it's of great use to rewrite URLs. So find the URL rewrite tools if you need them — or just find your webmaster.
Then, one more thing, the old URLs that have parameters should be "hidden" from Search Engines.