We all use "search engines" to find information
on the Internet, but what is a Search Engine? Yahoo!, Google, Ask,
or AOL may come to mind. You might find just what you're looking for
at any one of these, yet two of these search sites are "search engines"
and two are "directories" — and they each work differently.
Google and Ask are search engines. Here's how a search
engine works:
- A spider (or "crawler") visits, reads, and
follows links throughout the web, revisiting every one to two months
to try and keep it up to date.
- What the spider finds goes into the index part of the
engine which catalogs every page the spider finds. However, even though
a site is spidered regularly, it may take a while to be indexed and
if it's not yet indexed, it's not yet available through the search engine.
With some search engines, this can take several months.
- This brings us to the search engine software. This software
sifts through the millions of indexed pages to find matches and attempts
to rank them in some sort of relevant order. These are the results you
see when you do a search.
Other popular search engines include MetaCrawler, i-Won,
Lycos, WebTop, and AltaVista.
Yahoo! and Looksmart are examples of a directory. Others
include Netscape, Galaxy, Excite, and Magellan. Directories depend upon
human beings for its listings. When submitting a website to directories,
some require only the URL, others also want you to provide descriptions
and keywords. Some rely on human editors to search the web or look at
submissions and write a description or review of a site they think should
be included. This also means they may not include a site they feel isn't
somehow useful or important. At About.com, for example, you have to find
out who the editor is for the section you'd like to be included in, then
email them directly asking them to please add your website.
When these directories are searched by web users, it looks
for matches based solely on the descriptions. If your site changes, the
directory won't know about it unless the editor is somehow informed, and
even then, the changes will be made if they decide it needs to be changed.
Yahoo! is an example of this. They reserve "the right to edit or
refuse change requests." It can take months for a website change
to show up on some directories.
To make things more confusing, some search engines maintain
an associated directory. These are called "hybrid search engines."
Being included is usually a combination of luck and quality. As with directories,
a website can be submitted for review with no guarantees it will be included.
There's a search engine called Inktomi that can't be used as a search
engine directly online, but it provides database information for AOL,
MSN, i-Won, HotBot, LookSmart, GoTo.com (Overture), About.com, eoexchange.com,
powerize.com, NBCi, C|Net, GeoCities, i-Atlas, 4-Anything, ICQ, N2H2,
StarMedia, and eight other international search sites. Only very recently
were you even able to submit your site to Inktomi (cost: $30 for the first
page, $15 for additional pages — more about the new trend of charging
for submissions in an upcoming installment). Previously you had to see
that your site got a decent placement on one of the site's Inktomi served
in hopes that it would get back to Inktomi, then back out to all their
other sites ... whew.
Published 2001