Sending Search-Engine Traffic to Your Site
by Paul Boutin 27 Jul 1999

Paul Boutin discovered the Internet in 1980 and hasn't slept since. After stints at MIT in the ‘80s and Wired Digital in the ‘90s, Paul is now developing Web resources for Sendmail, because the only thing more challenging than Perl haiku is sendmail haiku.

In the three years I worked on HotBot, I read thousands of email messages and handled dozens of phone calls from people trying to get better placement for their sites in HotBot's search results. There is actually an entire consulting industry based on tweaking Web pages to get them to the top of search-engine results in the hope of capturing more traffic. The specialists call it search-engine optimization, but search-engine workers just label it spam because it usually involves some sort of trickery.

But after looking at search results for years, it's become obvious to me – and to the search engineers I talked to while writing this article – that most sites could get their pages much better placement on search engines without using deception.

Getting users to pick your site out of the sea of cluttered search results is a matter of how effectively you implement the three precepts of successful search placement: get crawled, get ranked, and get clicked.

Step 1: Get Crawled
Searchers won't find your pages if the search engines haven't found them first. Most search engines try to crawl as much of every site as they can, but it's harder than it sounds (especially if your site features dynamic content).

Imagine if you tried to compile a list of all the phone numbers in your hometown by calling everyone whose number you know and asking them (if they even answer) for the names and the phone numbers of all the people they know. Then repeat the process until you think you have all the numbers in town and the names that go with them. That's pretty much how ÒspideringÓ works. It's a process most search engines use to index your site.

Wouldn't it be easier if someone just handed you a phone book?

Actually, you can give search engines the equivalent of a phone list for your site by creating a crawler page, which is basically just a list of every URL you want indexed by search engines. Because the list is at the top of your site, the crawler is more likely to grab it (and therefore all your favorite URLs) before it gets lost chasing ad banners and links to other sites — and this is exactly what it will do once it starts hopping among the lower-level pages on your site.

With a crawler page, one URL is worth a thousand words. Take the master crawler page for all Wired News stories, for instance. Every story ever published by Wired News is listed one hop away from that page, with no images or other interruptions to block the way of a search spider intent on crawling the entire tree of story URLs. Once a search engine finds this crawler page, it will harvest every story within minutes, guaranteed.

A more complicated site like HotWired has a multilevel set of crawler pages, where the top crawler page points to other crawler pages or to the site's archive index pages. Each section is maintained by a different person, but a crawler will find all pages by going down the list.

Submit Early and Often
Once you've created your crawler pages, don't wait for the search engines to come to you. Most search engines allow you to submit URLs for them to crawl, so submit the URL of your crawler page. Be patient: It can take weeks and even months to get indexed for the first time.

You also need to be persistent; no search engine can index every URL submitted to it these days. All the Web, All the Time, for instance, should really be called "10-percent-of-the-web.com" (seeing as it's only 80 million pages strong, which is one-tenth of the estimated 1 billion Web pages out there). Search sites like these are running out of the storage space necessary to hold indexing info about the Web, which is huge and getting bigger by the hour. As a result, every time a search engine picks up new pages, it has to throw out other ones to make room. And it may toss yours if you haven't resubmitted them since the last crawl.

Since none of the search engines will penalize you for submitting your URL on a regular basis (as far as I know), I suggest you keep pestering them so they don't forget about you (though I don't recommend you make a full-time job of it – think once a week, not once an hour).

If you don't have the time or energy to submit regularly to every search engine you can find, just pick the ones with the most traffic. Site traffic follows a steep curve, so the top sites have as much traffic as all other sites on the Web put together. Sites listed on Yahoo! say it accounts for as much as 50 percent of their inbound traffic — as much as all other search engines combined. So just take the top 50 domains listed by Media Metrix and eliminate all but one of the engines powered by Inktomi. (By submitting to one, you submit to them all.) Diligently submit your crawler page to the short list that remains, and you should have all the traffic you can stand.

There are services that will submit your site to hundreds of search engines for a fee. But webmasters have told me that they often don't really work, and most of those search engines have no traffic to tap anyway, certainly when compared to the top 10 players. So as with many things in life, youÕre better off doing the job yourself. next page» Feedback | Help | About Us | Contribute | Jobs | Advertise Editorial Policy | Privacy Statement | Terms and Conditions Copyright © 1994-2000 Wired Digital Inc., a Lycos Network site. All rights reserved. Sending Search-Engine Traffic to Your Site Page 3 —

Step 2:
Get Ranked Playing the cops-and-robbers game of search engine optimization is where people tend to waste the majority of their time. There are three obvious ways to get your Web page to come up at the top of the search results for a specific search phrase. The funny thing is that the search engine managers I talked to said most people never try these. They are: keep it short, use title and meta tags, and make a separate page for each search phrase.

Keep it short.
Most pages have too much text in them to score well for one particular phrase. A page that has a few words, including "Web consulting," will score higher than one that goes on and on and doesn't mention Web consulting until the 10th paragraph. After all, which page is more likely to be only about Web consulting?

Use title and meta tags.
All search engines rank the title and meta descriptions and keywords highly, as well they should. So our page about Web consulting should contain these tags: <TITLE>Web consulting</TITLE> <META NAME="description" content+"web consulting"> <META NAME="keywords" content+"web consulting">

Many pages have the company or site name in the title and even some sort of marketing pitch in the description field. But a page titled "WebHedz: The ultimate Web consulting specialists for Java, dHTML, and e-commerce" will score lower than one that just says Web consulting.

Make a separate page for each search phrase.
The spammers know this one: If you want your site to come up for "Web consulting," "dHTML," and "Java," make three separate pages. Each should have title and meta tags that are optimized to score high for one of those phrases. Put all three of them in your crawler page. Submit the crawler page.

That said, there are a lot of tricks that search-engine spammers – er, optimization specialists – use to get their clients' pages to come out on top for search phrases that have nothing to do with the content on their clientsÕ Web sites. And it's for this reason that search engines spend a lot of time learning how to filter out these tricks. Spammers spent tens of thousands of hours creating pages with invisible text that only the search engines will see. Nevertheless, the search engineers figure out how to spot and ignore this kind of text within a few short months. So tempting though it may be, your best bet is to avoid blatant trickery.

However, no matter how you go about getting your pages on top, all that jockeying for position will be meaningless unless users actually click on your URL. next page» Sending Search-Engine Traffic to Your Site Page 4 —

Step 3:
Get Clicked Having result No. 1 for "Web development" won't do you any good unless people actually visit your site. But if you've followed our advice thus far about having a short, topical title tag and a description tag, you're guaranteed to improve your clickthrough performance. Almost all search engines use the contents of the title tag as the name of the link to your site, and many use your meta description as the abstract (the short blurb that runs below the title on the search engine's list of results). And what people click on most are pages that have either titles related to their search terms or abstracts that accurately describe what they're looking for.

For example, if a user looks at HotBot results for a search on "Web development," guess which one of the three results below gets clicked on 30 times more often than the other two?

1. Technology News from Wired News Wired News: top Internet news headlines, technology news, stories about the World Wide Web, business news, and coverage of digital culture and politics relating to the wired world

2. Sex MP3 porn real estate Web development HTML warez picx001.jpg picx002.jpg picx010.jpg picx011.jpg picx030.jpg picx032.jpg picx43.jpg

3. Cumberland Web Development Cumberland Web Development specializes in helping small businesses use the Internet, especially the Web, to market its products or services. We design and build Web sites

This may seem like blindingly obvious advice, but if you take a close look at most commercially produced Web pages, you'll see that more often than not they resemble result No. 1 above. Contrary to search logic, the title is the company's name and branding, while the description is some sort of marketing message.

The Wired News page has worked its way into the No. 1 spot for this search, because it was a story about Web development. However, users' eyeballs will skip right over it. Users have also learned to avoid "girls, girls, girls" sites like the second result. But Cumberland gets the serious traffic from HotBot, because users can see that itÕs what they were looking for even before they click on it.

And as more and more searchers click on your site, your ranking will only continue to improve.

It's Getting Better All the Time
Search spammers will always find ways to cheat around all three of these steps in an effort to capture traffic. They'll register a dozen domains and put the same pages on all of them. TheyÕll use invisible text (or whatever the next great spam invention is). They'll use fake titles, keywords, and descriptions, luring surfers who want Jeff Bezos to pages about Jenny McCarthy.

But I've spent a lot of time following up on complaints, and the pattern has been consistent: Sites that follow the above methods still manage to float to the top of the sea of spammers, and they get more traffic.

And the more searchers who visit your on-the-level site, the higher it will rise in the rankings. Just look at HotBot's top 10 results for Jeff Bezos, which are scored by tracking where surfers go after using HotBot. Also see HotBot's general Web results, scored by how many other sites have linked to each result. Note that most of the results have good titles and abstracts. You may also notice that none of them are spam pages from porn sites or real estate dealers. These new popularity-based ranking systems on HotBot have taught us something very important: Spam can trick search-engine users into clicking once, but they wonÕt click again unless they find exactly what theyÕre looking for.

You don't have to be Jeff Bezos to figure that out.