Where Are Search Engines Going? 
Paid Inclusion Trend Emerges
by Mike Banks Valentine 

The Search Engine Strategies conference and show, sponsored
by AltaVista, Search Engine Watch and Internet.com on August
16-17 provided a glimpse of several emerging search trends,
the biggest trend is toward "Paid Inclusion." The show, held
in San Francisco at the Fairmont Hotel on Nob Hill, provided
a look at where search is headed. 

Paid inclusion is the latest of several revenue producing
models considered by search engines as the cost of indexing
a much larger and more complex web increases and advertising
revenue declines. Paid inclusion was started quietly by several
previously free search engines as a revenue producing move
recently. 

Virtually everyone scrambled for new business models when
banner advertising effectiveness declined steeply in 2001.
Search engines had relied heavily on banner advertising to
generate the only income they were seeing in sparse times.
When the tech economy took a nosedive in 2001, the
justification given by advertising companies for continued
banner advertising was now as a "Branding" strategy rather
than a sales technique. 

To satisfy investor demands for such absurd things as "profit"
the search engines began probing for new cash infusions and
realized that they could no longer rely on advertising to
support free submissions. Since nobody was looking at banner
ads any more (or at least not clicking through) support for
free search listing nearly disappeared.  

Who benefits from search engines most? Those who receive the
traffic from searches. Simple. Who should pay for that traffic?
Those who gain that traffic. Simple. How do you charge them?
Ah! Now things get more complex. 

Goto.com was the first to introduce the novel idea of charging 
businesses for traffic generation directly with "Keyword Bids"
starting at a penny per click-through. Now companies began to
bid for top positions knowing that being at the top of the list
was worth more traffic than being further down the page. 

Reaction to the CPC or cost per click model in 1998 was nearly
unanimous from the web community, especially while banner
advertising revenues were still a viable business model for
search engines. It was, "You can't charge for search listings!
It'll never fly!" "Searchers won't trust the listings they
receive on a pay-per-click basis because they are now
'tainted' with commercial results!" At that time, in 1998,
nobody believed first, that people would pay for traffic and
second, that searchers would trust paid listings.

It very quickly became apparent after the decline in revenue
from banner advertising in 2000  and the dismal performance
of banners in delivering traffic that pay-per-click was
actually going to work for GoTo. At that point several new
pay for performance models were adopted by start-up PPC
search engines, and being first to market with the idea,
GoTo dominated the crowd.

It was then that the 900 pound gorilla, Yahoo stunned the
web with a $199 charge for a review of a commercial web
site. Wait, not $199 to be listed, but $199 to be LOOKED
AT by a reviewer. It was still not a guarantee of inclusion
in the directory! Search engines started thinking about
that and realized that Yahoo had the clout to demand money
to be reviewed, not listed, but reviewed.

 While the major search engines stewed on the dramatic
move by Yahoo, they quickly realized that demanding money
to be reviewed would not work for them because they actively
sought out sites by sending "spiders" out to "crawl" the web
and index pages. Search engines wanted to be exhaustive in
their coverage of the web and catalog the entire thing, not
just the good sites, but ALL of them!

When Google surpassed the 1 billion page mark they tooted
loudly that they had indexed more of the web than anyone
else. This was seen as a milestone in search history and
became the goal of many of the top search engines. Let's
index the entire web! That prospect becomes very expensive
for search companies and someone has to pay for it. But how?
We can't all become pay-per-click engines if we want to index
ALL the web, because only a limited number will pay to be
listed. So they all stewed about it some more.

Meanwhile, several other directories followed the lead of
Yahoo and began to charge to be listed. It became accepted
at multiple directory sites, notably at LookSmart and NBCi.
Several special interest and topical directories had been
charging for listings longer, but directories that listed
everyone had a harder time justifying those charges. Yahoo
can do it becuase the entire world knows of Yahoo and wants
to be seen there. Vertical portals can charge because they
draw a very targeted searcher seeking specific businesses.
It's worth paying for that targeted traffic.

When the goal is to index the entire web and results are far
less relevant and traffic less likely to result in sales to
the listed sites, how do you charge sites to be listed? Hmmmm.
The search engines said, "We've been getting complaints about
a couple of things from sites seeking listings, one that they
submit and never get listed, and two that when they are listed
it is only months later that they begin to benefit from
delivered search engine traffic." AHA! An opportunity presents
itself to charge money to someone for listings! 

Bingo! Paid inclusion is launched at Inktomi and becomes an
instant success with those who have been frustrated by slow
listing times and especially with those that can't seem to be
listed at all at important engines. Here is a way to get
listed and get listed sooner, I'll pay for that! But how much?
And are there any benefits above the free submission to entice
the reluctant web business into paying for what, until now has
been free. We'll guarantee, not only that you'll be listed,
but quickly, and to sweeten the pot, we'll recrawl your site
on a weekly basis to guarantee fresh listings.

Now it's time for everyone to leap into the new model and
begin more frequent crawls for preferred listees. Certainly
this will lead even further toward the commercialization of
the web, which angers many but is the only solution to
supporting the increasingly expensive proposistion of
providing complex search to millions and indexing a
dramatically larger web.

Time to innovate if you expect to attract those paid inclusion
listings. At Fast AllTheWeb, they've come up with an intra-
search facility that site owners with under a specified number
of pages can install as a branded, hosted search for their own
sites as an additional benefit. A hosted and branded site
search will likely cost them very little more to implement
for paid inclusion customers, while increasing revenue for
Fast. As Fast is a partner to the Lycos network, this paid
inclusion not only comes with intrasearch for your site and
regular site recrawls to keep search results fresh, but also
gets listings into the Lycos network as well. This program
is in conjunction with their partners at Lycos.

 http://searchservices.lycos.com/searchservices/select_overview.asp

A further enhancement in the race to paid inclusion will be
extra cost "partner" or "sponsor" links that turn up at the
top of the search results pages. This may affect some search
partners differently than others as some now use GoTo paid
listings as those "sponsor or partner" links. Certainly GoTo
will fight to maintain current partners by making themselves
more attractive to those search engines considering paid
inclusion programs.

Inktomi has the larger partner list to support their version
of paid inclusion and is more attractive for distribution and
visibility, but offers little in the way of "perks" for the
value driven crowd. Altavista offers their own version of
paid inclusion, dividing the program into two separate
categories, one for small businesses and sites under 500
pages and the other for the big boys of ebusiness with more
than 500 pages to index.

http://www.inktomi.com/products/search/pagesubmission.html

http://www.altavista.com/sites/search/express_incl

Pricing is just now being determined by many search engines
making the paid inclusion move, but it ranges anywhere from
$30 per page to be listed up to yearly pricing models which
come with the additional benefits, such as the intra-search
option from Fast, AllTheWeb.com.

I recommend keeping an eye out for short term sign-up bonuses
and perks as the paid inclusion model gains acceptance over
the next 6 months. Just as the pay-per-click model of search
took some time to gain traction, so too will paid inclusion.

Watch also for partnering changes in the near term as larger
networks battle to gain or maintain market share. Some will
be more agressive than others so bargains and benefits will
no doubt emerge within 6 to 8 months to build customer and
brand loyalty early in the race to paid inclusion dominance.