A Brief History of Search Engine Marketing on the Web
Jan 24, 2019
Changes in the search engine world happen at breakneck speed. If you’ve ever wondered why some clients still believe the meta keywords tag will boost their rankings, a quick history lesson might help. Anyone working in SEO and SEM today needs to understand where it all began—to manage client expectations, grasp current best practices, and anticipate what’s coming next. So dust off your time machines, and let’s travel back together…
1998: The Spam Era
Once upon a time, search engine marketing (SEM) was known simply as search engine optimization (SEO), and it relied heavily on formulaic algorithms and automatic submissions. Search engines used indexing programs to scan on-page text and applied a “secret formula” to determine rankings. This period could also be called the Golden Age of Spam, because repeating a keyword many times—or using other similar tricks—could easily earn a high ranking. (Yahoo! was an exception; it has always used human editors who could identify and ban spammy websites.)
Gradually, search engines began recognizing spam and penalizing it accordingly. Unscrupulous SEOs kept inventing new ways to fool indexing algorithms, and search engines were always one step behind. Clearly, innovation was needed to take control away from spammers and auto-submitters. One early anti-spam breakthrough was incorporating off-the-page information.
1999: Incorporating Off-the-Page Information
Around mid-1999, search engines started using the collective intelligence of users to improve search quality. DirectHit (later acquired by Ask Jeeves) introduced technology that tracked which sites searchers actually chose. In this system, a website frequently selected for a given keyword would rise in ranking for that keyword. More clicks = higher rank. Higher rank = more clicks. This gave large, popular sites an advantage over smaller competitors.
Another way to harness collective intelligence was to rank pages based on how many other pages linked to them. This became known as “link popularity” and remains a crucial ranking factor. Again, large, popular sites benefited.
Both of these parameters are called “off-the-page” factors because they aren’t directly based on your webpage’s content. Their appeal? Spam efforts couldn’t easily influence them. Google, for example, made excellent use of link popularity in its famous PageRank algorithm.
1999: Community-Edited Directories
Human-edited directories delivered quality results and resisted spam, but often lacked quantity due to the limited number of editors. Yahoo!, primarily a human-edited directory, was constrained by its editorial staff’s size and struggled to keep up with the vast number of sites deserving inclusion.
A solution emerged: the community-edited directory. This model allowed thousands of self-governing editors to continuously improve and expand a directory. The Netscape Open Directory Project was the first of its kind, and the now-defunct Go.com directory was another early leader. Zeal.com (which fed results into Looksmart.com and MSN Search) was a newer addition.
The Netscape Open Directory had another bright idea: being open source. That meant any website could display its directory content for free. As a result, by the year 2000, Open Directory listings began appearing on nearly every major search engine, along with many portals and other sites across the web.
The combination of scale and quality made community-edited directories a key player in the search world.
2000: Paid Listings
In 1999, AltaVista tried introducing paid listings, was universally rejected, and dropped the idea almost immediately. By the end of 2000, however, all major search engines offered some form of paid listings. Overture (formerly Goto.com) started this trend and remains a major player, offering keyword rankings based on an auction system: the higher your bid, the higher your rank. The bid amount is charged each time a user clicks your listing—a model known as pay-per-click (PPC) or “pay-for-performance.” Overture was later joined by Google AdWords, and together they became the two dominant forces in the PPC market.
Additionally, by 2001 most directories and search engines introduced various payment models for submissions: submission fees for major human-edited directories, indexing fees for major spiders, and several types of ad placement opportunities. Yahoo! launched a “Business Express” submission fee for commercial content (later changed to an annual fee). Looksmart switched from a one-time submission fee to a PPC model. Major spiders like Inktomi, AltaVista, and FAST Search introduced paid inclusion models—annual fees guaranteeing that a site would be listed and regularly re-crawled.
By 2001, a typical SEM campaign involved at least some form of spending, either annually or on a per-click basis.
2001: Google’s Rise to Prominence
The years 2000–2001 marked Google’s rise to prominence among search engines. While others focused on becoming “portals,” Google kept an extremely simple, fast interface focused solely on search results. Google also developed advanced features like PDF and Flash search. Moreover, Google’s use of off-the-page factors like link popularity made it highly resistant to spam. Google’s dominance was sealed when Yahoo! switched its secondary search results partnership from Inktomi to Google in July 2000 (Yahoo! later began using Google for its primary results). Many additional partners followed, including AOL and Ask Jeeves.
With Google’s success, several competitors tried to follow its lead: Wisenut, Teoma, and FAST Search offered simple, fast results. Other threats to Google included the Inktomi spider (now owned by Yahoo!). It remains to be seen whether any of these will succeed in chipping away at Google’s monopoly.
2002–2003: Hybridization and Partnerships
Hybridization of search results meant search engines could combine the best of all worlds: high-quality listings from editorially reviewed directories, a large quantity of pages indexed by major spiders (Inktomi, Google, and others), and revenue from PPC listings. By 2001, all major search engines delivered hybrid results. For example, Yahoo! mixed Yahoo! Directory results, Overture (PPC) results, and Google results. MSN delivered results from Overture (PPC), Looksmart, and Inktomi.
In 2002, several major acquisitions took place: Yahoo! bought Inktomi, Overture purchased FAST, and many search engine partnership contracts shifted. Yahoo!’s purchase of Overture in 2003 was another step toward industry consolidation.
Consolidation certainly simplified the SEM professional’s job—no longer did you need to worry about submissions and optimization for five or more major search engines. For novice searchers, consolidation may even have been an improvement. A beginner without the savvy to try multiple sources might have better luck with one large, hybrid search engine. However, this came at a loss to the searching public overall. Fewer independent editorial staffs and index databases meant fewer options when a searcher had special needs. For example, at the time, I often used Yahoo! Directory to find official company sites, Froogle for shopping searches, Google for most other searches, and AltaVista or FAST if Google didn’t deliver what I needed. Search engine consolidation threatened to take away much of that choice and control.
2003: The Little Guys Get In on PPC with Contextual Advertising
“Contextual advertising” is a general term for sponsored links that appear on content sites across the web, not just on search engines. Originally, it described systems (such as eZula and Gator) that created hyperlinks within the HTML of content pages. For example, if your webpage contained the word “car,” these systems would alter how your page appeared to users, turning every instance of “car” into a link to their search results. Before 2003, these systems didn’t obtain content providers’ permission, didn’t label links as ads, didn’t share revenue with content providers, and required browser plug-ins often installed without users’ knowledge or consent. Because of these flaws, the SEM community never embraced this approach.
In 2003, the “little guy” got a legitimate chance to join the PPC bandwagon with the launch of Google AdSense. AdSense allowed almost any web content provider to display Google AdWords (PPC links) on their site and receive a share of the revenue. The ads were clearly labeled and separated from content, similar to banner ads. Overture was expected to launch a similar service soon. The success of such services would depend on good editorial review (or excellent algorithms) to ensure ads appeared on well-matched pages. My hope was that the system would evolve to let content providers choose which ads ran on their pages—creating a natural, built-in editorial review to ensure relevant, well-matched advertising and maximum click-through rates.
2004 and Beyond: The Future of Search Engine Marketing
Search engines have evolved into sophisticated systems and will continue to enhance their technical capabilities: improved indexing of “deep” and dynamically generated pages, indexing of non-HTML content (PDFs, Flash, graphics), faster integration of new content (e.g., news via XML feeds), organization of results into logical categories, artificial intelligence, personalized search, and other advanced features. (For a glimpse of the future, visit Google Labs and Overture Research.) These advances will improve the search experience for everyone. Results will be more accurate, and spam techniques will become a thing of the past. The site optimization tricks of 1998–1999 will represent only a tiny fraction of SEM.
At the same time, corporate mergers, consolidation, and the old-fashioned “highest-bidder” approach are making search engines feel more and more like traditional advertising venues. SEM firms will need to offer services similar to ad agencies: understanding client business goals and messages, managing advertising channels, and designing campaigns that capture user interest and attention.
Finally, the SEM professional will need to take a holistic approach to the user experience of searching for, finding, and using a business website. This could include usability studies, conversion tracking, and research into the user experience on a website. When all is said and done, a business website’s success depends on much more than its search engine ranking.
Site optimization, advertising management, and holistic online marketing analysis will each be key components of online marketing in the future. The question is whether these services will be provided by standalone SEM firms or absorbed into ad agencies and marketing departments. I believe the SEM specialty firm will survive for at least the next several years. After that, some SEM firms may be bought by ad agencies, some work will be absorbed into corporate marketing departments, and some will be provided by PPC engine sales agents. My hope is that my favorite part of SEM—consulting with companies on their business goals and developing customized online marketing strategies, including usability, writing, and design—will survive and thrive as a successful, vital industry in its own right.