search engine optimization

Webmasters and content providers began optimizing websites for search engines in the mid-1990s, as the first search engines were cataloging the early Web. Initially, webmasters submitted the address of a page, or URL to the various search engines, which would send a web crawler to crawl that page, extract links to other pages from it, and return information found on the page to be indexed.{{cite web| url=http://www.thinkpink.com/bp/Thesis/Thesis.pdf| title=Finding What People Want: Experiences with the WebCrawler| access-date=May 7, 2007| publisher=The Second International WWW Conference Chicago, USA, October 17–20, 1994| author=Brian Pinkerton| archive-date=May 8, 2007| archive-url=https://web.archive.org/web/20070508124837/http://www.thinkpink.com/bp/Thesis/Thesis.pdf| url-status=live}}

According to a 2004 article by former industry analyst and current Google employee Danny Sullivan, the phrase "search engine optimization" came into use in 1997. Sullivan credits SEO practitioner Bruce Clay as one of the first people to popularize the term.{{cite web|url=http://forums.searchenginewatch.com/showpost.php?p=2119&postcount=10|title=Who Invented the Term "Search Engine Optimization"?|author=Danny Sullivan|date=June 14, 2004|publisher=Search Engine Watch|archive-url=https://web.archive.org/web/20100423051708/http://forums.searchenginewatch.com/showpost.php?p=2119|archive-date=23 April 2010|access-date=May 14, 2007}} See [https://groups.google.com/group/alt.current-events.net-abuse.spam/browse_thread/thread/6fee2777dc17b8ab/3858bff94e56aff3?lnk=st&q=%22search+engine+optimization%22&rnum=1#3858bff94e56aff3 Google groups thread] {{Webarchive|url=https://web.archive.org/web/20130617012709/http://groups.google.com/group/alt.current-events.net-abuse.spam/browse_thread/thread/6fee2777dc17b8ab/3858bff94e56aff3?lnk=st&q=%22search+engine+optimization%22&rnum=1#3858bff94e56aff3 |date=June 17, 2013 }}.

Early versions of search algorithms relied on webmaster-provided information such as the keyword meta tag or index files in engines like ALIWEB. Meta tags provide a guide to each page's content. Using metadata to index pages was found to be less than reliable, however, because the webmaster's choice of keywords in the meta tag could potentially be an inaccurate representation of the site's actual content. Flawed data in meta tags, such as those that were inaccurate or incomplete, created the potential for pages to be mischaracterized in irrelevant searches.{{Citation|chapter=The Challenge is Open|date=2020-11-17|title=Brain vs Computer|pages=189–211|publisher=WORLD SCIENTIFIC|doi=10.1142/9789811225017_0009|isbn=978-981-12-2500-0|s2cid=243130517}}{{dubious|date=October 2012}}

Web content providers also manipulated attributes within the HTML source of a page in an attempt to rank well in search engines.{{cite web |url=http://www.csse.monash.edu.au/~lloyd/tilde/InterNet/Search/1998_WWW7.html |title=What is a tall poppy among web pages? |date=April 1998 |website=Monash University |access-date=May 8, 2007 |author=Pringle, G. |author2=Allison, L. |author3=Dowe, D. |archive-date=April 27, 2007 |archive-url=https://web.archive.org/web/20070427161650/http://www.csse.monash.edu.au/~lloyd/tilde/InterNet/Search/1998_WWW7.html}} By 1997, search engine designers recognized that webmasters were making efforts to rank in search engines and that some webmasters were manipulating their rankings in search results by stuffing pages with excessive or irrelevant keywords. Early search engines, such as Altavista and Infoseek, adjusted their algorithms to prevent webmasters from manipulating rankings.{{cite news|url=https://query.nytimes.com/gst/fullpage.html?res=940DE0DF123BF932A25752C1A960958260|title=Desperately Seeking Surfers|date=November 11, 1996|newspaper=New York Times|author=Laurie J. Flynn|access-date=May 9, 2007|archive-date=October 30, 2007|archive-url=https://web.archive.org/web/20071030131226/http://query.nytimes.com/gst/fullpage.html?res=940DE0DF123BF932A25752C1A960958260|url-status=live}}

By relying on factors such as keyword density, which were exclusively within a webmaster's control, early search engines suffered from abuse and ranking manipulation. To provide better results to their users, search engines had to adapt to ensure their results pages showed the most relevant search results, rather than unrelated pages with numerous keywords by unscrupulous webmasters. This meant moving away from heavy reliance on term density to a more holistic process for scoring semantic signals.{{cite magazine|url=https://www.forbes.com/sites/jaysondemers/2016/01/20/is-keyword-density-still-important-for-seo/2/#2ef69ba36733|title=Is Keyword Density Still Important for SEO|author=Jason Demers|date=January 20, 2016|magazine=Forbes|access-date=August 15, 2016|archive-date=August 16, 2016|archive-url=https://web.archive.org/web/20160816221641/http://www.forbes.com/sites/jaysondemers/2016/01/20/is-keyword-density-still-important-for-seo/2/#2ef69ba36733|url-status=live}}

Search engines responded by developing more complex ranking algorithms, taking into account additional factors that were more difficult for webmasters to manipulate.{{Citation needed|date=January 2025}}

Some search engines have also reached out to the SEO industry and are frequent sponsors and guests at SEO conferences, webchats, and seminars. Major search engines provide information and guidelines to help with website optimization. Google has a Sitemaps program to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website.{{cite web|url=https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview|title=Sitemaps|access-date=July 4, 2012|archive-date=June 22, 2023|archive-url=https://web.archive.org/web/20230622175619/https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview|url-status=live}} Bing Webmaster Tools provides a way for webmasters to submit a sitemap and web feeds, allows users to determine the "crawl rate", and track the web pages index status.

In 2015, it was reported that Google was developing and promoting mobile search as a key feature within future products. In response, many brands began to take a different approach to their Internet marketing strategies.{{Cite web |url=https://www.startupgrind.com/blog/mobile-is-the-internet-for-consumers/ |title="By the Data: For Consumers, Mobile is the Internet" Google for Entrepreneurs Startup Grind September 20, 2015. |access-date=January 8, 2016 |archive-date=January 6, 2016 |archive-url=https://web.archive.org/web/20160106040341/https://www.startupgrind.com/blog/mobile-is-the-internet-for-consumers/ |url-status=live }}

In 1998, two graduate students at Stanford University, Larry Page and Sergey Brin, developed "Backrub", a search engine that relied on a mathematical algorithm to rate the prominence of web pages. The number calculated by the algorithm, PageRank, is a function of the quantity and strength of inbound links.{{cite web|author1=Brin, Sergey|author2=Page, Larry|name-list-style=amp|url=http://www-db.stanford.edu/~backrub/google.html|title=The Anatomy of a Large-Scale Hypertextual Web Search Engine|publisher=Proceedings of the seventh international conference on World Wide Web|year=1998|pages=107–117|access-date=May 8, 2007|archive-date=October 10, 2006|archive-url=https://web.archive.org/web/20061010084452/http://www-db.stanford.edu/~backrub/google.html|url-status=live}} PageRank estimates the likelihood that a given page will be reached by a web user who randomly surfs the web and follows links from one page to another. In effect, this means that some links are stronger than others, as a higher PageRank page is more likely to be reached by the random web surfer.

Page and Brin founded Google in 1998.{{cite web|title=Co-founders of Google - Google's co-founders may not have the name recognition of say, Bill Gates, but give them time: Google hasn't been around nearly as long as Microsoft. |website=Entrepreneur |url=http://www.entrepreneur.com/article/197848|date=2008-10-15|access-date=May 30, 2014|archive-date=May 31, 2014|archive-url=https://web.archive.org/web/20140531124147/http://www.entrepreneur.com/article/197848|url-status=live}} Google attracted a loyal following among the growing number of Internet users, who liked its simple design.{{cite news|author=Thompson, Bill|url=http://news.bbc.co.uk/1/hi/technology/3334531.stm|title=Is Google good for you?|work=BBC News|date=December 19, 2003|access-date=May 16, 2007|archive-date=January 25, 2009|archive-url=https://web.archive.org/web/20090125130328/http://news.bbc.co.uk/1/hi/technology/3334531.stm|url-status=live}} Off-page factors (such as PageRank and hyperlink analysis) were considered as well as on-page factors (such as keyword frequency, meta tags, headings, links and site structure) to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more difficult to game, webmasters had already developed link-building tools and schemes to influence the Inktomi search engine, and these methods proved similarly applicable to gaming PageRank. Many sites focus on exchanging, buying, and selling links, often on a massive scale. Some of these schemes involved the creation of thousands of sites for the sole purpose of link spamming.{{cite web|author1=Zoltan Gyongyi|author2=Hector Garcia-Molina|name-list-style=amp|url=http://infolab.stanford.edu/~zoltan/publications/gyongyi2005link.pdf|title=Link Spam Alliances|publisher=Proceedings of the 31st VLDB Conference, Trondheim, Norway|year=2005|access-date=May 9, 2007|archive-date=June 12, 2007|archive-url=https://web.archive.org/web/20070612023948/http://infolab.stanford.edu/~zoltan/publications/gyongyi2005link.pdf|url-status=live}}

By 2004, search engines had incorporated a wide range of undisclosed factors in their ranking algorithms to reduce the impact of link manipulation.{{cite news|newspaper=New York Times|access-date=June 6, 2007|url=https://www.nytimes.com/2007/06/03/business/yourmoney/03google.html|title=Google Keeps Tweaking Its Search Engine|date=June 3, 2007|first=Saul|last=Hansell|archive-date=November 10, 2017|archive-url=https://web.archive.org/web/20171110133529/https://www.nytimes.com/2007/06/03/business/yourmoney/03google.html|url-status=live}} The leading search engines, Google, Bing, and Yahoo, do not disclose the algorithms they use to rank pages. Some SEO practitioners have studied different approaches to search engine optimization and have shared their personal opinions.{{cite web |first=Danny |last=Sullivan |url=https://www.searchenginewatch.com/2005/09/29/rundown-on-search-ranking-factors/ |title=Rundown On Search Ranking Factors |publisher=Search Engine Watch |date=September 29, 2005 |access-date=May 8, 2007 |author-link=Danny Sullivan (technologist) |url-status=live |archive-url=https://web.archive.org/web/20070528133132/http://blog.searchenginewatch.com/blog/050929-072711 |archive-date=May 28, 2007 }} Patents related to search engines can provide information to better understand search engines.{{cite web|author=Christine Churchill|url=http://searchenginewatch.com/showPage.html?page=3564261|title=Understanding Search Engine Patents|publisher=Search Engine Watch|date=November 23, 2005|access-date=May 8, 2007|archive-url=https://web.archive.org/web/20070207222630/http://searchenginewatch.com/showPage.html?page=3564261|archive-date=February 7, 2007|df=mdy-all}} In 2005, Google began personalizing search results for each user. Depending on their history of previous searches, Google crafted results for logged in users.{{cite web|url=http://searchenginewatch.com/3563036|title=Google Personalized Search Leaves Google Labs|work=searchenginewatch.com|publisher=Search Engine Watch|archive-url=https://web.archive.org/web/20090125065500/https://www.searchenginewatch.com/3563036|archive-date=January 25, 2009|access-date=September 5, 2009}}

In 2007, Google announced a campaign against paid links that transfer PageRank.{{cite web|url=https://www.searchenginejournal.com/8-things-we-learned-about-google-pagerank/5897/|title=8 Things We Learned About Google PageRank|date=October 25, 2007|publisher=www.searchenginejournal.com|access-date=August 17, 2009|archive-date=August 19, 2009|archive-url=https://web.archive.org/web/20090819080745/http://www.searchenginejournal.com/8-things-we-learned-about-google-pagerank/5897/|url-status=live}} On June 15, 2009, Google disclosed that they had taken measures to mitigate the effects of PageRank sculpting by use of the nofollow attribute on links. Matt Cutts, a well-known software engineer at Google, announced that Google Bot would no longer treat any no follow links, in the same way, to prevent SEO service providers from using nofollow for PageRank sculpting.{{cite web|url=https://www.mattcutts.com/blog/pagerank-sculpting/|title=PageRank sculpting|publisher=Matt Cutts|access-date=January 12, 2010|archive-date=January 6, 2010|archive-url=https://web.archive.org/web/20100106120723/http://www.mattcutts.com/blog/pagerank-sculpting/|url-status=live}} As a result of this change, the usage of nofollow led to evaporation of PageRank. In order to avoid the above, SEO engineers developed alternative techniques that replace nofollowed tags with obfuscated JavaScript and thus permit PageRank sculpting. Additionally, several solutions have been suggested that include the usage of iframes, Flash, and JavaScript.{{cite web |url=http://searchengineland.com/google-loses-backwards-compatibility-on-paid-link-blocking-pagerank-sculpting-20408 |title=Google Loses "Backwards Compatibility" On Paid Link Blocking & PageRank Sculpting |date=June 3, 2009 |publisher=searchengineland.com |access-date=August 17, 2009 |archive-date=August 14, 2009 |archive-url=https://web.archive.org/web/20090814212229/http://searchengineland.com/google-loses-backwards-compatibility-on-paid-link-blocking-pagerank-sculpting-20408/ |url-status=live }}

In December 2009, Google announced it would be using the web search history of all its users in order to populate search results.{{cite web|url=https://googleblog.blogspot.com/2009/12/personalized-search-for-everyone.html|title=Personalized Search for everyone|access-date=December 14, 2009|archive-date=December 8, 2009|archive-url=https://web.archive.org/web/20091208140917/http://googleblog.blogspot.com/2009/12/personalized-search-for-everyone.html|url-status=live}} On June 8, 2010 a new web indexing system called Google Caffeine was announced. Designed to allow users to find news results, forum posts, and other content much sooner after publishing than before, Google Caffeine was a change to the way Google updated its index in order to make things show up quicker on Google than before. According to Carrie Grimes, the software engineer who announced Caffeine for Google, "Caffeine provides 50 percent fresher results for web searches than our last index..."{{cite web |url=http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html |title=Our new search index: Caffeine |publisher=Google: Official Blog |access-date=May 10, 2014 |archive-date=June 18, 2010 |archive-url=https://web.archive.org/web/20100618160021/http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html |url-status=live }} Google Instant, real-time-search, was introduced in late 2010 in an attempt to make search results more timely and relevant. Historically site administrators have spent months or even years optimizing a website to increase search rankings. With the growth in popularity of social media sites and blogs, the leading engines made changes to their algorithms to allow fresh content to rank quickly within the search results.{{cite web |title=Relevance Meets Real-Time Web |publisher=Google Blog |url=http://googleblog.blogspot.com/2009/12/relevance-meets-real-time-web.html |access-date=January 4, 2010 |archive-date=April 7, 2019 |archive-url=https://web.archive.org/web/20190407221454/http://googleblog.blogspot.com/2009/12/relevance-meets-real-time-web.html |url-status=live }}

In February 2011, Google announced the Panda update, which penalizes websites containing content duplicated from other websites and sources. Historically websites have copied content from one another and benefited in search engine rankings by engaging in this practice. However, Google implemented a new system that punishes sites whose content is not unique.{{cite web|title=Google Search Quality Updates|publisher=Google Blog|url=http://googleblog.blogspot.com/2011/02/finding-more-high-quality-sites-in.html|access-date=March 21, 2012|archive-date=April 23, 2022|archive-url=https://web.archive.org/web/20220423234246/https://googleblog.blogspot.com/2011/02/finding-more-high-quality-sites-in.html|url-status=live}} The 2012 Google Penguin attempted to penalize websites that used manipulative techniques to improve their rankings on the search engine.{{cite web|title=What You Need to Know About Google's Penguin Update|work=Inc |date=June 20, 2012|publisher=Inc.com|url=http://www.inc.com/aaron-aders/what-you-need-to-know-about-googles-penguin-update.html|access-date=December 6, 2012|archive-date=December 20, 2012|archive-url=https://web.archive.org/web/20121220235821/http://www.inc.com/aaron-aders/what-you-need-to-know-about-googles-penguin-update.html|url-status=live |last1=Aders |first1=Aaron }} Although Google Penguin has been presented as an algorithm aimed at fighting web spam, it really focuses on spammy links{{Cite news|url=http://searchengineland.com/google-penguin-looks-mostly-link-source-says-google-260902|title=Google Penguin looks mostly at your link source, says Google|date=2016-10-10|work=Search Engine Land|access-date=2017-04-20|language=en-US|archive-date=April 21, 2017|archive-url=https://web.archive.org/web/20170421001835/http://searchengineland.com/google-penguin-looks-mostly-link-source-says-google-260902|url-status=live}} by gauging the quality of the sites the links are coming from. The 2013 Google Hummingbird update featured an algorithm change designed to improve Google's natural language processing and semantic understanding of web pages. Hummingbird's language processing system falls under the newly recognized term of "conversational search", where the system pays more attention to each word in the query in order to better match the pages to the meaning of the query rather than a few words.{{cite web|title=FAQ: All About The New Google "Hummingbird" Algorithm|url=https://searchengineland.com/google-hummingbird-172816|website=www.searchengineland.com|date=September 26, 2013|access-date=17 March 2018|archive-date=December 23, 2018|archive-url=https://web.archive.org/web/20181223110045/https://searchengineland.com/google-hummingbird-172816|url-status=live}} With regards to the changes made to search engine optimization, for content publishers and writers, Hummingbird is intended to resolve issues by getting rid of irrelevant content and spam, allowing Google to produce high-quality content and rely on them to be 'trusted' authors.

In October 2019, Google announced they would start applying BERT models for English language search queries in the US. Bidirectional Encoder Representations from Transformers (BERT) was another attempt by Google to improve their natural language processing, but this time in order to better understand the search queries of their users.{{Cite web|title=Understanding searches better than ever before|url=https://blog.google/products/search/search-language-understanding-bert/|date=2019-10-25|website=Google|language=en|access-date=2020-05-12|archive-date=January 27, 2021|archive-url=https://web.archive.org/web/20210127042834/https://www.blog.google/products/search/search-language-understanding-bert/|url-status=live}} In terms of search engine optimization, BERT intended to connect users more easily to relevant content and increase the quality of traffic coming to websites that are ranking in the Search Engine Results Page.

File:PageRanks-Example.svg algorithm. Percentage shows the perceived importance.]]

The leading search engines, such as Google, Bing, Brave Search and Yahoo!, use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine-indexed pages do not need to be submitted because they are found automatically. The Yahoo! Directory and DMOZ, two major directories which closed in 2014 and 2017 respectively, both required manual submission and human editorial review.{{cite web|url=http://searchenginewatch.com/showPage.html?page=2167881|title=Submitting To Directories: Yahoo & The Open Directory|date=March 12, 2007|access-date=May 15, 2007|publisher=Search Engine Watch|archive-url=https://web.archive.org/web/20070519052103/http://searchenginewatch.com/showPage.html?page=2167881|archive-date=May 19, 2007|df=mdy-all}} Google offers Google Search Console, for which an XML Sitemap feed can be created and submitted for free to ensure that all pages are found, especially pages that are not discoverable by automatically following links{{cite web|url=http://www.google.com/support/webmasters/bin/answer.py?answer=40318&topic=8514|title=What is a Sitemap file and why should I have one?|access-date=March 19, 2007|archive-date=July 1, 2007|archive-url=https://web.archive.org/web/20070701232719/http://www.google.com/support/webmasters/bin/answer.py?answer=40318&topic=8514|url-status=live}} in addition to their URL submission console.{{cite web |url=https://www.google.com/webmasters/tools/submit-url |title=Search Console - Crawl URL |access-date=2015-12-18 |archive-date=August 14, 2022 |archive-url=https://web.archive.org/web/20220814100500/https://accounts.google.com/_/bscframe |url-status=live }} Yahoo! formerly operated a paid submission service that guaranteed to crawl for a cost per click;{{cite web|url=http://searchenginewatch.com/showPage.html?page=2167871|title=Submitting To Search Crawlers: Google, Yahoo, Ask & Microsoft's Live Search|date=March 12, 2007|access-date=May 15, 2007|publisher=Search Engine Watch |first1= Danny |last1=Sullivan |archive-url=https://web.archive.org/web/20070510090932/http://searchenginewatch.com/showPage.html?page=2167871#Teoma|archive-date=May 10, 2007|df=mdy-all}} however, this practice was discontinued in 2009.

Search engine crawlers may look at a number of different factors when crawling a site. Not every page is indexed by search engines. The distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled.{{cite web|url=http://ilpubs.stanford.edu:8090/347/|title=Efficient crawling through URL ordering|author1=Cho, J. |author2=Garcia-Molina, H. |author3=Page, L. |year=1998|work=Seventh International World-Wide Web Conference |location=Brisbane, Australia |publisher=Stanford InfoLab Publication Server |access-date=May 9, 2007|archive-date=July 14, 2019|archive-url=https://web.archive.org/web/20190714141416/http://ilpubs.stanford.edu:8090/347/}}

Mobile devices are used for the majority of Google searches.{{cite web|url=https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html|title=Mobile-first Index|access-date=March 19, 2018|archive-date=February 22, 2019|archive-url=https://web.archive.org/web/20190222000527/https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html|url-status=live}} In November 2016, Google announced a major change to the way they are crawling websites and started to make their index mobile-first, which means the mobile version of a given website becomes the starting point for what Google includes in their index.{{cite web |last1=Phan |first1=Doantam |title=Mobile-first Indexing |url=https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html |website=Official Google Webmaster Central Blog |access-date=16 January 2019 |date=4 November 2016 |archive-date=February 22, 2019 |archive-url=https://web.archive.org/web/20190222000527/https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html |url-status=live }} In May 2019, Google updated the rendering engine of their crawler to be the latest version of Chromium (74 at the time of the announcement). Google indicated that they would regularly update the Chromium rendering engine to the latest version.{{Cite web|url=https://webmasters.googleblog.com/2019/05/the-new-evergreen-googlebot.html|title=The new evergreen Googlebot|website=Official Google Webmaster Central Blog|language=en|access-date=2020-03-02|archive-date=November 6, 2020|archive-url=https://web.archive.org/web/20201106072307/https://webmasters.googleblog.com/2019/05/the-new-evergreen-googlebot.html|url-status=live}} In December 2019, Google began updating the User-Agent string of their crawler to reflect the latest Chrome version used by their rendering service. The delay was to allow webmasters time to update their code that responded to particular bot User-Agent strings. Google ran evaluations and felt confident the impact would be minor.{{Cite web|url=https://webmasters.googleblog.com/2019/10/updating-user-agent-of-googlebot.html|title=Updating the user agent of Googlebot|website=Official Google Webmaster Central Blog|language=en|access-date=2020-03-02|archive-date=March 2, 2020|archive-url=https://web.archive.org/web/20200302132028/https://webmasters.googleblog.com/2019/10/updating-user-agent-of-googlebot.html|url-status=live}}

{{main|Robots exclusion standard}}

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish to crawl. Pages typically prevented from being crawled include login-specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.{{cite web|url=http://searchengineland.com/newspapers-amok-new-york-times-spamming-google-la-times-hijacking-carscom-11169|title=Newspapers Amok! New York Times Spamming Google? LA Times Hijacking Cars.com?|publisher=Search Engine Land|date=May 8, 2007|access-date=May 9, 2007|archive-date=December 26, 2008|archive-url=https://web.archive.org/web/20081226161450/http://searchengineland.com/newspapers-amok-new-york-times-spamming-google-la-times-hijacking-carscom-11169|url-status=live}}

In 2020, Google sunsetted the standard (and open-sourced their code) and now treats it as a hint rather than a directive. To adequately ensure that pages are not indexed, a page-level robot's meta tag should be included.{{cite web|url=https://www.practicalecommerce.com/google-downgrades-nofollow-directive-now-what|title=Google Downgrades Nofollow Directive. Now What?|publisher=Practical Ecommerce|author=Jill Kocher Brown|date=February 24, 2020|access-date=2021-02-11|archive-date=January 25, 2021|archive-url=https://web.archive.org/web/20210125080754/https://www.practicalecommerce.com/google-downgrades-nofollow-directive-now-what|url-status=live}}

A variety of methods can increase the prominence of a webpage within the search results. Cross linking between pages of the same website to provide more links to important pages may improve its visibility. Page design makes users trust a site and want to stay once they find it. When people bounce off a site, it counts against the site and affects its credibility.{{Cite book|last=Morey|first=Sean|title=The Digital Writer.|publisher=Fountainhead Press|year=2008|pages=171–187}}

Writing content that includes frequently searched keyword phrases so as to be relevant to a wide variety of search queries will tend to increase traffic. Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a web page's metadata, including the title tag and meta description, will tend to improve the relevancy of a site's search listings, thus increasing traffic. URL canonicalization of web pages accessible via multiple URLs, using the canonical link element{{cite web|url=https://blogs.bing.com/webmaster/2009/02/12/partnering-to-help-solve-duplicate-content-issues|title=Bing – Partnering to help solve duplicate content issues – Webmaster Blog – Bing Community|date=February 12, 2009 |publisher=www.bing.com|access-date=October 30, 2009|archive-date=June 7, 2014|archive-url=https://web.archive.org/web/20140607164839/http://blogs.bing.com/webmaster/2009/02/12/partnering-to-help-solve-duplicate-content-issues/|url-status=live}} or via 301 redirects can help make sure links to different versions of the URL all count towards the page's link popularity score. These are known as incoming links, which point to the URL and can count towards the page link's popularity score, impacting the credibility of a website.

File:Demystifying Search Engine Optimization -viz notes- -wcyvr (8084222814).jpg

SEO techniques can be classified into two broad categories: techniques that search engine companies recommend as part of good design ("white hat"), and those techniques of which search engines do not approve ("black hat"). Search engines attempt to minimize the effect of the latter, among them spamdexing. Industry commentators have classified these methods and the practitioners who employ them as either white hat SEO or black hat SEO.{{cite web|author=Andrew Goodman|publisher=SearchEngineWatch|url=http://searchenginewatch.com/showPage.html?page=3483941|title=Search Engine Showdown: Black hats vs. White hats at SES|access-date=May 9, 2007|archive-url=https://web.archive.org/web/20070222004138/http://searchenginewatch.com/showPage.html?page=3483941|archive-date=February 22, 2007|df=mdy-all}} White hats tend to produce results that last a long time, whereas black hats anticipate that their sites may eventually be banned either temporarily or permanently once the search engines discover what they are doing.{{cite web|author=Jill Whalen|url=http://www.searchengineguide.com/whalen/2004/1116_jw1.html|title=Black Hat/White Hat Search Engine Optimization|publisher=searchengineguide.com|access-date=May 9, 2007|date=November 16, 2004|author-link=Jill Whalen

|archive-url=https://web.archive.org/web/20041117131237/http://www.searchengineguide.com/whalen/2004/1116_jw1.html

|archive-date=17 November 2004}}

An SEO technique is considered a white hat if it conforms to the search engines' guidelines and involves no deception. As the search engine guidelines{{cite web|url=http://www.google.com/webmasters/guidelines.html|title=Google's Guidelines on Site Design|access-date=April 18, 2007|archive-date=January 9, 2009|archive-url=https://web.archive.org/web/20090109073316/http://www.google.com./webmasters/guidelines.html|url-status=live}}{{cite web|url=http://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a|title=Bing Webmaster Guidelines|publisher=bing.com|access-date=September 11, 2014|archive-date=September 9, 2014|archive-url=https://web.archive.org/web/20140909120222/http://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a|url-status=live}}{{cite web|url=http://www.google.com/webmasters/seo.html|title=What's an SEO? Does Google recommend working with companies that offer to make my site Google-friendly?|access-date=April 18, 2007|archive-date=April 16, 2006|archive-url=https://web.archive.org/web/20060416054734/http://www.google.com/webmasters/seo.html|url-status=live}} are not written as a series of rules or commandments, this is an important distinction to note. White hat SEO is not just about following guidelines but is about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see. White hat advice is generally summed up as creating content for users, not for search engines, and then making that content easily accessible to the online "spider" algorithms, rather than attempting to trick the algorithm from its intended purpose. White hat SEO is in many ways similar to web development that promotes accessibility,{{cite web|author=Andy Hagans|publisher=A List Apart|url=http://alistapart.com/articles/accessibilityseo|title=High Accessibility Is Effective Search Engine Optimization|date=November 8, 2005|access-date=May 9, 2007|archive-date=May 4, 2007|archive-url=https://web.archive.org/web/20070504054044/http://www.alistapart.com/articles/accessibilityseo/|url-status=live}} although the two are not identical.

Black hat SEO attempts to improve rankings in ways that are disapproved of by the search engines or involve deception. One black hat technique uses hidden text, either as text colored similar to the background, in an invisible div, or positioned off-screen. Another method gives a different page depending on whether the page is being requested by a human visitor or a search engine, a technique known as cloaking. Another category sometimes used is grey hat SEO. This is in between the black hat and white hat approaches, where the methods employed avoid the site being penalized but do not act in producing the best content for users. Grey hat SEO is entirely focused on improving search engine rankings.

Search engines may penalize sites they discover using black or grey hat methods, either by reducing their rankings or eliminating their listings from their databases altogether. Such penalties can be applied either automatically by the search engines' algorithms or by a manual site review. One example was the February 2006 Google removal of both BMW Germany and Ricoh Germany for the use of deceptive practices.{{cite web|url=http://www.mattcutts.com/blog/ramping-up-on-international-webspam/|publisher=mattcutts.com/blog|title=Ramping up on international webspam|author=Matt Cutts|date=February 4, 2006|access-date=May 9, 2007|author-link=Matt Cutts|archive-date=June 29, 2012|archive-url=https://web.archive.org/web/20120629051407/http://www.mattcutts.com/blog/ramping-up-on-international-webspam/|url-status=live}} Both companies subsequently apologized, fixed the offending pages, and were restored to Google's search engine results page.{{cite web|access-date=May 9, 2007|url=http://www.mattcutts.com/blog/recent-reinclusions/|publisher=mattcutts.com/blog|title=Recent reinclusions|author=Matt Cutts|date=February 7, 2006|author-link=Matt Cutts|archive-date=May 22, 2007|archive-url=https://web.archive.org/web/20070522130714/http://www.mattcutts.com/blog/recent-reinclusions/|url-status=live}}

Companies that employ black hat techniques or other spammy tactics can get their client websites banned from the search results. In 2005, the Wall Street Journal reported on a company, Traffic Power, which allegedly used high-risk techniques and failed to disclose those risks to its clients.{{cite news|newspaper=Wall Street Journal|url=https://www.wsj.com/articles/SB112714166978744925?apl=y&r=947596|title=Sites Get Dropped by Search Engines After Trying to 'Optimize' Rankings|author=David Kesmodel|date=September 22, 2005|access-date=July 30, 2008|archive-date=August 4, 2020|archive-url=https://web.archive.org/web/20200804125356/https://www.wsj.com/articles/SB112714166978744925?apl=y&r=947596|url-status=live}} Wired magazine reported that the same company sued blogger and SEO Aaron Wall for writing about the ban.{{cite magazine|magazine=Wired Magazine|url=http://archive.wired.com/culture/lifestyle/news/2005/09/68799?currentPage=all|title=Legal Showdown in Search Fracas|date=September 8, 2005|author=Adam L. Penenberg|access-date=August 11, 2016|archive-date=March 4, 2016|archive-url=https://web.archive.org/web/20160304055056/http://archive.wired.com/culture/lifestyle/news/2005/09/68799?currentPage=all|url-status=live}} Google's Matt Cutts later confirmed that Google had banned Traffic Power and some of its clients.{{cite web|publisher=mattcutts.com/blog|author=Matt Cutts|url=http://www.mattcutts.com/blog/confirming-a-penalty/|title=Confirming a penalty|date=February 2, 2006|access-date=May 9, 2007|author-link=Matt Cutts|archive-date=June 26, 2012|archive-url=https://web.archive.org/web/20120626093828/http://www.mattcutts.com/blog/confirming-a-penalty/|url-status=live}}

SEO is not an appropriate strategy for every website, and other Internet marketing strategies can be more effective, such as paid advertising through pay-per-click (PPC) campaigns, depending on the site operator's goals.{{Editorializing|date=December 2024}} Search engine marketing (SEM) is the practice of designing, running, and optimizing search engine ad campaigns. Its difference from SEO is most simply depicted as the difference between paid and unpaid priority ranking in search results. SEM focuses on prominence more so than relevance; website developers should regard SEM with the utmost importance with consideration to visibility as most navigate to the primary listings of their search.{{Cite journal|last=Tapan|first=Panda|year=2013|title=Search Engine Marketing: Does the Knowledge Discovery Process Help Online Retailers?|journal=IUP Journal of Knowledge Management|volume=11|issue=3|pages=56–66|id={{ProQuest|1430517207}}}} A successful Internet marketing campaign may also depend upon building high-quality web pages to engage and persuade internet users, setting up analytics programs to enable site owners to measure results, and improving a site's conversion rate.{{cite web|url=http://www.grokdotcom.com/2007/03/13/the-battle-between-search-engine-optimization-and-conversion-who-wins/ |archive-url=https://web.archive.org/web/20080315221733/http://www.grokdotcom.com/2007/03/13/the-battle-between-search-engine-optimization-and-conversion-who-wins |archive-date=March 15, 2008 |title=The Battle Between Search Engine Optimization and Conversion: Who Wins? |author=Melissa Burdon |publisher=Grok.com |access-date=April 10, 2017 |date=March 13, 2007 }}{{Cite web |last= |first= |date= |title=SEO Tips and Marketing Strategies |url=https://skyrocketresultsseo.com/hvac/seo/ |access-date=2022-10-30 |website= |language=en-US |archive-date=October 30, 2022 |archive-url=https://web.archive.org/web/20221030122434/https://skyrocketresultsseo.com/hvac/seo/ |url-status=live }} In November 2015, Google released a full 160-page version of its Search Quality Rating Guidelines to the public,{{Cite web |url=http://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf |title="Search Quality Evaluator Guidelines" How Search Works November 12, 2015. |access-date=January 11, 2016 |archive-date=March 29, 2019 |archive-url=https://web.archive.org/web/20190329234138/http://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf |url-status=live }} which revealed a shift in their focus towards "usefulness" and mobile local search. In recent years the mobile market has exploded, overtaking the use of desktops, as shown in by StatCounter in October 2016, where they analyzed 2.5 million websites and found that 51.3% of the pages were loaded by a mobile device.{{cite news|last1=Titcomb|first1=James|title=Mobile web usage overtakes desktop for first time|url=https://www.telegraph.co.uk/technology/2016/11/01/mobile-web-usage-overtakes-desktop-for-first-time/ |archive-url=https://ghostarchive.org/archive/20220110/https://www.telegraph.co.uk/technology/2016/11/01/mobile-web-usage-overtakes-desktop-for-first-time/ |archive-date=January 10, 2022 |url-access=subscription |url-status=live|newspaper=The Telegraph|date=November 2016|access-date=17 March 2018}}{{cbignore}} Google has been one of the companies that are utilizing the popularity of mobile usage by encouraging websites to use their Google Search Console, the Mobile-Friendly Test, which allows companies to measure up their website to the search engine results and determine how user-friendly their websites are. The closer the keywords are together their ranking will improve based on key terms.

SEO may generate an adequate return on investment. However, search engines are not paid for organic search traffic, their algorithms change, and there are no guarantees of continued referrals. Due to this lack of guarantee and uncertainty, a business that relies heavily on search engine traffic can suffer major losses if the search engines stop sending visitors.{{cite magazine|magazine=Forbes |url=https://www.forbes.com/technology/2007/04/29/sanar-google-skyfacet-tech-cx_ag_0430googhell.html?partner=rss |title=Condemned To Google Hell |author=Andy Greenberg |date=April 30, 2007 |access-date=May 9, 2007 |archive-url=https://web.archive.org/web/20070502074629/http://www.forbes.com/technology/2007/04/29/sanar-google-skyfacet-tech-cx_ag_0430googhell.html?partner=rss |archive-date=May 2, 2007 |df=mdy-all }} Search engines can change their algorithms, impacting a website's search engine ranking, possibly resulting in a serious loss of traffic. According to Google's CEO, Eric Schmidt, in 2010, Google made over 500 algorithm changes – almost 1.5 per day.{{cite web|url=http://searchengineland.com/13000-precision-evaluations-schmidts-testimony-reveals-how-google-tests-algorithm-changes-93740|title=Schmidt's testimony reveals how Google tests algorithm changes|author=Matt McGee|date=September 21, 2011|access-date=January 4, 2012|archive-date=January 17, 2012|archive-url=https://web.archive.org/web/20120117152309/http://searchengineland.com/13000-precision-evaluations-schmidts-testimony-reveals-how-google-tests-algorithm-changes-93740|url-status=live}} It is considered a wise business practice for website operators to liberate themselves from dependence on search engine traffic.{{cite web|url=http://www.useit.com/alertbox/search_engines.html|publisher=useit.com|title=Search Engines as Leeches on the Web|date=January 9, 2006|access-date=May 14, 2007|author=Jakob Nielsen|author-link=Jakob Nielsen (usability consultant)|archive-date=August 25, 2012|archive-url=https://web.archive.org/web/20120825022222/http://www.useit.com/alertbox/search_engines.html|url-status=live}} In addition to accessibility in terms of web crawlers (addressed above), user web accessibility has become increasingly important for SEO.

Optimization techniques are highly tuned to the dominant search engines in the target market.

The search engines' market shares vary from market to market, as does competition.

In 2003, Danny Sullivan stated that Google represented about 75% of all searches.{{cite news|url=https://www.usatoday.com/tech/news/2003-08-25-google_x.htm|title=The search engine that could|newspaper=USA Today|access-date=May 15, 2007|date=August 26, 2003|first1=Jefferson|last1=Graham|archive-date=May 17, 2007|archive-url=https://web.archive.org/web/20070517051318/http://www.usatoday.com/tech/news/2003-08-25-google_x.htm|url-status=live}} In markets outside the United States, Google's share is often larger, and data showed Google was the dominant search engine worldwide as of 2007.{{cite web | url=http://searchenginewatch.com/article/2066064/Stats-Show-Google-Dominates-the-International-Search-Landscape | author=Greg Jarboe | title=Stats Show Google Dominates the International Search Landscape | publisher=Search Engine Watch | date=February 22, 2007 | access-date=May 15, 2007 | archive-date=May 23, 2011 | archive-url=https://web.archive.org/web/20110523154641/http://searchenginewatch.com/article/2066064/Stats-Show-Google-Dominates-the-International-Search-Landscape | url-status=live }} As of 2006, Google had an 85–90% market share in Germany.{{cite web|url=http://www.clickz.com/clickz/column/1702507/search-engine-optimizing-europe|title=Search Engine Optimizing for Europe|author=Mike Grehan|date=April 3, 2006|access-date=May 14, 2007|publisher=Click|archive-date=November 6, 2010|archive-url=https://web.archive.org/web/20101106014727/http://www.clickz.com/clickz/column/1702507/search-engine-optimizing-europe|url-status=live}} While there were hundreds of SEO firms in the US at that time, there were only about five in Germany. As of March 2024, Google still had a significant market share of 89.85% in Germany.{{Cite web |title=Germany search engine market share 2024 |url=https://www.statista.com/statistics/445974/search-engines-market-share-of-desktop-and-mobile-search-germany/#statisticContainer |access-date=2025-01-06 |website=Statista |language=en}} As of June 2008, the market share of Google in the UK was close to 90% according to Hitwise.{{cite news | url=https://www.theguardian.com/technology/blog/2008/jun/10/googleukclosesinon90mark | author=Jack Schofield | title=Google UK closes in on 90% market share | newspaper=Guardian | date=June 10, 2008 | access-date=June 10, 2008 | location=London | archive-date=December 17, 2013 | archive-url=https://web.archive.org/web/20131217023045/http://www.theguardian.com/technology/blog/2008/jun/10/googleukclosesinon90mark | url-status=live }}{{Obsolete source|date=December 2024}} As of March 2024, Google's market share in the UK was 93.61%.{{Cite web |title=UK search engines market share 2024 |url=https://www.statista.com/statistics/280269/market-share-held-by-search-engines-in-the-united-kingdom/ |access-date=2025-01-06 |website=Statista |language=en}}

Successful search engine optimization (SEO) for international markets requires more than just translating web pages. It may also involve registering a domain name with a country-code top-level domain (ccTLD) or a relevant top-level domain (TLD) for the target market, choosing web hosting with a local IP address or server, and using a Content Delivery Network (CDN) to improve website speed and performance globally. It is also important to understand the local culture so that the content feels relevant to the audience. This includes conducting keyword research for each market, using hreflang tags to target the right languages, and building local backlinks. However, the core SEO principles—such as creating high-quality content, improving user experience, and building links—remain the same, regardless of language or region.

Regional search engines have a strong presence in specific markets:

• China: Baidu leads the market, controlling about 70 to 80% market share.{{Cite web |title=China search engines market share 2024 |url=https://www.statista.com/statistics/253340/market-share-of-search-engines-in-china-pageviews/ |access-date=2025-01-06 |website=Statista |language=en}}
• South Korea: Since the end of 2021, Naver, a domestic web portal, has gained prominence in the country.{{Cite web |last1=cycles |first1=This text provides general information Statista assumes no liability for the information given being complete or correct Due to varying update |last2=Text |first2=Statistics Can Display More up-to-Date Data Than Referenced in the |title=Topic: Search engines in South Korea |url=https://www.statista.com/topics/10655/search-engines-in-south-korea/#topicOverview |access-date=2025-01-06 |website=Statista |language=en}}{{Cite web |title=South Korea: main service used to search for information 2024 |url=https://www.statista.com/statistics/1115944/south-korea-main-service-used-to-search-for-information/ |access-date=2025-01-06 |website=Statista |language=en}}
• Russia: Yandex is the leading search engine in Russia. As of December 2023, it accounted for at least 63.8% of the market share.{{Cite web |title=Most popular search engines in Russia 2023 |url=https://www.statista.com/statistics/1094920/leading-search-engines-by-visits-share-russia/ |access-date=2025-01-06 |website=Statista |language=en}}

By the early 2000s, businesses recognized that the web and search engines could help them reach global audiences. As a result, the need for multilingual SEO emerged.{{Cite journal |last1=Arora |first1=Sanjog |last2=Hemrajani |first2=Naveen |date=September 2023 |title=A REVIEW ON: MULTILINGUAL SEARCH TECHNIQUE |url=https://www.researchgate.net/publication/382274152 |journal=International Journal of Applied Engineering & Technology |volume=5 |issue=3 |pages=760–770 |via=ResearchGate}} In the early years of international SEO development, simple translation was seen as sufficient. However, over time, it became clear that localization and transcreation—adapting content to local language, culture, and emotional resonance—were far more effective than basic translation.{{Cite web |title=SEO Starter Guide: The Basics {{!}} Google Search Central {{!}} Documentation |url=https://developers.google.com/search/docs/fundamentals/seo-starter-guide |access-date=2025-01-13 |website=Google for Developers |language=en}}

On October 17, 2002, SearchKing filed suit in the United States District Court, Western District of Oklahoma, against the search engine Google. SearchKing's claim was that Google's tactics to prevent spamdexing constituted a tortious interference with contractual relations. On May 27, 2003, the court granted Google's motion to dismiss the complaint because SearchKing "failed to state a claim upon which relief may be granted."{{cite web|url=http://www.docstoc.com/docs/618281/Order-(Granting-Googles-Motion-to-Dismiss-Search-Kings-Complaint)|format=PDF|publisher=docstoc.com|title=Search King, Inc. v. Google Technology, Inc., CIV-02-1457-M|date=May 27, 2003|access-date=May 23, 2008|archive-date=May 27, 2008|archive-url=https://web.archive.org/web/20080527012138/http://www.docstoc.com/docs/618281/Order-(Granting-Googles-Motion-to-Dismiss-Search-Kings-Complaint)|url-status=live}}{{cite web|url=http://news.cnet.com/2100-1032_3-1011740.html|title=Judge dismisses suit against Google|website=CNET|author=Stefanie Olsen|access-date=May 10, 2007|date=May 30, 2003|archive-date=December 1, 2010|archive-url=https://web.archive.org/web/20101201180530/http://news.cnet.com/2100-1032_3-1011740.html|url-status=live}}

In March 2006, KinderStart filed a lawsuit against Google over search engine rankings. KinderStart's website was removed from Google's index prior to the lawsuit, and the amount of traffic to the site dropped by 70%. On March 16, 2007, the United States District Court for the Northern District of California (San Jose Division) dismissed KinderStart's complaint without leave to amend and partially granted Google's motion for Rule 11 sanctions against KinderStart's attorney, requiring him to pay part of Google's legal expenses.{{cite web|access-date=June 23, 2008|archive-date=May 11, 2008|archive-url=https://web.archive.org/web/20080511162049/http://blog.ericgoldman.org/archives/2007/03/kinderstart_v_g_2.htm|publisher=blog.ericgoldman.org|title=Technology & Marketing Law Blog: KinderStart v. Google Dismissed—With Sanctions Against KinderStart's Counsel|date=March 20, 2007 |url=http://blog.ericgoldman.org/archives/2007/03/kinderstart_v_g_2.htm|url-status=live}}{{cite web

|url=http://blog.ericgoldman.org/archives/2006/03/google_sued_ove.htm

|title=Technology & Marketing Law Blog: Google Sued Over Rankings—KinderStart.com v. Google

|publisher=blog.ericgoldman.org

|access-date=June 23, 2008

|archive-date=June 22, 2008

|archive-url=https://web.archive.org/web/20080622152019/http://blog.ericgoldman.org/archives/2006/03/google_sued_ove.htm

|url-status=live

}}

{{div col}}

• Competitor backlinking
• List of search engines
• Search engine marketing
• Search neutrality, the opposite of search manipulation
• User intent
• Website promotion
• Search engine results page
• Search engine scraping{{div col end}}

{{Reflist}}

{{Spoken Wikipedia|date=2008-05-20|Search engine optimization.ogg}}

• [https://support.google.com/webmasters/bin/answer.py?hl=en&answer=35769 Webmaster Guidelines] from Google
• [https://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf Google Search Quality Evaluators Guidelines (PDF)]
• [https://help.yahoo.com/kb/search/SLN2245.html Webmaster resources] from Yahoo!
• [http://www.bing.com/webmaster/help/webmaster-guidelines-30fba23a Webmaster Guidelines] from Microsoft Bing
• [https://www.nytimes.com/2011/02/13/business/13search.html The Dirty Little Secrets of Search] in The New York Times (February 12, 2011)

{{Search engine optimization}}

{{Authority control}}

Category:Digital marketing

Category:Web analytics

Category:Internet terminology

Category:Online advertising

Category:Promotion and marketing communications