Search engine optimization (SEO) is a key strategic component of any successful online business. Understanding how search engines work and how one can increase their visibility within the results of a given search is crucial to remaining competitive in the modern realm of e-commerce. There are a number of factors which affect the ranking of a page by a search engine and consequently how much traffic it can expect to generate from web searches. Understanding the underlying working of search engines opens the door for savvy web designers to make their page more appealing from the perspective of the search engine.
Why Do I Need SEO?
Without SEO, it is still possible that your page can become successful and show up in search results, high enough that users will click on the link. However, search results are divided into different pages, usually in order of perceived relevance according to complex algorithms. After the first page of results the number of users clicking on links falls dramatically, so much so that any link on page 3 or after is very unlikely to be seen, let alone followed. By adhering to the best practices of SEO, web owners can massively increase their chances of being noticed and, for many websites, this translates into substantially more money being generated through a larger customer base and/or advertising revenue.
Most web traffic, when a user moves from one web page to another, is the result of links provided by one of the major search engines, therefore it makes sense for website owners to focus their efforts at increasing traffic on search engine optimization.
Crawling
Search engines use special kinds of bots (small autonomous scripts) known as ‘crawlers’ which are constantly crawling across the internet, following every link that they encounter. As the crawler reaches a new page it adds information about that page to the search engine database. If it has already visited the page, it looks for any modifications and makes a note of them as appropriate. More modern crawlers used by major search engine providers save a cached copy of each webpage that they visit. This is essential for allowing search queries to be run quickly and results returned within a reasonable timeframe; if the search engine had to actually search the internet live and in real time it would take hours to respond to a query.
It might sound like an enormous task to crawl over the entirety of the internet (and it is!), but even in the very beginning when computing power was a fraction of what it is today, Google was able to crawl a couple of hundred pages every second. With more computing power and its mammoth infrastructure, Google can now crawl a significant portion of the internet in a relatively short amount of time.
Part of search engine optimization, in fact the first thing that is usually advised, is to update the robot.txt file that every website has and which tells bots which pages the site owner doesn’t want to be crawled. This helps prevent the site from showing up in irrelevant search queries and makes Google’s job that little bit easier by saving some time.
How Does a Search Engine Work?
To understand the difference SEO makes to a site’s visibility, we must take a quick look at how search engines work. The first stage of the process is known as ‘indexing’. Indexing is what those crawlers are doing; putting together a database of relevant information on as many web pages as the bots are able to find (and of course every page that those pages link to). The index, then, is essentially the most current (when you press the search button) snapshot that the search engine has of the entirety of the internet. It is able to very quickly run a user’s query against the information it has stored in its index and returns results accordingly.
Once the search engine has run the query against its most current index, it then needs a way of ordering the results so that the most likely pages to be useful to the user are at the top of the list. The most common way for this to be done is through ‘PageRank’. The PageRank, named after its creator and Google co-founder Larry Page, is an algorithm which has been refined over the years and which has now become the standard upon which all major search engines are based. In order to decide a page’s ranking, one of the criteria used is how many ‘votes of confidence’ the page has.
You can think of every link to a page from an external website as constituting a vote for that website. However, not all votes are equal; search engine crawlers are able to identify spam pages (pages which have been set up purely to increase the number of links to another page) and where these are found, the links in fact count against the site in terms of its ranking. Genuine links are known as natural, or organic, links and their value is decided by the originating site’s reputation.
Black Hat, White Hat
Legitimate ways of increasing a page’s visibility in search results are known as ‘white-hat’ methods, and those that directly contravene a search engine’s terms of use are labelled ‘black-hat’. Black hat methods, when discovered, result in penalties for the offending website and all the major search engine providers are becoming increasingly sophisticated in their methods of identifying these instances. White hat methods are deliberate actions on the part of the website owner that are designed to improve a site’s PageRank in a legitimate way. Google offers their own guide on SEO to help web developers stay within the guidelines.
Search engine optimization is an absolutely essential component of successfully marketing a website. The majority of internet traffic is generated as a result of searches performed through one of the handful of big names in the search engine industry. Understanding how search engines work and how one can use their workings to their advantage opens up whole new possibilities and could be the difference between a website’s success and its failure.
