A Search engine is our favourite place to go to learn about any topic. It has slowly and steadily become an integral part of our lives. Have you ever wondered how they work? Let’s find out.
Basically, a search engine is a software system, which provides answers for our queries, from the internet. We are already familiar with search engines such as Google, Yahoo!, Bing, Baidu, Duck Duck Go, etc. All of these engines follow the same procedure to return relevant results.
A web search engine performs the following three processes to find us answers.
- Crawling
- Indexing
- Ranking
Crawling
Search engines contain web crawlers or spider bots, which are programs accountable for browsing the World Wide Web to discover and index web pages. These spider bots are dispatched by the engines to visit the webpage URLs and download their robot.txt files.
A robot.txt file or robot exclusion standard/protocol is actually a set of instructions for the bots about what not to scan on the website. They can also contain sitemaps, which is the list of important pages on the website. Although these files are not necessary for the websites to undergo crawling, having them will help in avoiding scanning and indexing of each and every item on the website.
After following the instructions and processing the data on the website and the web pages listed on the sitemaps, the search engines get an idea about the contents of the site. It also discovers new URLs to be crawled later, via links on the crawled pages and direct URL submissions.
Since the contents on the web are ever evolving, it is necessary for the search engines to conduct frequent crawling and indexing to provide the users with appropriate results. Each web crawler has its own algorithm for determining when and what should be crawled, re-crawled, and indexed.
Indexing
Once the web pages are crawled, the search engine moves on to the next process, i.e., indexing. Each search engine has a huge database or index to store all the processed information and the process of doing so is called indexing. The index will contain all the crawled URLs and certain key signals such as the type, topic, user engagement, and freshness of the content.
The process of indexing is helpful for returning relevant results within a short span of time. Because when we enter a search query, the search engine doesn’t search the whole internet for answers; instead it searches the index and gives us results.
Search engines do not index all those pages available on the web. Low quality pages with duplicate or plagiarised content are often not indexed. Likewise, pages with no index tag and URLs returning error pages aren’t indexed either.
The search index is updated continuously. Some pages or contents may be added, and some may be removed.
Ranking
After crawling and indexing, the next step is to rank the pages in the index. It is the rank of the page or content that determines its position on the search engine result page (SERP).
Every search engine has a certain algorithm to rank pages according to the search query. Although we are not completely aware of all the ranking factors, we could still list some of them by adhering to the fact that the ultimate aim of a search engine is to provide its users with matching results.
Primarily, keywords from the user’s query will be used to find matching results. But depending on just the keyword leaves room for providing erroneous information. So search engines nowadays use aggregated and anonymised user data to return relevant results. Other ranking factors include presence of backlinks, level of authority, search intent, content type, depth and freshness, page speed, and user experience. Along with these, there are hundreds of other factors too.
SERP
Finally, after all these procedures, we are served with a long list of answers to our query on a webpage called a search engine result page (SERP). Each answer generally contains the title of the page, the link to the webpage, and also a small description. Usually, there will be a number of result pages since there is a huge amount of data spread on the web, with succeeding pages containing less relevant results.