How Search Engines Work – The Beginners Guide to SEO

Imagine the World Wide Web as a network of stops in a big city subway system.

Each stop is a unique document (usually a web page, but sometimes a PDF, JPG, or other file). The search engines need a way to “crawl” the entire city and find all the stops along the way, so they use the best path available—links.

  1. Crawling and IndexingCrawling and indexing the billions of documents, pages, files, news, videos, and media on the World Wide Web.
  2. Providing AnswersProviding answers to user queries, most frequently through lists of relevant pages that they’ve retrieved and ranked for relevancy.

The link structure of the web serves to bind all of the pages together.

Links allow the search engines’ automated robots, called “crawlers” or “spiders,” to reach the many billions of interconnected documents on the web.

Once the engines find these pages, they decipher the code from them and store selected pieces in massive databases, to be recalled later when needed for a search query. To accomplish the monumental task of holding billions of pages that can be accessed in a fraction of a second, the search engine companies have constructed datacenters all over the world.

These monstrous storage facilities hold thousands of machines processing large quantities of information very quickly. When a person performs a search at any of the major engines, they demand results instantaneously; even a one- or two-second delay can cause dissatisfaction, so the engines work hard to provide answers as fast as possible.

Large Hard Drive
Providing AnswersSearch engines are answer machines. When a person performs an online search, the search engine scours its corpus of billions of documents and does two things: first, it returns only those results that are relevant or useful to the searcher’s query; second, it ranks those results according to the popularity of the websites serving the information. It is both relevance and popularity that the process of SEO is meant to influence.

How do search engines determine relevance and popularity?

To a search engine, relevance means more than finding a page with the right words. In the early days of the web, search engines didn’t go much further than this simplistic step, and search results were of limited value. Over the years, smart engineers have devised better ways to match results to searchers’ queries. Today, hundreds of factors influence relevance, and we’ll discuss the most important of these in this guide.

Search engines typically assume that the more popular a site, page, or document, the more valuable the information it contains must be. This assumption has proven fairly successful in terms of user satisfaction with search results.

Popularity and relevance aren’t determined manually. Instead, the engines employ mathematical equations (algorithms) to sort the wheat from the chaff (relevance), and then to rank the wheat in order of quality (popularity).

These algorithms often comprise hundreds of variables. In the search marketing field, we refer to them as “ranking factors.” Moz crafted a resource specifically on this subject

What is a search engine?

Basically, a search engine is a software program that searches for sites based on the words that you designate as search terms. Search engines look through their own databases of information in order to find what it is that you are looking for.

Are Search Engines and Directories The Same Thing?

Search engines and Web directories are not the same things; although the term “search engine” often is used interchangeably. Search engines automatically create website listings by using spiders that “crawl” web pages, index their information, and optimally follows that site’s links to other pages. Spiders return to already-crawled sites on a pretty regular basis in order to check for updates or changes, and everything that these spiders find goes into the search engine database.

A spider, also known as a robot or a crawler, is actually just a program that follows, or “crawls”, links throughout the Internet, grabbing content from sites and adding it to search engine indexes.

Spiders only can follow links from one page to another and from one site to another. That is the primary reason why links to your site (inbound links) are so important SEO Services.

Links to your website from other websites will give the search engine spiders more “food” to chew on. The more times they find links to your site, the more times they will stop by and visit. Google especially relies on its spiders to create their vast index of listings.

Spiders find Web pages by following links from other Web pages, but users can also submit web pages directly to a search engine or directory and request a visit by their spiders.

In fact, it’s a good idea to manually submit your site to a human-edited directory such as Yahoo, and usually spiders from other search engines (such as Google) will find it and add it to their database. It can be useful to submit your URLstraight to the various search engines as well; but spider-based engines will usually pick up your site regardless of whether or not you’ve submitted it to a search engine. Much more about search engine submission can be found in this article titled Free Search Engine Submission: Six Places You Can Submit Your Site For Free. It should be noted that most sites are picked up automatically upon publishing by search engine spiders, but manual submission is still practiced.

How Do Search Engines Process Searches?

Please note: search engines are not simple. They include incredibly detailed processes and methodologies, and are updated all the time. This is a bare bones look at how search engines work to retrieve your search results. All search engines go by this basic process when conducting search processes, but because there are differences in search engines, there are bound to be different results depending on which engine you use.

  1. The searcher types a query into a search engine.
  1. Search engine software quickly sorts through literally millions of pages in its database to find matches to this query.
  2. The search engine’s results are ranked in order of relevancy.

Examples of Search Engines

There are a TON of great search engines out there for you to choose from. Whatever your search need might be, you’ll find a search engine to meet it.

  • 100 Search Engines in 100 Days: All-purpose search engines, visual search engines, people search engines…you’ll find all these and more in this list of search engines, a comprehensive guide to the best search engines on the Web.
  • How to Pick a Search Engine: Pick the best search engine for your searching needs with Search Engines 101, a great way to explore more of your search topic, try a new search engine, and search more of the Web

A search engine makes this index using a program called a ‘web crawler’. This automatically browses the web and stores information about the pages it visits.

Every time a web crawler visits a webpage, it makes a copy of it and adds its URL to an index. Once this is done, the web crawler follows all the links on the page, repeating the process of copying, indexing and then following the links. It keeps doing this, building up a huge index of many webpages as it goes.

Some websites stop web crawlers from visiting them. These pages will be left out of the index, along with pages that no-one links to.

The information that the web crawler puts together is then used by search engines. It becomes the search engine’s index. Every webpage recommended by a search engine has been visited by a web crawler.

5.How do search engines order results?

Search engines sort results to show you the ones they think are the most useful.

PageRank is the best known algorithm which is used to improve web search results. In simple terms, PageRank is a popularity contest. The more links that point to a webpage, the more useful it will seem. This means it will appear higher up in the results.

The webpages on the first page of results are those that PageRank thinks are the best.

Search engines also pay attention to lots of other ‘signals’ when working out the order to show you results. For example how often the page is updated and if it is from a trustworthy domain.

There are many search engines to choose from. Different search engines use different algorithms. This means that some sites will give their results in a different order, or they may even show completely


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s