How Do Search Engines Work?: From Spiders to Web Surfers

Search engines – if you are like many Americans, then you probably use one almost every day, but you might sometimes find yourself wondering, how do search engines work? To explain better the inner workings of this everyday phenomenon, let’s go through a step by step of the search engine process, starting with spiders…

1. That’s right, spiders. Spiders are special software programs that a search engine uses to comb through the internet and compile lists of words found on individual Web sites. The process of building this list is called Web crawling.
2. As a spider’s Web crawling, it starts compiling lists from most visited sites first and then also collects every possible subsequent link while indexing along the way. This gives us a much clearer understanding of the World Wide Web, since a spider essentially creates a strand of connecting pages and words, each made up of large amounts of information.
3. To better understand how do search engines work, one form of information that spiders are particularly fond of is something called a meta tag. Meta tags allow the owner or creator of a particular internet page to specify how they want their page to be classified. For example, an internet page about traveling to Costa Rica may include meta tags such as “Costa Rica travel,” “travelling in Costa Rica,” or other forms and variations of that same concept.
4. While crawling the web, information gathered by spiders is stored according to ranking. That is, each word found on a page is assigned a weight that might be determined by the number of times it is found in a page, whether it is used in a heading, a link, meta tag, or if it appears near the top of the page.
5. To save storage space, each piece of information is encoded, or condensed into a smaller form. After this occurs, the piece of information is indexed by a hash table. In hashing, each piece of information is assigned a numerical value according to its weight in relation to other words, and then stored accordingly. When wondering how do search engines work, it is this part of the process that creates the index your search engine utilizes to access the information you want.
6. Next, the search engine process requires you, the user, to build a query. This is whatever search you may enter in your search engine browser, be it Google, Yahoo, Bing, or AskJeeves.
7. Using literal, concept-based, or natural-language queries, search engines will organize matches to your search words according to all literal definitions, related pages of interest, or how you would ask a question of your neighbor, respectively.

We hope that this lesson gave you all the insight you need regarding how one of our greatest technological inventions is able to get us the answers we want all at the touch of our fingertips. Just remember though, if somebody asks you “how does a search engine work?”, tell them that it all starts with spiders.

The post How Do Search Engines Work?: From Spiders to Web Surfers appeared first on Farotech.

source https://farotech.com/blog/search-engines-work-spiders-web-surfers/