Many years back, search engines were super basic, but today they are crawled by complex spiders, which are also referred to as web crawlers, or bots.
That said, there is no doubt that the development of various search engines has changed how information is crawled, indexed, retrieved and ranked. Google, Bing, and Yahoo are key players as far as Search Engine Optimization (SEO) is concerned.
In this article, we are going to delve deeper into how search engines work plus the history and evolution of search engines.
How Do Search Engines Work?
The key functions of search engines are:
Search engines scan a website in order to collect details about each page including the keywords, titles, images, as well as other pages that have been linked to that particular website.
Pages are analyzed using web spiders, or bots and indexed. These are the pages that pop up as search results for relevant queries, or keywords.
A search engine returns results based on your search query. The information is then ranked hierarchically through the use of trust, as well as relevance signals.
In search engine optimization, ranking refers to the position of your website’s content on search engine results.
How Have Search Engines Changed Over the Years?
1990: Archie
Archie was the first search engine. It originated from the term “archives”, only that it does not have the letter “v”. Archie was created in 1990 by Alan Emtage, who was by then a student at the McGill University in Montreal.
The role of Archie search engine program was to download and index directory listings of all the public File Transfer Protocol (FTP) servers it could locate. Archie created a database of searchable filenames, which made it easier to find files.
1991: Gopher Protocol
Gopher protocol was created in 1991 by a team from University of Minnesota namely Mark McCahill, Paul Lindner, Farhad Anklesaria, Bob Alberti, and Daniel Torrey.
As a communication protocol, its main role was to distribute, search and retrieve documents in internet protocol networks. It was a form of anonymous FTP. Gopher is famous for having been the first search engine to utilize hypertext paradigm.
1992: Veronica
Veronica search engine was named after Veronica Lodge. It was created in 1992 by Steven Foster and Fred Barrier who were at the time based at the University of Nevada, Reno.
Its role was to search files stored in Gopher servers. Also, it was used to perform keyword search, particularly for Gopher menu titles found in Gopher listings. Veronica’s backronym was (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives).
1993: Jughead
The Jughead search engine was designed in 1993 by Rhett Jones who was at the time based at the University of Utah.
Jughead, (Jonzy’s Universal Gopher Hierarchy Excavation And Display) was a tool used to retrieve menu information from different Gopher servers.
1993: World Wide Web Wanderer
WWW Wanderer was the first robot designed in 1993 by Matthew Gray who was at time an MIT student. The role of Wanderer was to count web servers in order to determine the size of the web. However, it was at some point used to find URLs and form a database of websites referred to as Wandex.
1993: ALIWEB
Also known as Archie-Like Indexing of the Web, ALIWEB was created by Martijn Koster in 1993. It helped users submit their web pages for indexing. This search engine was based on automated meta-data collection.
1993: Excite
Excite search engine was introduced in 1993 by a group of six students from Stanford University. Excite helped to make the search process more efficient by incorporating statistical analysis of word relationships. Excite officially went online in 1995.
1994: Yahoo
Yahoo was started by Jerry Yang and David Filo in 1994. Initially, the two had created Yahoo to help them keep track of their favorite web pages, and it would also entail descriptions for the different pages. Two years later, the two received funds, which enabled them to officially launch Yahoo as a corporation.
1994: JumpStation
JumpStation was a crawler-based search engine, but it is no longer existent. It was created by Jonathan Fletcher who was at the time based at the University of Stirling in Scotland.
The main role of JumpStation was to use web crawlers to develop a searchable index of webpage titles and headings.
1994: World Wide Web WormThis search engine was a web crawler. It was developed by 1994 by Oliver McBryan who was at the time a professor at the University of Colorado. In December 1999, World Wide Web Worm was acquired by GoTo.com, and it is today part of Overture.
1994: WebCrawler
WebCrawler was developed in 1994 by Brian Pinkerton who was at the time based at the University of Washington. Brian was the first person to ever develop a full-text search engine, which helped to index text on each and every page on the Internet.
1994: Infoseek
Infoseek was developed in 1994 by Steve Kirsch. It was operated by Infoseek Corporation. Infoseek was not only a full-text search engine, but it was also used to categorize lists of web pages. In 1999, Infoseek was acquired by the Walt Disney Company.
1994: Lycos
Lycos was developed in 1994 by Michael Mauldin who was at the time based at Carnegie Mellon University. Lycos was famous for its huge catalog of over 57 million indexed documents. Also, it was famous for not only relevance retrieval, but also for word proximity, and its ability to match prefixes.
1995: Alta Vista
Alta Vista was a search engine developed in 1995 by DEC. This search engine enabled people to utilize a natural language user interface, as well as advanced search techniques. Additionally, Alta Vista allowed people to search for music, videos, as well as photos.
1996: Inktomi
Inktomi was developed in 1996 by Eric Brewer and Paul Gauthier during an experiment at the University of California at Berkeley. The crawler was famous for its ability to search more than 48 million pages on the internet. In 1999, Inktomi launched a directory search engine using the concept induction technology. It was acquired by Yahoo in 2003.
1996: HotBot
This application was developed in 1996 by Inktomi in collaboration with HotWired. It was famous for its ability to provide customized website pages and ads based on the user’s type of browser.
1996: Backrub
Backrub was developed in 1996 by Larry Page and Sergey Brin who were at the time students of Stanford University. The key role of Backrub search engine was to rank websites based on the relevancy of their inbound links, as well as their popularity. Backrub was given its name because it analyzed backlinks to websites. Later, BackRub became known as Google.
1996: LookSmart
LookSmart was developed in 1996 by Tracey Ellery and Evan Thornley. Initially, LookSmart was known as NatGet. LookSmart was created as a directory service. Today, LookSmart was the most notable paid insertion directory on the World Wide Web.
1996: Netscape
Netscape was originally known as Mosaic Communications, which was founded by Jim Clark and Marc Andreessen of the University of Illinois at Urbana–Champaign. Mosaic was Andreessen’s idea, and he started working on the search engine in 1993 when he was still at the University.
Netscape was acquired by AOL in 1998. Today, Netscape works in partnership with LookSmart, Lycos, Overture, Ask Jeeves and Google.
1997: Search Engine Watch
Search Engine Watch was developed in 1997 by Danny Sullivan. The key role of this online search engine magazine was to offer critical information regarding the search industry, how search engine work and how to search the website. How to rank websites was also part of the information offered by Search Engine Watch.
Search Engine Land was another brainchild of Danny Sullivan which is still alive and thriving today even after Sullivan left the company in 2017.
1997: Ask Jeeves
Askjeeves was developed in 1997 by Garrett Greuner and David Warthen who were at the time based in Berkeley, California. When Garrett and David created Askjeeves, they wanted to help people find answers for their most pressing issues more so because they were aware that not everyone knew about Boolean search and syntax.
That said, they came up with Ask Jeeves, a search engine that featured Natural Language Processing, or NLP. Ask Jeeves, now Ask.com, enabled its users to search for answers using their day-to-day language including asking questions.
Other Major Search Engines worth noting include:
1997: Google.com
Google search is the brainchild of Larry Page and Sergey Brin and was once named BackRub. Today, Google is the most dominant search engine on the planet.
1998: DMOZ
DMOZ was originally developed in 1998 by Bob Truel and Rich Skrenta. Also known as Open Directory Project, DMOZ operated as a web directory for 19, but it stopped its operation in 2017.
1998: MSN Search
MSN was developed in 1998 by Microsoft. It was launched using results from previous search engines like LookSmart, Inktomi, and AltaVista. However, it was rebranded to Live Search in 2006. Live Search was rebranded to Bing in 2009.
1998: AllTheWeb
AllTheWeb (also called FASTSearch) was developed in 1999 and acquired by Yahoo in 2003. After many transformations, Yahoo shut down AllTheWeb in 2011.
2005: Snap
Snap was developed in 2005 by Bill Gross. It was famous for its ability to display search volumes and for its auto-completion feature.
2008: DuckDuckGo
DuckDuckGo, also known as DDG, is a search engine that was developed in 2008. Its main role was to protect your privacy by not profiling you, nor your search results and not following you around on the Internet. It shows same results for all users. It shows the top results from web properties such as Wikipedia, Bing and Yahoo.
Conclusion
In this article, we have delved into the history of search engines. We have looked at the various search engines including Archie, which was the first search engine, Gopher, Netscape, WebCrawler, Yahoo, Google, Bing, DuckDuckGo, just to mention a few.
The search engine with the largest share nowadays is Google, which is followed by search engines like Yahoo, Bing, Baidu and many small players in the field.
Our company was established in 2004. 
