Working of a Web Crawler
•
8 gostaram•9,109 visualizações
A presentation to give an idea as to how a web crawler works
![[object Object],[object Object],[object Object],[object Object],Steps Involved in the Crawling Process](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Working of a Web Crawler
Semelhante a Working of a Web Crawler (20)
Último
Último (20)
Working of a Web Crawler
- 1. How does a Web Crawler work
- 3. Starting URL is specified here WebSPHINX Web Crawler’s GUI
- 4. Starting URL or Root of the tree The Crawler “checks” if the URL exists, parses through it and retrieves all the links then repeats this process on the links, hence obtained. It checks this by sending HTTP requests to the server
- 5. Root of the tree The process is done recursively Son of the root Son of the previous son of the root The crawler has to constantly check the links for duplication. This is done to avoid redundancies, which otherwise will take a toll on the efficiency of the crawling process
- 6. Theoretically this process should continue till all the links have been retrieved. But practically the crawler goes only to 5 levels of depth from the home page of the URL it visits . After this, it is concluded that there is no need of going further. But the crawling can still continue from other URLs Here, the process stops after five depths
- 7. The red crosses signify that crawling cannot be continued from that particular URL. This can arise in the following cases- 1) The server containing the URL is taking too long to respond 2) Server is not allowing access to the crawler 3) URL is left out to avoid duplication 4) The crawler has been specifically been designed to ignore such pages (“ Politeness ” of a crawler)