Abstract: This paper describes mutual exclusion principle for multithreaded web crawlers. The existing web crawlers use data structures to hold frontier set in local address space. This space could be used to run more crawler threads for faster operation. All crawler threads fetch the URL to crawl from the centralized frontier. The mutual exclusion principle is used to provide access to frontier for each crawler thread in synchronized manner to avoid deadlock. The approach to utilize the waiting time on mutual exclusion lock in efficient manner has been discussed in detail.

Kartik Kumar Perisetla, “Mutual Exclusion Principle for Multithreaded Web Crawlers” International Journal of Advanced Computer Science and Applications(IJACSA), 3(9), 2012. http://dx.doi.org/10.14569/IJACSA.2012.030925

@article{Perisetla2012,
title = {Mutual Exclusion Principle for Multithreaded Web Crawlers},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2012.030925},
url = {http://dx.doi.org/10.14569/IJACSA.2012.030925},
year = {2012},
publisher = {The Science and Information Organization},
volume = {3},
number = {9},
author = {Kartik Kumar Perisetla}
}