Mining of Web Server Logs in a Distributed Cluster Using Big Data Technologies

Big Data is an emerging growing dataset beyond the ability of a traditional database tool. Hadoop rides the big data where the massive quantity of information is processed using cluster of commodity hardware. Web server logs are semi-structured files generated by the computer in large volume usually of flat text files. It is utilized efficiently by Mapreduce as it process one line at a time. This paper performs the session identification in log files using Hadoop in a distributed cluster. Apache Hadoop Mapreduce a data processing platform is used in pseudo distributed mode and in fully distributed mode. The framework effectively identifies the session utilized by the web surfer to recognize the unique users and pages accessed by the users. The identified session is analyzed in R to produce a statistical report based on total count of visit per day. The results are compared with non-hadoop approach a java environment, and it results in a better time efficiency, storage and processing speed of the proposed work.

@article{K2014,
  title     = {Mining of Web Server Logs in a Distributed Cluster Using Big Data Technologies},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {5},
  number    = {1},
  year      = {2014},
  publisher = {The Science and Information Organization},
  author    = {Savitha K and Vijaya MS},
  doi       = {10.14569/IJACSA.2014.050119},
  url       = {https://doi.org/10.14569/IJACSA.2014.050119}
}