BotSeer
{{Short description|Web-based information system}}
BotSeer was a Web-based information system and search tool used for research on Web robots and trends in Robot Exclusion Protocol deployment and adherence. It was created and designed by Yang Sun,{{Cite web |url=http://yangsun.com/ |title=Yang Sun |access-date=2019-06-13 |archive-url=https://web.archive.org/web/20140104042519/http://yangsun.com/ |archive-date=2014-01-04 |url-status=dead }} Isaac G. Councill,[http://theaussieseo.com/2013/05/15/isaac-g-councill/ Isaac G. Councill] {{webarchive |url=https://web.archive.org/web/20140517153559/http://theaussieseo.com/2013/05/15/isaac-g-councill/ |date=May 17, 2014 }} Ziming Zhuang[http://www.personal.psu.edu/users/z/x/zxz127/ Ziming Zhuang] {{webarchive |url=https://web.archive.org/web/20071228095114/http://www.personal.psu.edu/users/z/x/zxz127/ |date=December 28, 2007 }} and C. Lee Giles. BotSeer was in operation from 2007 to 2010, approximately.
History
BotSeer served as a resource for studying the regulation and behavior of Web robots as well as information about the creation of effective robots.txt files and crawler implementations. It was publicly available on the World Wide Web at the College of Information Sciences and Technology at the Pennsylvania State University.
BotSeer provided services including robots.txt searching, robot bias analysis,Yang Sun, Z. Zhuang, I. Councill, C.L. Giles, [http://nitch.marketing/determining-bias-search-engines-robots-txt/ Determining Bias to Search Engines from Robots.txt] {{webarchive|url=https://web.archive.org/web/20150402104202/http://nitch.marketing/determining-bias-search-engines-robots-txt/ |date=2015-04-02 }}, Proceedings of IEEE/WIC/ACM International Conference on Web Intelligence (WI 2007), 149-155, 2007.{{cite web |url=http://www.zoomwebmedia.com/search-engine-optimization.php |title=Zoom Web Media Offers Affordable Web Design, Development and SEO Services |website=www.zoomwebmedia.com |url-status=dead |archive-url=https://web.archive.org/web/20121130103340/http://www.zoomwebmedia.com/search-engine-optimization.php |archive-date=2012-11-30}} and robot-generated log analysis. The prototype of BotSeer also allowed users to search 6,000 documentation files and source codes from 18 open source crawler projects.
BotSeer had indexed and analyzed 2.2 million robots.txt files obtained from 13.2 million websites, as well as a large Web server log of real-world robot behavior and related analysis. BotSeer's goals were to assist researchers, webmasters, web crawler developers and others with web robots related research and information needs. However, some people received BotSeer negatively, arguing that it contradicted the purpose of the robots.txt convention.[https://web.archive.org/web/20120420234917/http://www.ihelpyou.com/forums/archive/index.php/t-25747.html BotSeer? - SEO Best Practices Search Engine Forums]
BotSeer had also had set up a honeypot{{webarchive |url=https://web.archive.org/web/20081222203547/http://www.v4d.net/ |date=December 22, 2008 |title=Web Robot Behavior and Performance Test }} (instead of [http://www.v4d.net/ unrelated current site] ) to test the ethics, performance and behavior of web crawlers.
References
- {{cite web
|url= http://www.newsvine.com/_news/2007/11/28/1128150-webmasters-may-shape-search-results
|accessdate= 2011-12-11
|title= Webmasters May Shape Search Results
|agency= Associated Press
|publisher= Newsvine
|date= November 28, 2007
}}
- {{cite web
|url= http://www.networkworld.com/news/2007/111507-google-favored.html
|accessdate= 2007-12-19
|title= Google favored by Web admins
|publisher= Network World
|date= November 15, 2007
|url-status= dead
|archiveurl= https://web.archive.org/web/20071218072802/http://www.networkworld.com/news/2007/111507-google-favored.html
|archivedate= December 18, 2007
}}
Notes
{{reflist}}
External links
- {{cite web|url=http://botseer.ist.psu.edu/ |title=BotSeer: Robots.txt and Web Crawler Search Engine |accessdate=2011-12-11 |url-status=dead |archiveurl=https://web.archive.org/web/20100208214818/http://botseer.ist.psu.edu/ |archivedate=2010-02-08 }}