Does it respect robots.txt file?


Yes. Absolutely. We strictly respect robots.txt, both disallow and allow rules.


To change the frequency of LocalGlobalBot visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file


Please also note that if your robots.txt contains errors and LocalGlobalBot won’t be able to recognize your commands it will continue crawling your website.