Forums

Daidu Spider (Chinese robot) is crawling my web app and collect ... everything. I don't see any interest to be registered on the Chinese seach engine.

Does someone have a suggest to return a 503 Error (service not available) to Baidu with Flask? (or PythonAnywhere config?)

Regards, Dominique

Have you tried adding a robots.txt to your site? I can't imagine that BaiduSpider would ignore it.