Forums

Is PA can use for web crawling? Like every hours to a few websites? Any limitation from PA?

@vinus

Shouldn't be a problem, within the limits of your PA subscription and any applicable law.

Lots of us are doing it quite happily!

HTH, Jim

Yup, that should work fine! One caveat -- if you want to use selenium to crawl (rather than, say, requests/BeautifulSoup, because you want to crawl JavaScript-heavy pages) then:

• Make sure you close any selenium instances and browsers, even if there are errors. If you don't, you'll probably hit problems with processes piling up and stopping you from starting more.
• You'll need to use the Firefox browser, and we have version 17 installed. This should be fine for most websites, though we've had reports that some sites are no longer working with it.