Forums

Hi,

For some reason I can’t seem to get Scrapy to crawl any pages.

I’ve tried running the dmoz tutorial

http://doc.scrapy.org/en/latest/intro/tutorial.html

and this very basic (up-to-date) example

http://mherman.org/blog/2012/11/05/scraping-web-pages-with-scrapy/

from the Bash console.

In both cases a project is created but "0 items are crawled and 0 times are scraped".

When running (from the craigslist example in the second link)

scrapy crawl craig -o items.csv -t csv

no items are returned and the csv file created is empy.

Am I missing something?

Free accounts don't get general access to the Internet. We have a whitelist of sites that we allow - they are chosen by having a public API and being generally useful to users.

I see.

So in that case should I be able to crawl any page on the whitelist with Scrapy right now?

That's correct.

Is this still valid?

Yes, it is.