Forums

As the title suggests, is anyone here using or used Scrapyd?

From a brief look at the docs for scrapyd, I don't believe that it will work on PythonAnywhere.

Is this likely to have support at some point?

Hi there, this would come under our eventual plans for "long-running services". We're working on it, but there are several major infrastructure changes we need to make in order to get there, so we can't say how soon we'll be able to support it. Months not weeks.

I am interested in this functionality too.

Hey! Wondering if there was any update/timeline on this? Curious as to how to use scrapy with my PythonAnywhere project and exploring options/trying to see what's feasible.

We're still working towards the infrastructure changes needed for it -- the first step (always-on tasks) is in beta right now, but there are further things we'll need to implement before we can fully support it.

Thanks for responding definitely appreciate it - one thing I explored was running recurring tasks on the hour though figured I'd explore if this was an option. Will stay tuned, if you're looking for someone to try it out in when it's beta I'd love to be considered.

Sure! I'll add you to the list of people to invite when we have a beta.

Is scrapyd supported by now?

No, it is not.