Hello,
I am using Beautifulsoup and requests to crawl some data from indeed and eBay. The code works fine locally but once I deployed in Pythonanywhere I am getting timeout issue. Do the IP Address of Pythonanywhere blocked by eBay and indeed? Could you please help me?
thanks
It's certainly possible that the server that you're connecting to is blocking incoming requests, either from PythonAnywhere or perhaps from all cloud computing environments. Sites that don't want to be scraped will sometimes do that.
Sometimes changing the user-agent header that you send will help -- that is, they might not be blocking the IP addresses specifically, but just obviously-automated requests from those IPs. If you want to try changing it, this Stack Overflow answer has the details. It's not guaranteed to work, though -- it's just something that has helped with some other servers that were blocking scraping in the past.
Thanks but changing the user-agent doesn't work. Could anyone recommend a proxy or help me fix this issue?
If it is a remote server blocking you, we can't do anything about that.
PythonAnywhere blocks this
giles
|
11788
posts
|
PythonAnywhere staff
|
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2077, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1525, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1523, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1509, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/MFGGaming/data/web.py", line 72, in turnOnServer
requests.post(url, headers=headers, data=data)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 559, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='panel.magmanode.com', port=443): Max retries exceeded with url: /api/client/servers/dd83444a/power (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))
I see that you've posted in a separate forum thread so I'll reply there.
