Forums

So I have an always-on task I've been trying to get working, but as I was working on it last night it seemed that python anywhere had some downtime... I woke up this morning hoping to get it up and running - I can still see the task but it says 'Preparing to start' and is just stuck there.

The main issue though is that, I can't edit it and I can't delete it - those buttons are disabled, and when I go into task log it gives a 404 error :-/

Any solution to this would be great as I've people waiting on me today!

Thanks :-)

Your task appears to have an error that causes it to error out early. When that happens we have a back-off algorithm that means that attempts to restart it become further and further apart. It appears you were in a state where it was just a really long time until we'd try to run it again. Fix your code so that it does not immediately exit and your task should be fine.

Ah, thanks! Apologies - I hadn't come across that error before where I couldn't even get into the logs, so I was wondering if it was related to the downtime.

Cheers!!

Glad to hear that you would be able to fix it!

Thanks! First time using pythonanywhere.

JUST wondering, rather than starting a new thread if there is any advice possible, as there is an issue specifically with the always on task I cannot seem to solve

I have a file called twitter.py that is using tweepy to listen for tweets live, and add them to a mongoDB database with pymongo.

It is working perfectly when I run it manually in pythonanywhere, and also if I run it in the bash console with:

workon venv

/they_think_its_all_clover/twitter.py

However when I run it as an always-on task, it is hitting tons of errors (which I can post in a spearate comment as they are pretty lengthy)

I have tried every method I can find of running it as an always-on task, and I can get it to run but as soon as I send a tweet with the hashtag it is listening for it give me the following errors (which again only happens as an always-on task)

``` Mar 5 12:45:47 Traceback (most recent call last):

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 1278, in _get_socket

Mar 5 12:45:47 sock_info = self.sockets.popleft()

Mar 5 12:45:47 IndexError: pop from an empty deque

Mar 5 12:45:47

Mar 5 12:45:47 During handling of the above exception, another exception occurred:

Mar 5 12:45:47

Mar 5 12:45:47 Traceback (most recent call last):

Mar 5 12:45:47 File "they_think_its_all_clover/twitter.py", line 43, in <module>

Mar 5 12:45:47 stream.filter(track=['#theyThinkItsAllClover'])

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/tweepy/streaming.py", line 474, in filter

Mar 5 12:45:47 self._start(is_async)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/tweepy/streaming.py", line 389, in _start

Mar 5 12:45:47 self._run()

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/tweepy/streaming.py", line 320, in _run

Mar 5 12:45:47 six.reraise(*exc_info)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/six.py", line 703, in reraise

Mar 5 12:45:47 raise value

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/tweepy/streaming.py", line 289, in _run

Mar 5 12:45:47 self._read_loop(resp)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/tweepy/streaming.py", line 351, in _read_loop

Mar 5 12:45:47 self._data(next_status_obj)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/tweepy/streaming.py", line 323, in _data

Mar 5 12:45:47 if self.listener.on_data(data) is False:

Mar 5 12:45:47 File "they_think_its_all_clover/twitter.py", line 32, in on_data

Mar 5 12:45:47 collection.insert_one(tweet_data)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/collection.py", line 701, in insert_one

Mar 5 12:45:47 session=session),

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/collection.py", line 615, in _insert

Mar 5 12:45:47 bypass_doc_val, session)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/collection.py", line 603, in _insert_one

Mar 5 12:45:47 acknowledged, _insert_command, session)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write

Mar 5 12:45:47 return self._retry_with_session(retryable, func, s, None)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session

Mar 5 12:45:47 return self._retry_internal(retryable, func, session, bulk)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1408, in _retry_internal

Mar 5 12:45:47 with self._get_socket(server, session) as sock_info:

Mar 5 12:45:47 File "/usr/lib/python3.6/contextlib.py", line 81, in enter

Mar 5 12:45:47 return next(self.gen)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1247, in _get_socket

Mar 5 12:45:47 self.__all_credentials, checkout=exhaust) as sock_info:

Mar 5 12:45:47 File "/usr/lib/python3.6/contextlib.py", line 81, in enter

Mar 5 12:45:47 return next(self.gen)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 1231, in get_socket

Mar 5 12:45:47 sock_info = self._get_socket(all_credentials)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 1281, in _get_socket

Mar 5 12:45:47 sock_info = self.connect(all_credentials)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 1197, in connect

Mar 5 12:45:47 sock_info.check_auth(all_credentials)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 793, in check_auth

Mar 5 12:45:47 self.authenticate(credentials)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 810, in authenticate

Mar 5 12:45:47 auth.authenticate(credentials, self)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/auth.py", line 673, in authenticate

Mar 5 12:45:47 auth_func(credentials, sock_info)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/auth.py", line 591, in _authenticate_default

Mar 5 12:45:47 return _authenticate_scram(credentials, sock_info, 'SCRAM-SHA-1')

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/auth.py", line 295, in _authenticate_scram

Mar 5 12:45:47 res = sock_info.command(source, cmd)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 694, in command

Mar 5 12:45:47 exhaust_allowed=exhaust_allowed)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/network.py", line 161, in command

Mar 5 12:45:47 parse_write_concern_error=parse_write_concern_error)

Mar 5 12:45:47 File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/helpers.py", line 164, in _check_command_response

Mar 5 12:45:47 raise OperationFailure(errmsg, code, response, max_wire_version)

Mar 5 12:45:47 pymongo.errors.OperationFailure: Authentication failed., full error: {'ok': 0, 'errmsg': 'Authentication failed.', 'code': 8000, 'codeName': 'AtlasError'} ```

Do you use the same credentials in both cases?

Hi fjl,

Yep! I haven't changed anything in the file at all between running it manually in python anywhere, or running it as an always on task. Even when it fails as a task, I go in and it runs fine manually.

I've tried multiple methods to run it as n always on task

workon venv && python3 they_think_its_all_clover/twitter.py

or

/home/cjcon90/.virtualenvs/venv/bin/python3 /home/cjcon90/they_think_its_all_clover/twitter.py

I also tried cd-ing to the folder and using source/bin/activate and then running it, all with a different combination of python / python3 / python3.8

No idea why it is bugging out specifically in the always-on task

From seeing the IndexError: pop from an empty deque error reported in different services, it might be a threading problem?? But I don't fully understand how that would work or be an issue in pythonanywhere

If you're trying to pop from an empty datastructure, trace back from there to see if you can find out why it's empty when you expect it to have something in it.

Hiya Glenn, that's the strange thing I'm not popping from a datastructure, the only functioning is a very simple adding to a MongoDB database... It only seems to happen in the always-on so I don't know if it is a network issue or something?

So what the code is doing is using tweepy's StreamListener method, defining a fucntion that on a tweet it will add it to the database, and then is just authorisation and setting the listener. So I have no idea why the always-on task is creating an IndexError: pop from an empty deque

When run normally it works perfectly fine!

from tweepy.streaming import StreamListener from tweepy import OAuthHandler from tweepy import Stream from pymongo import MongoClient import json from credentials import DB_USER, DB_PASSWORD, DB_CLUSTER, DB_COLL, API_KEY, API_SECRET_KEY, ACCESS_TOKEN, ACCESS_TOKEN_SECRET

client = MongoClient(f"mongodb+srv://{DB_USER}:{DB_PASSWORD}@{DB_CLUSTER}.wijab.mongodb.net/{DB_COLL}?retryWrites=true&w=majority")
db = client['they_think_its_all_clover']
collection = db['tweet']

# Override the Std Class (https://www.kite.com/python/docs/examples.streaming.StdOutListener)
class StdOutListener(StreamListener):
# https://www.kite.com/python/docs/tweepy.streaming.StreamListener.on_data
def on_data(self, data):
    text = json.loads(data)
    content = text["text"]
    username = text["user"]['name']
    avatar = text["user"]["profile_image_url"]
    url = f'https://twitter.com/{text["user"]["screen_name"]}'
    try:
        location = text["place"]["bounding_box"]["coordinates"]
    except:
        location = text["user"]["location"]
    tweet_data = {
            "content": content,
            "username": username,
            "avatar": avatar,
            "url": url,
            "location": location
        }
    collection.insert_one(tweet_data)


auth = OAuthHandler(API_KEY, API_SECRET_KEY)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
# Listen for a stream
listen = StdOutListener()
stream = Stream(auth, listen)
# Filter the stream to tweets containing #theyThinkItsAllClover:
stream.filter(track=['#theyThinkItsAllClover'])

Are you stiil getting the rest of the exception - that is the auth error? That is probably the root of your problem. If you're reading your credentials from a file, make sure that you are actually reading the file you think you're reading: https://help.pythonanywhere.com/pages/NoSuchFileOrDirectory/

Have you checked that the credentials that you're importing are the ones that you expect them to be when you're running that as a task?

Yep! I even just ran it in the most explicit way possible, I exported each of the environment variables in my always-on command i.e. export API_KEY=ajkndfso8a7sdfuad;exportAPI_SECRET_KEY=adfja98sdf7asdf etc (those aren't the actual keys! :-p)

Once again, it says running but as soon as I actually send a tweet that the stream is listening for, I get the same errors, starting with:

File "/home/cjcon90/.virtualenvs/venv/lib/python3.6/site-packages/pymongo/pool.py", line 1278, in _get_socket
sock_info = self.sockets.popleft()
IndexError: pop from an empty deque

and then the long series of other errors. As always the file works fine otherwise it just seems to be an impossibility as an always-on task, which is the whole point, unfortunately!!

I think after that attempt I'll just have to accept it ain't gonna work :-/

This is your problem:

Mar 5 12:45:47 pymongo.errors.OperationFailure: Authentication failed., full error: {'ok': 0, 'errmsg': 'Authentication failed.', 'code': 8000, 'codeName': 'AtlasError'} ```

Make sure the the values of the credentials that you're using in the always on task are actually the ones that you think you're setting. Print them out in the code so that you can make sure you're using the ones that you think you're using.