Forums

Having trouble with schedules tasks running correctly and return code 254

We are getting errors, our code will stop working several hours in after we are running scheduled tasks every hour to send data to our mysql database we have created on pythonanywhere... So we are getting return code 1 and return code 254, I think that 254 is the one that is killing us. So basically we are dealing with an issue were even though our script doesn't have a loop, sometime our console processes there are like tons of them open. We are doing an ssh tunnel into our db from our code because we can't get it to work any other way. We tried not using a tunnel and it never worked. We keep getting these sshd processes like this sshd: samwins 2021-07-29 at 05:22:56 0. They aren't using any power up but I do think they are causing a problem.

Here is a part of our code you would probably want to see. I took out sensitive info. Can anyone explain why the db task code log is giving an error saying we are out of resources? We do not have it on a loop, other than it's a scheduled task on pythonanywhere. Thanks for the help!

sshtunnel.SSH_TIMEOUT = 21000000.0
sshtunnel.TUNNEL_TIMEOUT = 21000000.0
print("Attempting to connect to database")
with sshtunnel.SSHTunnelForwarder(
    ('ssh.pythonanywhere.com'),
    ssh_username='', ssh_password='"',
    remote_bind_address=('s.mysql.pythonanywhere-services.com', 3306)
) as tunnel:
    connection = MySQLdb.connect(
        user='',
        passwd='',
        host='127.0.0.1', port=tunnel.local_bind_port,
        db='',
    )
    print("Database Connected!")
    #Top 50 Performer Script & Block Winners
    for x in addresses:
        response = r.get("https://api.helium.io/v1/hotspots/"+x+"/rewards/sum?min_time=-1%20day&bucket=day").text
        response_info = json.loads(response)
        list_hnt_earned.append(response_info['data'][0]['total'])
        response2 = r.get("https://api.helium.io/v1/hotspots/"+x).text
        response_info2 = json.loads(response2)
        allnodes[response_info2['data']['name']] = response_info['data'][0]['total']
    #Go through the dict to filter out top 50 and top 1 performing node
    N = 50
    top_fifty = dict(sorted(allnodes.items(), key = itemgetter(1), reverse = True)[:N])
    M = 1
    top_one = dict(sorted(top_fifty.items(), key = itemgetter(1), reverse = True)[:M])
    hotspotcursor = connection.cursor()
    for key in top_one:  #Print out top 1 and top 50 nodes with name and amount of hnt perday
        print(key+ " Won the block!")
        hotspotcursor.execute("insert into top_performers db mysql info went here")

====Scheduled Task Log===== 2021-07-28 11:46:13 -- Completed task, took 9.54 seconds, return code was 1.

/etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: Resource temporarily unavailable

2021-07-28 12:46:20 -- Completed task, took 17.12 seconds, return code was 254.

/etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: Resource temporarily unavailable

2021-07-28 13:46:20 -- Completed task, took 17.11 seconds, return code was 254.

/etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: Resource temporarily unavailable

2021-07-28 14:46:21 -- Completed task, took 17.68 seconds, return code was 254.

/etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: Resource temporarily unavailable

2021-07-28 15:46:21 -- Completed task, took 17.57 seconds, return code was 254.

/etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: retry: No child processes /etc/bashrc: fork: Resource temporarily unavailable

2021-07-28 16:46:21 -- Completed task, took 17.60 seconds, return code was 254.

Loading Top Performance List... Attempting to connect to database Database Connected!

2021-07-28 17:19:32 -- Completed task, took 148.80 seconds, return code was 0.

Looks like you spawned too many processes. 128 is the per-user limit.

@fji how do I make my code not spawn so many processes? I have no clue how or why it's doing that

I think I might be able to fix the problem alternatively by connecting to my db without sshtunnel. Every time I have tried though, it has failed miserably. Does anyone have a format they can show me to interface with mysql database inside pythonanywhere without an ssh tunnel?

If your code is running inside PythonAnywhere, it definitely doesn't need to use an SSH tunnel -- those are only required if you're trying to connect from outside our system.

What errors do you get if you try to connect without using one? What config are you using to do the connection (with any passwords masked out, of course)?

we tried something like this

conn = await aiomysql.connect(host='127.0.0.1', port=3306,
                                       user='', password='', db='')

also I adjusted the timeout to 15 minutes and the discord bot code to hourly to stop so many processes from loading and it worked. We don't have all those processes anymore, but now we have new errors. What is this return code 1 error and what does it mean? Why would the scripts not be running now? They all stopped working over the last hour

2021-07-29 23:55:24,239| ERROR   | Could not establish connection from local ('127.0.0.1', 37577) to remote ('samwins.mysql.pythonanywhere-services.com', 3306) side of the tunnel: can't start new thread

rm: cannot remove '/mnt/chroots/samwins/fishnchips/dev/shm': Device or resource busy
2021-07-29 23:32:14,555| ERROR   | Unknown exception: can't start new thread
2021-07-29 23:32:14,563| ERROR   | Traceback (most recent call last):
2021-07-29 23:32:14,563| ERROR   |   File "/home/samwins/.local/lib/python3.8/site-packages/paramiko/transport.py", line 2047, in run
2021-07-29 23:32:14,563| ERROR   |     self.packetizer.start_handshake(self.handshake_timeout)
2021-07-29 23:32:14,563| ERROR   |   File "/home/samwins/.local/lib/python3.8/site-packages/paramiko/packet.py", line 252, in start_handshake
2021-07-29 23:32:14,563| ERROR   |     self.__timer.start()
2021-07-29 23:32:14,563| ERROR   |   File "/usr/lib/python3.8/threading.py", line 852, in start
2021-07-29 23:32:14,563| ERROR   |     _start_new_thread(self._bootstrap, ())
2021-07-29 23:32:14,563| ERROR   | RuntimeError: can't start new thread
2021-07-29 23:32:14,563| ERROR   | 
An open stream object is being garbage collected; call "stream.close()" explicitly.
An open stream object is being garbage collected; call "stream.close()" explicitly.
An open stream object is being garbage collected; call "stream.close()" explicitly.
An open stream object is being garbage collected; call "stream.close()" explicitly.
rm: cannot remove '/mnt/chroots/samwins/fishnchips/dev/shm': Device or resource busy
Traceback (most recent call last):
  File "/home/anywhere/djangoproject/jails/pa_sudo.py", line 160, in <module>
  File "/home/anywhere/djangoproject/jails/pa_sudo.py", line 156, in main
  File "/home/anywhere/djangoproject/jails/pa_sudo.py", line 136, in pa_sudo
BlockingIOError: [Errno 11] Resource temporarily unavailable

2021-07-30 00:32:11 -- Completed task, took 6.57 seconds, return code was 1.

Traceback (most recent call last):
  File "/home/samwins/neo_site/python_scripts/under_performing_bot.py", line 75, in <module>
    bot.run('')
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/client.py", line 723, in run
    return future.result()
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/client.py", line 702, in runner
    await self.start(*args, **kwargs)
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/client.py", line 666, in start
    await self.connect(reconnect=reconnect)
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/client.py", line 563, in connect
    self.ws = await asyncio.wait_for(coro, timeout=60.0)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/gateway.py", line 329, in from_client
    await ws.poll_event()
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/gateway.py", line 559, in poll_event
    await self.received_message(msg.data)
  File "/home/samwins/.local/lib/python3.8/site-packages/discord/gateway.py", line 469, in received_message
    self._keep_alive.start()
  File "/usr/lib/python3.8/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

2021-07-30 00:08:16 -- Completed task, took 190.30 seconds, return code was 1.

I think the main problem is the host='127.0.0.1'. That's specific to using an SSH tunnel; if you're connecting without one, rather than 127.0.0.1, you should use the database hostname from the "MySQL" tab on the "Databases" page -- it will start with your username.

I tried

async def test():
    connection = await aiomysql.connect(host='samwins.mysql.pythonanywhere-services.com', port=3306,
                                       user='', password=', db='samwins$mydbname')
    print("Database Connected!")

It's never getting to the Database connected print line

Do I assume correctly that user password and mydbname are replaced with correct values?

Yes they are the correct values

How do you call test()?

I don't use it again after this

But how do you call it the first time? async def is not calling anything, it's just a defintion.