Forums

Why so many scheduled processes?

I have a script, run-web2py-scheduler-for-CABLE.py, to launch the web2py scheduler for my application, called CABLE:

#/usr/bin/env python
import os
import socket
import subprocess
import sys
filename = os.path.abspath(__file__)  # we use this to generate a unique socket name

try:
    # we use a local socket as a lock. 
    # it can only be bound once, and will be released if the process dies
    socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM).bind('\0' + filename)
except socket.error:
    print("Failed to acquire lock, task must already be running")
    sys.exit()

subprocess.call(["python", "web2py/web2py.py", "-K", "CABLE"])

I have set up an hourly scheduled task: python run-web2py-scheduler-for-CABLE.py

It appears to be working: my website can process tasks using web2py's scheduler.

The strange thing to me is that when I view the list of Running Scheduled Tasks (by clicking Fetch Process List on the Schedule tab) I get a long list of processes, each with a "Kill" button attached. I thought the script would exit if it couldn't find a socket. I am confused: are multiple processes still running and consuming time? Are multiple workers engaged? That would be a problem because my scheduled tasks assume a single threaded operation. Can I safely ignore these multiple processes?

Here is the last few lines of the log file associated with the schedule:

ERROR:web2py.scheduler.giles-livetask1#24718:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#1659:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#17831:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#14512:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#426:Error retrieving status
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2015
Version 2.9.12-stable+timestamp.2015.01.17.06.11.03
Database drivers available: sqlite3, pymysql, MySQLdb, mysqlconnector, psycopg2, pg8000, pyodbc, pymongo, imaplib
starting single-scheduler for "CABLE"...
ERROR:web2py.scheduler.giles-livetask1#24088:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#31150:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#31150:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#17831:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#426:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#24088:Error retrieving status
ERROR:web2py.scheduler.giles-livetask1#6635:Error retrieving status
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2015
Version 2.9.12-stable+timestamp.2015.01.17.06.11.03
Database drivers available: sqlite3, pymysql, MySQLdb, mysqlconnector, psycopg2, pg8000, pyodbc, pymongo, imaplib
starting single-scheduler for "CABLE"...
ERROR:web2py.scheduler.giles-livetask1#24088:Error retrieving status

Will someone educate me? Thank you!

Beloved premium user GDRSolutions | 5 posts | Sept. 17, 2015, 4:02 p.m. | permalink

That looks like the socket lock isn't doing its job. The only thing I can think of is that an abspath of file is going to have slashes in it, the socket may not be behaving correctly when there are slashes in the name.

I would suggest adding some logging about the socket name and the attachment to it and then try changing the socket name to something simple and see how that affects things.

Staff glenn | 9498 posts | PythonAnywhere staff | Sept. 18, 2015, 12:23 p.m. | permalink

Thanks! I changed the script along the lines of http://stackoverflow.com/a/20886670/366221. That seems to have fixed it. Now only two processes are running (the bash and the python). THe log file still shows lots of "error retrieving status" messages but the scheduler is working. I'll consider the problem solved for now. Thanks again.

Beloved premium user GDRSolutions | 5 posts | Sept. 18, 2015, 7:21 p.m. | permalink