Forums

I have issues with your schedule tasks slow down since Jan 5.

I have started the same run of schedule tasks which normally completed within 2 hours but now it is taking 8 hours since Jan 5 2018.

Your team does not take a look in the matter and just give excuses of variance of schedule tasks. I use your service to run important schedule tasks alone and do not use any console run from terminal unless schedule tasks fail and interrupted.

Run from terminal console 0.5 hours

Before Jan 5, schedule tasks run taking 2 hours

After Jan 5, schedule tasks takes 8 hours

PA team should ask yourself if this is acceptable? No investigation after my complaint on Jan 6 till now. Really disappointed.

Please fix it now. It is not something from me(my script always the same) but your resources form PA side.

We are having a look at this, but any information that you can give us about eg. Which tasks show this, a little about what the tasks are doing (you can send it to the supprt email if you don't want it in the forums) etc. would be useful in diagnosing it.

I have reply in feedback but no response

Anyway, all the schedule tasks is slow down since Jan 5. For last two days, none of the schedule tasks able to complete. It means the condition is worsen day by day. It is now more than 16 times slower.

Something seriously wrong. PA should look into your system

Hi there, thanks for letting us know (by email) which tasks are slow. We've identified that the server they are running on is actually one of the least loaded, and we haven't made any changes on or around Jan 5th, so whilst we're not ruling out the possibility that the problem is at our end, and we will keep looking into it, we think it's at least worth looking into the possibility that the problem is elsewhere.

What are your scripts doing? Do you have any way of measuring and finding out which specific things are taking a long time? For example, is it CPU-intensive calculations, or read and writes from the filesystem, or database queries, or connections to external Internet sites?

There are some tips on this page about slow web apps -- not all are applicable to scheduled tasks, but some are...

Harry,

How you explain I use 30 minutes to complete the tasks if it run in console but taking 16 times in schedule tasks (and from Jan 7, all tasks never completed)?

This already ruling out the problem is from my side.

It is computation and query from api only. Again, this is no relation as it is running perfectly in console.

SO, it is down to your poor service in schedule tasks. I really do not see it is from my side. No logic at all if things going smooth in console and only slow in schedule tasks.

Again, it is top urgent to me. Please check it out. As I said, you may login and see my tasks. I need it fix today

Ah. One of my colleagues just informed me that your processes are spending most of their time in the tarpit. Once you've gone over your CPU quota, then we make no guarantees about how fast your code runs, since you're essentially using resources you haven't paid for.

From today's usage it looks like you need a minimum of 8,000 seconds (versus your current 2,000). So the first thing to do is to upgrade your account. You can do that from the accounts page. We'll happily refund you if it doesn't improve things...

Refer back of what I said. The usage is HIGH BECAUSE I have to rerun all those tasks that unable to complete.

SO DO YOU UNDERSTAND WHY THE USAGE IS HIGH NOW?

Run from terminal console 0.5 hours Before Jan 5, schedule tasks run taking 2 hours After Jan 5, schedule tasks takes 8 hours

STOP blaming me but check carefully what cause the problem. Or please move me from what you said a least loaded to another more loaded server.

Just to clarify things regarding the tarpit, and explain the variance you're seeing between consoles and scheduled tasks: each month, you pay for a certain amount of CPU power each day. Once you've used that up, you've used up everything that you have paid for.

We do not kill your processes, however -- instead, we keep them running, but they get any CPU power that's left over after people who have not used up all of the CPU power that they have paid for.

This means that if your code is running on a machine that has lots of unused CPU power after the paid-for power is accounted for, then you get extra CPU for free. We're happy with this, as it's CPU power that would be "wasted" otherwise.

But if you're on a machine that has less unused CPU power after the paid-for power has been used up, then your code will run more slowly.

Again, once you've used up your CPU allowance, you're getting something for free, which we provide as a courtesy to you. We do not guarantee that you'll get unlimited CPU power above what you've paid for; you just get what's left over.

Regarding this point:

The usage is HIGH BECAUSE I have to rerun all those tasks that unable to complete.

You have been using more than your CPU allowance every day for over a year, so I don't think that your high usage is due to anything you've changed since 5 January.

I usual usage is around 2500s unless I have run the code if it is fail in scheduled tasks.

I do not have any changes of the code for the part in scheduled tasks. It is for sure in January 5 or around that time.

If you claim I have used more than what is January 5 than days before is not true as well.

To solve the problem, why not just move me to another server to see if things improved?

I usual usage is around 2500s unless I have run the code if it is fail in scheduled tasks.

I do not have any changes of the code for the part in scheduled tasks. It is for sure in January 5 or around that time.

If you claim I have used more than what is January 5 than days before is not true as well.

To solve the problem, why not just move me to another server to see if things improved?

I'm afraid we don't have any easy way to move you to another server. And, as I say, the one you're running on is actually one of the least loaded.

Based on what you're saying (that you used to use approx 2,500 seconds), my best guess for what's been happening is that, before, your code was only just over its CPU-limit, so it would be slowed down at the end, but not enough to cause the tasks to fail / restart. But at some point, they got just a little bit slower, and that caused some of them to be restarted, which caused more CPU usage, which caused you to go into the tarpit sooner, which caused more of them to be restarted, which caused more CPU usage, and so on.

In any case, the solution is to upgrade your account and start paying for the CPU-seconds you're using. Currently that's about 8,000. Maybe you'll find that once you're no longer in the tarpit that number will drop, and you can downgrade again, maybe to approx 3,000. But the only way to find out is to upgrade first. Let us know when you've done that, and what results you see.

I stop all the tasks before the four important tasks from 18 to 19 UTC.

My cpu usage is just reset. As what you said, my schedule task should be fast and completed in 30 minutes. If it takes longer or not completed, then it is something wrong at your side.

I can't guarantee your tasks will run as fast on the tasks server as they do on the console servers, because they're different server types. But as long as you don't go into the tarpit, it should at least be within the same order of magnitude. certainly they shouldn't be 16x slower. If you do go into the tarpit of course, all bets are off (I see you've already used 50% of your quota in less than an hour. Hopefully that will slow down if you only just switched off most of your tasks?)

let us know what happens, in any case...

Hi there, according to our logs you've now exceeded your quota again (it looks like it took you just over 3 hours to do so). Did any of your tasks complete in the meantime?

[edit]: PS - at the time of writing, you had only one task running, longrun_aus.py, and it had used about 600 CPU-seconds, so I assume the other 1400 or so seconds were used by other tasks. It may be worth re-running this experiment with all your tasks except one disabled.

I have a similar situation last year on a hacker account and had to pay someone to migrate to a USD 5 dollar VPS and it works fine.

I run 3 tasks hourly which takes 90s each in console. But then in schedule tasks, it takes 180s to sometimes 400s.

I am checking price movement for soy bean, coffee and sugar and 5 minutes of delay is serious. I am not in tarpit anyway. I believe it can be due to slow to start rather than slow to compute.

I do not understand what is different of console and schedule tasks, but I feel schedule tasks server is significantly slower. I gave up after 3 months although I like PA and come back to use the free account.

You should enhance your schedule tasks system. People like me who writes some code but do not know how to configure a VPS and strong IT background loves PA user friendly environment rather than DigitalOcean, Linode and etc. I am much willing to pay PA than other VPS if the performance is acceptable.

yesterday, some tasks which is rerun not completed before it is reset. Today will test again without those distraction.

However, I want to point out how much schedule tasks has been slow down with my two short tasks in recent months.

stcoklist_th1.py

Dec 13, 376 s

Jan 1, 525s

Jan 2, 1380s

Jan 5, 517s

Jan 6, 1197s

Jan 8, 5336s

Jan 9, 2667s

the same for stocklist_asx1.py

It started at 2s in June, but from December, it takes 8 to 9s until January 8, it takes 66s for the tasks.

You can check the log

Understood. Let's see how it goes today, when there's only one task running.

re: those dates, I see that you were actually in the tarpit on every single one of them. Here are the times at which your tarpit states changed on those days:

2017-12-13 12:40:22 Removed from daily tarpit
2017-12-13 19:17:36 Placed in daily tarpit
2018-01-01 13:12:18 Removed from daily tarpit
2018-01-01 19:46:17 Placed in daily tarpit
2018-01-02 13:14:20 Removed from daily tarpit
2018-01-02 19:11:37 Placed in daily tarpit
2018-01-05 13:17:21 Removed from daily tarpit
2018-01-05 16:32:27 Placed in daily tarpit
2018-01-06 13:19:19 Removed from daily tarpit
2018-01-06 17:12:21 Placed in daily tarpit
2018-01-08 13:21:20 Removed from daily tarpit
2018-01-08 17:03:43 Placed in daily tarpit
2018-01-09 13:23:19 Removed from daily tarpit
2018-01-09 16:27:40 Placed in daily tarpit

So you're consistently going into the tarpit within 7 hours, and sometimes as soon as 3 hours, so unless your tasks are running during the (brief) time where you're actually using your allocated quota, then we're not going to be held to any promises about execution time.

I can see that, from your point of view, something that used to run fast has gotten more slow, so that seems unfair. But, from our point of view, you've been consistently using resources that you haven't paid for, and we've been sending you warnings about that,every single day.

You're currently on our cheapest plan, at $5 per month. Upgrading to a sufficient CPU quota would only cost you $11/month. Possibly less, if it means your scripts need to do less retries.

It completed but 2 times slower even not in tarpit

tasks console scheduled how many times slower

sgx 1800 4284 2.38

KLSE 1800 3389 1.882777778

SET 1080 2212 2.048148148

IDX 1020 1907 1.869607843

It is obviously not what Harry mention "the same magnitude between console and schedule tasks if I am not in tarpit."

another one is stocklist_asx1, it also shown it is 2x slower than in June 2017.4s now compare 2s previously.

Noted than when I am wirting this, I only used 77% of CPU

OK, thanks, that's useful data. Harry said "the same order of magnitude", which technically means between one and ten times the same speed, but I agree that two times slower when you're not in the tarpit doesn't sound good.

I've made a couple of tweaks to the configuration of our task servers, and I'd be interested in knowing if you see the same level of slowdown now.

One other question -- what does your task do? I'm not asking in order to assign blame, like you thought when my colleague Glenn asked before. I'm asking because it will help us work out which part of the system might be causing the slowdown. The actions we need to take will differ based on whether you're writing lots of stuff to files, or making lots of network connections, or doing lots of number-crunching.

After your tweaking at your side, it become even worse.. None of the tasks is completed even I am not in tarpit.

two main type tasks.

1. runSGX.sh and the others 3 tasks consists two parts. First, use pandas to query data. After that in another python script use pands,numpy and talib to compute finance model.

2. marketdata.py is using lxml,request to scrap some data.

Honestly, I have no ideas what is going wrong since my script has been almost the same, except change a some parameters inside which not affected my usage.

I have tested and compare the speed in console and scheduled tasks when I deploy in April last year. At that time, I can said console time is like 10 to 15% faster than scheduled tasks. Even in tarpit, the schedule tasks is just 30% to 40% slower.

I believe something is changed in the backend gradually since april last year that caused it slower and slower. The evidence is even not in tarpit, it is 2x slower and like today, not even manage to complete.

Attached the picture as evidence.

That's really weird. I can think of any way the tweaks could have made things slower; at the very least I'd have expected no change. I see that one of your tasks has completed since then, however -- the one that starts at 6:06am. Could you see if that's running any faster or slower than it was?

Regarding what your scripts do -- so it sounds like your scripts in the first group are doing a bit of everything -- querying data (either via pandas or directly using requests), then doing some number crunching. I assume they're also writing the models out as files, is that right?

Is marketdata.py script doing anything apart from the requests stuff? For example, is it writing to disk?

Today's the tasks completed. it is about 50% slower than console or 1.5X compare 2X previously. Marketdata do a calculation at the end and create a csv file as output.

Pandas script create a csv for each query(create around 500 files). Another python script then read the csv to analysis and create a csv as output in the end.

OK, so that's an improvement at least. There's one other thing I'd like to try, but I'll have to do it in the office on Monday. It will involve shutting down any running tasks you have -- is there a specific time of day that would be best for that?

You can shut down schedule tasks between 12.30 UTC to 13.30UTC or before 06.00 UTC

May I know what have you tweaked to get a better result?

Thanks! The tweak I did before slightly increased the CPU capacity available to your tasks. What I'm considering doing tomorrow is temporarily starting a new task server and putting just your tasks on it. This won't work for a long-term solution (the server costs about $200 / month) but if you get a sudden speedup, it'll tell us that the slowdown is CPU-related. If that's the case, we'll move other people's tasks onto it (so that it's paying its way) and monitor.

It turned out that putting just one user on a new server wasn't doable. However, we've added more servers to the scheduled task pool -- it would be good to know if that's made things any faster for you.

Good. It is now about 15% slower than console and it is acceptable for my tasks. Hope it can continue to perform the same.

We'll keep an eye out. If you start seeing large differences between runtimes when you're not in the tarpit, please do let us know. As ever, the in-tarpit runtime may vary.