Forums

It appears to me that everything is so sluggish with PAW. For example, it takes more and more time to run a script (judging from cron record). SSH login is slow as hell. Sometimes just can't connect and freeze at connecting phase. Even very easy command costs much time to run in the bash console. The last time when I opened an Ipython console, it was like half a minute passing then I saw the prompt.

any issue with the site performance?

Hi Robbie,

We were having some performance issues overnight. A whole heap of run away processes. Basically scaling issues that we need to be solving faster.

We've been working on it this morning and console responsiveness seems to be back to normal. The slow starting IPython console might have been a different issue. Put simply the very first time you open a console after we have deployed a new version there is quite a lot of work to be done. We prepare the entire execution environment. So that first start up can be very slow. The next one should be snappy however. What are things looking like now for you?

hi, hansel

it is better now according to cron log.

and still, I think the loading speed of the website and ssh needs some improvement. It's not like totally unusable or very bad. but sometimes could impair user experience. I am not sure whether geography location needs to be factored here. I live in Netherlands.

Ah yes, all the servers are currently located in the US which does make a difference. We are based in the UK so we do notice the trans-Atlantic lag. One of our next big jobs is improving performance and generally working on speed fixes. Any specific areas that you notice are oddly slow you should always mention to us so we can optimise them.

I'm in California, and have been noticing an additional 60 millisecond lag from PAW's AWS instance not present from my own personal AWS instance in us-east-1. It may help to assign higher priority "nice" values to online editors like vim and its downstream thoroughput -- clearly the latency is I/O bound.

Hi rsvp -- what are you seeing the lag in? That is, is it in a ping to the webserver or something over the console?

robbielj -- we're working on that right now; we've moving the website over to nginx from Apache, which should definitely help matters, and we're looking at ways of improving console server performance. Stay tuned!

hi giles, it was both: via ping and latency in the console. Didn't get precise timing on the latter though... typing vim commands I wondered if things were registering... then whamm, three or four previous ones would execute. The delay could be noticed even for backspace to erase characters -- comparable to a 300-baud modem :-) so that's the qualitiative speed. Wish I could report back "load averages" on the system, but both "top" and "uptime" seem to be unavailable.

Thanks, rsvp.

The ping slowness sounds odd, we don't do anything special there so I'd expect it to be pretty much the same as your instance. Not sure what the problem is there, we'll investigate.

The console lagginess -- well, there's a certain amount of overhead in the web-based console that's just inherent in using WebSockets and JavaScript, so that will probably never be as fast as ssh. But it should certainly be faster than it is. Right now we're having a few scaling issues as more and more people start using the system, and we need to work out some better ways of isolating stuff so that one person's heavy use doesn't impact others -- especially so that people using ssh (which after all is a paid feature) don't get hit.

Rest assured that sorting this all out is our top priority -- we can't expect people to want to use our service if it's too slow to be usable! And 300-baud modem speed definitely isn't usable unless you're doing 80s-retro development ;-) Normally it's much better than that, though -- I suspect that the time you saw that was when we were slow because of other stuff happening on the server.

Hello,

I'm having similar issues.. My ping is quite alright (roundabout 180ms from Germany) but trying to mount relatively small projects via ssh (~10mb currently) is taking minutes. I could definitely see some performance improvements with hosted projects (when interacting with them via http(s)), those seem much more responsive than a few months back, but maybe there's something you could do to speed up ssh.

Thanks & regards, Philipp

That's a bit weird, we're using sshfs mounts for our dev work and it's pretty fast. What client are you using to mount over ssh? Do you have a git repo in the projects?

Yes, I have a git repo there. I'm using sshfs/Macfusion.

sshfs and git don't get along very well. Git makes an enormous number of stat calls to the file system and they all go across the network individually. If you have a nice prompt that shows your git status or even just your branch, it can take forever. This is a known issue with git and networked filesystems. Even NFS has trouble - it's not as bad as sshfs, though.

Here's a much more specific performance non-issue.

I've found that when I try "save and run" on a unit test file the console takes an extremely long time to run (whereas the same task a couple of seconds on my local machine, for example). It is erratic. Sometimes it runs immediately, other times it seems to hang forever.

Next I tried running the very same unit test file directly from a bash console on PAW. It runs immediately, providing an obvious workaround (which I why I called it a non-issue).

Still, there is just a little glitch in there somewhere.

It's mainly because "Save & Run" is doing more. In order to prevent free accounts from running out of consoles really quickly (or paid accounts from having a gigantic list of defunct consoles), we have to ensure that we use an existing console if there is one that's been used for that particular file. Then we have to clean it up so we can run the code in it. All of that takes time and then we usually have to start a console process, which also takes time.

Another possibility (in addition to what Glenn mentioned above) is that due to chance, your consoles from Save and Run might be on a different, more loaded console server to your existing bash console. The stuff that Glenn describes should take less than ten seconds to happen -- so if the additional slowdown you're seeing is larger than that, then the system load might be the cause.

That's consistent with my observation (i.e., it sometimes runs in a time consistent with a relatively free machine). I'm running on a web dev account, btw.

Hmm, with a web dev account then you should be OK -- your processes get higher priority than free users, and it should all be load-balanced. Perhaps there's a bug in our load-balancing algorithm that means that some of our servers are running too much stuff.

We'll take a look, but in the meantime I guess the best option is to keep trying consoles until you get a fast one :-(