Forums

Web servers down: 502 - backend

enter image description hereAnyone else getting this error in thier website? I've been getting it for an hour or so.

Second time in as many weeks there has been a prolonged unscheduled outage for me. Not great service PA.

We're investigating, but we're not aware of any general system problems at the moment. If anyone else is seeing issues, please do let us know!

Thanks for the update Giles. I'm getting intermittent service, but performance is terrible. 20+ seconds load time

Thanks for the reply. Is your site really database-heavy? There was a report earlier of someone getting bad MySQL performance. The DB server your app uses is quite loaded at the moment (we're actually working on improving our handling of that kind of thing in our next release), but it shouldn't be loaded enough to cause server errors -- or indeed performance issues of the 20s-per-page level.

There are a fair few DB calls, but even the pages that don't call the db aren't loading

I think I may have tracked down the culprit. Someone on the same database instance as you was hammering the server with a quite incredible number of queries. Somehow (I'm sure innocently) they'd worked out a way of structuring their code that worked around our normal resource-limiting system.

I've put in place some temporary hacks to limit how much damage they can do, and it looks like your site is reasonably fast now -- at least, the index page responded immediately when I hit it just now. I'd be grateful if you would take a look and let me know if it looks like it's fixed.

All good. Lickity split again :)

Appreciate the quick resolution Giles. I'll have a good test and let you know if there is anything else

Fantastic! Thanks for confirming. Now we "just" need to work out a long-term solution to deal with this new way people can exceed their server resources so that it doesn't affect other customers...