Downtime Options

[95CelicaST]

I've noticed since the chat room deal that the site has been down quite a bit. Did the server crash again? Or is there some testing going on?


It also looks like we're running a backup - probably taken from the 1st of the month. Did SQL take a dump, Coomer?

This post has been edited by 95CelicaST: Aug 4, 2009 - 8:21 PM

[Coomer]

Definitely some horrible crashing. I've gone through hell the last couple days trying to figure stuff out.

What was happening was that the caching engine on the new server would crash randomly, using up every spare byte of memory and swap, bringing the server down instantly. A couple times when this happened, it corrupted the database, so that even a hard server restart wouldn't fix things.

So after two nights without sleep and tons of troubleshooting, I couldn't figure it out and bit the bullet and moved everything over to a different host (where I wasn't responsible for installing everything...I think something I did installing the server software and whatnot messed things up).

Now the site should be back up reliably. Unfortunately, I had to roll back to the last clean database/website backup, which was August 1. But at least the site is back up, and should be reliable once again. And I'm looking into fixing the chat.

I'm really sorry guys...this isn't acceptable and I apologize. I really try hard to keep this place online, and the last few days have been nothing but problems. But it should be back online...let me know if you see anything weird or broken.