Yesterday, early morning (India time) servers hosting Visual Website Optimizer stopped responding to queris. This may have slowed down your website(s). Please accept my sincere apologies for that. You cannot imagine how concerned we (Wingify team) all are for this. The servers are up at the moment. To clarify, the servers have NOT been compromised – it is just that they got slowed down.
We are transparent with customers on such issues and take responsibility for informing you on what happened, why it happened and what steps we are we taking to ensure it never happens again.
For people who understand server-admin technical terms: our server load typically ranges between 10%-15%. Somehow, yesterday during the outage it shot upto 400% which slowed down the servers. We are still investigating what exactly caused such high server activity but there are three likely causes:
1. A bug in code which resulted in repeated endless queries on database
2. A VERY, VERY highly trafficked site sending requests to the server
3. A DDoS attack on servers
We had taken proper precautions for cases like these: a) server monitoring is in place; b) daily backups are taken at an offsite secure location; c) your passwords are stored in salted hash format (and not plaintext) to ensure that nobody (not even us) can know what your password is.
All in all, we are REALLY concerned about safety of your data and smooth functioning of your websites where Visual Website Optimizer code is installed.
Just as a precaution, so that such a server outage doesn’t happen again, we are taking two immediate steps:
1. Upgrading our servers – essentially doubling the capacity
2. Upgrading the server monitoring service so that response time for such an outage can be brought down to minutes
As mentioned earlier, the servers are up now. So, if you had removed the code due to the slow response, please re-insert it. The situation is back to normal.
Once again, I personally apologise for this outage.
Thanks for understanding.
Regards,
Paras Chopra
RSS feed for comments on this post. TrackBack URL
This can happen to everyone. I’ll put the code back on
Thanks for the information.
/Kim A
Comment by Kim Andersen — January 27, 2010 @ 10:03 am
[...] more downtime – Amazon S3 based external JavaScript hosting During the last server outage, some of the websites where Visual Website Optimizer was running slowed down because our servers [...]
Pingback by No more downtime – Amazon S3 based external JavaScript hosting « I love split testing – Visual Website Optimizer Blog — April 28, 2010 @ 7:52 pm