Jan 15 2008
THC Resolves Unexpected Database Issue After Security Patch, Server Reboot
On January 8, 2008, we announced to our customers at Top Hosting Center through our Message Center that we are going to apply security patches and reboot our servers in the evening of January 11. The advisory reads:
Please take note that Friday January 11th at 09:00 pm Eastern time, we are going to apply security patches and reboot our Linux servers.
That night, without any negative expectations associated to the applied security patches and server reboot, we encountered a serious connectivity issue between our web server and the MySQL database engine. Database-reliant web pages of websites running under THC completely failed regardless of plan and platform. There was no exception: even on THC’s official site, all dynamic pages were down as well.
The assessments made by the tech department revealed that the database connectivity issue was severe. This prompted me to declare an “emergency status” after realizing the scope of the problem, and directed our support department to issue another advisory to our customers about the situation.
We are currently facing an issue with MySQL databases after our reboot.
Please bear with us as we are currently working on the issue.
All my technical staff quickly responded to the problems and checked the software configurations of the web server and database engine. After hours of analysis, they came up with a solution to implement “load balancing” which I agreed. With it, should our primary MySQL server fail again, site operations will not be interrupted as the secondary server automatically replaces and resumes the database operations. After implementation and several successful confirmation tests, the issue was resolved and our support department rolled another email to our customers.
We are glad to inform you that our MySQL database connectivity has been fully restored. All database-reliant web pages and files are now working well.
As you read from our previous advisory, we applied security patches and did a server reboot last night. After such, we encountered unexpected connectivity issue with our MySQL server in a high severity.
The time we spent during the restoration process included a major revamp with our MySQL server configuration. We have applied load balancing as our permanent solution for this issue. With it, should our primary MySQL server fails again, site operations will not be interrupted as the secondary MySQL server automatically takes care of the database operations.
Please note that all your data and files are backed-up, safeguarded and remained intact after recovery.
If you ever encounter problems with your website that are non-existent before, please write to this e-mail customercare@tophostingcenter.com
We are very grateful for your forbearance and understanding.
It was a quite long night for everyone at THC, and we were all astonished when we realized how many cups of coffee we consumed. I was able to congratulate all of them for doing their dedication and for delivering their best service in catering rare emergency situation such as this.
I am more grateful to all our customers who were very patient and understanding, as reported by our customer support department. Please be informed that this event is isolated and the people of THC are strongly committed to give you the best hosting experience ever.
Update: Twelve hours later, we made some optimizations and adjustments to our load balancing configuration at our data center. This caused short, intermittent time outs to all websites. The end results, however, are improved server performance and assured stability.