I’m performing some maintenance of the server that houses the remote backups.
I expect it tol be offline until sometime this evening…
Will update this thread when it’s back.
Update: It came back up during the evening and all backups are again available.
11/6/08: GT24-1 is ready for your VPSes. We’ve opened up the ordering side to accept orders for it. Note though, that we will be setting up VPS accounts by hand not instantaneously at least for now.
Win2008-2 is now named HyperVM… It’s been reconfigured as the HyperVM manager server. It’s the centralized server for controlling your VPS regardless of where it’s located. It also serves as the backup manager handling the backups for the VPSes. The backups themselves are actually stored on another machine with RAID5 data protection.
Last night we experienced a problem with an upstream provider for the Dallas based servers.
I’ve copied the email from our Dallas provider descibing what happened.
Hello Customer,
This is an email to advise you of an incident that was experienced on the network on the night of Thursday, November 20, 2008. We began experiencing intermittent loss of BGP routing from our upstream providers which alternated across our fiber and RF networks. The incident began as transient packet loss and eventually progressed to a full loss of routing taking place for intermittent times through approximately 3:45am. At that time our technicians de-peered from our upstream providers in order to isolate the problem.Indications initially were that the issue was due to bad media (ie: fiber, interface). After de-peering and removing BGP sessions we found the trouble went away but recurred when BGP was turned up again. This pointed to a DDOS attack.
We were able to restore customer networks relatively soon after this, with full service restoration occurring around 5:00am. That was after we implemented a tool created to trace the source and destination of the packets. We found a single customer to be the target of the attack from a network of systems worldwide. We have logged this information in order to assist law enforcement and our attorneys with further investigation.
We kept our office staffed overnight in order to receive calls from concerned customers. The technicians responding to the issue as well as the customer service reps on duty at the time are scheduled for a follow up meeting with management to talk about improvements to our systems, policies and procedures that are in place to deal with this type of situation. If you have concerns or questions please let our business office know. The phone number is 972-485-4125. If you need further technical support, please open a support ticket at https://support.virtbiz.com
Stuart G
VIRTBIZ Support
GT24-1 is up and running. The maintenance and monitoring processes have been setup, such as automatic centralized backups to a remote server. We’ll watch the server over the next couple of days just to be sure, but all looks well.
GT24-1 arrived safely in Atlanta. We’re removing KCVPS from the monitoring and replacing it with
GT24-1. KCVPS was never a customer hosting site, but did perform backup DNS. With the switch, we are resetting
the uptime statistics.
We’re experiencing a problem with the linux1 server. It is being looked into and will post an update as soon as we we know something.
This affects our site, the clientexec panel, and ns1 (ns2 and ns3 are running so people should still be able to find your site).
Update – At about 8:10am central, the linux1 was back online.
Comments Off