But also notice that they need to do more work tonight. According to the virtbiz site, it will start at 8pm Central and go through midnight. If I hear anything different, I’ll post.

I understand and apologize for your frustration. Last night we had a
maintenance window to upgrade software in the row distribution switches
and upgrade the Supervisor modules and MSFC’s in the gateway switches.
We and announced and anticipated interruptions in service for about 30
seconds or so but due to a conflict in software versions it went on
longer than that as we were forced to roll-back.

We’ve documented what went happened and identified where we had the
issue, which ultimately resulted in broken STP between the gateway and
distribution switches, to be a conflict in software versioning. We’ve
obtained alternate versions from Cisco as of last night and have the new
software loaded and running. However, once we got everything back
online, we were not comfortable sending things down for another reset.
The network (or more accurately, the users) had been through enough
trauma for one night.

Another window will be opened overnight tonight to address the failure
and complete the work that we set out to do last night. We are starting
early to construct a lab environment using equipment that we sourced
overnight and today to test with first, then apply to the network.

I understand that a good part of your concern stems from the
communication before and during the event. I’d like to bullet-point a
few things for you to help address that.

- We opened last night’s window to address an issue that came about in a
failure from last week that we had made temporary work-arounds for.

- Last night’s window was announced on our website, Twitter feed and
email. I know the website was unavailable for a while during the event.

- With regard to the phone system, we recognize that as an area that
needs improvement. A few minutes into the event we disabled the
answering system and had staff in to live-answer the phones to help
explain to customers what was going on. We felt that to be important
given the fact that the website was unavailable. Unfortunately, we
missed you.

You should know that as a company that takes its service very seriously,
we’re not satisfied with how last night’s window played out. We do feel
that there are lessons we can take that will help us continue to be
better. I certainly appreciate your comments for us, as well as your
understanding, as we continue working to be the guys you can always rely on.

Thanks,

Chris Gebhardt

Xen VPSes have a greater isolation level than OpenVZ and allow other Operating Systems to be run – like Windows. We will be allowing Windows VPSes, but you will need to provide an ISO of the OS version you want (will not count to your disk space and will remain on the server as a CD/DVD drive) and a scan of the License key.

Pricing will be announced shortly. Contact us if you are interested in a Xen VPS.

Edit: the support.virtbiz.com came up for a minute or so… Tried to put in a ticket, not sure if it made it or not.

3:30am edit: Was finally able to put in a ticket, last one did not take – network crapped out before the submit took. But it’s there now: Ticket ID: RXU-696907 I’ve also setup a binarycanary monitor for support.virtbiz.com to track it (how I could tell it was back up to hit submit again)

3:45 : Virtbiz claimed they notified customers of…

Engineers will be performing emergency network maintenance tonight, September 4 2010.

VIRTBIZ customers receiving service at the DAL-1 facility (2805 Canton St) in Dallas TX may be affected by this maintenance. The purpose of this work is to perform important software upgrades to distribution switches to help prevent flood attacks and unplanned outages.

EMERGENCY MAINTENANCE WINDOW: September 4, 2010, 8:00PM – 11:59PM CDT

Customers in the “Bakers Rack” area (ie: non-rack-mountable equipment) will experience an interruption in service that may last up to 30 minutes. Customers in rows 11-16 may experience brief interruptions in service lasting no longer than 30 seconds.

Only problem, couldn’t get to a website to know anything about it. No email was sent. And 30 minutes… try 2 hours plus! Not a happy camper. I understand the need, but the execution was pretty poor.

Oh wait, we’re in row 11… Should have been 30 SECONDS not 30 minutes… I think what gets me the most was the the “you should have known” attitude that came from the response and no apology, explanation, or anything else. How are you supposed to know if you don’t get notice, can’t get to the support site, and the phone’s network status message was “all networks operational”… Uh, NO they weren’t.

As a follow-up, I would like to share with you a postmortem of the
routing event you experienced this morning.

At about 6:30AM we received a preliminary alarm of a failure on one of
our customer gateway routers. The event was logged and the system was
enabled for failover to the redundant processing card on that gateway
router. A short time later, the routing fabric of the gateway failed
and caused a disruption in network service. Our technician brought the
router back up and ran a diagnostic, which ran clear. At that time, the
system was brought back online and routing resumed to your equipment.

After running stably for about 30 minutes, the system and its backup
failed again. Senior technicians implemented a hard-failure plan and
brought up a “warm spare” gateway router and began loading the
configuration. Network service was restored individually to each
customer affected by the outage as the configuration was checked-in and
loaded.

I should note that while you are one of several customers that was
affected by this incident, this was not a full-scale routing outage.
Our network architecture makes extensive use of sandboxing in order to
not put all eggs in one basket. Nevertheless, I understand that while
not everybody was affected, YOU were affected, and I do apologize for
the incident.

At this time, our warm spare has been placed into production and
functioning normally. We have brought in a cold spare and activated it
into standby so that redundancy is still in place. We anticipate that
we will replace the affected gateway router with new hardware. When
that happens, we will migrate your routing onto the new system, place
the warm spare back into standby and pull power from the cold spare.
All this will be seamless and will go unnoticed from a connectivity
standpoint.

Please be assured that your VIRTBIZ team will continue to review this
incident so that we can further improve our service to you.

I hope that you have a pleasant remainder of your weekend.

Original posts and updates:

We are experiencing a network issue between cogent and virtbiz this morning. First shortly after 6am, and lasted apx. 2 minutes. By the time I started to look into it and contact the DC, it was back up.

It is occurring again now. This time I was already on the servers looking into a spam report from a VPS account. Check checks showed the link between cogent and virtbiz to be down. It back up to the point I could send virtbiz a message (sure they already knew but just in case…), and am waiting to hear back. It had come back up at 7:50. Actually the virtbiz support site came up about 10 minutes before that.

At 8:01, it’s out again… Still waiting on an answer from vb…

As you can see by this, it’s having a problem finding a route.

[root@gt24-1 ~]# traceroute support.virtbiz.com
traceroute to support.virtbiz.com (208.77.216.244), 30 hops max, 40 byte packets
1 208.75.228.193 (208.75.228.193) 0.482 ms 0.809 ms 0.952 ms
2 tulip-core-2-ge3-8.tshost.com (208.75.224.5) 0.326 ms 0.318 ms 0.335 ms
3 core-1-gi7-2.tshost.com (208.75.224.13) 0.314 ms 0.344 ms 0.380 ms
4 te8-3.mpd01.atl01.atlas.cogentco.com (38.104.182.45) 0.310 ms 0.274 ms 0.287 ms
5 te0-0-0-6.mpd21.iah01.atlas.cogentco.com (154.54.28.254) 14.704 ms 14.816 ms te0-2-0-1.mpd21.iah01.atlas.cogentco.com (154.54.2.146) 14.697 ms
6 te2-1.mpd01.dfw01.atlas.cogentco.com (154.54.5.133) 20.387 ms 20.471 ms te3-4.mpd01.dfw01.atlas.cogentco.com (154.54.25.93) 20.560 ms
7 vl3834.na01.b000868-0.dfw01.atlas.cogentco.com (38.112.35.54) 21.105 ms 21.519 ms vl3534.na01.b000868-0.dfw01.atlas.cogentco.com (66.250.13.178) 20.402 ms
8 38.107.227.210 (38.107.227.210) 20.385 ms 20.646 ms 20.818 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 25.759 ms 25.751 ms 25.755 ms
16 * * *
17 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 25.950 ms 25.892 ms 25.902 ms
18 * * *
19 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 26.025 ms 25.996 ms 26.005 ms
20 * * *
21 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 26.172 ms 26.126 ms 26.668 ms
22 * * *
23 ge8-0.brdr2.dal1.virtbiz.com (64.125.196.45) 26.741 ms 26.330 ms 26.374 ms
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

As of 9am, we are back up at the moment… Just received this from VB:

We have been having an issue with one of our routers. Our technicians are correcting the issue and services should be fully restored shortly.

I’ll close this ticket now. If we can be of further service, just respond to this message.

Thank you
Jack B. – VIRTBIZ Internet Support

The plan right now is to replace the IBM X335 server. We expect this will be sometime in September – so we don’t rush things. We’d rather see it done properly than quickly. Since all public facing servers are virtualized, downtime should be able to be measured in seconds while the VM migrates.

Early the morning, GT24-2 experienced an issue, we remotely reset the machine and brought it back up. About an hour later, it happened a second time. The about 1pm we lost connectivity to all equipment there. Servers are coming back online and it appears they had lost power.

Edit -
Virtbiz says they had an issue with the power to the rack and had to shutdown the power temporarily.

As the /var/log/messages shows nothing out of the ordinary up until it stopped, we’ll also be logged into the node to see if we can see what is occurring in real-time. It also means it the delay until notification will be gone. We’ll also be sending out a replacement server to the DC in case we need to migrate the VMs off of it. If it does happen again we’ll migrate a couple VMs off the server as well, but trying not to as the server they will be migrated to has less RAM, CPU and disk to begin with. If we do it’s a stop gap measure while the replacement is setup.