I received this email in response to my ticket about what happened last night… Short answer = Cisco issues. Been there myself, so I understand it. My biggest issue however was no notice since this apparently has been scheduled beforehand. Virtbiz says they sent an email, and I have no reason to doubt it. But I never got it. I don’t look to twitter as a support mechanism so never even thought to go there. So we still have a couple things to work out communication-wise.

But also notice that they need to do more work tonight. According to the virtbiz site, it will start at 8pm Central and go through midnight. If I hear anything different, I’ll post.

I understand and apologize for your frustration. Last night we had a
maintenance window to upgrade software in the row distribution switches
and upgrade the Supervisor modules and MSFC’s in the gateway switches.
We and announced and anticipated interruptions in service for about 30
seconds or so but due to a conflict in software versions it went on
longer than that as we were forced to roll-back.

We’ve documented what went happened and identified where we had the
issue, which ultimately resulted in broken STP between the gateway and
distribution switches, to be a conflict in software versioning. We’ve
obtained alternate versions from Cisco as of last night and have the new
software loaded and running. However, once we got everything back
online, we were not comfortable sending things down for another reset.
The network (or more accurately, the users) had been through enough
trauma for one night.

Another window will be opened overnight tonight to address the failure
and complete the work that we set out to do last night. We are starting
early to construct a lab environment using equipment that we sourced
overnight and today to test with first, then apply to the network.

I understand that a good part of your concern stems from the
communication before and during the event. I’d like to bullet-point a
few things for you to help address that.

- We opened last night’s window to address an issue that came about in a
failure from last week that we had made temporary work-arounds for.

- Last night’s window was announced on our website, Twitter feed and
email. I know the website was unavailable for a while during the event.

- With regard to the phone system, we recognize that as an area that
needs improvement. A few minutes into the event we disabled the
answering system and had staff in to live-answer the phones to help
explain to customers what was going on. We felt that to be important
given the fact that the website was unavailable. Unfortunately, we
missed you.

You should know that as a company that takes its service very seriously,
we’re not satisfied with how last night’s window played out. We do feel
that there are lessons we can take that will help us continue to be
better. I certainly appreciate your comments for us, as well as your
understanding, as we continue working to be the guys you can always rely on.

Thanks,

Chris Gebhardt

Robert

Robert Porter is the founder and managing member of Lagniappe Internet LLC. Robert holds multiple certifications incluging Oracle Certified Professional-Java 6 SE (OCP-J6), MCSE, A+, Net+, Project+, Security+, and multiple CIW certifications. He has been in the hosting industry for more than a decade and is founded Lagniappe Internet L.L.C. as a privately owned, completely debt free, hosting company based out of New Orleans. Robert's background includes 25+ years in programming, databases, networking and systems administration.

We’re finalizing a Xen VPS server. It’s in Dallas – has 2 Quad Core AMD Opteron processors, 16GB of ECC RAM and 4x 750 WD Raid Edition (RE2) Enterprise class drives connected to a Dell/LSI PERC RAID controller in HARDWARE RAID-10 for speed and reliability.

Xen VPSes have a greater isolation level than OpenVZ and allow other Operating Systems to be run – like Windows. We will be allowing Windows VPSes, but you will need to provide an ISO of the OS version you want (will not count to your disk space and will remain on the server as a CD/DVD drive) and a scan of the License key.

Pricing will be announced shortly. Contact us if you are interested in a Xen VPS.

Robert

Robert Porter is the founder and managing member of Lagniappe Internet LLC. Robert holds multiple certifications incluging Oracle Certified Professional-Java 6 SE (OCP-J6), MCSE, A+, Net+, Project+, Security+, and multiple CIW certifications. He has been in the hosting industry for more than a decade and is founded Lagniappe Internet L.L.C. as a privately owned, completely debt free, hosting company based out of New Orleans. Robert's background includes 25+ years in programming, databases, networking and systems administration.

We now have a cPanel/WHM reseller vServer available in Atlanta. Same prices as Dallas, but use this link to sign up: http://www.lagniappeinternet.com/members/signup.php?clienttype=16

Robert

Robert Porter is the founder and managing member of Lagniappe Internet LLC. Robert holds multiple certifications incluging Oracle Certified Professional-Java 6 SE (OCP-J6), MCSE, A+, Net+, Project+, Security+, and multiple CIW certifications. He has been in the hosting industry for more than a decade and is founded Lagniappe Internet L.L.C. as a privately owned, completely debt free, hosting company based out of New Orleans. Robert's background includes 25+ years in programming, databases, networking and systems administration.

We are considering a Enkompass server for resellers. Basically, Enkompass is cPanel for Windows. What we’d like to know is how many people would be interested in an Enkompass Reseller account. Enkompass licenses are based on the number of accounts and is more expensive… so pricing would be different than Linux cPanel/WHM licenses. But you would have access to ASP.NET and PHP… Please email rob -at- lagniappeinternet.com or put in a support ticket to let us know if you’re interested.

Robert

Robert Porter is the founder and managing member of Lagniappe Internet LLC. Robert holds multiple certifications incluging Oracle Certified Professional-Java 6 SE (OCP-J6), MCSE, A+, Net+, Project+, Security+, and multiple CIW certifications. He has been in the hosting industry for more than a decade and is founded Lagniappe Internet L.L.C. as a privately owned, completely debt free, hosting company based out of New Orleans. Robert's background includes 25+ years in programming, databases, networking and systems administration.

At 1:38am I started receiving pages from binarycanary about servers in Dallas being unreachable. Two minutes later 1 server reportedly passed, but was down 1 minute later again. The support.virtbis.com site is down as well, which leads me to believe it is a repeat of the incident on the 28th of August – all the servers were functioning fine, but not reachable because of a router issue. I’ve emailed Chris with Virtbiz a – though I’m not sure it can get through right now as well as called and left voicemail. I’m sure they have their own monitoring setup and have been altered. I will update this as I know anything.

Edit: the support.virtbiz.com came up for a minute or so… Tried to put in a ticket, not sure if it made it or not.

3:30am edit: Was finally able to put in a ticket, last one did not take – network crapped out before the submit took. But it’s there now: Ticket ID: RXU-696907 I’ve also setup a binarycanary monitor for support.virtbiz.com to track it (how I could tell it was back up to hit submit again)

3:45 : Virtbiz claimed they notified customers of…

Engineers will be performing emergency network maintenance tonight, September 4 2010.

VIRTBIZ customers receiving service at the DAL-1 facility (2805 Canton St) in Dallas TX may be affected by this maintenance. The purpose of this work is to perform important software upgrades to distribution switches to help prevent flood attacks and unplanned outages.

EMERGENCY MAINTENANCE WINDOW: September 4, 2010, 8:00PM – 11:59PM CDT

Customers in the “Bakers Rack” area (ie: non-rack-mountable equipment) will experience an interruption in service that may last up to 30 minutes. Customers in rows 11-16 may experience brief interruptions in service lasting no longer than 30 seconds.

Only problem, couldn’t get to a website to know anything about it. No email was sent. And 30 minutes… try 2 hours plus! Not a happy camper. I understand the need, but the execution was pretty poor.

Oh wait, we’re in row 11… Should have been 30 SECONDS not 30 minutes… I think what gets me the most was the the “you should have known” attitude that came from the response and no apology, explanation, or anything else. How are you supposed to know if you don’t get notice, can’t get to the support site, and the phone’s network status message was “all networks operational”… Uh, NO they weren’t.

Robert

Robert Porter is the founder and managing member of Lagniappe Internet LLC. Robert holds multiple certifications incluging Oracle Certified Professional-Java 6 SE (OCP-J6), MCSE, A+, Net+, Project+, Security+, and multiple CIW certifications. He has been in the hosting industry for more than a decade and is founded Lagniappe Internet L.L.C. as a privately owned, completely debt free, hosting company based out of New Orleans. Robert's background includes 25+ years in programming, databases, networking and systems administration.