A few minutes ago Ie started getting alerts for the Dallas servers. Initial checks showed not being able to connect to them. A few minutes later and I could get back in. Pings at that time were intermittent. They are back to 100%… I’ve contacted Virtbiz and will let you know if I find anything more out.
The gt24-2 server is currently having an issue… it started as several VMs residing there were unavailable from the network though the VMs appeared to be running. The server is being cycled and we’ll update as we know more. This also affects the DA1 directadmin reseller VM.
5:10p – Asked Virtbiz to take a look… They say it is coming up but running an fsck.
With 1.5TB of space (4x 750GB drives in RAID10) that can take a while.
Will update when I know more.
5:13pm – Node is back up. fsck is finished. Virtual machines starting to come back up now…
5:15p – da1 directadmin reseller is back up
5:16p – All virtual servers back up. All looks well.
If you’re seeing this post, all is well… the http://news.lagniappeinternet.com/ site has been moved to a new virtual server to enable some new features… More information will be coming soon.
We’re doing some quick patching and reboots to address a Linux kernel vulnerability that has been announced. The servers should be down less than 5 minutes and virtual servers they house less than 20 minutes. Sorry for the inconvenience but it
We received notice from Virtbiz that they will need to cut the power off tomorrow night at 11pm Central time.
[quote]As you should be aware from our previous announcements, we have recently replaced a primary UPS that provides filtered power to your equipment. The change took place because of certain errors that could not be cleared from the replaced unit. Accordingly, a new system was ordered and installed over the weekend. This was done without interruption to service.
Unfortunately, due to technical limitations of the outgoing maintenance bypass and the new UPS, we are forced to take a small outage in order to transfer back to fully protected power. In order to limit possible exposure to failure from utility power service, we have elected to take this emergency maintenance window as fast as practically possible.
We are scheduling the emergency maintenance window for Tuesday, September 21 2010, after 11:00PM Central Time.
Customers located in Rows 10, 11 & 12 will experience a brief power outage while power is switched to the new UPS. The interruption will last no longer than 1 minute, and may in fact be contained to under 5 seconds. Of course, that will likely be long enough to cause an outage to your connected equipment.
…
[/quote]
I received this email in response to my ticket about what happened last night… Short answer = Cisco issues. Been there myself, so I understand it. My biggest issue however was no notice since this apparently has been scheduled beforehand. Virtbiz says they sent an email, and I have no reason to doubt it. But I never got it. I don’t look to twitter as a support mechanism so never even thought to go there. So we still have a couple things to work out communication-wise.
But also notice that they need to do more work tonight. According to the virtbiz site, it will start at 8pm Central and go through midnight. If I hear anything different, I’ll post.
I understand and apologize for your frustration. Last night we had a
maintenance window to upgrade software in the row distribution switches
and upgrade the Supervisor modules and MSFC’s in the gateway switches.
We and announced and anticipated interruptions in service for about 30
seconds or so but due to a conflict in software versions it went on
longer than that as we were forced to roll-back.We’ve documented what went happened and identified where we had the
issue, which ultimately resulted in broken STP between the gateway and
distribution switches, to be a conflict in software versioning. We’ve
obtained alternate versions from Cisco as of last night and have the new
software loaded and running. However, once we got everything back
online, we were not comfortable sending things down for another reset.
The network (or more accurately, the users) had been through enough
trauma for one night.Another window will be opened overnight tonight to address the failure
and complete the work that we set out to do last night. We are starting
early to construct a lab environment using equipment that we sourced
overnight and today to test with first, then apply to the network.I understand that a good part of your concern stems from the
communication before and during the event. I’d like to bullet-point a
few things for you to help address that.- We opened last night’s window to address an issue that came about in a
failure from last week that we had made temporary work-arounds for.- Last night’s window was announced on our website, Twitter feed and
email. I know the website was unavailable for a while during the event.- With regard to the phone system, we recognize that as an area that
needs improvement. A few minutes into the event we disabled the
answering system and had staff in to live-answer the phones to help
explain to customers what was going on. We felt that to be important
given the fact that the website was unavailable. Unfortunately, we
missed you.You should know that as a company that takes its service very seriously,
we’re not satisfied with how last night’s window played out. We do feel
that there are lessons we can take that will help us continue to be
better. I certainly appreciate your comments for us, as well as your
understanding, as we continue working to be the guys you can always rely on.Thanks,
Chris Gebhardt
We’re finalizing a Xen VPS server. It’s in Dallas – has 2 Quad Core AMD Opteron processors, 16GB of ECC RAM and 4x 750 WD Raid Edition (RE2) Enterprise class drives connected to a Dell/LSI PERC RAID controller in HARDWARE RAID-10 for speed and reliability.
Xen VPSes have a greater isolation level than OpenVZ and allow other Operating Systems to be run – like Windows. We will be allowing Windows VPSes, but you will need to provide an ISO of the OS version you want (will not count to your disk space and will remain on the server as a CD/DVD drive) and a scan of the License key.
Pricing will be announced shortly. Contact us if you are interested in a Xen VPS.
We now have a cPanel/WHM reseller vServer available in Atlanta. Same prices as Dallas, but use this link to sign up: http://www.lagniappeinternet.com/members/signup.php?clienttype=16
We are considering a Enkompass server for resellers. Basically, Enkompass is cPanel for Windows. What we’d like to know is how many people would be interested in an Enkompass Reseller account. Enkompass licenses are based on the number of accounts and is more expensive… so pricing would be different than Linux cPanel/WHM licenses. But you would have access to ASP.NET and PHP… Please email rob -at- lagniappeinternet.com or put in a support ticket to let us know if you’re interested.
At 1:38am I started receiving pages from binarycanary about servers in Dallas being unreachable. Two minutes later 1 server reportedly passed, but was down 1 minute later again. The support.virtbis.com site is down as well, which leads me to believe it is a repeat of the incident on the 28th of August – all the servers were functioning fine, but not reachable because of a router issue. I’ve emailed Chris with Virtbiz a – though I’m not sure it can get through right now as well as called and left voicemail. I’m sure they have their own monitoring setup and have been altered. I will update this as I know anything.
Edit: the support.virtbiz.com came up for a minute or so… Tried to put in a ticket, not sure if it made it or not.
3:30am edit: Was finally able to put in a ticket, last one did not take – network crapped out before the submit took. But it’s there now: Ticket ID: RXU-696907 I’ve also setup a binarycanary monitor for support.virtbiz.com to track it (how I could tell it was back up to hit submit again)
3:45 : Virtbiz claimed they notified customers of…
Engineers will be performing emergency network maintenance tonight, September 4 2010.
VIRTBIZ customers receiving service at the DAL-1 facility (2805 Canton St) in Dallas TX may be affected by this maintenance. The purpose of this work is to perform important software upgrades to distribution switches to help prevent flood attacks and unplanned outages.
EMERGENCY MAINTENANCE WINDOW: September 4, 2010, 8:00PM – 11:59PM CDT
Customers in the “Bakers Rack” area (ie: non-rack-mountable equipment) will experience an interruption in service that may last up to 30 minutes. Customers in rows 11-16 may experience brief interruptions in service lasting no longer than 30 seconds.
Only problem, couldn’t get to a website to know anything about it. No email was sent. And 30 minutes… try 2 hours plus! Not a happy camper. I understand the need, but the execution was pretty poor.
Oh wait, we’re in row 11… Should have been 30 SECONDS not 30 minutes… I think what gets me the most was the the “you should have known” attitude that came from the response and no apology, explanation, or anything else. How are you supposed to know if you don’t get notice, can’t get to the support site, and the phone’s network status message was “all networks operational”… Uh, NO they weren’t.
Comments Off