All Dallas servers are reporting outages. Ticket has been placed with the DC. Will update when we get more information.
Update:
There was, as the A/C techs are installing a new 20 ton Liebert and they have been in the 480 panel. They seem to have bumped a breaker which was reset.
The SR1435 did not come up and I booted it just a minute ago.
Thanks for the heads-up on this, and all should be well now.
Let me know,
Jack B. – VIRTBIZ Internet Support
I received this email in response to my ticket about what happened last night… Short answer = Cisco issues. Been there myself, so I understand it. My biggest issue however was no notice since this apparently has been scheduled beforehand. Virtbiz says they sent an email, and I have no reason to doubt it. But I never got it. I don’t look to twitter as a support mechanism so never even thought to go there. So we still have a couple things to work out communication-wise.
But also notice that they need to do more work tonight. According to the virtbiz site, it will start at 8pm Central and go through midnight. If I hear anything different, I’ll post.
I understand and apologize for your frustration. Last night we had a
maintenance window to upgrade software in the row distribution switches
and upgrade the Supervisor modules and MSFC’s in the gateway switches.
We and announced and anticipated interruptions in service for about 30
seconds or so but due to a conflict in software versions it went on
longer than that as we were forced to roll-back.We’ve documented what went happened and identified where we had the
issue, which ultimately resulted in broken STP between the gateway and
distribution switches, to be a conflict in software versioning. We’ve
obtained alternate versions from Cisco as of last night and have the new
software loaded and running. However, once we got everything back
online, we were not comfortable sending things down for another reset.
The network (or more accurately, the users) had been through enough
trauma for one night.Another window will be opened overnight tonight to address the failure
and complete the work that we set out to do last night. We are starting
early to construct a lab environment using equipment that we sourced
overnight and today to test with first, then apply to the network.I understand that a good part of your concern stems from the
communication before and during the event. I’d like to bullet-point a
few things for you to help address that.- We opened last night’s window to address an issue that came about in a
failure from last week that we had made temporary work-arounds for.- Last night’s window was announced on our website, Twitter feed and
email. I know the website was unavailable for a while during the event.- With regard to the phone system, we recognize that as an area that
needs improvement. A few minutes into the event we disabled the
answering system and had staff in to live-answer the phones to help
explain to customers what was going on. We felt that to be important
given the fact that the website was unavailable. Unfortunately, we
missed you.You should know that as a company that takes its service very seriously,
we’re not satisfied with how last night’s window played out. We do feel
that there are lessons we can take that will help us continue to be
better. I certainly appreciate your comments for us, as well as your
understanding, as we continue working to be the guys you can always rely on.Thanks,
Chris Gebhardt
We are considering a Enkompass server for resellers. Basically, Enkompass is cPanel for Windows. What we’d like to know is how many people would be interested in an Enkompass Reseller account. Enkompass licenses are based on the number of accounts and is more expensive… so pricing would be different than Linux cPanel/WHM licenses. But you would have access to ASP.NET and PHP… Please email rob -at- lagniappeinternet.com or put in a support ticket to let us know if you’re interested.
At 1:38am I started receiving pages from binarycanary about servers in Dallas being unreachable. Two minutes later 1 server reportedly passed, but was down 1 minute later again. The support.virtbis.com site is down as well, which leads me to believe it is a repeat of the incident on the 28th of August – all the servers were functioning fine, but not reachable because of a router issue. I’ve emailed Chris with Virtbiz a – though I’m not sure it can get through right now as well as called and left voicemail. I’m sure they have their own monitoring setup and have been altered. I will update this as I know anything.
Edit: the support.virtbiz.com came up for a minute or so… Tried to put in a ticket, not sure if it made it or not.
3:30am edit: Was finally able to put in a ticket, last one did not take – network crapped out before the submit took. But it’s there now: Ticket ID: RXU-696907 I’ve also setup a binarycanary monitor for support.virtbiz.com to track it (how I could tell it was back up to hit submit again)
3:45 : Virtbiz claimed they notified customers of…
Engineers will be performing emergency network maintenance tonight, September 4 2010.
VIRTBIZ customers receiving service at the DAL-1 facility (2805 Canton St) in Dallas TX may be affected by this maintenance. The purpose of this work is to perform important software upgrades to distribution switches to help prevent flood attacks and unplanned outages.
EMERGENCY MAINTENANCE WINDOW: September 4, 2010, 8:00PM – 11:59PM CDT
Customers in the “Bakers Rack” area (ie: non-rack-mountable equipment) will experience an interruption in service that may last up to 30 minutes. Customers in rows 11-16 may experience brief interruptions in service lasting no longer than 30 seconds.
Only problem, couldn’t get to a website to know anything about it. No email was sent. And 30 minutes… try 2 hours plus! Not a happy camper. I understand the need, but the execution was pretty poor.
Oh wait, we’re in row 11… Should have been 30 SECONDS not 30 minutes… I think what gets me the most was the the “you should have known” attitude that came from the response and no apology, explanation, or anything else. How are you supposed to know if you don’t get notice, can’t get to the support site, and the phone’s network status message was “all networks operational”… Uh, NO they weren’t.
At about 5:15pm central time, we had an issue where gt24-1 went offline. We can be pretty sure of when it went offline as I was logged into the server as part of the migration of vps accounts to SolusVM. I had gone to get something to drink. When I came back I had an instant message about email on an account on a vps on that server not working… I went to check and my SSH session was dead. After some initial testing and determining the server was completely unresponsive, I put in a ticket with Tulix support. They had to come out to reboot the server. The server was at a kernel panic. I suspect it had to do with the tweaking we had to do to convert the server over to a SolusVM node. I’ll continue to monitor the node. A second Atlanta node server is currently being prepped for deployment later this year… We may have to alter our timelines. I have a email in to Tulix management asking about the new charge for a reboot, as well as how we can best go forward… for example, is a remote PDU is order.
As of 7:15 the server was coming back up. By 7:25pm all vps containers we up and running.
GT24-1 has alerted us to a hard drive error… The server uses a RAID-10 array, and the other 3 drives are functional. So the server is still online. We are in the process of running a manual secondary rsync of the nightly vzdump files to another server as a precaution. A replacement drive has been ordered for overnight delivery to the datacenter. We’ll update this post as information is available.
9/23 2pm: The drive has been replaced and we are actively resyncing the arrays at this time. We did have to reboot the server a couple of times during the procedure as linux had decided to ‘renumber’ the drive and we needed back at /dev/sda.
The DirectAdmin virtual server da1 was restarted due to a runaway process count.
At 8:42am, we received a nagios warning about the number of processes running on the gt24-2 node (almost 3 times its normal).
Upon closer inspection, 1/3rd of all processes belonged to the da1 VM.
At at 8:47am, we elected to restart the VM — some held connections were evident during the restart.
By 8:49am, the da1 VM was available and the number of processes back to normal.
An incorrect dns zone file for lagniappeinternet.com itself got pushed out by mistake. We’ve restored the file within minutes, but the dns cache time may cause issues with reaching the hosts by lagniappeinternet.com based hostnames. This is only affecting the lagniappeinternet.com domain – customer domains should not be affected. Servers themselves are not affected – only the name resolution for lagniappeinternet.com hostnames
The Installatron module for the DirectAdmin reseller server has been migrated to the new server as well. If you experience and issues with Installatron, or other aspects of the new server, please let us know.
All accounts have been migrated! You should now login with the new server address not the old.
If you are using external nameservers, you MUST update them for the new ip address sent to you in email. Your sites WILL NOT WORK otherwise.
Installatron will be added to the new server shortly.
Comments Off